Metareinforcement learning and domain randomization training method for model prediction task sampling

Abstract

The invention relates to a meta reinforcement learning and domain randomization training method for model prediction task sampling, and the method comprises the steps: constructing a risk random function of an online constructed depth generation model, determining the risk function distribution of the risk random function, and deducing the approximate posteriori of the risk function distribution; estimating function posterior distribution of the depth generation model to construct a target acquisition function, and performing random sampling in a target task space by using the target acquisition function to obtain a predicted value of posterior task adaptive loss and corresponding acquisition data; generating a target plot optimization task batch according to the prediction value and the collection score of the collected data predicted by the risk learner; and determining a domain randomization and meta reinforcement learning updating rule of the machine learning device so as to execute decision model updating training operation in the adaptive model of the target zero sample or the target small sample on the machine learning device. Therefore, the problem that it is difficult to improve the calculation efficiency of the model while improving the adaptive robustness in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the technical field of meta-reinforcement learning and domain randomization, and in particular to a meta-reinforcement learning and domain randomization training method for model prediction task sampling. Background Art

[0002] Meta Reinforcement Learning (MRL) and Domain Randomization (DR) are two very important technologies in the field of reinforcement learning, which can help improve the small-sample and zero-sample generalization capabilities of intelligent agents in different environments.

[0003] The core idea of ​​meta-reinforcement learning is to enable the agent to quickly adapt to new tasks or environments through a small amount of experience or environmental interaction. Unlike traditional reinforcement learning methods, meta-reinforcement learning not only focuses on how to optimize behavioral strategies in the current environment, but also focuses on how to learn the commonalities between different tasks so that the agent can quickly adjust and effectively learn when faced with new tasks. The core of this method is to train the agent through "meta-learning" so that it can accumulate experience in multiple tasks and learn how to quickly find the right strategy in new tasks. For example, in a typical meta-reinforcement learning framework, the agent will experience multiple tasks, each with different states and reward structures. Through meta-learning, the agent can learn a higher-level strategy that enables it to quickly adapt and make reasonable decisions when encountering new tasks with only a small amount of samples or experience.

[0004] Domain randomization is a commonly used technique, especially in robotics and simulation environments, to solve the overfitting problem encountered by intelligent agents during training. It can randomize various factors in the environment (such as physical parameters, object shape, texture, lighting, etc.) during training, so that the intelligent agent will not be overly dependent on a specific environment setting, thereby improving its generalization ability in unknown environments.

[0005] For example, in a robot control task, if the agent is only trained in a fixed simulated environment, it may encounter situations that it cannot cope with in the real environment. By introducing domain randomization in training, the agent can be trained under a variety of environmental changes, enabling it to better cope with different changes in the real world, thus having the ability to go from simulation to reality (Sim2Real).

[0006] Meta-reinforcement learning and domain randomization are often used for meta-training or pre-training of fast adaptive decision makers, especially in complex robotics tasks or reinforcement learning applications. Meta-reinforcement learning can help agents quickly adapt to different tasks and improve their overall learning efficiency; while domain randomization increases the diversity of training environments, avoiding overfitting of agents to specific environments, allowing agents to perform more stably in the ever-changing real world. In summary, both meta-reinforcement learning and domain randomization are committed to improving the generalization ability of reinforcement learning, especially in real-world applications, which can significantly improve the performance of agents, reduce dependence on training data and environment, and promote the application of reinforcement learning technology in practical scenarios.

[0007] However, it is worth noting that adaptive robustness may become an increasingly pressing consideration. Some simple sampling strategies, such as uniform sampling, tend to overlook adaptive robustness in some risk-sensitive scenarios, for example, traffic accidents are more informative than general cases when training autonomous driving systems. In addition, although there have been some progress in robust adaptiveness, the demand for high-quality task datasets or the additional computational overhead required for data collation is still large in practical applications. To achieve data-driven robust control, the robot must interact with multiple Markov decision processes (MDPs), collect a large number of training rounds, and retain a portion of the data for policy search after evaluating the adaptive difficulty. These constitute the key steps in the standard robust adaptive optimization process - evaluate-rank-prune. In robust task episode learning, some commonly used strategies take into account the risk information in the task batch, such as scoring the difficulty of randomly sampled tasks during the adaptation phase and then selecting a subset for optimization. However, the following challenges still exist:

[0008] (1) It incurs additional sample overhead during evaluation, especially when the decision environment is complex, and the evaluation of the policy function in deep reinforcement learning can be very expensive;

[0009] (2) The limited uniform sampling results in the task batch weaken the effective exploration of the agent's decision-making and control task space.

[0010] In summary, it is difficult for existing technologies to improve the computational efficiency of the model while improving adaptive robustness, which needs to be solved urgently. Summary of the invention

[0011] The present application provides a meta-reinforcement learning and domain randomization training method for model prediction task sampling to solve the problem that the existing technology is difficult to improve the computational efficiency of the model while improving the adaptive robustness.

[0012] The first aspect of the present application provides a meta-reinforcement learning and domain randomization training method for model prediction task sampling, comprising the following steps: based on a deep generative model constructed online, and in combination with a target latent variable and a target learning data set, generating a risk random function corresponding to the deep generative model, and determining a risk function distribution corresponding to the risk random function, so as to use a preset streaming variational inference strategy to infer an approximate posterior of the risk function distribution, so as to obtain a corresponding posterior inference result; based on the posterior inference result, estimating the posterior distribution of the function corresponding to the deep generative model, and constructing a target acquisition function according to the posterior distribution of the function and a preset upper confidence bound principle, and using the target acquisition function to obtain a corresponding posterior inference result; The set function performs random sampling in the target task space to obtain the predicted value of the posterior task adaptive loss and the corresponding collected data, and predicts the collection score of the collected data through the predicted value of the posterior task adaptive loss and the preset risk learner, so as to generate a target episode optimization task batch according to the collection score and the preset task screening strategy; determine the domain randomization update rule and meta-reinforcement learning update rule of the preset machine learning machine, and perform a decision model update training operation on the machine learning machine in the target zero-sample adaptive model or the target small-sample adaptive model according to the target episode optimization task batch, the domain randomization update rule and the meta-reinforcement learning update rule.

[0013] Optionally, in one embodiment of the present application, the deep generative model constructed online is combined with the target latent variable and the target learning data set to generate a risk random function corresponding to the deep generative model, and determine the risk function distribution corresponding to the risk random function, so as to use the preset streaming variational inference strategy to infer the approximate posterior of the risk function distribution to obtain the corresponding posterior inference result, including: based on the target latent variable, constructing the risk function distribution corresponding to the risk random function; encoding historical risk information through the target latent variable to determine the conditional prior corresponding to the risk function distribution; processing the target streaming task batch using the conditional prior and the preset recursive strategy to update the function posterior in the risk function distribution to obtain the corresponding approximate function posterior; determining the variational distribution according to the preset parameterized Gaussian distribution, so as to convert the target evidence lower bound into a target operational optimization objective function based on the approximate function posterior and the variational distribution, and obtaining the posterior inference result through the target operational optimization objective function.

[0014] Optionally, in one embodiment of the present application, the target acquisition function is used to perform random sampling in the target task space to obtain the predicted value of the posterior task adaptive loss and the corresponding acquisition data, and the acquisition score of the acquisition data is predicted by the predicted value of the posterior task adaptive loss and a preset risk learner to generate a target plot optimization task batch according to the acquisition score and a preset task screening strategy, including: based on a preset simulation operation, performing an adaptability evaluation on the task identifiers corresponding to the sequential task batches in the target task space to obtain the corresponding acquisition scores; sorting the task identifiers according to the acquisition scores to obtain the sorting results of the task identifiers; and screening the task identifiers that meet the preset candidate requirements from the sorting results of the task identifiers to construct the target plot optimization task batches.

[0015] Optionally, in one embodiment of the present application, the mathematical expression of the deep generative model is:

[0016]

[0017] Among them, p(z0) represents the prior of the initial task risk characterization; p ψ (H t |z t ,θ t ) means that in a given decision model θ t and the posterior representation z t The conditional probability likelihood of the task batch adaptive risk; θ t represents the decision model parameters at time t; z t represents the target latent variable; H t represents the target learning dataset; p(z t+1 |z t ) represents the conditional migration of batch task risk representation due to decision model optimization; T represents the number of iterations at the final convergence.

[0018] Optionally, in one embodiment of the present application, the mathematical expression of the target acquisition function is:

[0019]

[0020] Among them, m(l i ) represents the risk mean; σ(l i ) represents the standard deviation of the task-adaptive risk estimated by random forward propagation; z t represents the target latent variable; τ represents the task identifier; q φ (z t |H t) represents the approximate posterior distribution encoded by the neural network; γ0,γ1 are hyperconstants; p(τ) represents the distribution of tasks or task identifiers; p ψ (l|τ i ,z t ) represents the optimization iterative input task identifier τ at time t i and batch task risk characterization z t The corresponding conditional probability of the adaptive risk l; Represents the input τ i Predict the expected or mean value of l; Represents the batch tasks and identifiers filtered by the Top-K criterion; Represents the input τ i The standard deviation of l is predicted; φ represents the neural network parameters of the approximate posterior encoder; ψ represents the neural network parameters of the adaptive risk decoder.

[0021] The second aspect of the present application provides a meta-reinforcement learning and domain randomization training device for model prediction task sampling, including: a posterior inference module, which is used to generate a risk random function corresponding to the deep generative model based on an online constructed deep generative model, and in combination with a target latent variable and a target learning data set, and determine the risk function distribution corresponding to the risk random function, so as to use a preset streaming variational inference strategy to infer the approximate posterior of the risk function distribution, so as to obtain the corresponding posterior inference result; an acquisition module, which is used to estimate the posterior distribution of the function corresponding to the deep generative model based on the posterior inference result, and construct a target acquisition function according to the posterior distribution of the function and a preset upper confidence bound principle, and use the The target acquisition function performs random sampling in the target task space to obtain the predicted value of the posterior task adaptive loss and the corresponding acquisition data, and predicts the acquisition score of the acquisition data through the predicted value of the posterior task adaptive loss and the preset risk learner, so as to generate a target episode optimization task batch according to the acquisition score and the preset task screening strategy; an update module is used to determine the domain randomization update rule and meta-reinforcement learning update rule of the preset machine learning machine, and according to the target episode optimization task batch, the domain randomization update rule and the meta-reinforcement learning update rule, the decision model update training operation is performed on the machine learning machine in the target zero-sample adaptive model or the target small-sample adaptive model.

[0022] Optionally, in one embodiment of the present application, the posterior inference module includes: a construction unit, used to construct a risk function distribution corresponding to the risk random function based on the target latent variable; an encoding unit, used to encode historical risk information through the target latent variable to determine the conditional prior corresponding to the risk function distribution; a processing unit, used to process the target streaming task batch using the conditional prior and a preset recursive strategy to update the function posterior in the risk function distribution to obtain the corresponding approximate function posterior; a conversion unit, used to determine the variational distribution according to a preset parameterized Gaussian distribution, so as to convert the target evidence lower bound into a target operational optimization objective function based on the approximate function posterior and the variational distribution, and obtain the posterior inference result through the target operational optimization objective function.

[0023] Optionally, in one embodiment of the present application, the acquisition module includes: an adaptability evaluation unit, used to perform adaptability evaluation on the task identifiers corresponding to the sequential task batches of the target task space based on preset simulation operations to obtain corresponding acquisition scores; a sorting unit, used to sort the task identifiers according to the acquisition scores to obtain the sorting results of the task identifiers; and a screening unit, used to screen the task identifiers that meet preset candidate requirements from the sorting results of the task identifiers to construct the target plot optimization task batches.

[0024] Optionally, in one embodiment of the present application, the mathematical expression of the deep generative model is:

[0025]

[0026] Among them, p(z0) represents the prior of the initial task risk characterization; p ψ (H t |z t ,θ t ) means that in a given decision model θ t and the posterior representation z t The conditional probability likelihood of the task batch adaptive risk; θ t represents the decision model parameters at time t; z t represents the target latent variable; H t represents the target learning dataset; p(z t+1 |z t ) represents the conditional migration of batch task risk representation due to decision model optimization; T represents the number of iterations at the final convergence.

[0027] Optionally, in one embodiment of the present application, the mathematical expression of the target acquisition function is:

[0028]

[0029] Among them, m(l i ) represents the risk mean; σ(l i ) represents the standard deviation of the task-adaptive risk estimated by random forward propagation; z t represents the target latent variable; τ represents the task identifier; q φ (z t |H t ) represents the approximate posterior distribution encoded by the neural network; γ0,γ1 are hyperconstants; p(τ) represents the distribution of tasks or task identifiers; p ψ (l|τ i ,z t ) represents the optimization iterative input task identifier τ at time t i and batch task risk characterization z t The corresponding conditional probability of the adaptive risk l; Represents the input τ i Predict the expected or mean value of l; Represents the batch tasks and identifiers filtered by the Top-K criterion; Represents the input τ i The standard deviation of l is predicted; φ represents the neural network parameters of the approximate posterior encoder; ψ represents the neural network parameters of the adaptive risk decoder.

[0030] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the meta-reinforcement learning and domain randomization training method for model prediction task sampling as described in the above embodiments.

[0031] The fourth aspect embodiment of the present application provides a computer-readable storage medium, which stores a computer program, which, when executed by a processor, implements the meta-reinforcement learning and domain randomization training method for the above model prediction task sampling.

[0032] The fifth aspect embodiment of the present application provides a computer program product, including a computer program, which is executed to implement the above-mentioned meta-reinforcement learning and domain randomization training method for model prediction task sampling.

[0033] Therefore, the embodiments of the present application have the following beneficial effects:

[0034] The embodiments of the present application can generate a risk random function corresponding to the deep generative model based on an online constructed deep generative model and in combination with a target latent variable and a target learning data set, and determine the risk function distribution corresponding to the risk random function, so as to use a preset streaming variational inference strategy to infer an approximate posterior of the risk function distribution to obtain the corresponding posterior inference result; based on the posterior inference result, estimate the posterior distribution of the function corresponding to the deep generative model, and construct a target acquisition function according to the posterior distribution of the function and a preset upper confidence bound principle, and use the target acquisition function to perform random sampling in the target task space to obtain the predicted value of the posterior task adaptive loss and the corresponding acquisition data, and through the posterior The predicted value of the adaptive loss of the test task and the collection score of the preset risk learner predicting the collection data are used to generate a target episode optimization task batch according to the collection score and the preset task screening strategy; the domain randomization update rule and meta-reinforcement learning update rule of the preset machine learning machine are determined, and according to the target episode optimization task batch, the domain randomization update rule and the meta-reinforcement learning update rule, the machine learning machine is executed in the adaptive model or algorithm of the target zero-shot or target few-shot to update the decision model training operation, thereby reducing the interaction with the decision environment, and ensuring the learning efficiency of the model while achieving adaptive robustness. Thus, the problem that the existing technology is difficult to improve the computational efficiency of the model while improving the adaptive robustness is solved.

[0035] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0037] Figure 1 A flowchart of a meta-reinforcement learning and domain randomization training method for model prediction task sampling provided according to an embodiment of the present application;

[0038] Figure 2 An example diagram of a meta-reinforcement learning and domain randomization training device for sampling a model prediction task according to an embodiment of the present application;

[0039] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0040] Among them, 10-meta-reinforcement learning and domain randomization training device for model prediction task sampling; 100-posteriori inference module, 200-acquisition module, 300-update module; 301-memory, 302-processor, 303-communication interface. DETAILED DESCRIPTION

[0041] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0042] The following describes the meta-reinforcement learning and domain randomization training method for model prediction task sampling of an embodiment of the present application with reference to the accompanying drawings. In response to the problems mentioned in the above background technology, the present application provides a meta-reinforcement learning and domain randomization training method for model prediction task sampling, in which a risk random function corresponding to the deep generative model is generated based on a deep generative model constructed online and combined with a target latent variable and a target learning data set, and the risk function distribution corresponding to the risk random function is determined, so as to use a preset streaming variational inference strategy to infer the approximate posterior of the risk function distribution to obtain the corresponding posterior inference result; based on the posterior inference result, the function posterior distribution corresponding to the deep generative model is estimated, and a target acquisition function is constructed according to the function posterior distribution and a preset upper confidence bound principle, and the target acquisition function is used to perform random sampling in the target task space. Sampling to obtain the predicted value of the adaptive loss of the posterior task and the corresponding collected data, and predicting the collection score of the collected data through the predicted value of the adaptive loss of the posterior task and the preset risk learner, so as to generate the target plot optimization task batch according to the collection score and the preset task screening strategy; determining the domain randomization update rule and meta-reinforcement learning update rule of the preset machine learning machine, and performing the decision model update training operation in the target zero-sample adaptive model or the target small-sample adaptive model on the machine learning machine according to the target plot optimization task batch, domain randomization update rule and meta-reinforcement learning update rule, thereby reducing the interaction with the decision environment, and ensuring the learning efficiency of the model while achieving adaptive robustness. Thus, the problem that the existing technology is difficult to improve the computational efficiency of the model while improving the adaptive robustness is solved.

[0043] In order to facilitate those skilled in the art to understand the execution process of the present application, the execution logic of the present application and the role objects involved are briefly described as follows:

[0044] Based on actual needs, this application focuses on achieving robust adaptability while ensuring learning efficiency, such as avoiding uninformative task selection steps (such as labeling or evaluation) and eliminating unnecessary calculations, so as to achieve adaptability in real-world scenarios and obtain the Model Predictive Task Sampling (MPTS) strategy. The optimization process of this application mainly involves the following roles:

[0045] (1) The model optimizer specifies an adaptive machine learning machine and learns it in any number of times;

[0046] (2) The risk learner acts as a critic to assess and predict task-specific risks;

[0047] (3) The task sampler acts as an actor to screen the task batch for the next iteration. These components operate in a cyclic manner until convergence.

[0048] Specifically, Figure 1 A flowchart of a meta-reinforcement learning and domain randomization training method for model prediction task sampling provided in an embodiment of the present application.

[0049] like Figure 1 As shown in Figure 1, the meta-reinforcement learning and domain randomization training method for model prediction task sampling includes the following steps:

[0050] In step S101, based on the deep generative model constructed online, a risk random function corresponding to the deep generative model is generated in combination with the target latent variable and the target learning data set, and the risk function distribution corresponding to the risk random function is determined, so as to use the preset streaming variational inference strategy to infer the approximate posterior of the risk function distribution, so as to obtain the corresponding posterior inference result.

[0051] It should be understood by those skilled in the art that in task scenario training, the task selection strategy is closely related to the optimization process and the generalization ability of the machine learning machine. Therefore, the embodiment of the present application can adopt a sampling strategy based on risk landscape, and As a learning dataset to model the risk random function.

[0052] In order to describe the episode-adaptive risk in the batch optimization process, the embodiment of the present application introduces a latent variable z to summarize the historical information of the risk and construct a general deep generative model online.

[0053] Optionally, in one embodiment of the present application, the mathematical expression of the deep generative model is:

[0054]

[0055] Among them, p(z0) represents the prior of the initial task risk characterization; p ψ (H t |z t ,θ t ) means that in a given decision model θ t and the posterior representation z t The conditional probability likelihood of the task batch adaptive risk; θ t represents the decision model parameters at time t; z t represents the target latent variable; H trepresents the target learning dataset; p(z t+1 |z t ) represents the conditional migration of batch task risk representation due to decision model optimization; T represents the number of iterations at the final convergence.

[0056] It should be noted that the mathematical expression of the deep generative model constructed in the embodiment of the present application is as follows:

[0057]

[0058] Among them, p(z0) represents the prior of the initial task risk characterization; p ψ (H t |z t ,θ t ) means that in a given decision model θ t and the posterior representation z t The conditional probability likelihood of the task batch adaptive risk; θ t represents the decision model parameters at time t; z t represents the target latent variable; H t represents the target learning dataset, which contains the identifier set and adaptive risk value of the collected batch tasks; p(z t+1 |z t ) represents the conditional migration of batch task risk representation due to decision model optimization; T represents the number of iterations at the final convergence.

[0059] In the Bayesian framework, the embodiments of the present application can approximate the potential function distribution through latent variables, where the posterior p(z t |H t ) summarizes the historical risk information and takes into account the uncertainty of the distribution. According to the Bayesian formula, p(z t |H t ) is of the following form:

[0060]

[0061] Where p(z t |H 1:t-1 ) as a conditional prior, encoding the results of past collection and evaluation; and p(H t |z t ) expresses the probability of generating a batch risk value observation in the tth iteration; it should be noted that due to the complexity of the integral in the denominator, the exact calculation of the posterior is not feasible.

[0062] Optionally, in one embodiment of the present application, based on the deep generative model constructed online, a risk random function corresponding to the deep generative model is generated in combination with the target latent variable and the target learning data set, and the risk function distribution corresponding to the risk random function is determined, so as to use the preset streaming variational inference strategy to infer the approximate posterior of the risk function distribution to obtain the corresponding posterior inference result, including: constructing the risk function distribution corresponding to the risk random function based on the target latent variable; encoding historical risk information through the target latent variable to determine the conditional prior corresponding to the risk function distribution; processing the target streaming task batch using the conditional prior and the preset recursive strategy to update the function posterior in the risk function distribution to obtain the corresponding approximate function posterior; determining the variational distribution according to the preset parameterized Gaussian distribution, so as to convert the target evidence lower bound into a target operational optimization objective function based on the approximate function posterior and the variational distribution, and obtaining the posterior inference result through the target operational optimization objective function.

[0063] In the embodiment of the present application, the task batch risk value of interest is related to the parameter θ of the meta-learner. Specifically, the variables related to sequence batch optimization are decomposed as follows:

[0064]

[0065] In the probabilistic graphical model, the latent variable z t The distribution of the risk function is constituted (for the sake of simplicity, other variables not related to the learning purpose are omitted in this embodiment of the application). Here, the embodiment of the application can assume that given the task identifier τ, the latent variable z and the model parameter θ, the task-specific risk value is conditionally independent, and the main optimization goal is Used for stochastic risk function learning.

[0066] In addition, since the mode of episode training is that the task batches and evaluation values ​​arrive in sequence, the embodiments of the present application can use these correlations and learn some quantities that are helpful for selecting task batches. However, since the posterior distribution lacks explicit structural information, it is difficult to accurately infer p(z t |H t ) is not feasible.

[0067] It should be noted that in each iteration, the risk function distribution depends on the updated machine learning parameters θ; therefore, this non-stationarity in the risk function distribution enables the embodiments of the present application to use streaming variational inference to obtain approximate posteriors.

[0068] Specifically, the embodiment of the present application processes the streaming task batch in a recursive manner and updates the posterior, as shown in the following formula:

[0069]

[0070] Among them, p(zt |H 1:t-1 ) represents the conditional prior of the last updated posterior. The estimated function posterior is used to provide uncertainty-aware predictions on the stochastic risk function and serves for task sampling strategy design.

[0071] Therefore, through approximate inference, the embodiment of the present application can formalize the evidence lower bound (ELBO) as an operational optimization target, as shown in the following formula:

[0072]

[0073] For ease of practical implementation, the embodiments of the present application may use a parameterized Gaussian distribution with a diagonal covariance matrix as a variational distribution, similar to the Vanilla VAE. The parameters of these distributions are learned through a neural network, for example And use the reparameterization trick for stochastic gradient estimation.

[0074] In terms of neural network structure, the embodiment of the present application can use the Deepset encoding module to process the collection data set. In addition, in the context of streaming variational inference, q φ (z t |H t-1 ) is usually used as a default conditional prior proxy. Therefore, the embodiment of the present application can modify the precise ELBO formula and transform it into an actual optimization process through the Lagrange multiplier β:

[0075]

[0076]

[0077] in, This means that the gradient will not be calculated through φ in this term, and Used to constrain the parameter search range of the machine learning machine in the next iteration.

[0078] In step S102, based on the posterior inference result, the posterior distribution of the function corresponding to the deep generative model is estimated, and the target acquisition function is constructed according to the posterior distribution of the function and the preset upper confidence bound principle, and the target acquisition function is used to perform random sampling in the target task space to obtain the predicted value of the posterior task adaptive loss and the corresponding collected data, and the collection score of the collected data is predicted by the predicted value of the posterior task adaptive loss and the preset risk learner, so as to generate a target plot optimization task batch according to the collection score and the preset task screening strategy.

[0079] Understandably, in robust task episode learning, some commonly used strategies take into account the risk information in the task batch, such as scoring the difficulty of randomly sampled tasks during the adaptation phase and then selecting a subset for optimization. However, the following challenges still exist:

[0080] (1) It brings additional computational overhead during evaluation, especially when the model complexity is very high;

[0081] (2) In data-driven control tasks, evaluating interactions with the environment can be very expensive (e.g., deep reinforcement learning);

[0082] (3) The limited uniform sampling results in the task batch weaken the exploration of the task space.

[0083] In order to solve the above problems, the embodiment of the present application assumes that the plot optimization process of the task is predictable, and constructs a risk learner to overcome the learning resource limitations encountered in the robust optimization process. The random risk function generated by the generation model can be used to sequentially query task batches. The basic principle is that in the embodiment of the present application, the learned risk learner can store rich historical task risk information in the iterative process, and can integrate specific rules into the acquisition function to achieve effective sampling.

[0084] It should be noted that an important application of risk learners is to support MPTS, which replaces the precise evaluation in all test scenarios with a pseudo-evaluation of the adaptive performance of the machine learning learner; in other words, MPTS does not require additional manual or high-overhead runtime evaluation, nor does it require interaction with the real world to modify the task sampling results.

[0085] Optionally, in one embodiment of the present application, a target acquisition function is used to perform random sampling in the target task space to obtain a predicted value of the posterior task adaptive loss and the corresponding acquisition data, and the acquisition score of the acquisition data is predicted by the predicted value of the posterior task adaptive loss and a preset risk learner to generate a target plot optimization task batch according to the acquisition score and a preset task screening strategy, including: based on a preset simulation operation, performing an adaptability evaluation on the task identifiers corresponding to the sequential task batches in the target task space to obtain the corresponding acquisition scores; sorting the task identifiers according to the acquisition scores to obtain the sorting results of the task identifiers; and screening the task identifiers that meet the preset candidate requirements from the sorting results of the task identifiers to construct the target plot optimization task batches.

[0086] In the actual implementation process, the embodiments of the present application can use the above-constructed deep generative model and the estimated function posterior distribution as a tool for actively selecting task batches. Specifically, the predicted distribution in the form of function posterior can be expressed as:

[0087]

[0088] It can be understood that the above prediction distribution p ψ (l|τ,H 1:t )Thanks to Bayesian modeling, it provides a feasible way to evaluate task difficulty across the entire task space without expensive computation.

[0089] Afterwards, the embodiment of the present application may perform sorting and pruning on the next task batch to perform episode training.

[0090] Specifically, when p ψ (l|τ,H 1:t ), the embodiment of the present application formulates a task batch sampling strategy based on its quantitative statistical data, wherein the corresponding standard can refer to the acquisition function in the classical Bayesian optimization, including evaluation criteria such as expected improvement, output information theory index or upper confidence bound.

[0091] However, it should be noted that the search space is on sequential task batches rather than on the parameters of the machine learning machine, which is different from the ultimate goal in classical Bayesian optimization. The core idea in the embodiments of the present application is the optimistic principle in the face of uncertainty (Optimismin the Face of Uncertainty, OFU); in addition, since the worst performing task subset should receive extra attention to optimize adaptive robustness; and in the last few rounds of meta-training, the task area with high prediction uncertainty is often underestimated, therefore, the embodiments of the present application can consider the worst case in adaptation and the uncertainty in the prediction function as the key elements for developing the acquisition function, therefore, the embodiments of the present application can construct the corresponding acquisition function based on the upper confidence bound principle.

[0092] Optionally, in one embodiment of the present application, the mathematical expression of the target acquisition function is:

[0093]

[0094] Among them, m(l i ) represents the risk mean; σ(l i ) represents the standard deviation of the task-adaptive risk estimated by random forward propagation; z t represents the target latent variable; τ represents the task identifier; q φ (z t |H t ) represents the approximate posterior distribution encoded by the neural network; γ0,γ1 are hyperconstants; p(τ) represents the distribution of tasks or task identifiers; p ψ (l|τ i ,z t ) represents the optimization iterative input task identifier τ at time t iand batch task risk characterization z t The corresponding conditional probability of the adaptive risk l; Represents the input τ i Predict the expected or mean value of l; Represents the batch tasks and identifiers filtered by the Top-K criterion; Represents the input τ i The standard deviation of l is predicted; φ represents the neural network parameters of the approximate posterior encoder; ψ represents the neural network parameters of the adaptive risk decoder.

[0095] It should be noted that the mathematical expression of the corresponding acquisition function (i.e., the target acquisition function) based on the upper confidence bound principle is as follows:

[0096]

[0097] Among them, m(l i ) represents the risk mean; σ(l i ) represents the standard deviation of the task-adaptive risk estimated by random forward propagation; z t represents the target latent variable; τ represents the task identifier; q φ (z t |H t ) represents the approximate posterior distribution encoded by the neural network; γ0,γ1 are hyperconstants; p(τ) represents the distribution of tasks or task identifiers; p ψ (l|τ i ,z t ) represents the optimization iterative input task identifier τ at time t i and batch task risk characterization z t The corresponding conditional probability of the adaptive risk l; Represents the input τ i Predict the expected or mean value of l; Indicates the batch tasks and identifiers selected by the Top-K criterion; Represents the input τ i The standard deviation of l is predicted; φ represents the neural network parameters of the approximate posterior encoder; ψ represents the neural network parameters of the adaptive risk decoder.

[0098] Afterwards, the embodiment of the present application may perform simulation-sort-pruning operations in the task batch in the t+1 iteration:

[0099]

[0100] The above formula represents the step of actively selecting task batches and has no analytical form in maximization.

[0101] In the specific implementation process, the embodiment of the present application can pseudo-evaluate the adaptability performance of a batch of task identifiers through simulation, sort these identifiers according to their respective collection scores, and prune the identifier batches to further query the optimization of the task data set and the machine learning machine. The above steps approximate the solution of the above formula in a heuristic way and obtain

[0102] In other words, the embodiment of the present application still performs random sampling in the task space and predicts the corresponding collection score from the risk learner And filter out Candidates in utility values, thereby generating task batches for plot optimization

[0103] It can be understood that the embodiment of the present application can theoretically improve the adaptive performance of the worst-case proportionality through the design of the MPTS acquisition function, and can also improve the robustness even when the task distribution is severely shifted; in addition, the lightweight risk learner replaces the computationally expensive part of the optimization and evaluation, thereby greatly reducing the computational overhead and balancing the utilization and exploration in the task space in specific scenarios.

[0104] Therefore, the embodiments of the present application establish a mapping relationship between the task space and the adaptive loss through a model prediction task sampling method, thereby reducing unnecessary interactions and strategy evaluations with the decision-making environment and lowering the amount of task sampling; in addition, the embodiments of the present application construct a task acquisition function and an adaptive risk prediction agent model based on stream Bayesian inference, thereby providing strong support and guidance for the design of task sampling criteria.

[0105] In step S103, the domain randomization update rule and meta-reinforcement learning update rule of the preset machine learning machine are determined, and according to the target episode optimization task batch, the domain randomization update rule and the meta-reinforcement learning update rule, the decision model update training operation in the target zero-sample adaptive model or the target small-sample adaptive model is performed on the machine learning machine.

[0106] After getting the pruned task batch The embodiment of the present application can perform a standard model optimization step to update the parameters of the machine learning machine. The task-specific adaptation risk in the t+1 iteration is expressed as for the sampled task τ i l t+1,i (θ); Thus, in the embodiments of the present application, MPTS can approximately predict the adaptation risk and enable it to be seamlessly integrated into the adaptive learning paradigm of domain randomization (zero-shot) and meta-reinforcement learning (few-shot), as described below:

[0107] (1) Update of machine learning machines in zero-shot adaptive models or algorithms (domain randomization)

[0108] Since the zero-shot setting does not require a supporting data set to identify the task, taking the traditional domain randomization method as an example, the embodiment of the present application can obtain the following update rules:

[0109]

[0110] Among them, θ represents the parameters of the zero-shot learning model and λ is the learning rate.

[0111] (2) Updates of machine learning machines in few-shot adaptive models or algorithms (meta-reinforcement learning)

[0112] The present application embodiment takes the typical optimization-based method MAML as an example and provides the following update rules:

[0113]

[0114] Among them, θ meta represents the meta-initialization of the model, λ 1,1 and λ 1,2 Represent the learning rates in the inner and outer loops respectively.

[0115] Therefore, the embodiments of the present application require fewer learning resources, thereby reducing the interaction with the decision-making environment and well retaining stronger zero-sample and small-sample adaptive robustness.

[0116] According to the meta-reinforcement learning and domain randomization training method for model prediction task sampling proposed in the embodiment of the present application, a risk random function corresponding to the deep generative model is generated based on a deep generative model constructed online and in combination with a target latent variable and a target learning data set, and the risk function distribution corresponding to the risk random function is determined, so as to use a preset streaming variational inference strategy to infer an approximate posterior of the risk function distribution to obtain the corresponding posterior inference result; based on the posterior inference result, the posterior distribution of the function corresponding to the deep generative model is estimated, and a target acquisition function is constructed according to the posterior distribution of the function and a preset upper confidence bound principle, and the target acquisition function is used to perform random sampling in the target task space to obtain the posterior task. The predicted value of the task adaptive loss and the corresponding collected data are obtained, and the collection score of the collected data is predicted by the predicted value of the posterior task adaptive loss and the preset risk learner, so as to generate a target episode optimization task batch according to the collection score and the preset task screening strategy; the domain randomization update rule and meta-reinforcement learning update rule of the preset machine learning machine are determined, and according to the target episode optimization task batch, the domain randomization update rule and the meta-reinforcement learning update rule, the decision model update training operation is performed on the machine learning machine in the target zero-sample adaptive model or the target small-sample adaptive model, thereby reducing the interaction with the decision environment, and ensuring the learning efficiency of the model while achieving adaptive robustness.

[0117] Secondly, the meta-reinforcement learning and domain randomization training device for model prediction task sampling proposed in an embodiment of the present application is described with reference to the accompanying drawings.

[0118] Figure 2 It is a block diagram of a meta-reinforcement learning and domain randomization training device for model prediction task sampling in an embodiment of the present application.

[0119] like Figure 2 As shown, the meta-reinforcement learning and domain randomization training device 10 for model prediction task sampling includes: a posterior inference module 100, a collection module 200 and an update module 300.

[0120] Among them, the posterior inference module 100 is used to generate a risk random function corresponding to the deep generative model based on the online constructed deep generative model, and combine the target latent variables and the target learning data set, and determine the risk function distribution corresponding to the risk random function, so as to use the preset streaming variational inference strategy to infer the approximate posterior of the risk function distribution, so as to obtain the corresponding posterior inference result.

[0121] The acquisition module 200 is used to estimate the posterior distribution of the function corresponding to the deep generative model based on the posterior inference result, and construct a target acquisition function according to the posterior distribution of the function and a preset upper confidence bound principle, and use the target acquisition function to perform random sampling in the target task space to obtain the predicted value of the posterior task adaptive loss and the corresponding acquisition data, and predict the acquisition score of the acquisition data through the predicted value of the posterior task adaptive loss and the preset risk learner, so as to generate a target plot optimization task batch according to the acquisition score and the preset task screening strategy.

[0122] The updating module 300 is used to determine the domain randomization update rule and meta-reinforcement learning update rule of the preset machine learning machine, and perform a decision model update training operation on the machine learning machine in the target zero-sample adaptive model or the target small-sample adaptive model according to the target plot optimization task batch, the domain randomization update rule and the meta-reinforcement learning update rule.

[0123] Optionally, in one embodiment of the present application, the a posteriori inference module 100 includes: a construction unit, an encoding unit, a processing unit and a conversion unit.

[0124] The construction unit is used to construct a risk function distribution corresponding to the risk random function based on the target latent variable.

[0125] The encoding unit is used to encode historical risk information through the target latent variable to determine the conditional prior corresponding to the risk function distribution.

[0126] The processing unit is used to process the target streaming task batch using the conditional prior and the preset recursive strategy to update the function posterior in the risk function distribution and obtain the corresponding approximate function posterior.

[0127] The conversion unit is used to determine the variational distribution according to the preset parameterized Gaussian distribution, so as to convert the target evidence lower bound into a target operational optimization objective function based on the approximate function posterior and the variational distribution, and obtain the posterior inference result through the target operational optimization objective function.

[0128] Optionally, in one embodiment of the present application, the acquisition module 200 includes: an adaptability evaluation unit, a sorting unit and a screening unit.

[0129] Among them, the adaptability evaluation unit is used to perform adaptability evaluation on the task identifiers corresponding to the sequential task batches in the target task space based on the preset simulation operation to obtain the corresponding collection score.

[0130] The sorting unit is used to sort the task identifiers according to the collection scores to obtain a sorting result of the task identifiers.

[0131] The screening unit is used to screen the task identifiers that meet the preset candidate requirements from the sorting results of the task identifiers to construct a target plot optimization task batch.

[0132] Optionally, in one embodiment of the present application, the mathematical expression of the deep generative model is:

[0133]

[0134] Among them, p(z0) represents the prior of the initial task risk characterization; p ψ (H t |z t ,θ t ) means that in a given decision model θ t and the posterior representation z t The conditional probability likelihood of the task batch adaptive risk; θ t represents the decision model parameters at time t; z t represents the target latent variable; H t represents the target learning dataset; p(z t+1 |z t ) represents the conditional migration of batch task risk representation due to decision model optimization; T represents the number of iterations at the final convergence.

[0135] Optionally, in one embodiment of the present application, the mathematical expression of the target acquisition function is:

[0136]

[0137] Among them, m(l i ) represents the risk mean; σ(l i ) represents the standard deviation of the task-adaptive risk estimated by random forward propagation; z t represents the target latent variable; τ represents the task identifier; q φ (z t |H t ) represents the approximate posterior distribution encoded by the neural network; γ0,γ1 are hyperconstants; p(τ) represents the distribution of tasks or task identifiers; p ψ (l|τ i ,z t ) represents the optimization iterative input task identifier τ at time t i and batch task risk characterization z t The corresponding conditional probability of the adaptive risk l; Represents the input τ i Predict the expected or mean value of l; Represents the batch tasks and identifiers filtered by the Top-K criterion; Represents the input τ iThe standard deviation of l is predicted; φ represents the neural network parameters of the approximate posterior encoder; ψ represents the neural network parameters of the adaptive risk decoder.

[0138] It should be noted that the explanation of the above-mentioned embodiment of the meta-reinforcement learning and domain randomization training method for model prediction task sampling is also applicable to the meta-reinforcement learning and domain randomization training device for model prediction task sampling in this embodiment, and will not be repeated here.

[0139] The meta-reinforcement learning and domain randomization training device for model prediction task sampling proposed in the embodiment of the present application includes a posterior inference module, which is used to generate a risk random function corresponding to the deep generative model based on the online constructed deep generative model, and combine the target latent variable and the target learning data set, and determine the risk function distribution corresponding to the risk random function, so as to use the preset streaming variational inference strategy to infer the approximate posterior of the risk function distribution, so as to obtain the corresponding posterior inference result; an acquisition module, which is used to estimate the posterior distribution of the function corresponding to the deep generative model based on the posterior inference result, and construct a target acquisition function according to the function posterior distribution and the preset upper confidence bound principle, and use the target acquisition function to perform random sampling in the target task space, so as to obtain the corresponding posterior inference result. The predicted value of the posterior task adaptive loss and the corresponding collected data are obtained, and the collection score of the collected data is predicted by the predicted value of the posterior task adaptive loss and the preset risk learner, so as to generate a target episode optimization task batch according to the collection score and the preset task screening strategy; an update module is used to determine the domain randomization update rule and meta-reinforcement learning update rule of the preset machine learning machine, and according to the target episode optimization task batch, the domain randomization update rule and the meta-reinforcement learning update rule, the decision model update training operation is performed on the machine learning machine in the target zero-sample adaptive model or the target small-sample adaptive model, thereby reducing the interaction with the decision environment, and ensuring the learning efficiency of the model while achieving adaptive robustness.

[0140] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0141] A memory 301 , a processor 302 , and a computer program stored in the memory 301 and executable on the processor 302 .

[0142] When the processor 302 executes the program, the meta-reinforcement learning and domain randomization training methods for model prediction task sampling provided in the above embodiments are implemented.

[0143] Furthermore, the electronic device further comprises:

[0144] The communication interface 303 is used for communication between the memory 301 and the processor 302 .

[0145] The memory 301 is used to store computer programs that can be run on the processor 302 .

[0146] The memory 301 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0147] If the memory 301, the processor 302 and the communication interface 303 are implemented independently, the communication interface 303, the memory 301 and the processor 302 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0148] Optionally, in a specific implementation, if the memory 301, the processor 302 and the communication interface 303 are integrated on a chip, the memory 301, the processor 302 and the communication interface 303 can communicate with each other through an internal interface.

[0149] The processor 302 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0150] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the meta-reinforcement learning and domain randomization training method for the above-mentioned model prediction task sampling.

[0151] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned meta-reinforcement learning and domain randomization training method for model prediction task sampling.

[0152] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0153] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0154] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0155] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0156] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0157] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0158] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0159] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A meta-reinforcement learning and domain randomization training method for model prediction task sampling, characterized in that: The following steps are involved: Based on the deep generative model constructed online, a risk stochastic function corresponding to the deep generative model is generated in combination with the target latent variable and the target learning data set, and the risk function distribution corresponding to the risk stochastic function is determined, so as to use the preset streaming variational inference strategy to infer the approximate posterior of the risk function distribution, so as to obtain the corresponding posterior inference result; Based on the posterior inference result, the posterior distribution of the function corresponding to the deep generative model is estimated, and a target acquisition function is constructed according to the posterior distribution of the function and a preset upper confidence bound principle, and the target acquisition function is used to perform random sampling in the target task space to obtain the predicted value of the posterior task adaptive loss and the corresponding collected data, and the collection score of the collected data is predicted by the predicted value of the posterior task adaptive loss and a preset risk learner, so as to generate a target plot optimization task batch according to the collection score and a preset task screening strategy; Determine a domain randomization update rule and a meta-reinforcement learning update rule of a preset machine learning machine, and perform a decision model update training operation in a target zero-shot adaptive model or a target small-shot adaptive model on the machine learning machine according to the target episode optimization task batch, the domain randomization update rule and the meta-reinforcement learning update rule.

2. The method according to claim 1, characterized in that: The deep generative model constructed online is combined with the target latent variable and the target learning data set to generate the risk stochastic function corresponding to the deep generative model, and the risk function distribution corresponding to the risk stochastic function is determined, so as to use the preset streaming variational inference strategy to infer the approximate posterior of the risk function distribution, so as to obtain the corresponding posterior inference result, including: Based on the target latent variable, construct a risk function distribution corresponding to the risk stochastic function; Encoding historical risk information through the target latent variable to determine the conditional prior corresponding to the risk function distribution; Processing the target streaming task batch using the conditional prior and a preset recursive strategy to update the function posterior in the risk function distribution to obtain a corresponding approximate function posterior; The variational distribution is determined according to a preset parameterized Gaussian distribution, so as to convert the target evidence lower bound into a target operational optimization objective function based on the approximate function posterior and the variational distribution, and the posterior inference result is obtained through the target operational optimization objective function.

3. The method according to claim 1, characterized in that The target acquisition function is used to perform random sampling in the target task space to obtain the predicted value of the posterior task adaptive loss and the corresponding acquisition data, and the acquisition score of the acquisition data is predicted by the predicted value of the posterior task adaptive loss and a preset risk learner, so as to generate a target plot optimization task batch according to the acquisition score and a preset task screening strategy, including: Based on a preset simulation operation, an adaptability evaluation is performed on the task identifiers corresponding to the sequential task batches of the target task space to obtain corresponding acquisition scores; sorting the task identifiers according to the collection scores to obtain a sorting result of the task identifiers; Task identifiers that meet preset candidate requirements are screened from the sorting results of the task identifiers to construct the target plot optimization task batch.

4. The method according to claim 1, characterized in that The mathematical expression of the deep generative model is: Among them, p(z0) represents the prior of the initial task risk characterization; p ψ (H t |z t ,θ t ) means that in a given decision model θ t and the posterior representation z t The conditional probability likelihood of the task batch adaptive risk; θ t represents the decision model parameters at time t; z t represents the target latent variable; H t represents the target learning dataset; p(z t+1 |z t ) represents the conditional migration of batch task risk representation due to decision model optimization; T represents the number of iterations at the final convergence.

5. The method according to claim 1, characterized in that The mathematical expression of the target acquisition function is: Among them, m(l i ) represents the risk mean; σ(l i ) represents the standard deviation of the task-adaptive risk estimated by random forward propagation; z t represents the target latent variable; τ represents the task identifier; q φ (z t |H t ) represents the approximate posterior distribution encoded by the neural network; γ0, γ1 are hyperconstants; p(τ) represents the distribution of tasks or task identifiers; p ψ , (l|τ i , z t ) represents the optimization iterative input task identifier τ at time t i and batch task risk characterization z t The corresponding conditional probability of the adaptive risk l; Represents the input τ i Predict the expected or mean value of l; Represents the batch tasks and identifiers filtered by the Top-K criterion; Represents the input τ i The standard deviation of l is predicted; φ represents the neural network parameters of the approximate posterior encoder; ψ represents the neural network parameters of the adaptive risk decoder.

6. A meta-reinforcement learning and domain randomization training device for model prediction task sampling, characterized in that: include: A posterior inference module is used to generate a risk stochastic function corresponding to the deep generative model based on the online constructed deep generative model and in combination with the target latent variable and the target learning data set, and determine the risk function distribution corresponding to the risk stochastic function, so as to use the preset streaming variational inference strategy to infer the approximate posterior of the risk function distribution, so as to obtain the corresponding posterior inference result; A collection module, used to estimate the posterior distribution of the function corresponding to the deep generative model based on the posterior inference result, and construct a target collection function according to the posterior distribution of the function and a preset upper confidence bound principle, and use the target collection function to perform random sampling in the target task space to obtain the predicted value of the posterior task adaptive loss and the corresponding collection data, and predict the collection score of the collection data through the predicted value of the posterior task adaptive loss and a preset risk learner, so as to generate a target plot optimization task batch according to the collection score and a preset task screening strategy; An updating module is used to determine a domain randomization update rule and a meta-reinforcement learning update rule of a preset machine learning machine, and perform a decision model update training operation in a target zero-shot adaptive model or a target small-shot adaptive model on the machine learning machine according to the target episode optimization task batch, the domain randomization update rule and the meta-reinforcement learning update rule.

7. The device according to claim 6, characterized in that The posterior inference module comprises: A construction unit, configured to construct a risk function distribution corresponding to the risk stochastic function based on the target latent variable; An encoding unit, configured to encode historical risk information through the target latent variable to determine a conditional prior corresponding to the risk function distribution; A processing unit, configured to process the target streaming task batch using the conditional prior and a preset recursive strategy to update the function posterior in the risk function distribution to obtain a corresponding approximate function posterior; A conversion unit is used to determine a variational distribution according to a preset parameterized Gaussian distribution, so as to convert a target evidence lower bound into a target operational optimization objective function based on the approximate function posterior and the variational distribution, and obtain the posterior inference result through the target operational optimization objective function.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the meta-reinforcement learning and domain randomization training method for model prediction task sampling as described in any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the meta-reinforcement learning and domain randomization training method for model prediction task sampling as described in any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that The computer program is executed to implement the meta-reinforcement learning and domain randomization training method for model prediction task sampling as described in any one of claims 1-5.

Images