Meta-reinforcement learning and domain randomization training method for model prediction task sampling

Abstract

The application relates to a model prediction task sampling meta-reinforcement learning and domain randomization training method, wherein the method comprises the following steps: constructing a risk random function of an online constructed deep generative model, determining a risk function distribution of the deep generative model, and inferring an approximate posterior of the risk function distribution; estimating a function posterior distribution of the deep generative model to construct a target collection function, and performing random sampling in a target task space by using the target collection function to obtain a prediction value of a posterior task adaptive loss and corresponding collection data, and generating a target scenario optimization task batch by using the prediction value and a risk learner to predict a collection score of the collection data; determining a domain randomization and meta-reinforcement learning update rule of a machine learner, so as to perform a decision model update training operation in an adaptive model of a target zero sample or a target small sample on the machine learner. Therefore, the problems that the prior art is difficult to improve the adaptive robustness and improve the calculation efficiency of the model are solved.

Description

Technical Field

[0001] This application relates to the fields of meta-reinforcement learning and domain randomization, and in particular to a meta-reinforcement learning and domain randomization training method for model prediction task sampling. Background Technology

[0002] Meta-reinforcement learning (MRL) and domain randomization (DR) are two very important techniques in the field of reinforcement learning, which can help improve the ability of agents to generalize with few and zero samples in different environments.

[0003] The core idea of ​​meta-reinforcement learning is to enable an agent to quickly adapt to new tasks or environments based on limited experience or environmental interactions. Unlike traditional reinforcement learning methods, meta-reinforcement learning not only focuses on optimizing behavioral strategies in the current environment but also on learning commonalities across different tasks, allowing the agent to quickly adjust and learn effectively when faced with new tasks. The core of this method is training the agent through "meta-learning," allowing it to accumulate experience across multiple tasks and learn how to quickly find appropriate strategies for new tasks. For example, in a typical meta-reinforcement learning framework, the agent experiences multiple tasks, each with different states and reward structures. Through meta-learning, the agent can learn a higher-level strategy, enabling it to quickly adapt and make reasonable decisions when encountering new tasks based on only a limited number of samples or experiences.

[0004] Domain randomization is a commonly used technique, especially in robotics and simulation environments, to address the overfitting problem encountered by agents during training. It can randomize various factors in the environment (such as physical parameters, object shapes, textures, lighting, etc.) during training, so that the agent does not become overly dependent on a specific environmental setting, thereby improving its generalization ability in unknown environments.

[0005] For example, in robot control tasks, if an agent is trained only in a fixed simulation environment, it may encounter situations that it cannot cope with in the real environment. By introducing domain randomization in the training, the agent can be trained under various environmental changes, enabling it to better cope with different changes in the real world, thus having the ability to go from simulation to reality (Sim2Real).

[0006] Meta-reinforcement learning and domain randomization are frequently used for the meta-training or pre-training of fast adaptive decision-makers, especially in applications requiring complex robotic tasks or reinforcement learning. Meta-reinforcement learning helps agents improve their overall learning efficiency by quickly adapting to different tasks. Domain randomization, on the other hand, increases the diversity of the training environment, preventing overfitting to specific environments and enabling the agent to perform more stably in the ever-changing real world. In summary, both meta-reinforcement learning and domain randomization aim to improve the generalization ability of reinforcement learning, particularly in real-world applications. They significantly improve agent performance, reduce dependence on training data and environment, and promote the application of reinforcement learning techniques in practical scenarios.

[0007] However, it is worth noting that adaptive robustness may become an increasingly pressing consideration. Some simple sampling strategies, such as uniform sampling, easily overlook adaptive robustness in some risk-sensitive scenarios; for example, traffic accidents are more informative than general situations when training autonomous driving systems. Furthermore, despite some progress in robust adaptation, the need for high-quality task datasets or the additional computational overhead required for data preparation remains significant in practical applications. To achieve data-driven robust control, the robot must interact with multiple Markov Decision Processes (MDPs), collect a large number of training epochs, and retain a portion of the data for policy search after evaluating the adaptation difficulty. These constitute the key steps in the standard robust adaptive optimization process—evaluation-ranking-pruning. In robust task episode learning, some commonly used strategies consider risk information in task batches, such as scoring the difficulty of randomly sampled tasks during the adaptation phase and then selecting a subset for optimization. However, the following challenges remain:

[0008] (1) It incurs additional sample overhead during evaluation, especially when the decision-making environment is complex, and the evaluation of policy functions in deep reinforcement learning can be very expensive;

[0009] (2) The finite uniform sampling results in the task batch weaken the agent's effective exploration of the decision control task space.

[0010] In summary, existing technologies struggle to improve both adaptive robustness and computational efficiency of the model, a problem that urgently needs to be addressed. Summary of the Invention

[0011] This application provides a meta-reinforcement learning and domain randomization training method for model prediction task sampling, in order to solve the problem that existing technologies are unable to improve both adaptive robustness and computational efficiency of the model.

[0012] The first aspect of this application provides a meta-reinforcement learning and domain randomization training method for model prediction task sampling, comprising the following steps: generating a risk random function corresponding to the deep generative model based on an online constructed deep generative model, and combining the target latent variable and the target learning dataset, and determining the risk function distribution corresponding to the risk random function, so as to infer the approximate posterior of the risk function distribution using a preset streaming variational inference strategy, so as to obtain the corresponding posterior inference result; estimating the function posterior distribution corresponding to the deep generative model based on the posterior distribution of the function and a preset upper confidence bound principle, and constructing a target acquisition function using the target acquisition... The set function performs random sampling in the target task space to obtain the predicted value of the posterior task adaptive loss and the corresponding collected data. The collected data is then predicted using the predicted value of the posterior task adaptive loss and a preset risk learner to generate a target plot optimization task batch based on the collected data score and a preset task selection strategy. A preset domain randomization update rule and meta-reinforcement learning update rule for the machine learning are determined. Based on the target plot optimization task batch, the domain randomization update rule, and the meta-reinforcement learning update rule, the machine learning is subjected to a decision model update training operation in the target zero-shot adaptive model or the target few-shot adaptive model.

[0013] Optionally, in one embodiment of this application, the step of generating a risk random function corresponding to the deep generative model based on an online-constructed deep generative model, and combining the target latent variable and the target learning dataset to determine the risk function distribution corresponding to the risk random function, and using a preset streaming variational inference strategy to infer the approximate posterior of the risk function distribution to obtain the corresponding posterior inference result, includes: constructing the risk function distribution corresponding to the risk random function based on the target latent variable; encoding historical risk information through the target latent variable to determine the conditional prior corresponding to the risk function distribution; processing the target streaming task batch using the conditional prior and a preset recursive strategy to update the function posterior in the risk function distribution to obtain the corresponding approximate function posterior; determining the variational distribution according to a preset parameterized Gaussian distribution, and converting the target evidence lower bound into a target operable optimization objective function based on the approximate function posterior and the variational distribution, and obtaining the posterior inference result through the target operable optimization objective function.

[0014] Optionally, in one embodiment of this application, the step of randomly sampling in the target task space using the target acquisition function to obtain the predicted value of the posterior task adaptive loss and the corresponding acquisition data, and predicting the acquisition score of the acquisition data using the predicted value of the posterior task adaptive loss and a preset risk learner, and generating a target plot optimization task batch based on the acquisition score and a preset task screening strategy, includes: performing adaptive evaluation on task identifiers corresponding to sequential task batches in the target task space based on a preset simulation operation to obtain the corresponding acquisition score; sorting the task identifiers according to the acquisition score to obtain the sorting result of the task identifiers; and screening task identifiers that meet preset candidate requirements from the sorting result of the task identifiers to construct the target plot optimization task batch.

[0015] Optionally, in one embodiment of this application, the mathematical expression of the deep generative model is:

[0016]

[0017] Where p(z0) represents the prior of the initial task risk representation; p ψ (H t |z t ,θ t ) represents the situation where, given a decision model θ t and posterior representation z t The conditional probability likelihood of adaptive risk in task batches; θ t z represents the decision model parameters at time t; t H represents the target latent variable; t p(z) represents the target learning dataset; t+1 |z t ) represents the conditional transition of the batch task risk representation due to the optimization of the decision model; T represents the number of iterations when the final convergence occurs.

[0018] Optionally, in one embodiment of this application, the mathematical expression of the target acquisition function is:

[0019]

[0020] Where, m(l) i ) represents the mean risk; σ(l) i ) represents the standard deviation of the task adaptive risk estimated through random forward propagation; z t τ represents the target latent variable; q represents the task identifier; φ (z t |H t) represents the approximate posterior distribution encoded by the neural network; γ0, γ1 are hyperconstants; p(τ) represents the distribution of the task or task identifier; p ψ (l|τ i ,z t ) represents the input task identifier τ at time t for the optimization iteration. i and batch task risk characterization z t The conditional probability of the corresponding adaptive risk l; Indicates input τ i The expected or mean value of l is predicted. This represents the batch of tasks and identifiers selected using the Top-K criteria; Indicates input τ i The standard deviation of l is predicted; φ represents the neural network parameters of the approximate posterior encoder; ψ represents the neural network parameters of the adaptive risk decoder.

[0021] A second aspect of this application provides a meta-reinforcement learning and domain randomization training apparatus for model prediction task sampling, comprising: a posterior inference module, configured to generate a risk random function corresponding to the deep generative model based on an online constructed deep generative model, and in combination with a target latent variable and a target learning dataset, and determine the risk function distribution corresponding to the risk random function, so as to infer the approximate posterior of the risk function distribution using a preset streaming variational inference strategy, so as to obtain the corresponding posterior inference result; and a collection module, configured to estimate the function posterior distribution corresponding to the deep generative model based on the posterior inference result, and construct a target collection function according to the function posterior distribution and a preset upper confidence bound principle, and utilize the... The target acquisition function randomly samples in the target task space to obtain the predicted value of the posterior task adaptive loss and the corresponding acquisition data. It then uses the predicted value of the posterior task adaptive loss and a preset risk learner to predict the acquisition score of the acquisition data. Based on the acquisition score and a preset task selection strategy, it generates a target scenario optimization task batch. The update module determines preset domain randomization update rules and meta-reinforcement learning update rules for the machine learning machine. Based on the target scenario optimization task batch, the domain randomization update rules, and the meta-reinforcement learning update rules, it performs a decision model update training operation on the machine learning machine in either the target zero-shot adaptive model or the target few-shot adaptive model.

[0022] Optionally, in one embodiment of this application, the posterior inference module includes: a construction unit, configured to construct a risk function distribution corresponding to the risk random function based on the target latent variable; an encoding unit, configured to encode historical risk information through the target latent variable to determine the conditional prior corresponding to the risk function distribution; a processing unit, configured to process the target streaming task batch using the conditional prior and a preset recursive strategy to update the function posterior in the risk function distribution and obtain the corresponding approximate function posterior; and a transformation unit, configured to determine a variational distribution according to a preset parameterized Gaussian distribution, to transform the target evidence lower bound into a target operable optimization objective function based on the approximate function posterior and the variational distribution, and to obtain the posterior inference result through the target operable optimization objective function.

[0023] Optionally, in one embodiment of this application, the acquisition module includes: an adaptive evaluation unit, used to perform an adaptive evaluation on the task identifiers corresponding to the sequential task batches in the target task space based on a preset simulation operation, so as to obtain the corresponding acquisition score; a sorting unit, used to sort the task identifiers according to the acquisition score, so as to obtain the sorting result of the task identifiers; and a filtering unit, used to filter task identifiers that meet preset candidate requirements from the sorting result of the task identifiers, so as to construct the target plot optimization task batch.

[0024] Optionally, in one embodiment of this application, the mathematical expression of the deep generative model is:

[0025]

[0026] Where p(z0) represents the prior of the initial task risk representation; p ψ (H t |z t ,θ t ) represents the situation where, given a decision model θ t and posterior representation z t The conditional probability likelihood of adaptive risk in task batches; θ t z represents the decision model parameters at time t; t H represents the target latent variable; t p(z) represents the target learning dataset; t+1 |z t ) represents the conditional transition of the batch task risk representation due to the optimization of the decision model; T represents the number of iterations when the final convergence occurs.

[0027] Optionally, in one embodiment of this application, the mathematical expression of the target acquisition function is:

[0028]

[0029] Where, m(l) i ) represents the mean risk; σ(l) i ) represents the standard deviation of the task adaptive risk estimated through random forward propagation; z t τ represents the target latent variable; q represents the task identifier; φ (z t |H t ) represents the approximate posterior distribution encoded by the neural network; γ0, γ1 are hyperconstants; p(τ) represents the distribution of the task or task identifier; p ψ (l|τ i ,z t ) represents the input task identifier τ at time t for the optimization iteration. i and batch task risk characterization z t The conditional probability of the corresponding adaptive risk l; Indicates input τ i The expected or mean value of l is predicted. This represents the batch of tasks and identifiers selected using the Top-K criteria; Indicates input τ i The standard deviation of l is predicted; φ represents the neural network parameters of the approximate posterior encoder; ψ represents the neural network parameters of the adaptive risk decoder.

[0030] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the meta-reinforcement learning and domain randomization training method for model prediction task sampling as described in the above embodiments.

[0031] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the meta-reinforcement learning and domain randomization training method for model prediction task sampling as described above.

[0032] A fifth aspect of this application provides a computer program product, including a computer program that is executed to implement the above-described meta-reinforcement learning and domain randomization training method for model prediction task sampling.

[0033] Therefore, the embodiments of this application have the following beneficial effects:

[0034] The embodiments of this application can generate a risk random function corresponding to the deep generative model based on an online-built deep generative model, combined with target latent variables and target learning datasets, and determine the risk function distribution corresponding to the risk random function. An approximate posterior of the risk function distribution is then inferred using a preset streaming variational inference strategy to obtain the corresponding posterior inference result. Based on the posterior inference result, the posterior distribution of the function corresponding to the deep generative model is estimated, and a target acquisition function is constructed according to the function posterior distribution and a preset upper confidence bound principle. Random sampling is then performed in the target task space using the target acquisition function to obtain the predicted value of the posterior task adaptive loss and the corresponding acquired data. This data is then processed through... The method uses the predicted value of the adaptive loss for the experimental task and the collection score of the collected data predicted by the pre-set risk learner to generate target plot optimization task batches based on the collection scores and the pre-set task selection strategy. It determines the pre-set domain randomization update rule and meta-reinforcement learning update rule for the machine learning, and performs decision model update training operations on the machine learning in the adaptive model or algorithm with zero-shot or few-shot targets based on the target plot optimization task batches, domain randomization update rules, and meta-reinforcement learning update rules. This reduces interaction with the decision-making environment, achieving adaptive robustness while ensuring model learning efficiency. Therefore, it solves the problem that existing technologies struggle to improve adaptive robustness while simultaneously improving model computational efficiency.

[0035] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0036] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0037] Figure 1 This is a flowchart of a meta-reinforcement learning and domain randomization training method for model prediction task sampling according to an embodiment of this application;

[0038] Figure 2 This is an example diagram of a meta-reinforcement learning and domain randomization training apparatus for model prediction task sampling according to an embodiment of this application;

[0039] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0040] Among them, 10 is the meta-reinforcement learning and domain randomization training device for model prediction task sampling; 100 is the posterior inference module, 200 is the acquisition module, and 300 is the update module; 301 is the memory, 302 is the processor, and 303 is the communication interface. Detailed Implementation

[0041] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0042] The following describes a meta-reinforcement learning and domain randomization training method for model prediction task sampling according to embodiments of this application, with reference to the accompanying drawings. Addressing the problems mentioned in the background art, this application provides a meta-reinforcement learning and domain randomization training method for model prediction task sampling. In this method, a risk random function corresponding to the deep generative model is generated based on an online-constructed deep generative model, combined with the target latent variable and the target learning dataset. The risk function distribution corresponding to the risk random function is determined, and an approximate posterior of the risk function distribution is inferred using a preset streaming variational inference strategy to obtain the corresponding posterior inference result. Based on the posterior inference result, the posterior distribution of the function corresponding to the deep generative model is estimated, and a target acquisition function is constructed according to the function posterior distribution and a preset upper confidence bound principle. The target acquisition function is then used to perform randomization in the target task space. Sampling is performed to obtain the predicted value of the posterior task adaptive loss and the corresponding collected data. The predicted value of the posterior task adaptive loss and a pre-defined risk learner are used to predict the collection score of the collected data. Based on the collection score and a pre-defined task selection strategy, a target scenario optimization task batch is generated. Pre-defined domain randomization update rules and meta-reinforcement learning update rules for the machine learning are determined. Based on the target scenario optimization task batch, domain randomization update rules, and meta-reinforcement learning update rules, the machine learning performs decision model update training operations in either the target zero-shot adaptive model or the target few-shot adaptive model. This reduces interaction with the decision-making environment, achieving adaptive robustness while ensuring model learning efficiency. Therefore, this solves the problem that existing technologies struggle to improve adaptive robustness while simultaneously improving model computational efficiency.

[0043] To facilitate understanding of the execution process of this application by those skilled in the art, the execution logic and the roles involved in this application are briefly described below:

[0044] This application focuses on achieving robust adaptability based on practical needs, while ensuring learning efficiency. For example, it avoids task selection steps with little information (such as labeling or evaluation) and eliminates unnecessary computations, thereby achieving adaptability in real-world scenarios and resulting in the Model Predictive Task Sampling (MPTS) strategy. The optimization process in this application mainly involves the following roles:

[0045] (1) The model optimizer specifies an adaptive machine learning machine and learns in any number of ways;

[0046] (2) Risk learners act as critics to assess and predict task-specific risks;

[0047] (3) The task sampler acts as an actor to select the task batch for the next iteration. These components operate in a cyclical manner until convergence.

[0048] Specifically, Figure 1 This is a flowchart illustrating a meta-reinforcement learning and domain randomization training method for model prediction task sampling provided in an embodiment of this application.

[0049] like Figure 1 As shown, the meta-reinforcement learning and domain randomization training method for the model's prediction task sampling includes the following steps:

[0050] In step S101, based on the online constructed deep generative model, and combined with the target latent variable and the target learning dataset, a risk random function corresponding to the deep generative model is generated, and the risk function distribution corresponding to the risk random function is determined. An approximate posterior of the risk function distribution is inferred using a preset streaming variational inference strategy to obtain the corresponding posterior inference result.

[0051] Those skilled in the art will understand that in task scenario training, the task selection strategy is closely related to the optimization process and the generalization ability of the machine learning machine. Therefore, embodiments of this application may employ a risk landscape-based sampling strategy, and through... As a training dataset, it is used to model a risk stochastic function.

[0052] To describe the adaptive risks during batch optimization, this application introduces a latent variable z to summarize historical information about the risks and to build a general deep generative model online.

[0053] Optionally, in one embodiment of this application, the mathematical expression of the deep generative model is:

[0054]

[0055] Where p(z0) represents the prior of the initial task risk representation; p ψ (H t |z t ,θ t ) represents the situation where, given a decision model θ t and posterior representation z t The conditional probability likelihood of adaptive risk in task batches; θ t z represents the decision model parameters at time t; t H represents the target latent variable; tp(z) represents the target learning dataset; t+1 |z t ) represents the conditional transition of the batch task risk representation due to the optimization of the decision model; T represents the number of iterations when the final convergence occurs.

[0056] It should be noted that the mathematical expression of the deep generative model constructed in this embodiment is as follows:

[0057]

[0058] Where p(z0) represents the prior of the initial task risk representation; p ψ (H t |z t ,θ t ) represents the situation where, given a decision model θ t and posterior representation z t The conditional probability likelihood of adaptive risk in task batches; θ t z represents the decision model parameters at time t; t H represents the target latent variable; t This represents the target learning dataset, which contains the set of identifiers and adaptive risk values ​​for the batch of tasks collected; p(z) t+1 |z t ) represents the conditional transition of the batch task risk representation due to the optimization of the decision model; T represents the number of iterations when the final convergence occurs.

[0059] Within the Bayesian framework, embodiments of this application can approximate the latent function distribution through latent variables, where the posterior p(z) t |H t This summarizes historical risk information and considers the uncertainty of the distribution. According to Bayes' theorem, p(z) t |H t The form is as follows:

[0060]

[0061] Where p(z) t |H 1:t-1 As a conditional prior, p(H) encodes the results of past collection and evaluation; while p(H) t |z t The expression represents the probability of generating the task batch risk value observation in the t-th iteration; it should be noted that due to the complexity of the integral in the denominator, the precise calculation of the posterior is not feasible.

[0062] Optionally, in one embodiment of this application, a risk random function corresponding to the deep generative model is generated based on an online-constructed deep generative model, combined with the target latent variables and the target learning dataset. The risk function distribution corresponding to the risk random function is determined, and an approximate posterior of the risk function distribution is inferred using a preset streaming variational inference strategy to obtain the corresponding posterior inference result. This includes: constructing a risk function distribution corresponding to the risk random function based on the target latent variables; encoding historical risk information through the target latent variables to determine the conditional prior corresponding to the risk function distribution; processing the target streaming task batch using the conditional prior and a preset recursive strategy to update the function posterior in the risk function distribution and obtain the corresponding approximate function posterior; determining a variational distribution according to a preset parameterized Gaussian distribution; converting the target evidence lower bound into a target operable optimization objective function based on the approximate function posterior and the variational distribution; and obtaining the posterior inference result through the target operable optimization objective function.

[0063] In the embodiments of this application, the risk value of the task batch of interest is related to the parameter θ of the meta-learner. Specifically, the variable decomposition related to sequence batch optimization is as follows:

[0064]

[0065] In a probabilistic graphical model, the latent variable z t The distribution constituting the risk function (for simplicity, other variables unrelated to the learning objective are omitted in this embodiment). Here, this embodiment may assume that, given the task identifier τ, latent variable z, and model parameters θ, the task-specific risk values ​​are conditionally independent, and the primary optimization objective is... Used for learning stochastic risk functions.

[0066] Furthermore, since the training of the plot involves task batches and evaluation values ​​arriving sequentially, embodiments of this application can utilize these correlations and learn quantities that aid in selecting task batches. However, due to the lack of explicit structural information in the posterior distribution, accurate inference of p(z) is challenging. t |H t This is not feasible.

[0067] It should be noted that in each iteration, the risk function distribution depends on the updated machine learning machine parameters θ; therefore, this non-stationarity in the risk function distribution allows the embodiments of this application to use streaming variational inference to obtain an approximate posterior.

[0068] Specifically, embodiments of this application process streaming task batches and update post-tests recursively, as shown in the following equation:

[0069]

[0070] Wherein, p(z)t |H 1:t-1 ) represents the conditional prior of the last updated posterior. The estimated functional posterior is used to provide uncertainty-aware predictions on stochastic risk functions and to serve the design of task sampling strategies.

[0071] Therefore, through approximate inference, the embodiments of this application can formalize the Evidence Lower Bound (ELBO) into an operable optimization objective, as shown in the following formula:

[0072]

[0073] For ease of practical implementation, embodiments of this application may use a parameterized Gaussian distribution with a diagonal covariance matrix as the variational distribution, similar to the Vanilla VAE. The parameters of these distributions are learned through a neural network, for example... And use the reparameterization technique for stochastic gradient estimation.

[0074] Regarding the neural network architecture, embodiments of this application can employ a Deepset encoding module to process ensemble datasets. Furthermore, in the context of streaming variational inference, q φ (z t |H t-1 This is typically used as a default conditional prior agent. Therefore, embodiments of this application can modify the exact ELBO formula and transform it into an actual optimization process using the Lagrange multiplier β:

[0075]

[0076]

[0077] in, This indicates that the gradient will not be calculated using φ in this term, and Used to constrain the range of parameters searched by the machine learning machine in the next iteration.

[0078] In step S102, based on the posterior inference results, the posterior distribution of the function corresponding to the deep generative model is estimated, and a target acquisition function is constructed according to the posterior distribution of the function and the preset upper confidence bound principle. The target acquisition function is used to randomly sample in the target task space to obtain the predicted value of the posterior task adaptive loss and the corresponding acquisition data. The acquisition score of the acquisition data is predicted by the predicted value of the posterior task adaptive loss and the preset risk learner, so as to generate the target plot optimization task batch according to the acquisition score and the preset task screening strategy.

[0079] Understandably, in robust task-based learning, some common strategies consider risk information in task batches, such as scoring the difficulty of randomly sampled tasks during the adaptation phase and then selecting a subset for optimization. However, the following challenges remain:

[0080] (1) It introduces additional computational overhead during evaluation, especially when the model complexity is extremely high;

[0081] (2) In data-driven control tasks, evaluation through interaction with the environment can be very expensive (e.g., deep reinforcement learning).

[0082] (3) The finite uniform sampling results in the task batch weaken the exploration of the task space.

[0083] To address the aforementioned issues, this application assumes that the task's plot optimization process is predictable and constructs a risk learner to overcome the learning resource limitations encountered during robust optimization. The stochastic risk function generated by the generative model can be used to sequentially query task batches. The basic principle is that, in this application's embodiments, the learned risk learner can store rich historical task risk information during the iteration process and can incorporate specific rules into the sampling function, thereby achieving effective sampling.

[0084] It should be noted that an important application of risk learners is to support MPTS, which replaces the precise evaluation in all test scenarios by providing a pseudo-evaluation of the adaptive performance of the machine learning machine; in other words, MPTS does not require additional manual or high-overhead runtime evaluation, nor does it require interaction with the real world to modify the task sampling results.

[0085] Optionally, in one embodiment of this application, random sampling is performed in the target task space using a target acquisition function to obtain the predicted value of the posterior task adaptive loss and the corresponding acquisition data. The acquisition score of the acquisition data is predicted by the predicted value of the posterior task adaptive loss and a preset risk learner. The target plot optimization task batch is generated based on the acquisition score and a preset task selection strategy. This includes: performing adaptive evaluation on the task identifiers corresponding to the sequential task batches in the target task space based on preset simulation operations to obtain the corresponding acquisition scores; sorting the task identifiers according to the acquisition scores to obtain the sorting results of the task identifiers; and selecting task identifiers that meet preset candidate requirements from the sorting results of the task identifiers to construct the target plot optimization task batch.

[0086] In practical implementation, embodiments of this application can use the constructed deep generative model and the estimated posterior distribution of the function as tools for actively selecting task batches. Specifically, the prediction distribution in the form of the posterior function can be expressed as:

[0087]

[0088] It is understandable that the above predicted distribution p ψ (l|τ,H 1:t Thanks to Bayesian modeling, a feasible way is provided to assess the task difficulty of the entire task space without requiring expensive computation.

[0089] Subsequently, embodiments of this application may sort and prune the next batch of tasks for plot training.

[0090] Specifically, upon obtaining p ψ (l|τ,H 1:t After that, the embodiments of this application formulate a task batch sampling strategy based on its quantitative statistical data. The corresponding criteria can refer to the sampling function in classical Bayesian optimization, including evaluation criteria such as expected improvement, output information theoretical indicators or upper confidence bound.

[0091] However, it is important to note that the search space is on sequential task batches, not on the parameters of the machine learning machine, which differs from the final objective in classic Bayesian optimization. The core idea of ​​this application's embodiments is the Optimism of the Face of Uncertainty (OFU) principle; furthermore, since the worst-performing subset of tasks should receive extra attention to optimize adaptive robustness; and since high-prediction-uncertainty task regions are often underestimated in the last few rounds of meta-training, this application's embodiments can consider the worst-case scenario in adaptation and the uncertainty in the prediction function as key elements in developing the acquisition function. Therefore, this application's embodiments can construct a corresponding acquisition function based on the upper confidence bound principle.

[0092] Optionally, in one embodiment of this application, the mathematical expression of the target acquisition function is:

[0093]

[0094] Where, m(l) i ) represents the mean risk; σ(l) i ) represents the standard deviation of the task adaptive risk estimated through random forward propagation; z t τ represents the latent variable of the target; τ represents the task identifier; q φ (z t |H t ) represents the approximate posterior distribution encoded by the neural network; γ0, γ1 are hyperconstants; p(τ) represents the distribution of the task or task identifier; p ψ (l|τ i ,z t ) represents the input task identifier τ at time t for the optimization iteration. iand batch task risk characterization z t The conditional probability of the corresponding adaptive risk l; Indicates input τ i The expected or mean value of l is predicted. This represents the batch of tasks and identifiers selected using the Top-K criteria; Indicates input τ i The standard deviation of l is predicted; φ represents the neural network parameters of the approximate posterior encoder; ψ represents the neural network parameters of the adaptive risk decoder.

[0095] It should be noted that the mathematical expression of the corresponding acquisition function (i.e., the target acquisition function) based on the upper confidence bound principle is shown in the following equation:

[0096]

[0097] Where, m(l) i ) represents the mean risk; σ(l) i ) represents the standard deviation of the task adaptive risk estimated through random forward propagation; z t τ represents the latent variable of the target; τ represents the task identifier; q φ (z t |H t ) represents the approximate posterior distribution encoded by the neural network; γ0, γ1 are hyperconstants; p(τ) represents the distribution of the task or task identifier; p ψ (l|τ i ,z t ) represents the input task identifier τ at time t for the optimization iteration. i and batch task risk characterization z t The conditional probability of the corresponding adaptive risk l; Indicates input τ i The expected or mean value of l is predicted. This represents the batch of tasks and identifiers selected using the Top-K criteria; Indicates input τ i The standard deviation of l is predicted; φ represents the neural network parameters of the approximate posterior encoder; ψ represents the neural network parameters of the adaptive risk decoder.

[0098] Subsequently, embodiments of this application can perform simulation-sorting-pruning operations on task batches in iteration t+1:

[0099]

[0100] The above formula represents the steps of actively selecting task batches, and there is no parsed form in the maximization.

[0101] In the specific implementation process, embodiments of this application can simulate and pseudo-evaluate the adaptive performance of a batch of task identifiers, sort these identifiers according to their respective collection scores, and prune the identifier batch to further query the task dataset and optimize the machine learning machine. The above steps approximate the solution to the above formula in a heuristic manner, and obtain...

[0102] In other words, the embodiments of this application still perform random sampling in the task space and predict the corresponding sampling scores from the risk learner. And filter out Candidates from utility values ​​are used to generate task batches for plot optimization.

[0103] It is understood that the design of the MPTS acquisition function in this application embodiment can theoretically improve the adaptive performance in the worst-case scenario, and improve robustness even when the task distribution is severely shifted. In addition, the lightweight risk learner replaces the computationally expensive parts of optimization and evaluation, thereby effectively reducing computational overhead and balancing utilization and exploration in the task space in specific scenarios.

[0104] Therefore, the embodiments of this application establish a mapping relationship between the task space and the adaptive loss through the model prediction task sampling method, thereby reducing unnecessary interactions with the decision-making environment and policy evaluation, and resulting in a lower task sampling volume. In addition, the embodiments of this application strongly support the design of task sampling criteria by constructing a task sampling function and an adaptive risk prediction proxy model based on flow Bayesian inference.

[0105] In step S103, the preset domain randomization update rule and meta-reinforcement learning update rule of the machine learning machine are determined, and the task batch, domain randomization update rule and meta-reinforcement learning update rule are optimized according to the target plot. The machine learning machine is then subjected to decision model update training operations in the target zero-shot adaptive model or the target few-shot adaptive model.

[0106] After the task batch was trimmed Embodiments of this application can perform standard model optimization steps to update the parameters of the machine learning engine. The task-specific adaptation risk in iteration t+1 is represented as τ for the sampled task. i l t+1,i (θ); Thus, in the embodiments of this application, MPTS can approximately predict adaptation risk and can be seamlessly integrated into the adaptive learning paradigm of domain randomization (zero-shot) and meta-reinforcement learning (few-shot), as described below:

[0107] (1) Updates of machine learning machines in zero-shot adaptive models or algorithms (domain randomization)

[0108] Since the zero-shot setting does not require a supporting dataset to identify the task, the following update rules can be obtained in this application embodiment, taking the traditional domain randomization method as an example:

[0109]

[0110] Where θ represents the parameters of the zero-shot learning model, and λ is the learning rate.

[0111] (2) Machine learning machine updates in few-shot adaptive models or algorithms (meta-reinforcement learning)

[0112] This application uses the typical optimization-based method MAML as an example and provides the following update rules:

[0113]

[0114] Where, θ meta Denotes the meta-initialization of the model, λ 1,1 and λ 1,2 These represent the learning rates in the inner and outer loops, respectively.

[0115] Therefore, the embodiments of this application require fewer learning resources, thereby reducing interaction with the decision-making environment and better preserving the adaptive robustness of zero-shot and few-shot datasets.

[0116] According to the meta-reinforcement learning and domain randomization training method for model prediction task sampling proposed in this application, a risk random function corresponding to the deep generative model is generated based on an online constructed deep generative model, combined with the target latent variable and the target learning dataset. The risk function distribution corresponding to the risk random function is determined, and an approximate posterior of the risk function distribution is inferred using a preset streaming variational inference strategy to obtain the corresponding posterior inference result. Based on the posterior inference result, the posterior distribution of the function corresponding to the deep generative model is estimated, and a target acquisition function is constructed according to the function posterior distribution and a preset upper confidence bound principle. Random sampling is then performed in the target task space using the target acquisition function to obtain the posterior task... The system predicts the adaptive loss value and the corresponding collected data, and uses the predicted value of the posterior adaptive loss and the preset risk learner to predict the collection score of the collected data. Based on the collection score and the preset task selection strategy, it generates target plot optimization task batches. It determines the preset domain randomization update rule and meta-reinforcement learning update rule of the machine learning machine, and performs decision model update training operations on the machine learning machine in the target zero-shot adaptive model or the target few-shot adaptive model according to the target plot optimization task batch, domain randomization update rule and meta-reinforcement learning update rule. This reduces the interaction with the decision environment and ensures the learning efficiency of the model while achieving adaptive robustness.

[0117] Secondly, with reference to the accompanying drawings, a meta-reinforcement learning and domain randomization training apparatus for model prediction task sampling according to an embodiment of this application is described.

[0118] Figure 2 This is a block diagram of a meta-reinforcement learning and domain randomization training apparatus for model prediction task sampling according to an embodiment of this application.

[0119] like Figure 2 As shown, the meta-reinforcement learning and domain randomization training device 10 for the model prediction task sampling includes: a posterior inference module 100, a collection module 200, and an update module 300.

[0120] The posterior inference module 100 is used to generate a risk random function corresponding to the deep generative model based on the online constructed deep generative model, and in combination with the target latent variable and the target learning dataset. It also determines the risk function distribution corresponding to the risk random function, and uses a preset streaming variational inference strategy to infer the approximate posterior of the risk function distribution to obtain the corresponding posterior inference result.

[0121] The acquisition module 200 is used to estimate the posterior distribution of the function corresponding to the deep generative model based on the posterior inference results, construct the target acquisition function according to the posterior distribution of the function and the preset upper confidence bound principle, and use the target acquisition function to randomly sample in the target task space to obtain the predicted value of the posterior task adaptive loss and the corresponding acquisition data. The acquisition score of the acquisition data is predicted by the predicted value of the posterior task adaptive loss and the preset risk learner, so as to generate the target plot optimization task batch according to the acquisition score and the preset task screening strategy.

[0122] The update module 300 is used to determine the preset domain randomization update rule and meta-reinforcement learning update rule of the machine learning machine, and optimize the task batch, domain randomization update rule and meta-reinforcement learning update rule according to the target plot, and perform decision model update training operation on the machine learning machine in the target zero-shot adaptive model or the target few-shot adaptive model.

[0123] Optionally, in one embodiment of this application, the posterior inference module 100 includes: a construction unit, an encoding unit, a processing unit, and a conversion unit.

[0124] The construction unit is used to construct the risk function distribution corresponding to the risk random function based on the target latent variable.

[0125] The encoding unit is used to encode historical risk information through target latent variables in order to determine the conditional priors corresponding to the risk function distribution.

[0126] The processing unit is used to process the target streaming task batch using conditional priors and a preset recursive strategy to update the function posterior in the risk function distribution and obtain the corresponding approximate function posterior.

[0127] The transformation unit is used to determine the variational distribution based on the preset parameterized Gaussian distribution, so as to transform the lower bound of the target evidence into a target operable optimization objective function based on the posterior of the approximate function and the variational distribution, and obtain the posterior inference result through the target operable optimization objective function.

[0128] Optionally, in one embodiment of this application, the acquisition module 200 includes: an adaptability evaluation unit, a sorting unit, and a filtering unit.

[0129] The adaptive evaluation unit is used to adaptively evaluate the task identifiers corresponding to the sequential task batches in the target task space based on preset simulation operations, so as to obtain the corresponding acquisition scores.

[0130] The sorting unit is used to sort the task identifiers according to the collection scores to obtain the sorting result of the task identifiers.

[0131] The filtering unit is used to filter task identifiers that meet preset candidate requirements from the sorting results of task identifiers in order to construct a target plot optimization task batch.

[0132] Optionally, in one embodiment of this application, the mathematical expression of the deep generative model is:

[0133]

[0134] Where p(z0) represents the prior of the initial task risk representation; p ψ (H t |z t ,θ t ) represents the situation where, given a decision model θ t and posterior representation z t The conditional probability likelihood of adaptive risk in task batches; θ t z represents the decision model parameters at time t; t H represents the target latent variable; t p(z) represents the target learning dataset; t+1 |z t ) represents the conditional transition of the batch task risk representation due to the optimization of the decision model; T represents the number of iterations when the final convergence occurs.

[0135] Optionally, in one embodiment of this application, the mathematical expression of the target acquisition function is:

[0136]

[0137] Where, m(l) i ) represents the mean risk; σ(l) i ) represents the standard deviation of the task adaptive risk estimated through random forward propagation; z t τ represents the latent variable of the target; τ represents the task identifier; q φ (z t |H t ) represents the approximate posterior distribution encoded by the neural network; γ0, γ1 are hyperconstants; p(τ) represents the distribution of the task or task identifier; p ψ (l|τ i ,z t ) represents the input task identifier τ at time t for the optimization iteration. i and batch task risk characterization z t The conditional probability of the corresponding adaptive risk l; Indicates input τ i The expected or mean value of l is predicted. This represents the batch of tasks and identifiers selected using the Top-K criteria; Indicates input τ iThe standard deviation of l is predicted; φ represents the neural network parameters of the approximate posterior encoder; ψ represents the neural network parameters of the adaptive risk decoder.

[0138] It should be noted that the explanation of the aforementioned embodiment of the meta-reinforcement learning and domain randomization training method for sampling model prediction tasks also applies to the meta-reinforcement learning and domain randomization training device for sampling model prediction tasks in this embodiment, and will not be repeated here.

[0139] The meta-reinforcement learning and domain randomization training device for model prediction task sampling proposed in this application includes a posterior inference module, used to generate a risk random function corresponding to the deep generative model based on an online constructed deep generative model, combined with the target latent variable and the target learning dataset, and determine the risk function distribution corresponding to the risk random function, so as to infer the approximate posterior of the risk function distribution using a preset streaming variational inference strategy to obtain the corresponding posterior inference result; and a collection module, used to estimate the posterior distribution of the function corresponding to the deep generative model based on the posterior inference result, and construct a target collection function according to the function posterior distribution and a preset upper confidence bound principle, and use the target collection function to perform random sampling in the target task space, so as to obtain the approximate posterior of the risk function distribution. The system obtains the predicted value of the posterior task adaptive loss and the corresponding collected data. It then uses the predicted value of the posterior task adaptive loss and the preset risk learner to predict the collection score of the collected data. Based on the collection score and the preset task selection strategy, it generates a target plot optimization task batch. The update module is used to determine the preset domain randomization update rule and meta-reinforcement learning update rule of the machine learning machine. Based on the target plot optimization task batch, domain randomization update rule, and meta-reinforcement learning update rule, it performs decision model update training operations on the machine learning machine in the target zero-shot adaptive model or the target few-shot adaptive model. This reduces the interaction with the decision environment and ensures the learning efficiency of the model while achieving adaptive robustness.

[0140] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0141] The memory 301, the processor 302, and the computer program stored on the memory 301 and capable of running on the processor 302.

[0142] When the processor 302 executes the program, it implements the meta-reinforcement learning and domain randomization training method for model prediction task sampling provided in the above embodiments.

[0143] Furthermore, electronic devices also include:

[0144] Communication interface 303 is used for communication between memory 301 and processor 302.

[0145] The memory 301 is used to store computer programs that can run on the processor 302.

[0146] The memory 301 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0147] If the memory 301, processor 302, and communication interface 303 are implemented independently, then the communication interface 303, memory 301, and processor 302 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0148] Optionally, in a specific implementation, if the memory 301, processor 302, and communication interface 303 are integrated on a single chip, then the memory 301, processor 302, and communication interface 303 can communicate with each other through an internal interface.

[0149] Processor 302 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0150] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for meta-reinforcement learning and domain randomization training of model prediction task sampling.

[0151] This application also provides a computer program product, including a computer program that, when executed, is used to implement the above-described meta-reinforcement learning and domain randomization training method for model prediction task sampling.

[0152] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0153] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0154] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0155] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0156] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0157] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0158] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0159] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A meta-reinforcement learning and domain randomization training method for model prediction task sampling applied to robot control tasks, characterized in that, Includes the following steps: The robot collects task batch data by interacting with multiple Markov decision processes and constructs a target learning dataset containing a set of identifiers for the collected batches of tasks and adaptive risk values. Based on the online constructed deep generative model, and combined with the target latent variable and the target learning dataset, a risk random function corresponding to the deep generative model is generated, and the risk function distribution corresponding to the risk random function is determined. Then, an approximate posterior of the risk function distribution is inferred using a preset streaming variational inference strategy to obtain the corresponding posterior inference result. Based on the posterior inference results, the posterior distribution of the function corresponding to the deep generation model is estimated, and a target acquisition function is constructed according to the posterior distribution of the function and a preset upper confidence bound principle. The target acquisition function is used to randomly sample in the target task space to obtain the predicted value of the posterior task adaptive loss and the corresponding acquisition data. The acquisition score of the acquisition data is predicted by the predicted value of the posterior task adaptive loss and a preset risk learner. The target plot optimization task batch is generated according to the acquisition score and a preset task screening strategy. Determine the preset domain randomization update rule and meta-reinforcement learning update rule for the machine learning machine, and optimize the task batch, the domain randomization update rule and the meta-reinforcement learning update rule according to the target plot, and perform decision model update training operation on the machine learning machine in the target zero-shot adaptive model or the target few-shot adaptive model. The domain randomization update rule is used to randomize at least one of the physical parameters, object shape, texture and lighting in the training environment in which the robot is located, so that the robot has the ability to transfer from simulation to reality. The mathematical expression for the target acquisition function is: in, Indicates the mean risk; This represents the adaptive risk standard deviation of the task estimated through random forward propagation; Indicates the target latent variable; Indicates the task identifier; This represents the approximate posterior distribution encoded by a neural network; It is a superconstant; Represents the distribution of tasks or task identifiers; Optimize the input task identifier in real time. Batch task risk characterization Corresponding adaptive risk The conditional probability; Indicates input Predicted The expected or mean value; This represents the batch of tasks and identifiers selected using the Top-K criteria; Indicates input Predicted Standard deviation; The neural network parameters represent the approximate posterior encoder; This represents the neural network parameters of the adaptive risk decoder.

2. The method according to claim 1, characterized in that, The method involves using an online-constructed deep generative model, combining the target latent variables and the target learning dataset to generate a risk random function corresponding to the deep generative model, and determining the risk function distribution corresponding to the risk random function. A preset streaming variational inference strategy is then used to infer an approximate posterior of the risk function distribution to obtain the corresponding posterior inference result, including: Based on the target latent variables, construct the risk function distribution corresponding to the risk random function; Historical risk information is encoded by the target latent variables to determine the conditional priors corresponding to the risk function distribution; The target streaming task batch is processed using the aforementioned conditional prior and a preset recursive strategy to update the function posterior in the risk function distribution and obtain the corresponding approximate function posterior. The variational distribution is determined based on the preset parameterized Gaussian distribution. Based on the posterior of the approximate function and the variational distribution, the lower bound of the target evidence is transformed into a target operable optimization objective function, and the posterior inference result is obtained through the target operable optimization objective function.

3. The method according to claim 1, characterized in that, The process involves randomly sampling in the target task space using the target acquisition function to obtain the predicted value of the posterior task adaptive loss and the corresponding acquired data. The acquired data is then predicted using the predicted value of the posterior task adaptive loss and a preset risk learner to generate a target scenario optimization task batch based on the acquired data score and a preset task selection strategy. This includes: Based on a preset simulation operation, an adaptive evaluation is performed on the task identifiers corresponding to the sequential task batches in the target task space to obtain the corresponding collection scores. The task identifiers are sorted according to the collected scores to obtain the sorting result of the task identifiers; Task identifiers that meet preset candidate requirements are selected from the sorting results of the task identifiers to construct the target plot optimization task batch.

4. The method according to claim 1, characterized in that, The mathematical expression for the deep generative model is: in, The prior information represents the initial risk characterization of the task. In a given decision model and posterior representation The conditional probability likelihood of adaptive risk in task batches; express Parameters of the decision model at any given time; Indicates the target latent variable; This represents the target learning dataset; This indicates the conditional shift in the risk representation of batch tasks due to the optimization of the decision model; This indicates the number of iterations required for final convergence.

5. A meta-reinforcement learning and domain randomization training apparatus for model prediction task sampling applied to robot control tasks, as described in any one of claims 1-4, characterized in that, include: The posterior inference module is used to generate a risk random function corresponding to the deep generative model based on the online constructed deep generative model, and in combination with the target latent variable and the target learning dataset, and to determine the risk function distribution corresponding to the risk random function, so as to infer the approximate posterior of the risk function distribution using a preset streaming variational inference strategy, so as to obtain the corresponding posterior inference result. The acquisition module is used to estimate the posterior distribution of the function corresponding to the deep generative model based on the posterior inference result, construct a target acquisition function according to the posterior distribution of the function and a preset upper confidence bound principle, and use the target acquisition function to randomly sample in the target task space to obtain the predicted value of the posterior task adaptive loss and the corresponding acquisition data. The module also uses the predicted value of the posterior task adaptive loss and a preset risk learner to predict the acquisition score of the acquisition data, and generates a target plot optimization task batch according to the acquisition score and a preset task screening strategy. The update module is used to determine the preset domain randomization update rule and meta-reinforcement learning update rule of the machine learning machine, and to perform decision model update training operations on the machine learning machine in the target zero-shot adaptive model or the target few-shot adaptive model according to the target plot optimization task batch, the domain randomization update rule and the meta-reinforcement learning update rule.

6. The apparatus according to claim 5, characterized in that, The posterior inference module includes: The construction unit is used to construct the risk function distribution corresponding to the risk random function based on the target latent variable; The encoding unit is used to encode historical risk information through the target latent variable in order to determine the conditional prior corresponding to the risk function distribution; The processing unit is used to process the target streaming task batch using the conditional prior and the preset recursive strategy, so as to update the function posterior in the risk function distribution and obtain the corresponding approximate function posterior. The transformation unit is used to determine the variational distribution according to the preset parameterized Gaussian distribution, so as to transform the lower bound of the target evidence into a target operable optimization objective function based on the posterior of the approximate function and the variational distribution, and obtain the posterior inference result through the target operable optimization objective function.

7. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, the processor executing the program to implement the meta-reinforcement learning and domain randomization training method for model prediction task sampling applied to robot control tasks as described in any one of claims 1-4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the meta-reinforcement learning and domain randomization training method for model prediction task sampling applied to robot control tasks as described in any one of claims 1-4.

9. A computer program product, comprising a computer program, characterized in that, The computer program is executed to implement the meta-reinforcement learning and domain randomization training method for model prediction task sampling applied to robot control tasks as described in any one of claims 1-4.

Images