Model learning method, device and equipment based on variational discriminator and storage medium

Abstract

The embodiment of the invention provides a model learning method and device based on a variational discriminator, equipment and a storage medium. The method comprises the following steps: acquiring a skill variable from skill prior distribution information, and acquiring a discriminator parameter from parameter prior distribution information; the skill prior distribution information represents probability distribution of skill variables of the current time step, the skill variables are skills to be learned by the model, and the parameter prior distribution information represents probability distribution of parameters of a variation discriminator of the current time step; according to the skill variables and the discriminator parameters, based on the current environment state and the execution strategy, determining skill posterior distribution information; a scene applied by the environment state representation model executes a strategy of the strategy representation model for skill learning, and the skill posterior distribution information represents probability distribution of skill variables of the next time step; according to the skill prior distribution information and the skill posterior distribution information, updating the execution strategy to obtain a target strategy; the target strategy is used for indicating the model to learn skills.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model learning method, apparatus, device and storage medium based on a variational discriminator. Background Technology

[0002] In the marketing field, intelligent agents need to interact dynamically with the environment through language models to complete complex tasks, such as predicting the effectiveness of marketing campaigns, correcting campaign configurations, generating marketing copy, and planning campaigns.

[0003] However, traditional methods struggle to drive agents to autonomously learn diverse skills in the absence of explicit reward signals. Therefore, improving the model learning ability of agents is a pressing issue that needs to be addressed. Summary of the Invention

[0004] This application provides a model learning method, apparatus, device, and storage medium based on a variational discriminator to improve the learning ability of a model.

[0005] In a first aspect, embodiments of this application provide a model learning method based on a variational discriminator, including:

[0006] Skill variables are obtained from prior skill distribution information, and discriminator parameters are obtained from prior parameter distribution information; wherein, the prior skill distribution information represents the probability distribution of skill variables at the current time step, the skill variables represent the skills that the model wants to learn, the prior parameter distribution information represents the probability distribution of the parameters of the variational discriminator at the current time step, and the discriminator parameters represent the parameters of the variational discriminator.

[0007] Based on the skill variables and the discriminator parameters, and considering the current environmental state and the current execution strategy, the skill posterior distribution information is determined; wherein, the environmental state represents the scenario in which the model is applied, the execution strategy represents the strategy by which the model learns skills, and the skill posterior distribution information represents the probability distribution of the skill variables at the next time step;

[0008] Based on the prior skill distribution information and the posterior skill distribution information, the current execution strategy is updated to obtain the target strategy; wherein, the target strategy is used to instruct the model to learn skills.

[0009] Secondly, embodiments of this application provide a model learning device based on a variational discriminator, comprising:

[0010] An information acquisition unit is used to acquire skill variables from skill prior distribution information and discriminator parameters from parameter prior distribution information; wherein, the skill prior distribution information represents the probability distribution of skill variables at the current time step, the skill variables represent the skills that the model wants to learn, the parameter prior distribution information represents the probability distribution of the parameters of the variational discriminator at the current time step, and the discriminator parameters represent the parameters of the variational discriminator;

[0011] The information determination unit is used to determine the skill posterior distribution information based on the skill variables and the discriminator parameters, and on the current environmental state and the current execution strategy; wherein the environmental state represents the scenario in which the model is applied, the execution strategy represents the strategy by which the model performs skill learning, and the skill posterior distribution information represents the probability distribution of the skill variables at the next time step;

[0012] The strategy update unit is used to update the current execution strategy according to the prior skill distribution information and the posterior skill distribution information to obtain a target strategy; wherein the target strategy is used to instruct the model to learn skills.

[0013] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0014] The memory stores computer-executed instructions;

[0015] The processor executes computer execution instructions stored in the memory, causing the processor to perform the implementation method described in the first aspect above.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the embodiments described in the first aspect above.

[0017] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the implementation methods described in the first aspect above.

[0018] This application provides a model learning method, apparatus, device, and storage medium based on a variational discriminator. By introducing prior skill distribution information and prior parameter distribution information, skill variables and discriminator parameters are sampled from both. Based on the skill variables, discriminator parameters, current environmental state, and execution strategy, the posterior skill distribution information is jointly inferred, enabling updates to the probability distribution of skills and improving the flexibility of skill learning. The difference between the prior and posterior skill distribution information then drives the update of the execution strategy. Since the discriminator parameters are modeled as probability distributions rather than fixed values, and the posterior distribution of skill distributions is updated, the model can learn new skills and integrate old knowledge while learning new skills. This avoids catastrophic forgetting caused by parameter overwriting, improves the robustness and generalization ability of the discriminator to complex environmental states, and enables the agent to discover richer, more diverse, and more discriminative potential skills, significantly enhancing exploration efficiency and downstream task transfer performance. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] Figure 1 A flowchart illustrating a model learning method based on a variational discriminator provided in an embodiment of this application;

[0021] Figure 2 A flowchart illustrating a model learning method based on a variational discriminator provided in an embodiment of this application;

[0022] Figure 3 A flowchart illustrating a model learning method based on a variational discriminator provided in an embodiment of this application;

[0023] Figure 4 A schematic diagram of a model learning device based on a variational discriminator provided in this application embodiment;

[0024] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0025] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0027] First, let me explain the terms used in this application:

[0028] Environmental State: The environmental state s consists of multiple hierarchical variables s={s1,s2,s3,s4,s5}, used to characterize the dynamic changes of marketing tasks at the linguistic level. s1 can represent the market context description, describing the external macro-marketing environment, such as "during a major promotion," "strong holiday atmosphere," and "intense market competition." s2 can represent the user group description, reflecting the characteristics of the target audience, such as "young women prefer fashion products" and "the proportion of returning users is increasing." s3 can represent the activity configuration variable, i.e., the current marketing activity configuration, such as "20 off for every 100 spent," "App payment only," and "budget of 1 million yuan." s4 can represent historical semantic variables, storing the historical dialogue content between the agent and the environment, such as "the conversion rate of the last activity was 15%," "user feedback copy was too complex," and "there is a contradiction between the activity redemption rules and usage details." s5 can represent linguistic semantic variables, semantic embeddings generated by a large language model, used to capture contextual information, such as "this activity needs to highlight the discount" and "the brand tone is high-end."

[0029] Skill: The skill learned by the agent during unsupervised learning is a latent variable z. By embedding environmental variables and interacting with the environment, multi-skill mining can be achieved. For example, z1 can represent the skill of generating event copy; z2 can represent the skill of event prediction; z3 can represent the skill of event planning; and z4 can represent the skill of event recommendation.

[0030] Actions: The language behaviors that the agent can output. Each action 'a' is represented in the form of a natural language sentence and is converted into a semantic vector representation by the language model. For example, a1 is a marketing strategy generation action, used to generate a description of a marketing campaign plan; a2 is an activity configuration adjustment action, used to modify existing activity parameters or budget; a3 is a marketing copy generation action, used to output copy that conforms to user profiles and brand tone; a4 is a proactive inquiry action, used to ask questions to obtain feedback on the environmental status, such as "What was the conversion rate last time?"; a5 is an error correction action, used to determine whether an activity configuration conforms to the rules.

[0031] Reinforcement learning is an important branch of machine learning. Its core goal is to enable agents to learn strategies through interaction with the environment in order to maximize long-term cumulative rewards. Unsupervised reinforcement learning, as an important evolutionary direction, has freed itself from dependence on external reward signals and instead drives exploration and skill discovery through intrinsic incentive mechanisms. Among them, DIAYN (Diversity Is All You Need) is a classic unsupervised reinforcement learning algorithm that encourages agents to learn a diverse and distinguishable set of potential skills by maximizing the mutual information between skill variables and states.

[0032] However, existing unsupervised reinforcement learning methods such as DIAYN generally use discriminators with deterministic parameters, where the weights and biases are fixed scalar values. When continuously facing new skills or new environmental distributions, the discriminator's ability to distinguish old skills is easily degraded due to parameter overlay updates, leading to catastrophic forgetting. In addition, fixed parameters limit the discriminator's ability to model the uncertainty of state distribution, weakening its generalization and robustness, and thus restricting skill diversity and exploration efficiency.

[0033] This application provides a model learning method, apparatus, device, and storage medium based on a variational discriminator, which is applied in the field of cloud computing technology to improve the efficiency and accuracy of fault handling.

[0034] It should be noted that the data in this embodiment is not specific to any particular user and does not reflect the personal information of any particular user. It should also be noted that the data in this embodiment comes from a publicly available dataset.

[0035] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0036] To enable readers to have a deeper understanding of the implementation principles of this application, the embodiments are further detailed.

[0037] Figure 1 This is a flowchart illustrating a model learning method based on a variational discriminator according to an embodiment of this disclosure. This method can be executed by a model learning device based on a variational discriminator. Figure 1 As shown, the method includes the following steps:

[0038] S101. Obtain skill variables from skill prior distribution information and discriminator parameters from parameter prior distribution information; wherein, skill prior distribution information represents the probability distribution of skill variables at the current time step, skill variables represent the skills that the model wants to learn, parameter prior distribution information represents the probability distribution of the parameters of the variational discriminator at the current time step, and discriminator parameters represent the parameters of the variational discriminator.

[0039] For example, prior skill distribution information refers to the prior probability distribution p(z) defined for the potential skill space at the current time step t, where different z represent different skills, i.e., skill variables. Prior skill distribution information can be a standard normal distribution, a Gaussian mixture distribution, or a learnable parameterized distribution, etc. Its function is to provide a statistical basis for skill sampling, ensuring that the sampled skills have initial diversity and coverage. The skill variable z is a random sampling result that follows the prior skill distribution, i.e., z... p(z).

[0040] The prior distribution information of the parameters can refer to the initial probability distribution q(θ) set for the discriminator parameters θ of the variational discriminator. The variational discriminator is modeled as a neural network, and the discriminator parameters can include the neural network weights and bias terms. In this embodiment, the discriminator parameters are not fixed, but are modeled through the probability distribution q(θ), which includes the mean and variance of the neural network weights and bias terms, thereby expressing the uncertainty of the network. The prior distribution information of the parameters encodes the Bayesian belief in the initial expressive power of the discriminator. The discriminator parameters are specific instances sampled from this prior information, used to construct the variational discriminator available at the current moment. The output of the variational discriminator is a softmax (normalized exponential function) probability vector consistent with the skill space dimension, representing the confidence of each skill's attribution in a given environmental state.

[0041] The sampling process in this embodiment is non-deterministic; each interaction may yield different parameter configurations, thus endowing the discriminator with inherent exploratory capabilities and resistance to overfitting. During the initial learning process, initial skill prior distribution information and parameter prior distribution information can be preset. In subsequent iterations, new skill prior distribution information and parameter prior distribution information can be output, thus obtaining the skill prior distribution information and parameter prior distribution information for the next learning iteration.

[0042] Prior skill distribution information and prior parameter distribution information constitute a dual-source probabilistic modeling framework. The former drives the semantic diversity of the skill space, while the latter ensures the structural robustness of the discrimination mechanism. Together, they form the starting point for Bayesian inference, providing a computable benchmark for subsequent posterior updates.

[0043] S102. Based on the skill variables and discriminator parameters, and considering the current environmental state and execution strategy, determine the skill posterior distribution information; wherein, the environmental state represents the scenario in which the model is applied, the execution strategy represents the strategy by which the model learns skills, and the skill posterior distribution information represents the probability distribution of the skill variables at the next time step.

[0044] For example, the current environmental state refers to the contextual representation observed by the agent at the current time step t, and its composition varies depending on the application scenario. The environmental state must satisfy the Markov property and be normalized or standardized to adapt to the neural network input. A set of environmental states can be preset, and during skill learning, an environmental state can be randomly selected from the set as the preset environmental state.

[0045] The current execution policy refers to the policy network used to learn skills at the current time step t, which is the policy function π. A typical architecture is an Actor network with skill embeddings. The current environmental state is determined, and based on this state, the current execution policy is executed, i.e., the policy network is run to learn the skill variables. The policy network can output the learning result for the skill variables, i.e., the action performed by the model for that skill variable. Based on the output of the policy network and the sampled discriminator parameters, the variational discriminator can update the prior skill distribution information. The updated information is the posterior skill distribution information. In other words, the posterior skill distribution information is the skill distribution obtained through Bayesian inversion; its essence is the forward output of the variational discriminator, which is the allocation of the skill assignment probability for the next time step.

[0046] S103. Update the current execution strategy based on the prior and posterior skill distribution information to obtain the target strategy; wherein, the target strategy is used to instruct the model to learn skills.

[0047] For example, the difference between the prior and posterior distributions of skills constitutes the intrinsic driving force for policy optimization, and its quantification can take the form of KL divergence or log-likelihood ratio. KL divergence emphasizes the degree of deviation of the posterior from the prior, while log-likelihood ratio directly provides a scalar reward signal. The two are mathematically consistent. When the prior is uniformly distributed, KL divergence is proportional to log-likelihood ratio.

[0048] The difference between prior and posterior skill distributions can reflect the novelty and distinguishability of the current skill variable at adjacent time steps. If the posterior is concentrated on a single skill variable and deviates significantly from the prior, it indicates that the skill variable has successfully led to a unique state distribution and should be reinforced; if the posterior is diffuse or close to the prior, it indicates that the skill variable has not produced effective behavioral differentiation and the execution strategy needs to be adjusted.

[0049] The current execution policy can be updated using standard policy gradient methods, such as the automatic entropy adjustment of policy loss in SAC (SoftActor-Critic). The target policy is a policy function that converges after multiple iterations, and its output action can stably trigger highly discriminative state transitions, thereby autonomously constructing a skill hierarchy structure without external rewards. In this embodiment, the optimization algorithm for the execution policy is not specifically limited.

[0050] The comparison mechanism between skill priors and posteriors enables self-supervised learning without the need for manually designed reward functions. The prior provides a benchmark, while the posterior reflects the actual behavioral output; the difference between the two is the intrinsic value signal. This signal does not depend on environmental feedback but is generated solely by the model's own cognitive structure, thus possessing strong generalization and cross-task transfer potential. In this embodiment, the prior distribution can be set to a task-adaptive form, such as a prior constructed based on the posterior of historical tasks, to accelerate the cold start of new tasks.

[0051] This embodiment implements a closed-loop Bayesian learning paradigm. This paradigm models discriminator parameters as probability distributions rather than fixed values, enabling the model to express parameter-level uncertainty and integrate incremental knowledge, thus solving the catastrophic forgetting problem caused by fixed discriminator parameters. Because skill variables and discriminator parameters collaboratively participate in posterior inference, the robustness and distinguishability of the state-skill mapping are enhanced, alleviating the problems of insufficient skill diversity and low exploration efficiency. This embodiment does not rely on external reward signals, but drives optimization through intrinsic statistical differences, improving the model's generalization ability and deployment adaptability in reward-free scenarios, achieving the effect of continuous, diverse, and transferable skill autonomous learning without explicit rewards.

[0052] In this embodiment, after obtaining the target policy, the method further includes: determining the model to be trained based on the target policy; training the model to be trained based on the skill posterior distribution information and a preset reward function to obtain a trained model; wherein the trained model completes the learning of the skill.

[0053] Specifically, the target policy refers to the policy function converged after iterative optimization through the variational no-reward reinforcement learning process in this embodiment, possessing state-action mapping capability and skill-conditional generalization. The target policy can be directly used as the initial policy network for the downstream model, or as a teacher policy for knowledge distillation, or as the hidden layer representation output by a feature extractor to construct the input for the downstream model. The downstream model is the model to be trained in this embodiment; that is, the initial model to be trained can be determined based on the target policy.

[0054] In practical implementation, the model to be trained includes, but is not limited to, end-to-end language action generators, robot motion controllers, or combinations thereof, which can be dynamically adapted according to the type of downstream task. For example, in the marketing agent scenario, the model to be trained is a sequence-to-sequence generator that incorporates skill embeddings, which can output natural language actions through human-computer interaction.

[0055] Obtain the posterior distribution information of skills obtained when generating the target strategy. The dimension is equal to the spatial cardinality of the skill variables. For example, it can include four types of skills: copywriting, effect prediction, program planning, and event recommendation.

[0056] A reward function can be pre-set according to the application scenario of the model. The pre-set reward function is an explicit scalar function with domain knowledge designed for specific downstream tasks. For example, in the task of predicting the effect of marketing campaigns, the reward function R... t Defined as:

[0057] R t =w1*H+w2*L+w3*A-w4*C;

[0058] Where H is the redemption rate, L is the leverage ratio, A is the activation amount, C is the cost, and w1, w2, w3, and w4 are weight coefficients, which can be adjusted according to actual business needs. The model training process can adopt a standard supervised / rewarded reinforcement learning paradigm. The optimizer can be Adam, LAMB, or Nadam, etc., and the learning rate can be set to... to It supports warm-up and cosine decay scheduling. Based on a preset reward function, the model to be trained learns various skills by adapting to the posterior distribution information of the skills. For example, the model can stably output actions that conform to the semantics of the skill, such as the copywriting skill consistently producing grammatically correct, marketing-terminally accurate, and brand-consistent text. In this embodiment, after defining the reward function, based on the posterior distribution information of the skills, and following the traditional rewarded reinforcement learning paradigm, a specialized policy function is retrained to obtain the trained model, enabling rapid task adaptation through a limited number of reinforcement learning rounds.

[0059] The beneficial effect of this setup is that it realizes a closed-loop technical link from unrewarded pre-training to rewarded downstream adaptation, providing high-quality initialization for the model to be trained and improving the model's learning effect.

[0060] This application provides a model learning method based on a variational discriminator. By introducing prior skill distribution information and prior parameter distribution information, skill variables and discriminator parameters are sampled from both. Based on the skill variables, discriminator parameters, current environmental state, and execution policy, the posterior skill distribution information is jointly inferred, enabling updates to the probability distribution of skills and improving the flexibility of skill learning. The difference between the prior and posterior skill distribution information then drives the update of the execution policy. Since the discriminator parameters are modeled as probability distributions rather than fixed values, and the posterior distribution of skill distributions is updated, the model can learn new skills and integrate old knowledge while learning new skills. This avoids catastrophic forgetting caused by parameter overwriting, improves the discriminator's robustness and generalization ability to complex environmental states, and enables the agent to discover richer, more diverse, and more discriminative potential skills, significantly enhancing exploration efficiency and downstream task transfer performance.

[0061] Figure 2 A flowchart illustrating a model learning method based on a variational discriminator provided in this application embodiment is shown below. Figure 2 As shown, this embodiment, based on the above embodiments, provides a detailed description of a model learning method based on a variational discriminator. This method includes:

[0062] S201. Obtain skill variables from skill prior distribution information and discriminator parameters from parameter prior distribution information; wherein, skill prior distribution information represents the probability distribution of skill variables at the current time, skill variables represent the skills that the model wants to learn, parameter prior distribution information represents the probability distribution of the parameters of the variational discriminator at the current time, and discriminator parameters represent the parameters of the variational discriminator.

[0063] For example, this step can refer to step S101 above, and will not be repeated here.

[0064] S202. Based on the skill variables, and considering the current environmental state and execution strategy, determine the target's environmental state.

[0065] For example, the skill variable refers to the random latent variable z of the potential behavioral patterns that the model wants to learn, which follows a prior distribution p(z). Specific values ​​are taken in each sampling, for example, z1 represents the "copywriting" skill, and z3 represents the "proposal planning" skill.

[0066] The current environmental state s represents the scenario in which the model is to be located. The current execution strategy is a learnable policy network, whose network structure can adopt an Actor-Critic architecture, a pure Actor architecture, or embed a large language model as the policy head.

[0067] The target's environmental state is not the actual environmental state in the next moment, but rather a semantically enhanced intermediate state representation induced by the skill variable z and the current state s, and explicitly generated by the execution policy π. Essentially, it is the output of the policy network's modeling of the expected changes in the environment given the skill intention. For example, attention fusion can be performed on z and s to obtain the target's environmental state s. ， .

[0068] In this embodiment, the environmental state of the target is determined based on the skill variables, the current environmental state, and the current execution strategy. This includes: determining the enhanced state based on the skill variables and the current environmental state; wherein the enhanced state represents the skill variables and the current environmental state; and determining the environmental state of the target based on the enhanced state and the current execution strategy.

[0069] Specifically, the skill variable is a low-dimensional latent vector sampled from the skill prior distribution p(z), representing the abstract behavioral patterns the model intends to learn, such as semantically interpretable latent skill categories like "copywriting generation," "effect prediction," and "program planning." The current environmental state can represent marketing function configuration vectors, historical semantic variables, and linguistic semantic embeddings. The augmented state is an explicit joint representation of the skill variable z and the current environmental state s, which can be implemented by vector concatenation. The augmented state can be denoted as s. t s t = (s; z). By performing vector concatenation, we can ensure that the skill intent and the environmental context are aligned across modally at the input layer, avoiding semantic defocus caused by information decoupling.

[0070] The current execution strategy is a parameterized policy network π, which enhances the state s. t The input is fed into the policy network to obtain the target environment state. The target environment state refers to the new state obtained by the environment evolution. In this embodiment, the network architecture of the policy network is not specifically limited.

[0071] The beneficial effect of this setup is that by explicitly constructing the augmented state through the concatenation of skill variables and the current environmental state, the policy network obtains structured and interpretable joint observation inputs. By inferring the target environmental state through the current execution policy, the skill learning process acquires environmental responsiveness and semantic coherence, thereby improving the model's learning effectiveness.

[0072] In this embodiment, determining the target's environmental state based on the enhanced state and the current execution strategy includes: determining the model's action information based on the enhanced state and the current execution strategy; wherein, the action information represents the actions performed by the model during skill-based human-computer interaction; and adjusting the current environmental state based on the action information and the enhanced state to obtain the target's environmental state.

[0073] Specifically, the enhanced state refers to the joint representation formed by structurally combining the skill variable z with the current environmental state s, denoted as s. t The mathematical form can be a vector concatenation with dimensions dim(s) + dim(z). This concatenation operation can be replaced by additive fusion, gated fusion, or attention-weighted fusion, etc. The core role of the augmented state is to explicitly inject skill intent, making subsequent action generation skill-oriented and avoiding skill confusion caused by the policy network relying solely on the original state.

[0074] The current execution policy refers to the parameterized policy function π, whose input is the augmented state s. t The output is action a. t The policy function can be implemented using any policy gradient algorithm such as SAC (Soft Actor-Critic), DDPG (Deep Deterministic Policy Gradient), or PPO (Proximal Policy Optimization); the network structure of the policy network can be a fully connected MLP, a graph neural network, or a Transformer encoder.

[0075] Action information a t Action information is the specific output of the execution strategy under a given augmentation state, and its physical meaning varies depending on the application scenario. For example, in robot control scenarios, action information is the joint torque or end-effector pose increment; in marketing agents driven by large language models, action information is natural language sentences, such as "increase the budget from 500,000 yuan to 800,000 yuan" or "generate copy highlighting limited-time discounts." Action information meets preset task constraints; for example, in marketing scenarios, actions must conform to grammatical correctness, business logic consistency, and platform interface protocols. To ensure action feasibility, a constraint decoding module can be added after the policy network output layer. For example, clipping can be applied to continuous actions, a masking mechanism can be introduced to discrete actions (blocking illegal action IDs), or an LLM-based Safety Filter can be applied to language actions (based on fine-tuning a small model to score the compliance of the generated text and resample it).

[0076] Action information, as the driving signal for the dynamic evolution of the environment, is not applied in isolation, but rather participates in state transition modeling in conjunction with the enhanced state. Specifically, the generation process of the target environment state can be decomposed into two stages: the first stage, based on the enhanced state s... t With action State increments are generated through residual transformation. Where g is a lightweight neural network (such as a 2-layer MLP), and its output is the same as s. t Same-dimensional offset vector; Second stage, perform state update: s， =s t + .

[0077] The current environment state s is a structured representation of the scenario in which the model exists. Its composition can strictly be in the form of triples, for example, s = {s1, s2, s3}, where s1 is the current marketing function configuration, s2 is the historical semantic variable, and s3 is the linguistic semantic embedding. The environment state supports multimodal fusion, and each item in the triple can be updated independently.

[0078] The enhanced state provides contextual anchors for action generation, and the execution strategy decodes the abstract intent into action information. The action information and the enhanced state together constitute the joint driving source for environment updates. The final generated target environment state not only carries the action execution result but also retains the semantic imprint of the skill.

[0079] The beneficial effect of this setup is that the enhanced state explicitly integrates skill variables and environmental observations, and the execution strategy generates skill-oriented actions accordingly. These actions, together with the enhanced state, drive the environmental state to evolve towards the skill objective, enabling the skill learning process to have the function of environmental updating, and supporting the continuous discovery and stable maintenance of diverse skills under unrewarded conditions.

[0080] S203. Determine the skill posterior distribution information based on the target's environmental state and discriminator parameters.

[0081] For example, the environmental state of the target is s ， In actual calculations, these parameters serve as the input to the variational discriminator. The discriminator parameters are the network parameters of the variational discriminator, meaning a complete variational discriminator can be constructed based on these parameters.

[0082] Discriminator parameters refer to the set of learnable parameters of the variational discriminator, rather than the fixed neural network parameters in traditional DIAYN. During each forward propagation, specific parameters are sampled from q(θ) to construct the discriminator network. The discriminator network is typically a 2-3 layer MLP (Multi-Layer Perceptron), with a softmax output layer and a dimension equal to the skill space. This design enables the discriminator to model uncertainty, avoids overfitting to single-task data, and improves the model's learning ability.

[0083] The posterior distribution information of the skill refers to the conditional probability distribution output by the discriminator. The mathematical form is ,in This is the forward function of the variational discriminator. The posterior skill distribution information represents the observed target environment state. Under the premise of θ, the skill distribution obtained by the variational discriminator with θ as the discriminator parameter. This posterior distribution can be directly used for subsequent policy gradient updates, or it can be used as a supervision signal for fine-tuning of downstream tasks, such as freezing the variational discriminator and using its output as a training classifier head for skill labels.

[0084] Skill variables and environmental states are collaboratively generated into a target environmental state through the execution strategy. This state carries the observable consequences of skill intent. Discriminator parameters are modeled in the form of variational distribution and probabilistic inversion is performed based on the target environmental state to output the posterior distribution of skills. This avoids direct coupling between skill variables and discriminator parameters and improves training stability.

[0085] In this embodiment, skill posterior distribution information is determined based on the target's environmental state and discriminator parameters, including: generating a variational discriminator based on the discriminator parameters; and inputting the target's environmental state into the variational discriminator to obtain skill posterior distribution information.

[0086] Specifically, the discriminator parameters refer to the set of random variables consisting of learnable weights and bias vectors in the variational discriminator, which follows a variational approximation distribution q(θ) at the current time step. This distribution can be jointly characterized by the mean vector and the covariance matrix. The variational discriminator is a probabilistic neural network structure whose input is the environmental state and whose output is a conditional probability distribution in the skill space, i.e. The generation process involves sampling a set of specific parameters from the current variational distribution q(θ) and instantiating the sampling result as the weights and biases of a deterministic neural network model, thereby completing the dynamic construction of the discriminator. This generation mechanism differs from the static weight initialization of traditional deterministic discriminators, enabling the discriminator to possess Bayesian uncertainty modeling capabilities and supporting adaptive adjustment of the skill discrimination boundary at different task stages.

[0087] The network structure of the variational discriminator can be any of the following: a fully connected feedforward network, a convolutional neural network, a recurrent neural network, or a Transformer encoder.

[0088] The target's environmental state refers to the next observable state obtained after skill variables are guided, execution strategies are applied, and environmental dynamics evolve; specifically, it is the evolution of the enhanced state after action execution. The target's environmental state serves as the input to the variational discriminator, introducing no additional labels, rewards, or supervision signals, conforming to the no-reward reinforcement learning paradigm. The skill posterior distribution information is the conditional probability distribution of the discriminator's output. Its dimension is consistent with the prior skill distribution p(z), and each component corresponds to the posterior probability of a potential skill, representing the probability of observing s. ， Then, the agent determines the skill category to which the current behavior is most likely to belong and its confidence level.

[0089] The beneficial effects of this setup are that variational sampling of the discriminator parameters ensures that the model has the uncertainty representation capability required for continuous learning, avoiding parameter overfitting to a single task; while using the target environment state as input to drive discriminative reasoning strictly ensures that the skill posterior is only deduced from environmental feedback, without relying on external supervision signals, thereby supporting the mathematical legitimacy and interpretability of pseudo-rewards.

[0090] S204. Based on the prior and posterior skill distribution information, update the current execution strategy to obtain the target strategy; wherein, the target strategy is used to instruct the model to learn skills.

[0091] For example, this step can refer to step S103 above, and will not be repeated here.

[0092] This application provides a model learning method based on a variational discriminator. By introducing prior skill distribution information and prior parameter distribution information, skill variables and discriminator parameters are sampled from both. Based on the skill variables, discriminator parameters, current environmental state, and execution policy, the posterior skill distribution information is jointly inferred, enabling updates to the probability distribution of skills and improving the flexibility of skill learning. The difference between the prior and posterior skill distribution information then drives the update of the execution policy. Since the discriminator parameters are modeled as probability distributions rather than fixed values, and the posterior distribution of skill distributions is updated, the model can learn new skills and integrate old knowledge while learning new skills. This avoids catastrophic forgetting caused by parameter overwriting, improves the discriminator's robustness and generalization ability to complex environmental states, and enables the agent to discover richer, more diverse, and more discriminative potential skills, significantly enhancing exploration efficiency and downstream task transfer performance.

[0093] Figure 3 A flowchart illustrating a model learning method based on a variational discriminator provided in this application embodiment is shown below. Figure 3 As shown, this embodiment, based on the above embodiments, provides a detailed description of a model learning method based on a variational discriminator. This method includes:

[0094] S301. Obtain skill variables from skill prior distribution information and discriminator parameters from parameter prior distribution information; wherein, skill prior distribution information represents the probability distribution of skill variables at the current time, skill variables represent the skills that the model wants to learn, parameter prior distribution information represents the probability distribution of the parameters of the variational discriminator at the current time, and discriminator parameters represent the parameters of the variational discriminator.

[0095] For example, this step can refer to step S101 above, and will not be repeated here.

[0096] S302. Based on the skill variables and discriminator parameters, and considering the current environmental state and execution strategy, determine the skill posterior distribution information; wherein, the environmental state represents the scenario in which the model is applied, the execution strategy represents the strategy by which the model learns skills, and the skill posterior distribution information represents the probability distribution of the skill variables at the next time step.

[0097] For example, this step can refer to step S102 above, and will not be repeated here.

[0098] S303. Based on the prior distribution information and the posterior distribution information of skills, determine the pseudo-reward information; wherein, the pseudo-reward information represents the degree of skill differentiation between the prior distribution information and the posterior distribution information of skills.

[0099] For example, the prior distribution information of skills refers to the pre-defined probability distribution of the skill variable z at the current time step t, denoted as p(z). The posterior distribution information of skills refers to the inferred skill distribution after completing an environmental interaction, denoted as... The discriminant is generated by a variational discriminator. Unlike the fixed-weight discriminator in traditional DIAYN, in this embodiment, θ can be independently sampled from q(θ) for each forward propagation, so that the posterior estimate naturally carries Bayesian confidence and avoids overfitting to a single-parameter path.

[0100] Pseudo-reward information is used to characterize the distinguishability of skills and serves as a core bridge connecting probabilistic modeling and policy optimization. A pre-defined pseudo-reward function is used to calculate the pseudo-reward information based on the prior and posterior distributions of the skill. For example, when a skill variable z significantly improves its corresponding state s... ， The conditional likelihood, meaning the discriminator can identify the skill variable with high confidence, and the skill variable itself is not dominant in the prior, indicates that the pseudo-reward information is positive and has a large amplitude, signifying that the skill information has high discriminative power and high value. Conversely, if a skill variable is highly concentrated in the prior but the discriminator cannot reliably identify it, the pseudo-reward information is strongly negative, constituting a suppressive signal against policy exploration bias. Pseudo-reward information does not rely on external manual annotation; it is automatically generated entirely by the model's internal probability structure, belonging to the intrinsic incentive mechanism under the no-reward reinforcement learning paradigm.

[0101] S304. Update the current execution strategy based on the pseudo-reward information to obtain the target strategy.

[0102] For example, the current execution policy refers to the parameterized policy function π, whose input is the concatenated vector of the current environment state s and the sampled skill vector z, i.e., the augmented state s. t The output is action a. t .

[0103] The target policy is the new execution policy updated based on pseudo-reward information, and its update process can be achieved by maximizing the expected cumulative pseudo-reward. In this embodiment, the optimization algorithm for the execution policy is not specifically limited. For example, the automatic temperature adjustment mechanism of SAC can be used to balance reward maximization and policy entropy maximization, or the pruning objective function of PPO can be used to prevent the policy update step size from being too large. This update process does not modify the skill priors and discriminator parameter distributions themselves; it only adjusts the policy network's ability to model the skill-state-action mapping relationship, ensuring that the policy evolution is always anchored within the probabilistic inference framework.

[0104] This embodiment uses prior skill distribution information as the starting point for exploration and posterior skill distribution information output by the variational discriminator as the identification basis. The difference between the two is quantified into pseudo-reward information, which is then used as the optimization objective to drive the iterative update of the execution strategy. The pseudo-reward information directly reflects the distinguishability of skills in the state space, avoiding the skill pattern homogenization caused by blind strategy updates. Because the pseudo-reward calculation is entirely endogenous to the model's probability structure and has Bayesian uncertainty calibration, it enables the execution strategy to continuously focus on the learning objective of high-value, high-discrimination skills in the absence of explicit reward signals, thereby improving learning effectiveness.

[0105] In this embodiment, pseudo-reward information is represented as follows:

[0106] ;

[0107] Where r represents pseudo-reward information, p1 represents skill prior distribution information, and p2 represents skill posterior distribution information.

[0108] Specifically, the pseudo-reward function can be expressed as: p1 represents the prior skill distribution information, i.e., p(z), and p2 represents the posterior skill distribution information, i.e., Update the execution strategy with maximizing r as the constraint.

[0109] In this embodiment, updating the current execution strategy based on pseudo-reward information to obtain the target strategy includes: updating the current execution strategy based on pseudo-reward information and a preset reinforcement learning algorithm to obtain the updated execution strategy; determining the parameter posterior distribution information based on parameter prior distribution information and skill posterior distribution information; wherein the parameter posterior distribution information represents the probability distribution of the parameters of the variational discriminator at the next time step; if the parameter posterior distribution information satisfies a preset convergence condition, the updated execution strategy is determined as the target strategy.

[0110] Specifically, pseudo-reward information refers to the intrinsic incentive signal constructed from prior and posterior skill distribution information, which is the core scalar feedback driving policy optimization. Pseudo-reward information does not depend on the external reward function of the environment; it is generated solely through the discriminator's evaluation of the state-skill matching degree and the comparison with the prior skill distribution, reflecting the agent's self-evaluation ability regarding whether the current behavior effectively distinguishes and implements the selected skill.

[0111] The preset reinforcement learning algorithm refers to a policy optimization framework adapted to a policy without a reward setting, including but not limited to SAC, DDPG, or PPO algorithms. The preset reinforcement learning algorithm uses pseudo-reward information *r* as the immediate reward, backpropagates gradients to update the execution policy, and obtains the updated execution policy. In this embodiment, the optimization process of the execution policy is not specifically limited.

[0112] The prior distribution information of parameters refers to the posterior distribution of the variational discriminator parameters θ at the beginning of the current training phase, which is the parameter distribution determined at the end of the previous outer loop. This distribution carries the discriminative knowledge learned in the historical task.

[0113] Based on the prior and posterior distributions of the parameters, a Bayesian variational inference process is performed to maximize the lower bound of evidence and solve for the optimal approximation of the parameter distribution, i.e., the posterior distribution of the parameters. The posterior distribution of the parameters can be represented as... , The update can be performed using the following objective function:

[0114] ;

[0115] Where E represents the expected value and KL represents the KL divergence. This represents the posterior distribution of the parameters at the previous time step, which is equivalent to the prior distribution of the parameters at the current time step. The first term in the formula is the likelihood term given the discriminator parameters; maximizing this term encourages the discriminator to distinguish different skills as accurately as possible. The second term is the KL divergence between the prior and posterior; minimizing this term constrains the new distribution from deviating too far from the previous distribution, thus controlling the stationarity of parameter updates. In this embodiment, the Adam optimizer can be used to update the discriminator's parameter distribution through SGD (Stochastic Gradient Descent) to obtain the posterior distribution of the parameters.

[0116] A convergence condition is preset, which is a quantization threshold used to determine the stability of the variational discriminator parameters, including but not limited to any one or more combinations of the following: (1) the KL divergence is less than the preset threshold; (2) the relative rate of change of the parameter distribution mean is less than the preset rate of change threshold; (3) the absolute change of the parameter distribution variance is less than the preset change threshold; (4) in M ​​consecutive external loops, the fluctuation range of the discriminator's average skill recognition accuracy on the validation set is less than The above conditions can be enabled individually or in a weighted fusion. This convergence mechanism does not simply terminate training, but rather signifies that the discriminator has solidified its knowledge of the current skill distribution pattern, and its output parameter distribution has sufficient robustness to serve as a stable anchor point for subsequent policy optimization. If the posterior distribution information of the parameters satisfies the preset convergence conditions, the currently determined updated execution policy can be used as the target policy for subsequent model learning.

[0117] The beneficial effects of this setup are that it achieves coordinated rhythmic control between policy updates and discriminator evolution, solving the training oscillation problem caused by the asynchrony between policy and discriminator updates in traditional rewardless reinforcement learning. The posterior parameter distribution is tightly coupled with historical priors through KL constraints, resolving the catastrophic forgetting problem caused by discriminator parameters overwriting old knowledge. Ultimately, this achieves the effect of enabling the execution policy and variational discriminator to evolve synchronously and converge stably without explicit rewards, while providing a reusable parameter base for continuous learning.

[0118] In this embodiment, the method further includes: if the posterior distribution information of the parameters does not meet the preset convergence condition, then the posterior distribution information of the skills is determined as the new prior distribution information of the skills, and the posterior distribution information of the parameters is determined as the new prior distribution information of the parameters, and the method of obtaining new skill variables from the new prior distribution information of the skills and new discriminator parameters from the new prior distribution information of the parameters is continued until the determined posterior distribution information of the parameters meets the preset convergence condition.

[0119] Specifically, the posterior distribution information of the parameters refers to the probability distribution obtained by Bayesian updating of the variational discriminator parameters after completing the current task batch interaction. Its form can be a Gaussian distribution. , where the mean with standard deviation These are learnable parameters that characterize the uncertainty estimates of the discriminator weights and biases. The posterior distribution information of the parameters is obtained through variational lower bound optimization, specifically constrained by the expected log-likelihood term and the KL divergence regularization term.

[0120] If the posterior distribution information of the parameters meets the preset convergence condition, it is determined whether the posterior distribution information of the skills determined at the current time step meets the prior distribution information of the skills at the next time step. The posterior distribution information of the parameters determined at the current time step meets the prior distribution information of the parameters at the next time step. The steps of this embodiment are then executed again to obtain the skill variables from the prior distribution information of the skills and the discriminator parameters from the prior distribution information of the parameters. That is, a new round of iteration is started until the determined posterior distribution information of the parameters meets the preset convergence condition.

[0121] The beneficial effect of this setup is that the convergence determination mechanism of the parameter posterior distribution and the prior rolling update mechanism form a feedback loop. When the parameter distribution fails to converge, the system does not terminate training, but instead upgrades the skill posterior and parameter posterior learned in the current round to new priors, thereby solidifying the phased learning results into prior knowledge. This enables the entire learning process to have online Bayesian inference capabilities, realizing the construction of a continuous skill learning loop for sequential task flows under the condition of no explicit reward signal.

[0122] This application provides a model learning method based on a variational discriminator. By introducing prior skill distribution information and prior parameter distribution information, skill variables and discriminator parameters are sampled from both. Based on the skill variables, discriminator parameters, current environmental state, and execution policy, the posterior skill distribution information is jointly inferred, enabling updates to the probability distribution of skills and improving the flexibility of skill learning. The difference between the prior and posterior skill distribution information then drives the update of the execution policy. Since the discriminator parameters are modeled as probability distributions rather than fixed values, and the posterior distribution of skill distributions is updated, the model can learn new skills and integrate old knowledge while learning new skills. This avoids catastrophic forgetting caused by parameter overwriting, improves the discriminator's robustness and generalization ability to complex environmental states, and enables the agent to discover richer, more diverse, and more discriminative potential skills, significantly enhancing exploration efficiency and downstream task transfer performance.

[0123] Figure 4 A schematic diagram of a model learning device based on a variational discriminator provided in this application embodiment is shown below. Figure 4 As shown, the model learning device 40 based on a variational discriminator provided in this embodiment includes:

[0124] The information acquisition unit 401 is used to acquire skill variables from skill prior distribution information and discriminator parameters from parameter prior distribution information; wherein, skill prior distribution information represents the probability distribution of skill variables at the current time step, skill variables represent the skills that the model wants to learn, parameter prior distribution information represents the probability distribution of the parameters of the variational discriminator at the current time step, and discriminator parameters represent the parameters of the variational discriminator.

[0125] The information determination unit 402 is used to determine the posterior distribution information of skills based on the skill variables and discriminator parameters, the current environmental state, and the current execution strategy. The environmental state represents the scenario in which the model is applied, the execution strategy represents the strategy by which the model learns skills, and the posterior distribution information of skills represents the probability distribution of the skill variables at the next time step.

[0126] The strategy update unit 403 is used to update the current execution strategy based on the prior skill distribution information and the posterior skill distribution information to obtain the target strategy; wherein, the target strategy is used to instruct the model to learn skills.

[0127] In one possible implementation, the information determining unit 402 includes:

[0128] The environment determination module is used to determine the target's environmental state based on skill variables, the current environmental state, and the current execution strategy.

[0129] The skill posterior module is used to determine the skill posterior distribution information based on the target's environmental state and discriminator parameters.

[0130] In one possible implementation, the environment determination module is specifically used for:

[0131] The enhancement state is determined based on the skill variables and the current environmental state; where the enhancement state represents the skill variables and the current environmental state.

[0132] Determine the target's environmental state based on the enhanced state and the current execution strategy.

[0133] In one possible implementation, the environment determination module is specifically used for:

[0134] Based on the enhanced state and the current execution strategy, the model's action information is determined; whereby the action information represents the actions performed by the model during skill-based human-computer interaction.

[0135] Based on action information and enhanced status, the current environmental state is adjusted to obtain the target's environmental state.

[0136] In one possible implementation, the skill post-test module is specifically used for:

[0137] Generate a variational discriminant based on the discriminant parameters;

[0138] The target's environmental state is input into the variational discriminator to obtain the skill posterior distribution information.

[0139] In one possible implementation, the policy update unit 403 includes:

[0140] The pseudo-reward determination module is used to determine pseudo-reward information based on prior skill distribution information and posterior skill distribution information; wherein, pseudo-reward information represents the degree of skill differentiation between prior skill distribution information and posterior skill distribution information.

[0141] The strategy update module is used to update the current execution strategy based on the pseudo-reward information to obtain the target strategy.

[0142] In one possible implementation, the pseudo-reward information is represented as:

[0143] ;

[0144] Where r represents pseudo-reward information, p1 represents skill prior distribution information, and p2 represents skill posterior distribution information.

[0145] In one possible implementation, the policy update module is specifically used for:

[0146] Based on the pseudo-reward information, the current execution policy is updated using a preset reinforcement learning algorithm to obtain the updated execution policy.

[0147] Based on the prior distribution information of the parameters and the posterior distribution information of the skills, the posterior distribution information of the parameters is determined; wherein, the posterior distribution information of the parameters represents the probability distribution of the parameters of the downvariant discriminator in the next time step;

[0148] If the posterior distribution information of the parameters meets the preset convergence condition, the updated execution strategy will be determined as the target strategy.

[0149] One possible implementation also includes:

[0150] The loop unit is used to determine the skill posterior distribution information as the new skill prior distribution information and the parameter posterior distribution information as the new parameter prior distribution information if the parameter posterior distribution information does not meet the preset convergence condition. It continues to obtain new skill variables from the new skill prior distribution information and new discriminator parameters from the new parameter prior distribution information until the determined parameter posterior distribution information meets the preset convergence condition.

[0151] This embodiment provides a model learning device based on a variational discriminator, which can execute the method provided in the above-described method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0152] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application. Figure 5As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the device 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0153] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0154] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0155] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0156] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0157] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0158] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0159] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0160] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0161] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0162] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0163] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0164] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0165] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0166] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0167] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A model learning method based on a variational discriminator, characterized in that, include: Skill variables are obtained from prior skill distribution information, and discriminator parameters are obtained from prior parameter distribution information; wherein, the prior skill distribution information represents the probability distribution of skill variables at the current time step, the skill variables represent the skills that the model wants to learn, the prior parameter distribution information represents the probability distribution of the parameters of the variational discriminator at the current time step, and the discriminator parameters represent the parameters of the variational discriminator. Based on the skill variables and the discriminator parameters, and considering the current environmental state and the current execution strategy, the skill posterior distribution information is determined; wherein, the environmental state represents the scenario in which the model is applied, the execution strategy represents the strategy by which the model learns skills, and the skill posterior distribution information represents the probability distribution of the skill variables at the next time step; Based on the prior skill distribution information and the posterior skill distribution information, the current execution strategy is updated to obtain the target strategy; wherein, the target strategy is used to instruct the model to learn skills.

2. The method according to claim 1, characterized in that, Based on the skill variables and the discriminator parameters, and considering the current environmental state and execution strategy, the skill posterior distribution information is determined, including: Based on the skill variables, the target's environmental state is determined according to the current environmental state and the current execution strategy; The skill posterior distribution information is determined based on the target's environmental state and the discriminator parameters.

3. The method according to claim 2, characterized in that, Based on the skill variables, and considering the current environmental state and execution strategy, the target's environmental state is determined, including: An enhanced state is determined based on the skill variables and the current environmental state; wherein the enhanced state represents the skill variables and the current environmental state. The environmental state of the target is determined based on the enhanced state and the current execution strategy.

4. The method according to claim 3, characterized in that, Based on the enhanced state and the current execution strategy, the environmental state of the target is determined, including: Based on the enhanced state and the current execution strategy, the action information of the model is determined; wherein, the action information represents the actions performed by the model during skill-based human-computer interaction; Based on the action information and the enhanced state, the current environmental state is adjusted to obtain the environmental state of the target.

5. The method according to claim 2, characterized in that, Based on the environmental state of the target and the discriminator parameters, the skill posterior distribution information is determined, including: The variational discriminator is generated based on the discriminator parameters; The environmental state of the target is input into the variational discriminator to obtain the skill posterior distribution information.

6. The method according to claim 1, characterized in that, The current execution strategy is updated based on the prior skill distribution information and the posterior skill distribution information to obtain the target strategy, including: Based on the prior skill distribution information and the posterior skill distribution information, pseudo-reward information is determined; wherein, the pseudo-reward information characterizes the degree of skill differentiation between the prior skill distribution information and the posterior skill distribution information. Based on the pseudo-reward information, the current execution strategy is updated to obtain the target strategy.

7. The method according to claim 6, characterized in that, The pseudo-reward information is characterized as follows: ； Where r represents pseudo-reward information, p1 represents skill prior distribution information, and p2 represents skill posterior distribution information.

8. The method according to claim 6, characterized in that, Based on the pseudo-reward information, the current execution strategy is updated to obtain the target strategy, including: Based on the pseudo-reward information, the current execution policy is updated using a preset reinforcement learning algorithm to obtain the updated execution policy. Based on the prior distribution information of the parameters and the posterior distribution information of the skills, the posterior distribution information of the parameters is determined; wherein, the posterior distribution information of the parameters represents the probability distribution of the parameters of the downvariant discriminator in the next time step; If the posterior distribution information of the parameters satisfies the preset convergence condition, then the updated execution strategy is determined as the target strategy.

9. The method according to claim 8, characterized in that, Also includes: If the parameter posterior distribution information does not meet the preset convergence condition, then the skill posterior distribution information is determined as the new skill prior distribution information, and the parameter posterior distribution information is determined as the new parameter prior distribution information. The process of obtaining new skill variables from the new skill prior distribution information and obtaining new discriminator parameters from the new parameter prior distribution information continues until the determined parameter posterior distribution information meets the preset convergence condition.

10. The method according to any one of claims 1-9, characterized in that, After obtaining the target strategy, it also includes: Based on the target strategy, determine the model to be trained; Based on the skill posterior distribution information and a preset reward function, the model to be trained is trained to obtain a trained model; wherein the trained model has completed the learning of the skill.

11. A model learning device based on a variational discriminator, characterized in that, include: An information acquisition unit is used to acquire skill variables from skill prior distribution information and discriminator parameters from parameter prior distribution information; wherein, the skill prior distribution information represents the probability distribution of skill variables at the current time step, the skill variables represent the skills that the model wants to learn, the parameter prior distribution information represents the probability distribution of the parameters of the variational discriminator at the current time step, and the discriminator parameters represent the parameters of the variational discriminator; The information determination unit is used to determine the skill posterior distribution information based on the skill variables and the discriminator parameters, and on the current environmental state and the current execution strategy; wherein the environmental state represents the scenario in which the model is applied, the execution strategy represents the strategy by which the model performs skill learning, and the skill posterior distribution information represents the probability distribution of the skill variables at the next time step; The strategy update unit is used to update the current execution strategy according to the prior skill distribution information and the posterior skill distribution information to obtain a target strategy; wherein the target strategy is used to instruct the model to learn skills.

12. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-10.

14. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-10.

Images