A content generation method and system based on a probabilistic reward model

Abstract

The application provides a content generation method and system based on a probability reward model, which comprises collecting and processing paired preference samples to construct a preference training dataset; and constructing a probability reward model comprising an expert head and a router. The expert head comprises a plurality of parallel experts, each corresponding to a type of preference dimension. Each expert outputs the mean reward and standard deviation of the corresponding preference dimension. The router outputs a set of routing weights, each corresponding to an expert and the preference dimension to which the expert belongs. The preference training dataset is used to perform two-stage training on the probability reward model to obtain a trained probability reward model. The trained probability reward model is used to iteratively update a strategy model, and the updated strategy model generates content based on user input. The application can improve the stability of the reward model in the application of the strategy model, thereby improving the accuracy of the content output by the strategy model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a content generation method and system based on a probabilistic reward model. Background Technology

[0002] Large Language Models (LLMs) have been widely used in tasks such as open-ended question answering, dialogue generation, code generation, content creation, and complex reasoning. To make the output of LLMs more aligned with human expectations, current mainstream methods employ alignment training processes such as Reinforcement Learning from Human Feedback (RLHF) or Reinforcement Learning from AI Feedback (RLAIF). These processes typically require building a Reward Model (RM) or using a LLM evaluator to transform human or model preferences for several candidate answers into optimizable reward signals.

[0003] Existing reward models are mostly based on the Bradley-Terry Model (BT model) for pairwise preference learning. The basic idea is to assign a scalar score to each candidate answer and then calculate the probability that one answer is better than the other based on the score difference between the two answers. This method is simple in structure and easy to train, and has become a common foundation for reward model training in RLHF. A literature search of existing technologies revealed that Chinese patent CN122174884A proposes a method for constructing an interpretable reward model based on a sparse autoencoder. This method only requires adjusting the weights of specific features in the linear value head to change the reward signal in real time, guiding the policy model towards new preference optimization.

[0004] However, existing scalar reward models have structural flaws. Human preferences in open-ended generative tasks are not always unimodal, deterministic, or fully representable by a single scalar. The same problem may simultaneously involve multiple evaluation dimensions such as safety, usefulness, humor, accuracy, conciseness, and creativity. Different qualified annotators may have reasonable disagreements due to differences in their focus on different dimensions; such disagreements belong to subjective preference differences. At the same time, some samples may have judgment uncertainty due to problem ambiguity, insufficient information, external input, or insufficient knowledge of the annotator; such uncertainty belongs to cognitive uncertainty.

[0005] Existing reward models typically compress the aforementioned subjective preference differences and cognitive uncertainties into a single scalar score, meaning the model can only learn the preferences of an "average annotator." This approach leads to the following problems: First, it fails to preserve the multidimensional preference structure, resulting in unreasonable compression of conflicts between different evaluation dimensions; second, it cannot accurately measure the model's confidence in a given rating, allowing highly uncertain samples to still generate strong reward signals; third, in downstream reinforcement learning, the policy model may exploit vulnerabilities in the scalar reward model to obtain high scores without actually improving the quality of the response, exhibiting reward hacking or overoptimization; fourth, using fine-grained scoring or explicit attribute labeling to alleviate these problems requires extensive manual labeling, which is costly and difficult to scale.

[0006] Therefore, there is an urgent need for a method and system that uses a new reward model to drive the content generation of large language models. This new reward model can learn multidimensional preference structures from large-scale pairwise preference data without relying on a large amount of fine-grained manual annotation, and at the same time output the reward mean and uncertainty estimate. This allows the reward signal to not only express the diversity of subjective preferences, but also reduce the impact of unreliable rewards on training when the input is fuzzy or out of distribution. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the purpose of this application is to provide a content generation method based on a probabilistic reward model.

[0008] According to one aspect of this application, a content generation method based on a probabilistic reward model is provided, comprising: Collect and process paired preference samples to construct a preference training dataset; Construct a probabilistic reward model that includes an expert head and a router; the expert head includes several experts set in parallel, each expert corresponds to a preference dimension, and each expert outputs the mean and standard deviation of the reward under its corresponding preference dimension; the router outputs a set of routing weights, each routing weight corresponds to an expert and its corresponding preference dimension. Using the aforementioned preference training dataset, the probability reward model is trained in two stages to obtain the trained probability reward model. The policy model is iteratively updated based on the trained probability reward model and policy optimization algorithm. The updated policy model generates content based on user input, and the generated content is open-ended natural language text.

[0009] Optionally, the step of collecting and processing paired preference samples to construct a preference training dataset includes: Collect several pairs of preference samples, each pair of preference samples including input prompts and their corresponding candidate answers; perform text cleaning, text truncation, template formatting and data deduplication on the pairs of preference samples in sequence, and construct the preference training dataset from the processed pairs of preference samples.

[0010] Optionally, the probabilistic reward model uses a pre-trained large language model as its backbone network. The backbone network semantically encodes the input pairwise preference samples and outputs hidden feature representations. Several Gaussian experts and a router are configured in parallel at the output of the backbone network, wherein: Each Gaussian expert corresponds to a preference dimension. Each Gaussian expert receives the hidden feature representation. Each Gaussian expert includes a mean head and a standard deviation head. The mean head generates the reward mean under the corresponding preference dimension based on the input hidden feature representation. The standard deviation head generates the standard deviation under the corresponding preference dimension based on the input hidden feature representation. The Gaussian expert generates a Gaussian reward distribution based on the reward mean and the standard deviation. The router receives the hidden feature representation and outputs a number of routing weights equal to the number of Gaussian experts. Each routing weight corresponds one-to-one with an expert module and its generated reward mean and standard deviation. The outputs of each Gaussian expert are weighted and fused using routing weights to obtain the output of the probabilistic reward model, which is the reward evaluation result.

[0011] Optionally, the expression for the mean reward generated by the mean head is as follows: μ_k(x, y) = W_u^(k) h Where μ_k(x, y) is the mean reward of the k-th Gaussian expert for the input cue x-candidate answer y pair, W_u^(k) is the mean header parameter of the k-th Gaussian expert; h is the hidden feature representation; The expression for the standard deviation generated by the standard deviation head is as follows: σ_k(x, y) = σ_min + (σ_max - σ_min) · Sigmoid(z_k) z_k = W_σ^(k) h + b_σ^(k) Where σ_k(x, y) is the standard deviation of the input hint x-candidate answer y pair for the k-th Gaussian expert pair, σ_min is the lower bound of the standard deviation, σ_max is the upper bound of the standard deviation, Sigmoid() is the activation function, z_k is the intermediate latent variable, and W_σ^(k) and b_σ^(k) are both standard deviation head parameters; The expression for the Gaussian reward distribution is as follows: r_k(y | x) ~ N( μ_k(x, y), σ_k(x, y)^2 ) Where r_k(y | x) is the conditional probability distribution of the input cue x-candidate answer y pair for the k-th Gaussian expert pair.

[0012] Optionally, the expression for the routing weight output by the router is as follows: π(x) = Softmax(W_r h_x) Where π(x) is a set of K-dimensional routing weights under input cue x, K is the total number of Gaussian experts, W_r is the router parameters, Softmax() is the activation function, and h_x is the hidden feature representation corresponding to input cue x.

[0013] Optionally, the step of using the preference training dataset to perform two-stage training on the probabilistic reward model to obtain the trained probabilistic reward model includes: The probabilistic reward model is trained in two stages using the aforementioned preference training dataset, with the main parameters of the backbone network frozen throughout the training process. In the first stage, the parameters of the routers are frozen, and iterative parameter training is performed on all Gaussian experts until their training converges. After the Gaussian experts converge, the parameters of all Gaussian experts are frozen again. In the second stage, the state of the Gaussian experts with frozen parameters remains unchanged, and iterative parameter training is performed only on the routers until their training converges. Once the routers have completed convergent training, the two-stage training process ends, yielding the trained probabilistic reward model.

[0014] Optionally, during the first stage of training, a mixture likelihood loss function is used as the loss function for this stage of training. The first stage of training aims to minimize the mixture likelihood loss function, and all Gaussian experts are jointly trained. The expression for the mixture likelihood loss function is: L_stage1 = - E_(x,yw,yl) [ log( (1 / K) Σ_{k=1}^K P_k(yw yl | x) ) ] P_k(yw yl | Where L_stage1 represents the loss of the first stage of training, E_(x,yw,yl) represents the expected value of the pairwise preference samples, K represents the total number of Gaussian experts, and P_k(yw) represents the expected value of the pairwise preference samples. yl | x) represents the probability that the k-th Gaussian expert, given input cue x, determines that candidate answer yw is better than candidate answer yl; Sigmoid() represents the activation function, μ_k(x, yw) is the mean reward of the k-th Gaussian expert for the input cue x-candidate answer yw pair, μ_k(x, yl) is the mean reward of the k-th Gaussian expert for the input cue x-candidate answer yl pair, σ_k(x, yw) is the standard deviation of the k-th Gaussian expert for the input cue x-candidate answer yw pair, σ_k(x, yl) is the standard deviation of the k-th Gaussian expert for the input cue x-candidate answer yl pair, sqrt() represents the square root operation, and λ is the uncertainty weight coefficient.

[0015] Optionally, during the second stage of training, the router is trained with the optimization objective of minimizing the loss function of the second stage training. The expression for the loss function of the second stage training is: L_router = - E_(x,yw,yl) [ log P(yw yl | x) ] P(yw yl | x) = Σ_{k=1}^K Σ_{l=1}^K π_k(x) π_l(x) p_{kl} p_{kl}= Φ([μ_k(x, yw) - μ_l(x, yl) ] / sqrt( σ_k(x, yw)^2 +σ_l(x,yl)^2 + ε)) Where L_router represents the loss of the second-stage training, E_(x,yw,yl) represents the expected value of the pairwise preference samples, and P(yw) represents the expected value of the pairwise preference samples. Let yl | x) represent the probability that the router favors candidate answer yw over candidate answer yl under input cue x, K represent the total number of Gaussian experts, π_k(x) represent the routing weight of the k-th Gaussian expert under input cue x, π_l(x) represent the routing weight of the l-th Gaussian expert under input cue x, p_{kl} represent the probability that the reward of the k-th Gaussian expert for the favored candidate answer yw is greater than the reward of the l-th Gaussian expert for the unfavored candidate answer yl under input cue x, Φ() represent the cumulative distribution function of the standard normal distribution, μ_k(x,yw) represent the mean reward of the k-th Gaussian expert for the input cue x-candidate answer yw pair, μ_l(x, yl) represent the mean reward of the l-th Gaussian expert for the input cue x-candidate answer yl pair, σ_k(x, yw) represent the standard deviation of the input cue x-candidate answer yw pair by the k-th Gaussian expert, and σ_l(x, yl) is the standard deviation of the input hint x-candidate response yl for the l-th Gaussian expert pair, sqrt() represents the square root operation, and ε is the numerical stability term.

[0016] Optionally, the iterative update of the policy model based on the trained probabilistic reward model and policy optimization algorithm, wherein the updated policy model generates content based on user input, includes: The input prompt strategy model generates candidate answers; Input the input prompts and their corresponding candidate answers into the trained probability reward model to generate reward evaluation results; Based on the reward evaluation results and the strategy optimization algorithm, the strategy model is iteratively updated to obtain the updated strategy model; Input the user-specified input prompts into the updated strategy model, and output the target generated content.

[0017] According to another aspect of this application, a content generation system based on a probabilistic reward model is provided, comprising: The acquisition module is used to collect and process paired preference samples to construct a preference training dataset; The network construction module is used to construct a probabilistic reward model that includes an expert head and routers. The expert head includes several experts set in parallel, each expert corresponding to a preference dimension. Each expert outputs the mean and standard deviation of the reward under its corresponding preference dimension. The router outputs a set of routing weights, each routing weight corresponding to an expert and its preference dimension. The training module is used to perform two-stage training on the probability reward model using the preference training dataset to obtain the trained probability reward model. The generation module is used to iteratively update the policy model based on the trained probability reward model and policy optimization algorithm. The updated policy model generates content based on the user's input, and the generated content is open-ended natural language text.

[0018] This application provides a content generation method based on a probabilistic reward model. It utilizes different experts to represent different preference dimensions, avoiding the simplistic compression of complex human preferences into a single scalar. The experts' outputs include the reward mean and standard deviation. The reward mean represents the subjective preference dimension, and the standard deviation represents cognitive uncertainty, enabling the probabilistic reward model to distinguish between reasonable disagreements and low-confidence judgments. Through two-stage training, the probabilistic reward model can learn the latent preference structure from the preference training dataset, completing the training of the probabilistic reward model. By simultaneously outputting the reward mean and standard deviation, the stability of the probabilistic reward model in policy model applications can be improved, thereby increasing the accuracy of the policy model's output content.

[0019] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0020] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart of a content generation method based on a probability reward model in one embodiment of this application; Figure 2 This is a structural diagram of the PRISM system in one embodiment of this application; Figure 3 This is a flowchart of the two-stage training process of PRISM in one embodiment of this application; Figure 4 This is a flowchart illustrating the application of PRISM for downstream reinforcement learning in one embodiment of this application. Figure 5 This is a schematic diagram of a content generation system based on a probability reward model in one embodiment of this application. Detailed Implementation

[0021] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0022] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations.

[0023] To clearly illustrate the technical solution of this application, the relevant terms involved in this application are explained below: PRISM: Probabilistic Reward Model with Inherent Structural Modeling.

[0024] LLM: Large Language Model, used to encode input prompts and candidate answers.

[0025] RM: Reward Model, used to generate reward signals based on input and candidate answers.

[0026] RLHF: Reinforcement Learning from Human Feedback.

[0027] BT model: Bradley-Terry model, used to calculate preference probability based on the difference in implicit strength between two candidates.

[0028] MoG: Mixture of Gaussians, a model that uses multiple Gaussian components to represent the conditional reward distribution.

[0029] Subjective Preference: Reasonable differences in preferences among qualified evaluators due to different evaluation dimensions.

[0030] Cognitive uncertainty: low-confidence judgments caused by vague input, outside the domain, or insufficient information.

[0031] Large language models have been widely used in tasks such as open-ended question answering, dialogue generation, code generation, content creation, and complex reasoning. To make the output of large language models more in line with human expectations, current mainstream methods employ alignment training processes such as reinforcement learning based on human feedback or reinforcement learning based on artificial intelligence feedback. Such processes typically require building a reward model or using a large language model evaluator to transform human or model preferences for several candidate answers into optimizable reward signals.

[0032] Existing reward models compress complex, multidimensional, and potentially conflicting human preferences into a single scalar, making it difficult to represent the multimodal preference distribution formed by different evaluation dimensions. Existing models struggle to distinguish between "reasonable subjective disagreements among qualified annotators" and "cognitive uncertainty caused by sample ambiguity, insufficient knowledge, or out-of-distribution input." Existing BT-style reward models typically only output point estimates, lacking reliable modeling of sample difficulty, preference conflict, and prediction confidence, leading to reward hacking and training instability in downstream reinforcement learning. Existing fine-grained reward modeling methods often rely on extensive attribute-level annotation, resulting in high annotation costs and difficulty in generalizing to large-scale general preference data. Existing multi-expert or ensemble reward models, lacking input contextualized routing mechanisms, struggle to dynamically combine experts based on different input evaluation objectives, thus failing to balance specialization, generalization ability, and interpretability. Based on these issues, this application provides a content generation method based on a probabilistic reward model to address these problems.

[0033] Reference Figure 1 and Figure 2 As shown, this application provides a content generation method based on a probabilistic reward model, including: S1. Collect and process paired preference samples to construct a preference training dataset; S2. Construct a probabilistic reward model that includes an expert head and a router. The expert head includes several experts set in parallel. Each expert corresponds to a preference dimension. Each expert outputs the mean and standard deviation of the reward under its corresponding preference dimension. The router outputs a set of routing weights. Each routing weight corresponds to an expert and its corresponding preference dimension. S3. Using the preference training dataset, the probability reward model is trained in two stages to obtain the trained probability reward model. S4. The policy model is iteratively updated based on the trained probability reward model and policy optimization algorithm. The updated policy model generates content based on the user's input. The generated content is open-ended natural language text.

[0034] For example, open-ended natural language text includes at least one of text responses, dialogue replies, content summaries, code-generated content, inference results, and recommendation results.

[0035] The embodiments described above utilize different experts to represent different preference dimensions, avoiding the simplistic compression of complex human preferences into a single scalar. The experts' outputs include the reward mean and standard deviation. The reward mean represents the subjective preference dimension, and the standard deviation represents cognitive uncertainty, enabling the probabilistic reward model to distinguish between reasonable disagreements and low-confidence judgments. Through two-stage training, the probabilistic reward model can learn the latent preference structure from the preference training dataset, completing the training of the probabilistic reward model. By simultaneously outputting the reward mean and standard deviation, the stability of the probabilistic reward model in policy model applications can be improved, thereby enhancing the accuracy of the policy model's output.

[0036] In some specific embodiments of this application, collecting and processing paired preference samples to construct a preference training dataset may further include: Collect several pairs of preference samples, each pair of preference samples includes input prompts and their corresponding candidate answers; perform text cleaning, text truncation, template formatting and data deduplication on the pairs of preference samples in sequence, and construct the preference training dataset from the processed pairs of preference samples.

[0037] For example, the paired preference samples include input prompts x, preferred candidate answers yw, unpreferred candidate answers yl, and optional context or evaluation principle labels. The paired preference samples are cleaned, truncated, and templated, and the entire sample is processed through an integrated batch processing workflow to construct a preference training dataset.

[0038] In some specific embodiments of this application, the probabilistic reward model uses a pre-trained large language model as the backbone network. The backbone network performs semantic encoding on the input pairwise preference samples and outputs hidden feature representations. Several Gaussian experts and a router are set up in parallel at the output of the backbone network, wherein: Each Gaussian expert corresponds to a preference dimension. Each Gaussian expert receives a hidden feature representation. Each Gaussian expert includes a mean head and a standard deviation head. The mean head generates the mean reward under the corresponding preference dimension based on the input hidden feature representation. The standard deviation head generates the standard deviation under the corresponding preference dimension based on the input hidden feature representation. The Gaussian expert generates a Gaussian reward distribution based on the reward mean and standard deviation. The router receives the hidden feature representation and outputs a number of routing weights equal to the number of Gaussian experts. Each routing weight corresponds one-to-one with an expert module and its generated reward mean and standard deviation. The outputs of each Gaussian expert are weighted and fused by routing weights to obtain the output of the probabilistic reward model, which is the reward evaluation result.

[0039] For example, a pre-trained large language model is used to encode input cues and candidate answers into hidden feature representations. The pre-trained large language model is used as the backbone network, and its main parameters are frozen during the training of the reward head, which primarily consists of Gaussian experts and routers, to reduce training costs and maintain general language representation capabilities. The expert head contains K Gaussian experts, each including at least a mean head and a variance head or standard deviation head. The mean head outputs the reward mean along a certain latent preference dimension, and the standard deviation / variance head outputs the standard deviation / variance, learning the cognitive uncertainty of the Gaussian expert's rating of the sample (i.e., the reward mean). This application also pre-defines a lower bound σ_min and an upper bound σ_max for the standard deviation to constrain the output of the standard deviation, avoiding instability in the standard deviation value and allowing cognitive uncertainty to serve as a reliability gating signal. The router generates a mixture weight π(x) of the K Gaussian experts based on the input cues or the context of the input in the hidden feature representation, thereby dynamically selecting or combining Gaussian experts for different inputs.

[0040] In another embodiment of this application, the frozen pre-trained large language model can be replaced with partial fine-tuning, LoRA fine-tuning, or Adapter fine-tuning, as long as its output can be used to output the mean of the reward and the uncertainty estimate. The mean head and standard deviation head can be linear layers, multilayer perceptrons, low-rank adapters, or other lightweight neural network structures. The number of experts K can be set according to the data scale and application scenario, such as 4, 8, 16, or more. The Gaussian distribution can be replaced with a t-distribution, logistic distribution, Laplace distribution, or other probability distribution that can express uncertainty, but a Gaussian mixture is preferred to balance computability and expressiveness. The router can generate weights based on input prompts, input plus answer, scoring principles, user profiles, task types, or domain labels.

[0041] In some specific embodiments of this application, the expression for the mean reward generated by the mean header is as follows: μ_k(x, y) = W_u^(k) h Where μ_k(x, y) is the mean reward of the k-th Gaussian expert for the input cue x-candidate answer y pair, W_u^(k) is the mean header parameter of the k-th Gaussian expert; h is the hidden feature representation; The expression for the standard deviation generated by the standard deviation head is as follows: σ_k(x, y) = σ_min + (σ_max - σ_min) · Sigmoid(z_k) z_k = W_σ^(k) h + b_σ^(k) Where σ_k(x, y) is the standard deviation of the input hint x-candidate answer y pair for the k-th Gaussian expert pair, σ_min is the lower bound of the standard deviation, σ_max is the upper bound of the standard deviation, Sigmoid() is the activation function, z_k is the intermediate latent variable, and W_σ^(k) and b_σ^(k) are both standard deviation head parameters; The expression for the Gaussian reward distribution is as follows: r_k(y | x) ~ N( μ_k(x, y), σ_k(x, y)^2 ) Where r_k(y | x) is the conditional probability distribution of the input cue x-candidate answer y pair for the k-th Gaussian expert pair.

[0042] For example, the output of a Gaussian expert includes the mean reward, standard deviation, and Gaussian reward distribution. Specifically, given an input cue x and a candidate answer y, the frozen backbone network outputs the hidden feature representation h of the input cue x-candidate answer y pair. This application sets a mean header for each Gaussian expert k to output the mean reward μ_k(x, y) under the corresponding preference dimension of the Gaussian expert. This mean reward can be understood as the Gaussian expert's rating of the answer quality under its preference dimension. To characterize cognitive uncertainty, this application further sets a standard deviation header for each Gaussian expert k. The standard deviation output preferably adopts a bounded form, where σ_k(x, y) represents the uncertainty estimate / standard deviation of the k-th Gaussian expert for the input cue x-candidate answer y pair. Thus, each Gaussian expert outputs a Gaussian reward distribution for the candidate answer y: r_k(y | x) ~ N( μ_k(x, y), σ_k(x, y)^2 ) Multiple Gaussian experts collectively form a mixture of Gaussian reward distributions. The mean reward of different Gaussian experts is used to represent different latent preference dimensions or subjective preference patterns, and the standard deviation / variance of different Gaussian experts is used to represent the cognitive uncertainty under the corresponding preference dimension / variance.

[0043] In some specific embodiments of this application, the expression for the routing weight output by the router is as follows: π(x) = Softmax(W_r h_x) Where π(x) is a set of K-dimensional routing weights under input cue x, K is the total number of Gaussian experts, W_r is the router parameters, Softmax() is the activation function, and h_x is the hidden feature representation corresponding to input cue x.

[0044] In some specific embodiments of this application, a preference training dataset is used to perform two-stage training on the probabilistic reward model to obtain the trained probabilistic reward model, which may further include: The probabilistic reward model is trained in two stages using a preference training dataset, with the main parameters of the backbone network frozen throughout the training process. In the first stage, the router parameters are frozen, and an uncertainty adaptive gating method is designed based on training dynamics to iteratively train the parameters of all Gaussian experts until the training of each Gaussian expert converges. After the Gaussian experts converge, the parameters of all Gaussian experts are frozen. In the second stage, the state of the Gaussian experts with frozen parameters remains unchanged, and iterative parameter training is performed only on the routers until the routers converge. When the routers have completed convergent training, the two-stage training process ends, and the trained probabilistic reward model is obtained.

[0045] For example, in the first stage of the training process, multiple Gaussian experts are trained to form complementary preference dimensions; in the second stage of the training process, the Gaussian experts are frozen and the router is trained so that the probabilistic reward model can adaptively aggregate experts based on input cues.

[0046] Based on the training dynamics, an uncertainty adaptive gating method is designed to carry out parameter iterative training for all Gaussian experts. Specifically, this means: using the uncertainty adaptive gating method based on training dynamics to calculate the gating weights corresponding to each Gaussian expert, and performing parameter iterative training for all Gaussian experts based on the gating weights. Specifically, the uncertainty adaptive gating method based on training dynamics involves initializing each Gaussian expert with orthogonal parameters, designing the training loss as the denominator parameter for the difference in uncertainty and the numerator parameter for the difference in the mean. For each training data point, gradient information is adaptively weighted based on training dynamics. Specifically, Gaussian experts that fit the current data tend to train a mean representation for that type of data, while Gaussian experts that do not fit the current data tend to train a standard deviation representation for that type of data.

[0047] In another embodiment of this application, the first-stage training can employ a uniform prior, or it can employ temperature annealing, entropy regularization, expert load balancing, or expert diversity regularization to enhance expert division of labor. The second-stage training can train only the router, or it can fine-tune a small number of expert parameters while freezing most of them. Uncertainty gating can be applied to reward probability, training loss, sample weights, advantage estimation, gradient pruning, or policy update step size. The method of this application can be used to train reward models, as well as to construct model evaluators, sample filters, preference data cleaners, or multidimensional evaluation interpreters.

[0048] The embodiments described above in this application extend the output of the probabilistic reward model from a single scalar to a conditional mixture Gaussian distribution to simultaneously model multidimensional subjective preferences and cognitive uncertainty. A mean head and a bounded standard deviation head are simultaneously set for each expert, and the standard deviation is used as a reliability gating signal in training and inference. Uncertainty-aware BT preference probabilities are constructed based on Gaussian reward difference and MacKay approximation, automatically reducing gradient contributions from high-uncertainty samples. A two-stage training method is employed: the first stage forms a multi-expert preference structure under no-routing or uniform prior conditions; the second stage freezes the experts and trains the input-aware router. Pairwise preference probabilities are calculated using an outer product mixture approach, allowing preferred and inferior answers to be interpreted by different expert dimensions. In downstream reinforcement learning, the uncertainty of the PRISM output is used to modulate rewards, advantages, or gradients to mitigate reward hacking and overoptimization. Interpretable multidimensional reward evaluation results are provided through expert means, expert variances, and routing weights.

[0049] In some specific embodiments of this application, during the first stage of training, a mixture likelihood loss function is used as the loss function for this stage of training. The first stage of training aims to minimize the mixture likelihood loss function, and all Gaussian experts are jointly trained. The expression for the mixture likelihood loss function is: L_stage1 = - E_(x,yw,yl) [ log( (1 / K) Σ_{k=1}^K P_k(yw yl | x) ) ] P_k(yw yl | Where L_stage1 represents the loss of the first stage of training, E_(x,yw,yl) represents the expected value of the pairwise preference samples, K represents the total number of Gaussian experts, and P_k(yw) represents the expected value of the pairwise preference samples. yl | x) represents the probability that the k-th Gaussian expert, given input cue x, determines that candidate answer yw is better than candidate answer yl; Sigmoid() represents the activation function, μ_k(x, yw) is the mean reward of the k-th Gaussian expert for the input cue x-candidate answer yw pair, μ_k(x, yl) is the mean reward of the k-th Gaussian expert for the input cue x-candidate answer yl pair, σ_k(x, yw) is the standard deviation of the k-th Gaussian expert for the input cue x-candidate answer yw pair, σ_k(x, yl) is the standard deviation of the k-th Gaussian expert for the input cue x-candidate answer yl pair, sqrt() represents the square root operation, and λ is the uncertainty weight coefficient.

[0050] For example, the goal of the first phase of training is to automatically differentiate multiple Gaussian experts into different potential preference dimensions, relying solely on large-scale pairwise preference sample data, in the absence of a large amount of fine-grained attribute labels. To avoid the router dominating expert selection in the early stages of training, the first phase of training preferably bypasses the router and uses a uniform prior for the K Gaussian experts.

[0051] For a pairwise preference sample (x, yw, yl), where yw represents the preferred candidate answer and yl represents the unpreferred candidate answer, the reward difference D_k between the two candidate answers for the k-th Gaussian expert is: D_k = r_k(yw | x) - r_k(yl | x) Since both rewards are Gaussian distributed, their difference is also Gaussian distributed: D_k ~ N( μ_k(x, yw) - μ_k(x, yl), σ_k(x, yw)^2 + σ_k(x, yl)^2) In the BT model, the preference probability is typically obtained by applying a sigmoid function to the reward difference. Considering that the reward difference itself is a Gaussian variable, direct integration yields no closed-form solution. Therefore, this application uses the MacKay approximation to obtain the following computable form: P_k(yw yl | Where λ can be π / 8, The symbol for the preference relation is "better than". The uncertainty term in the above formula is in the denominator. When a Gaussian expert has high uncertainty about a certain preference dimension, the denominator increases, the reward difference decreases, and the preference probability approaches 0.5, thus reducing the gradient contribution of the paired preference sample to the Gaussian expert. This mechanism is equivalent to an uncertainty-induced reliability gating, which causes the Gaussian expert to actively learn only within its confident preference region, while automatically reducing the learning intensity for ambiguous, conflicting, or out-of-domain samples.

[0052] The first stage of training can employ the following mixed likelihood objective to train K Gaussian experts: L_stage1 = - E_(x,yw,yl) [ log( (1 / K) Σ_{k=1}^K P_k(yw yl | x) ) ] Through the aforementioned objective loss function, different Gaussian experts will competitively interpret different types (i.e., different preference dimensions) of paired preference samples during training. For some paired preference samples, high-confidence experts obtain larger effective gradients and learn the corresponding preference structures; low-confidence experts, due to higher uncertainty, have their gradients naturally weakened. Thus, Gaussian experts can form a division of labor even with no explicit attribute labels or only a few attribute labels.

[0053] In the above embodiments of this application, the preference probability between two candidate answers can be calculated based on the mixed Gaussian reward distribution, and training loss or inference score can be generated.

[0054] In some specific embodiments of this application, during the second-stage training process, the router is trained with the optimization objective of minimizing the loss function of the second-stage training. The expression for the loss function of the second-stage training is: L_router = - E_(x,yw,yl) [ log P(yw yl | x) ] P(yw yl | x) = Σ_{k=1}^K Σ_{l=1}^K π_k(x) π_l(x) p_{kl} p_{kl}= Φ([μ_k(x, yw) - μ_l(x, yl) ] / sqrt( σ_k(x, yw)^2 +σ_l(x,yl)^2 + ε)) Where L_router represents the loss of the second-stage training, E_(x,yw,yl) represents the expected value of the pairwise preference samples, and P(yw) represents the expected value of the pairwise preference samples. Let yl | x) represent the probability that the router favors candidate answer yw over candidate answer yl under input cue x, K represent the total number of Gaussian experts, π_k(x) represent the routing weight of the k-th Gaussian expert under input cue x, π_l(x) represent the routing weight of the l-th Gaussian expert under input cue x, p_{kl} represent the probability that the reward of the k-th Gaussian expert for the favored candidate answer yw is greater than the reward of the l-th Gaussian expert for the unfavored candidate answer yl under input cue x, Φ() represent the cumulative distribution function of the standard normal distribution, μ_k(x,yw) represent the mean reward of the k-th Gaussian expert for the input cue x-candidate answer yw pair, μ_l(x, yl) represent the mean reward of the l-th Gaussian expert for the input cue x-candidate answer yl pair, σ_k(x, yw) represent the standard deviation of the input cue x-candidate answer yw pair by the k-th Gaussian expert, and σ_l(x, yl) is the standard deviation of the input hint x-candidate response yl for the l-th Gaussian expert pair, sqrt() represents the square root operation, and ε is the numerical stability term.

[0055] For example, after the first phase of training, multiple Gaussian experts have formed different preference dimensions and uncertainty estimates. The goal of the second phase of training is to learn an input-aware dynamic aggregation mechanism, enabling the probabilistic reward model to infer the appropriate evaluation criterion based on the current input cue x and assign routing weights to each Gaussian expert. The second phase of training preferably freezes all expert heads obtained in the first phase and trains only the router parameters. The router extracts the hidden feature representation h_x of the input cue x from the frozen backbone network and outputs the routing weights: π(x) = Softmax(W_r h_x) Where π(x) is a K-dimensional probability vector, representing the degree of dependence of the current input cue x on each Gaussian expert, and W_r is the router parameter. The router can generate weights based solely on the input cue x, or it can generate weights in combination with task descriptions, scoring principles, context labels, domain labels, or user preference descriptions.

[0056] For paired preference samples (x, yw, yl), this application does not mandate the selection of a single Gaussian expert. Instead, it marginalizes both the preferred candidate answer (top choice) and the unpreferred candidate answer (bottom choice) along the preference dimension, obtaining the mixed preference probability in the form of an outer product: P(yw yl | x) = Σ_{k=1}^K Σ_{l=1}^K π_k(x) π_l(x) p_{kl} Where p_{kl} represents the probability that the reward for the best answer from the k-th expert is greater than the reward for the worst answer from the l-th expert. Since both rewards are Gaussian distributed, this probability can be written as: p_{kl}= Φ([μ_k(x, yw) - μ_l(x, yl) ] / sqrt( σ_k(x, yw)^2 + σ_l(x,yl)^2 + ε)) Where Φ represents the cumulative distribution function of the standard normal distribution, and ε is the numerically stable term. This outer product mixture form can simultaneously consider multiple preference dimensions that may exist in the same input cue x, and provides the router with a smooth and stable training gradient. The router training loss in the second stage of training is: L_router = - E_(x,yw,yl) [ log P(yw yl | x) ] After training, PRISM can output a conditional reward distribution (i.e., reward evaluation result) consisting of multiple Gaussian experts and their routing weights for any input cue-candidate answer pair, instead of a single reward point estimate.

[0057] In some specific embodiments of this application, the policy model is iteratively updated based on the trained probabilistic reward model and policy optimization algorithm. The updated policy model generates content based on user input and may further include: S41. Generate candidate answers from the input prompt input strategy model; S42. Input the input prompts and their corresponding candidate answers into the trained probability reward model to generate reward evaluation results; S43. Based on the reward evaluation results and the strategy optimization algorithm, iteratively update the strategy model to obtain the updated strategy model; S44. Input the user-specified input prompt into the updated strategy model and output the target generated content.

[0058] For example, during the inference phase, PRISM receives an input cue x and one or more candidate answers y, and outputs the following information: The mean reward for each expert, μ_k(x, y), is used to represent the quality score under that expert's preference dimension.

[0059] The standard deviation σ_k(x,y) for each expert is used to represent the uncertainty of that expert's quality score.

[0060] The routing weight π_k(x) is used to represent the degree of dependence of the current input prompt on different preference dimensions.

[0061] A mixed Gaussian reward distribution is used to preserve the multidimensional preference structure.

[0062] P(yw) is the preference probability among candidate answers. yl | x), used for sorting, filtering, or reinforcement learning rewards.

[0063] Optional scalar outputs, such as the mixed reward mean and the PRISM-MoG preference probability, where the mixed reward mean is a weighted average scalar calculated using the weights output by the router (i.e., the routing weights) and the reward mean of each expert, and the PRISM-MoG preference probability is a comparative victory probability calculated using the distribution of the reference answer.

[0064] In the application of PRISM, the Gaussian reward distribution, preference probability, and standard deviation output by PRISM are passed to policy optimization algorithms such as PPO and GRPO to train the policy model. Specifically, when compatibility with existing reinforcement learning algorithms is required, the reward evaluation results of PRISM can be converted into scalar rewards. For example, the preference probability P(yp) is calculated for the candidate answer yp and the baseline answer yb generated by the policy model. yb | x), where the preference probability is used as the reward.

[0065] The PRISM of this application can be used as a reward or evaluation module in processes such as PPO, GRPO, DPO variants, rejection sampling fine-tuning, Best-of-N selection, and model evaluation ranking. Taking GRPO as an example, the training process may include: For the same input prompt x, multiple candidate answers are sampled by the current policy model.

[0066] PRISM was used to calculate the preference probability, mean reward, and standard deviation of each candidate response relative to the benchmark response or other responses within the group.

[0067] The reward evaluation results are generated based on the PRISM output; for pairwise preference samples with high uncertainty, the routing weight or gradient weight is reduced.

[0068] Update the policy model using GRPO or other policy optimization algorithms.

[0069] Periodically evaluate the generalization performance of the policy model on human evaluation, LLM evaluator or public benchmarks, and prevent overoptimization caused by a single scalar reward, and finally obtain an updated policy model.

[0070] Compared to traditional scalar rewards, PRISM can provide a smoother and more reliable optimization landscape in downstream reinforcement learning. Its uncertainty gating mechanism can suppress overly strong reward signals in fuzzy or out-of-distribution regions, thereby mitigating reward hacking.

[0071] The PRISM proposed in this application redefines reward evaluation as a conditional probability distribution modeling problem. It simultaneously models multidimensional subjective preferences and cognitive uncertainty through a Gaussian mixture structure and achieves scalable learning through two-stage training. This application has the following characteristics: 1. Preserving the multidimensional structure of human preferences: This application models the reward as a mixture Gaussian distribution, which can represent different preference dimensions with different Gaussian experts, avoiding the crude compression of complex human preferences into a single scalar.

[0072] 2. Distinguishing between subjective preferences and cognitive uncertainty: The reward mean represents the subjective preference dimension, and the standard deviation represents cognitive uncertainty, enabling PRISM to distinguish between reasonable disagreement and low-confidence judgments.

[0073] 3. Reduced dependence on fine-grained annotation: Through two-stage training, this application is able to learn the latent preference structure from large-scale pairwise preference sample data, requiring only a small amount of context or evaluation principle labels to train the input router.

[0074] 4. Enhance the generalization ability of the reward model: Input conditional routing can dynamically combine experts according to different tasks, improving the adaptability of PRISM to multi-attribute, multi-domain and out-of-distribution data.

[0075] 5. Mitigating reward hacking: Uncertainty acts as a reliability gating mechanism, automatically reducing the reward or gradient strength on highly uncertain samples, thereby reducing the risk of policy models exploiting PRISM defects.

[0076] 6. Improve training stability: Mixed distribution and outer product preference probability provide a smoother training signal, which can reduce the performance collapse caused by over-optimization of a single scalar reward.

[0077] 7. Enhanced interpretability: Through expert relevance, expert attribute performance, routing weights, and uncertainty visualization, it is possible to explain why PRISM favors a certain answer and whether the current judgment is reliable.

[0078] 8. Lower computational and deployment costs: The preferred method is to freeze the backbone network of a large language model, train only lightweight expert heads and routers, which reduces training costs and facilitates integration into existing reward model systems.

[0079] The probabilistic reward model of this application can be used in scenarios that require multidimensional evaluation, ranking, or optimization of open-source generated results. Its core advantage lies in the fact that when evaluation criteria conflict, samples are difficult, or inputs deviate from the training distribution, the probabilistic reward model not only outputs the preference probability or expected reward, but also the expert dimension and uncertainty, thereby facilitating system reliability control.

[0080] (1) Open-ended dialogue and content generation alignment scenarios: In tasks such as chat assistants, writing assistants, summary generation, advertising copy generation, and educational Q&A, the same answer may simultaneously involve usefulness, authenticity, style, security, conciseness, and fun. The probabilistic reward model of this application can map the above criteria to different potential expert dimensions and automatically combine experts according to the current input, avoiding the use of a single scalar reward to represent complex preferences.

[0081] (2) Multi-attribute automatic evaluation and model ranking scenarios: In model benchmark testing, leaderboard evaluation, candidate answer ranking, A / B testing, and manual review assistance systems, the probability reward model of this application can output the preference probability among candidate answers, the contribution of each expert, and the uncertainty. For samples with high uncertainty, the system can require manual review, reduce the weight of automatic scoring, or trigger supplementary sampling.

[0082] (3) Reinforcement learning scenarios based on scoring rules: In policy optimization processes such as Rubric-based Reinforcement Learning, PPO, and GRPO, scoring rules often contain multiple dimensions. The probabilistic reward model in this application can use the complete mixture Gaussian distribution, routing weights, and uncertainty reduction mechanism as reward sources, so that the policy model can take into account multiple evaluation objectives during optimization and reduce overfitting to a single reward agent.

[0083] (4) Best-of-N selection, rejection sampling fine-tuning and data screening scenarios: When multiple candidate answers are generated for the same input, the probability reward model of this application can sort them according to the PRISM-MoG preference probability and select high-quality and low-uncertainty samples for output, fine-tuning or construction of new preference data; for samples with high uncertainty or large expert disagreement, they can be marked as samples to be reviewed.

[0084] (5) Reliability gating scenarios for out-of-distribution inputs and high-risk content: When the input belongs to a new domain outside the training data, is ambiguous, or requires careful judgment, the variance / standard deviation output of the probability reward model in this application can be used as a reliability indicator. The system can reduce rewards, reduce the policy update magnitude, refuse automatic decision-making, or switch to manual review based on uncertainty, thereby reducing reward hacking and erroneous automated judgments.

[0085] (6) Vertical Industry Large Model Quality Inspection Scenarios: In scenarios such as finance, healthcare, law, government affairs, customer service, and education, the quality of responses is usually determined by professionalism, compliance, factual accuracy, explainability, and user-friendliness. The probabilistic reward model in this application can drive the input perception router through domain labels, task types, or scoring principles, enabling the same reward model to adapt to different business objectives.

[0086] The present application will be further described below with reference to specific embodiments in order to better understand the above technical solutions of the present application. It should be understood that the following are only some examples and are not intended to limit the present application.

[0087] Example 1: Content Generation Method Based on Probabilistic Reward Model Step 1, refer to Figure 3 As shown, the PRISM reward model is trained based on 8 Gaussian experts: Collect a preference training dataset, where each pair of preference samples includes an input prompt x, a preferred answer yw, and a less preferred answer yl. Data sources can include general preference data, attribute preference data, manually labeled data, or AI feedback data.

[0088] Select a pre-trained large language model as the frozen backbone network, such as a 3B or 7B language model. Concatenate the input prompts and candidate answers and feed them into the backbone network to extract the last token or the hidden state representation h at a specified position.

[0089] Set up K=8 Gaussian experts. Each expert includes a mean header and a standard deviation header. The standard deviation header output is mapped to the range [σ_min, σ_max] using the Sigmoid function.

[0090] The first phase involves training the expert heads. For each preference pair, Gaussian BT preference probabilities are calculated for the eight experts, and the negative log-likelihood loss is calculated after averaging the expert probabilities. The input router is not enabled during training.

[0091] After the first phase convergence, all expert heads are frozen. The router is trained using data containing context, task type, or evaluation principle information, and outputs 8-dimensional expert weights based on input prompts. In the second phase, the overall preference probability is calculated using the outer product mixture formula, and only the router parameters are updated.

[0092] After training, PRISM outputs eight expert reward mean values, eight expert standard deviations, eight routing weights, and a Gaussian mixture reward distribution for any answer.

[0093] Step Two, Refer to Figure 4 As shown, PRISM is a reinforcement learning reward model. Given an input cue x for training, the policy model samples several candidate answers.

[0094] Choose a reference answer or a baseline answer within the group as the comparison object.

[0095] The probability P(yp) of a candidate answer being better than the reference answer is calculated using the PRISM strategy model. yb | x) = P(r | x), and calculate the mixed uncertainty (i.e., standard deviation).

[0096] The preference probability or its transformed value is used as the reward; when the mixed uncertainty is higher than a threshold, the mean reward is reduced or the advantage weight of the sample is decreased.

[0097] Update the policy model using PPO, GRPO, or other policy optimization methods.

[0098] Periodic validation is used to prevent reward hacking, such as monitoring human-assessed win rates, LLM-assessed win rates, out-of-distribution task performance, and uncertainty distribution, ultimately resulting in an updated policy model.

[0099] Example 2: PRISM is used for automatic evaluation and sample selection in multiple scenarios. The steps in Example 2 are the same as those in Example 1, except that PRISM is applied in the policy model, specifically including the following steps: Given an input cue x to be evaluated and multiple candidate answers, PRISM calculates the PRISM-MoG preference probability, mixed reward mean, mixed uncertainty, and input routing weights for each pair of candidate answers. PRISM ranks candidate answers based on preference probabilities and prioritizes answers with high preference probabilities and low uncertainty as the final output or training samples. When there is significant disagreement among multiple experts or the mixed uncertainty exceeds a threshold, PRISM marks the sample as a high-dispute sample for manual review, active learning, or preference data cleaning. In vertical industry applications, the router can combine task type, scoring principles, or domain labels to generate expert weights, allowing the model to conditionally balance input accuracy, compliance, professionalism, and user-friendliness. Finally, the policy model is iteratively updated based on the training samples and the policy optimization algorithm to obtain the updated policy model.

[0100] Experimental comparison results and technical effect verification: To demonstrate the technical performance of this application compared to traditional scalar reward models and existing multi-expert reward models, validation can be performed using a public preference evaluation set and downstream reinforcement learning tasks. All major reward model comparisons use the same 3B-level frozen backbone model or the same training data settings, thus showcasing the improvements brought by Gaussian mixture modeling, input-aware routing, and uncertainty gating. Regarding reward model prediction accuracy, PRISM achieves the highest average accuracy on HelpSteer2, RPR, and Reward-Bench. HelpSteer2 and RPR are used to validate multi-attribute preference modeling capabilities, while Reward-Bench is used to validate out-of-distribution generalization capabilities.

[0101] Table 1 Comparison of prediction accuracy of reward models Here, Single-BT represents the traditional reward model. The above results show that this application outperforms the single-head BT reward model, as well as static hybrid, multi-head ensemble, and MoE-style reward models in both multi-attribute subjective preference evaluation and out-of-distribution generalization evaluation.

[0102] In terms of uncertainty modeling, PRISM improves the mean variance of out-of-distribution Reward-Bench samples by approximately 25.28% compared to HelpSteer2 samples within the training distribution. This phenomenon indicates that the reward model in this application can proactively increase uncertainty on unfamiliar or more difficult-to-judge inputs, rather than forcibly giving an overconfident scalar reward, and therefore can serve as a reliability gating signal in downstream training and automatic evaluation.

[0103] In the downstream Rubric-based RL task, PRISM-MoG is used as the reward signal to train the policy model, trained using a subset of the HelpSteer3-Principle, and validated on the out-of-domain Arena-Hard v2.0 dataset. Comparison methods include: PRISM-Weight using only mixed means, PRISM-Uniform without input routing, the traditional Single-BT reward model, and the Base model not trained with this reward.

[0104] Table 2. Generalization results of downstream Rubric-based RL on Arena-Hard v2.0 The results show that the full PRISM-MoG reward outperforms the compressed scalar version, the unrouted version, and the Single-BT version on all three policy models, indicating that mixed distributions, uncertainty, and input conditional routing are all effective for policy optimization. Furthermore, the Single-BT scalar reward tends to peak early in training but then experiences a significant drop in win rate, indicating that the policy model exploits a scalar reward vulnerability rather than genuinely improving answer quality. In contrast, the PRISM-MoG proposed in this application maintains a higher and more stable win rate during training. These results validate the practical effectiveness of this application in mitigating reward hacking and improving training stability through probability distribution and uncertainty gating.

[0105] In summary, this application demonstrates from four aspects—prediction accuracy, out-of-distribution uncertainty, downstream reinforcement learning stability, and cross-model generalization—that it can not only improve the discriminative ability of the reward model itself, but also provide more reliable reward signals during the policy optimization stage.

[0106] Reference Figure 5 As shown, based on the same inventive concept, another embodiment of this application provides a content generation system based on a probabilistic reward model. The content generation system 100 includes: The acquisition module 110 is used to acquire and process paired preference samples to construct a preference training dataset; The network construction module 120 is used to construct a probabilistic reward model that includes an expert head and routers. The expert head includes several experts set in parallel, each expert corresponding to a preference dimension. Each expert outputs the mean and standard deviation of the reward under its corresponding preference dimension. The router outputs a set of routing weights, each routing weight corresponding to an expert and its preference dimension. Training module 130 is used to perform two-stage training on the probabilistic reward model using a preference training dataset to obtain the trained probabilistic reward model. The generation module 140 is used to iteratively update the policy model based on the trained probabilistic reward model and policy optimization algorithm. The updated policy model generates content based on the user's input, and the generated content is open natural language text.

[0107] It should be noted that the modules in the content generation system based on the probability reward model provided in the above embodiments of this application correspond to the steps of the content generation method based on the probability reward model in any of the above embodiments. Those skilled in the art can refer to the step features of the content generation method based on the probability reward model to implement the corresponding modules in the content generation system based on the probability reward model, which will not be elaborated here.

[0108] In another embodiment of this application, an electronic device is also provided, including a memory and a processor; the memory is used to store program instructions; the processor is used to call the program instructions stored in the memory and execute the steps of the above-described content generation method based on a probability reward model according to the obtained program instructions.

[0109] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0110] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0111] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0112] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0113] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0115] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0116] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0117] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0118] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A content generation method based on a probabilistic reward model, characterized in that, include: Collect and process paired preference samples to construct a preference training dataset; Construct a probabilistic reward model that includes an expert head and a router; the expert head includes several experts set in parallel, each expert corresponds to a preference dimension, and each expert outputs the mean and standard deviation of the reward under its corresponding preference dimension; the router outputs a set of routing weights, each routing weight corresponds to an expert and its corresponding preference dimension. Using the aforementioned preference training dataset, the probability reward model is trained in two stages to obtain the trained probability reward model. The policy model is iteratively updated based on the trained probability reward model and policy optimization algorithm. The updated policy model generates content based on user input, and the generated content is open-ended natural language text.

2. The content generation method based on a probabilistic reward model according to claim 1, characterized in that, The process of collecting and processing paired preference samples to construct a preference training dataset includes: Collect several pairs of preference samples, each pair of preference samples including input prompts and their corresponding candidate answers; perform text cleaning, text truncation, template formatting and data deduplication on the pairs of preference samples in sequence, and construct the preference training dataset from the processed pairs of preference samples.

3. The content generation method based on a probabilistic reward model according to claim 1, characterized in that, The probabilistic reward model uses a pre-trained large language model as its backbone network. The backbone network performs semantic encoding on the input pairwise preference samples and outputs hidden feature representations. Several Gaussian experts and one router are configured in parallel at the output end of the backbone network, wherein: Each Gaussian expert corresponds to a preference dimension. Each Gaussian expert receives the hidden feature representation. Each Gaussian expert includes a mean head and a standard deviation head. The mean head generates the reward mean under the corresponding preference dimension based on the input hidden feature representation. The standard deviation head generates the standard deviation under the corresponding preference dimension based on the input hidden feature representation. The Gaussian expert generates a Gaussian reward distribution based on the reward mean and the standard deviation. The router receives the hidden feature representation and outputs a number of routing weights equal to the number of Gaussian experts. Each routing weight corresponds one-to-one with an expert module and its generated reward mean and standard deviation. The outputs of each Gaussian expert are weighted and fused using routing weights to obtain the output of the probabilistic reward model, which is the reward evaluation result.

4. The content generation method based on a probabilistic reward model according to claim 3, characterized in that, The expression for the mean reward generated by the mean head is as follows: μ_k(x, y) = W_u^(k) h; Where μ_k(x, y) is the mean reward of the k-th Gaussian expert for the input cue x-candidate answer y pair, W_u^(k) is the mean header parameter of the k-th Gaussian expert; h is the hidden feature representation; The expression for the standard deviation generated by the standard deviation head is as follows: σ_k(x, y) = σ_min + (σ_max - σ_min) · Sigmoid(z_k); z_k = W_σ^(k) h + b_σ^(k); Where σ_k(x, y) is the standard deviation of the input hint x-candidate answer y pair for the k-th Gaussian expert pair, σ_min is the lower bound of the standard deviation, σ_max is the upper bound of the standard deviation, Sigmoid() is the activation function, z_k is the intermediate latent variable, and W_σ^(k) and b_σ^(k) are both standard deviation head parameters; The expression for the Gaussian reward distribution is as follows: r_k(y | x) ~ N( μ_k(x, y), σ_k(x, y)^2); Where r_k(y | x) is the conditional probability distribution of the input cue x-candidate answer y pair for the k-th Gaussian expert pair.

5. The content generation method based on a probabilistic reward model according to claim 3, characterized in that, The expression for the routing weight output by the router is as follows: π(x) = Softmax(W_r h_x); Where π(x) is a set of K-dimensional routing weights under input cue x, K is the total number of Gaussian experts, W_r is the router parameters, Softmax() is the activation function, and h_x is the hidden feature representation corresponding to input cue x.

6. The content generation method based on a probabilistic reward model according to claim 3, characterized in that, The step of using the preference training dataset to perform two-stage training on the probability reward model to obtain the trained probability reward model includes: The probabilistic reward model is trained in two stages using the aforementioned preference training dataset, with the main parameters of the backbone network frozen throughout the training process. In the first stage, the parameters of the routers are frozen, and iterative parameter training is performed on all Gaussian experts until their training converges. After the Gaussian experts converge, the parameters of all Gaussian experts are frozen again. In the second stage, the state of the Gaussian experts with frozen parameters remains unchanged, and iterative parameter training is performed only on the routers until their training converges. Once the routers have completed convergent training, the two-stage training process ends, yielding the trained probabilistic reward model.

7. The content generation method based on a probabilistic reward model according to claim 6, characterized in that, In the first stage of training, a mixture likelihood loss function is used as the loss function for this stage of training. The first stage of training aims to minimize the mixture likelihood loss function, and all Gaussian experts are jointly trained. The expression for the mixture likelihood loss function is: L_stage1 = - E_(x,yw,yl) [ log( (1 / K) Σ_{k=1}^K P_k(yw yl | x) ) ]; P_k(yw yl | x) =Sigmoid( [ μ_k(x, yw) - μ_k(x, yl) ] / sqrt( 1 + λ[σ_k(x, yw)^2 + σ_k(x, yl)^2] ) ); Where L_stage1 represents the loss of the first stage of training, E_(x,yw,yl) represents the expected value of the pairwise preference samples, K represents the total number of Gaussian experts, and P_k(yw) represents the expected value of the pairwise preference samples. yl | x) represents the probability that the k-th Gaussian expert, given input cue x, determines that candidate answer yw is better than candidate answer yl; Sigmoid() represents the activation function, μ_k(x, yw) is the mean reward of the k-th Gaussian expert for the input cue x-candidate answer yw pair, μ_k(x, yl) is the mean reward of the k-th Gaussian expert for the input cue x-candidate answer yl pair, σ_k(x, yw) is the standard deviation of the k-th Gaussian expert for the input cue x-candidate answer yw pair, σ_k(x, yl) is the standard deviation of the k-th Gaussian expert for the input cue x-candidate answer yl pair, sqrt() represents the square root operation, and λ is the uncertainty weight coefficient.

8. The content generation method based on a probabilistic reward model according to claim 6, characterized in that, In the second training phase, the router is trained with the optimization objective of minimizing the loss function of the second training phase. The expression for the loss function of the second training phase is as follows: L_router = - E_(x,yw,yl) [ log P(yw year | x) ]; P(yw yl | x) = Σ_{k=1}^K Σ_{l=1}^K π_k(x) π_l(x) p_{kl}; p_{kl}=Φ([μ_k(x, yw)-μ_l(x, yl)] / sqrt( σ_k(x, yw)^2 +σ_l(x, yl)^2+ε)); Where L_router represents the loss of the second-stage training, E_(x,yw,yl) represents the expected value of the pairwise preference samples, and P(yw) represents the expected value of the pairwise preference samples. Let yl | x) represent the probability that the router favors candidate answer yw over candidate answer yl under input cue x, K represent the total number of Gaussian experts, π_k(x) represent the routing weight of the k-th Gaussian expert under input cue x, π_l(x) represent the routing weight of the l-th Gaussian expert under input cue x, p_{kl} represent the probability that the reward of the k-th Gaussian expert for the favored candidate answer yw is greater than the reward of the l-th Gaussian expert for the unfavored candidate answer yl under input cue x, Φ() represent the cumulative distribution function of the standard normal distribution, μ_k(x, yw) represent the mean reward of the k-th Gaussian expert for the input cue x-candidate answer yw pair, μ_l(x, yl) represent the mean reward of the l-th Gaussian expert for the input cue x-candidate answer yl pair, σ_k(x, yw) represent the standard deviation of the input cue x-candidate answer yw pair by the k-th Gaussian expert, and σ_l(x, yl) is the standard deviation of the input hint x-candidate response yl for the l-th Gaussian expert pair, sqrt() represents the square root operation, and ε is the numerical stability term.

9. The content generation method based on a probabilistic reward model according to claim 1, characterized in that, The strategy model is iteratively updated based on the trained probability reward model and strategy optimization algorithm. The updated strategy model generates content based on user input, including: The input prompt strategy model generates candidate answers; Input the input prompts and their corresponding candidate answers into the trained probability reward model to generate reward evaluation results; Based on the reward evaluation results and the strategy optimization algorithm, the strategy model is iteratively updated to obtain the updated strategy model; Input the user-specified input prompts into the updated strategy model, and output the target generated content.

10. A content generation system based on a probabilistic reward model, characterized in that, include: The acquisition module is used to collect and process paired preference samples to construct a preference training dataset; The network construction module is used to construct a probabilistic reward model that includes an expert head and routers. The expert head includes several experts set in parallel, each expert corresponding to a preference dimension. Each expert outputs the mean and standard deviation of the reward under its corresponding preference dimension. The router outputs a set of routing weights, each routing weight corresponding to an expert and its preference dimension. The training module is used to perform two-stage training on the probability reward model using the preference training dataset to obtain the trained probability reward model. The generation module is used to iteratively update the policy model based on the trained probability reward model and policy optimization algorithm. The updated policy model generates content based on the user's input, and the generated content is open-ended natural language text.

Images