Large model value alignment method and system based on user dynamic cognitive motivation

Abstract

The application relates to the technical field of natural language generation, and discloses a large model value alignment method and system based on user dynamic cognitive motivation, which comprises the following steps: capturing cognitive signals generated in the interaction process of a user and a large model in real time, and quantifying the cognitive signals into numerical values or vector forms; weighting and fusing the quantified cognitive signals based on the weight vectors pre-trained offline for each user to generate a scalar reward signal; improving a near-end strategy optimization algorithm to realize parameter updating of the large model in an online single-sample scenario; and when the large model performs parameter updating, using full-amount parameter updating and an asynchronous parameter updating mechanism, so that the large model receives new user cognitive signals while completing full-amount parameter updating in the background. The method can effectively solve the problems of static preference, updating lag and high cost in the existing large model value alignment technology, and provides important technical support for safe and controllable application of the large model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language generation, in particular to a large model value alignment method and system based on user dynamic cognitive motivation. BACKGROUND

[0002] With the wide application of large language models in various fields, such as DeepSeek, GPT-5, etc. models show strong ability in text generation, dialogue system, content creation, etc. The alignment of artificial intelligence and human values is increasingly prominent. The value alignment of large language models is to ensure that the behavior and output of large language models conform to human values and social expectations.

[0003] The existing large model value alignment method mainly adopts reinforcement learning technology based on human feedback, collects preference data of human annotators on model output, trains reward model, and then uses reinforcement learning algorithm to optimize language model. However, the existing reinforcement learning based on human feedback (RLHF) depends on offline static preference and single sample reward: on the one hand, it is difficult to reflect the real-time cognition and individual differences of users in real interaction; on the other hand, in real online interaction, only one output and one reward are generated for each interaction, which is not suitable for algorithms that rely on relative comparison of multiple candidates under the same prompt. SUMMARY

[0004] To solve the above technical problems, the present application provides a large model value alignment method and system based on user dynamic cognitive motivation; the present application collects online multi-source cognitive signals, constructs personalized and time-varying reward functions, and uses proximal policy optimization (PPO) algorithm for online alignment update: uses value function as baseline and cooperates with advantage function (GAE) to obtain stable advantage, combines reference strategy to apply KL divergence constraint (adaptive coefficient), thereby realizing low delay, controllable and stable value alignment in single sample online scene.

[0005] To solve the above technical problems, the present application adopts the following technical solutions:

[0006] In the first aspect, the present application provides a large model value alignment method based on user dynamic cognitive motivation, comprising:

[0007] Real-time capture of cognitive signals generated in the interaction process of users and large models, and quantification into numerical or vector form; the cognitive signals include explicit feedback and implicit feedback;

[0008] Based on the weight vector pre-trained offline for each user, the quantified cognitive signals are weighted and fused to generate scalar reward signals;

[0009] The proximal policy optimization algorithm is improved in the following way to realize parameter updating of the large model in the online single-sample scene: the scalar reward signal is assigned to each generation step of the large model, the state value is estimated by the time difference method, the advantage function is calculated combined with the state value, a value network is trained as a baseline, and then the policy is optimized by cutting the target function and self-adaptive KL divergence constraint;

[0010] When the large model updates the parameters, full-quantity parameter updating is adopted, and an asynchronous parameter updating mechanism is adopted, so that the large model can receive new user cognitive signals while completing full-quantity parameter updating in the background.

[0011] In one embodiment, the explicit feedback is the direct interaction behavior of the user to the output content of the large model, including the numerical score of the user to the output content, binary feedback and the text feedback of the user; the text feedback is converted into a numerical evaluation score by a large language model with frozen parameters combined with a pre-defined score scale.

[0012] In one embodiment, the implicit feedback is the unconscious interaction behavior of the user to the output content of the large model, including the reading time of the user to the output content, the repeated viewing times, the subsequent inquiry behavior and the task completion degree in a specific scene; the interaction time is logarithmically transformed, the subsequent inquiry behavior is processed as a binary flag, and the task completion degree is converted into a percentage.

[0013] In one embodiment, the weight vector pre-trained for each user offline includes:

[0014] The historical cognitive signals of the user interacting with the large model are sampled, the sampled samples are evaluated by experts to generate reward scores; the historical cognitive signals of the user interacting with the large model are taken as input features, and the reward scores evaluated by experts are taken as supervision labels, and the weight vector of the current user is solved by training and solving the supervision learning task .

[0015] In one embodiment, the quantized cognitive signals are weighted and fused to generate a scalar reward signal, which specifically includes:

[0016] In the real-time interaction process between the user and the large model, the corresponding quantized cognitive signals are constructed into a feature vector, and the scalar reward signal The scalar reward signal is calculated by the dot product of the pre-trained weight vector of the current user and the feature vector.

[0017] ;

[0018] is the transpose, and represents the user At any moment For the output of large models The immediate cognitive characteristics, Indicates user Deadline The state of historical cognition This represents a feature extraction function that maps immediate cognitive features and historical cognitive states into a unified feature vector. This is the index of time t.

[0019] In one embodiment, the process of allocating the scalar reward signal to each generation step of the large model, estimating the state value using a time-difference method, and calculating the advantage function based on the state value specifically includes:

[0020] During the process of generating output from a large model, the previously generated large model will be used as the output. A sequence of tokens is defined as a state. , will generate The operation of each token is defined as an action; According to the set reward distribution rules, the rewards are broken down into a series of instant rewards arranged in time steps. For the state... The next generation Instant rewards after obtaining tokens The time difference method is used to estimate the state value;

[0021] The formula for estimating state value is:

[0022] ;

[0023] in, Indicates the state The value of the state under the following conditions Represents the mathematical expectation operator. Discount factor; advantage function Defined as:

[0024] .

[0025] In one embodiment, the optimization strategy is achieved by shearing the objective function and using adaptive KL divergence constraints:

[0026] Policy gradient objective function of the improved proximal policy optimization algorithm for:

[0027] ;

[0028] in, The importance sampling ratio, For reference strategy, To truncate parameters; denotes the i-th training sample input to the large model, denotes the output generated by the corresponding large model, denotes the parameters of the large model, denotes the mathematical expectation on when the training sample is input, denotes the clipping operation that truncates within the interval ( ), denotes the large model policy distribution defined by ;

[0029] To ensure the stability of training, a KL divergence constraint is introduced:

[0030] ;

[0031] wherein, is the loss used for training the large model finally, is the weight coefficient of the KL divergence constraint.

[0032] In one of the embodiments, the training of the value network as a baseline specifically includes:

[0033] The loss function of the value network is:

[0034] ;

[0035] denotes the parameters of the value network .

[0036] In one of the embodiments, when the large model performs parameter updating, full-amount parameter updating is adopted and an asynchronous parameter updating mechanism is adopted, so that the large model completes full-amount parameter updating in the background while receiving new user cognitive signals, specifically including:

[0037] The asynchronous parameter updating mechanism adopts a gradient accumulation strategy:

[0038] ;

[0039] wherein, denotes the gradient vector that has been accumulated by the asynchronous parameter updating mechanism, is the accumulation step number; when the accumulation reaches a preset step number, gradient clipping is performed first to prevent gradient explosion:

[0040] ;

[0041] denotes the gradient vector after gradient clipping, denotes the clipping operator, denotes the threshold hyperparameter of gradient clipping;

[0042] Then the large model performs a full parameter update:

[0043] ;

[0044] denotes the large model parameter before update, denotes the large model parameter after update, denotes the training learning rate, and the full parameter update will be set to zero, and enter the next round of accumulation.

[0045] In a second aspect, the present application provides a computer system comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method of any one of the embodiments of the first aspect when executing the computer program.

[0046] Compared with the prior art, the present application has the beneficial technical effects that:

[0047] (1) Dynamic personalization: By collecting explicit and implicit feedback from users in real time, the time-varying characteristics and individual differences of user preferences can be captured, the value alignment of personalization can be achieved, and the limitations of traditional methods relying on static preference data can be avoided.

[0048] (2) Efficient and stable and single-sample friendly: Improved proximal policy optimization algorithm (value baseline, advantage function, clipping target) and reference strategy KL constraint are adopted, stable advantage and controllable update can be obtained without intra-group relative comparison in a single output and single reward online scene, and the convergence stability and sample efficiency are improved.

[0049] (3) Real-time output: Through the online learning mechanism and asynchronous parameter update strategy, the user feedback can be responded to in real time, the low-delay value alignment adjustment can be realized, and the timeliness requirement in actual application can be met.

[0050] (4) Effectiveness: The full parameter fine-tuning strategy is adopted, the expression ability of the large model can be fully utilized, deep value alignment can be achieved, and the alignment effect and generalization ability are stronger than the parameter efficient method.

[0051] (5) Strong robustness: The multi-source cognitive signal fusion mechanism improves the robustness to single feedback noise, the KL divergence constraint ensures the stability of model update, and the risk of excessive deviation from pre-training knowledge is avoided.

[0052] The method of this invention can effectively solve the problems of static preference, update lag and high cost in existing large model value alignment technology, and provides important technical support for the safe and controllable application of large models. Attached Figure Description

[0053] Figure 1 This is an overall flowchart of the method in an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of the architecture in an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of the near-end optimization strategy algorithm in an embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram of asynchronous update in an embodiment of the present invention. Detailed Implementation

[0057] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0058] like Figure 1 As shown, a large-scale model value alignment method based on user dynamic cognitive incentives in this invention includes the following steps:

[0059] S1, captures cognitive signals generated during the user's interaction with the large model in real time, and quantifies them into numerical or vector form; the cognitive signals include explicit feedback and implicit feedback;

[0060] S2, based on the weight vectors pre-trained offline for each user, performs weighted fusion on the quantized cognitive signals to generate a scalar reward signal;

[0061] S3, the near-end policy optimization algorithm is improved in the following way to realize parameter update of large model in online single sample scenario: scalar reward signal is allocated to each generation step of large model, state value is estimated by time difference method, advantage function is calculated by combining state value, and a value network is trained as baseline, and then the policy is optimized by shearing objective function and adaptive KL divergence constraint.

[0062] S4 employs a full parameter update when updating parameters in the large model, optimizing all parameters of the large model as a whole, and uses an asynchronous parameter update mechanism to enable the large model to complete the full parameter update in the background while receiving new user cognitive signals.

[0063] The large model value alignment problem can be formally defined as: given a pre-trained large model and user groups The goal is to optimize large models through algorithmic improvements. The output generated when processing user input can best meet the user's value preferences and expectations.

[0064] Specifically, let the user input be , the output of the large model be , the user's cognitive evaluation (i.e., reward) of the output be , where is the index of time t, reflecting the dynamics of user cognition. The goal of value alignment is to maximize the expected reward:

[0065] ;

[0066] where the expectation is calculated over the user input distribution, the user cognition distribution, and the time distribution.

[0067] The technical solution proposed by the present application is shown in Figure 2 , which includes the following four core modules: (1) user cognition signal capture and quantification module, (2) personalized reward generation module, (3) online reinforcement learning optimization module, and (4) model parameter update module.

[0068] (1) User cognition signal capture and quantification module.

[0069] The goal of this module is to monitor and capture multi-source heterogeneous cognitive signals generated during user interaction with the large model in real time, and convert them into standardized numerical or vector form as direct input for the downstream personalized reward generation module. The capture and quantification of cognitive signals can be divided into two types: explicit and implicit:

[0070] Explicit feedback includes user's direct rating, like / dislike, text comments, and other behavioral data. Let the user 's explicit feedback at time be , which includes: numerical rating , binary feedback , and user's text feedback . Among them, the text feedback is converted into a numerical evaluation score through a large model with frozen parameters combined with a pre-defined rubrics, to facilitate subsequent calculations.

[0071] ​Implicit feedback is the indirect feedback inferred by analyzing the user's unconscious interaction behavior patterns, including reading duration, repeated viewing times, follow-up inquiry behavior, and task completion degree in specific scenarios. This module quantifies these behaviors, such as taking the interaction duration as a logarithmic transformation, treating follow-up inquiry behavior as a binary flag, and converting task completion degree into a percentage, to obtain calculable numerical signals. The specific processing method needs to be combined with the specific work scenario.

[0072] (2) Personalized reward generation module.

[0073] The core responsibility of this module is to combine the personalized preference weight vector learned offline from user historical data and the quantified cognitive signal captured in real-time from the current interaction, and fuse them into a single scalar reward signal that can guide the optimization of the large model . The implementation of this module includes two stages:

[0074] First stage: offline pre-training of personalized weight vector. To achieve deep modeling of user individual preferences, the invention trains and maintains a personalized weight vector for each user individually . This vector is not updated in real-time, but is periodically pre-trained offline based on the cognitive signals of all historical interactions accumulated by the user .

[0075] The training goal is to enable the weight vector to most accurately fit a small number of high-precision reward signals given by human expert annotators according to pre-set evaluation standards from the user's historical implicit feedback combination. Specifically, a portion of the user's historical interaction cognitive signals is sampled and refined by trained expert annotators to generate an objective, high-dimensional true reward score for these samples. Subsequently, the entire historical interaction cognitive signals of the user (including cheap implicit feedback and user self-evaluated implicit feedback) are used as input features, and the high-precision reward scores annotated by experts are used as supervision labels to train and solve the weight vector through a supervised learning task.

[0076] Second stage: real-time calculation of online reward. During real-time user interaction, this module first obtains the quantified cognitive signals (including implicit feedback and implicit feedback) of the current interaction session from the upstream module and constructs an instant feature vector . Then, the pre-trained and fixed personalized weight vector for the user is retrieved. The final scalar reward signal is calculated by the dot product of the weight vector and the instant feature vector:

[0077] ;

[0078] This design ensures that the reward signal reflects both the user's personalized preferences and aligns with higher standards of value judgment, providing a high-quality and personalized incentive signal for subsequent reinforcement learning.

[0079] (3) Online reinforcement learning optimization module.

[0080] This module employs an improved Proximal Policy Optimization (PPO) algorithm for online model updates. For example... Figure 3 As shown, to adapt to the scenario of online single-sample generation, this invention employs a value network. As a baseline, it effectively reduces the variance of policy gradient estimation. An advantage function is constructed using real-time user perception signals, and combined with KL divergence constraints, to achieve stable and efficient online policy optimization. The scalar reward signal is used as a baseline. The steps assigned to the sequence generation are as follows: First, for a complete interaction between the user and the large model, a sequence-level final scalar reward signal is calculated. Since the large model outputs tokens one by one, it is necessary to... According to a preset reward distribution rule, it is broken down into a series of instant rewards arranged in time steps. Equal-weight allocation can be adopted. , This is the sequence length output by the large model. During the reinforcement learning update phase, this series of instantaneous rewards, divided by time step, is used as the basis. The Generalized Advantage Estimation (GAE) method is used to calculate the value function and advantage function for each state, and the policy probability is updated at each time step within the PPO framework. In this way, the user cognitive reward, which was originally defined only at the sequence level, is structured and fed back to each step of the large model in generating the token sequence.

[0081] Output generated from large model During the process, the state Defined as the sequence of tokens already generated by the large model, i.e., the previous... Each token defines an action as the operation of generating the next token. For the state... The next generation Instant rewards after obtaining tokens This invention employs the time difference (TD) method to estimate state value.

[0082] The formula for estimating state value is:

[0083] ;

[0084] in This is the discount factor.

[0085] Based on this, the advantage function is defined as:

[0086] .

[0087] The policy gradient objective function of improved PPO is:

[0088] ;

[0089] wherein is the importance sampling ratio, is the reference policy, is the truncation parameter.

[0090] To ensure the stability of training, the KL divergence constraint is introduced:

[0091] ;

[0092] wherein is the weight coefficient of the KL divergence constraint, which is dynamically set through an adaptive adjustment mechanism.

[0093] To reduce the estimation bias of the value function, a simplified value network is trained simultaneously, and its loss function is:

[0094] .

[0095] (4) Model parameter update module.

[0096] To achieve accurate value alignment for large models, this module uses a full- parameter fine-tuning strategy to update all parameters of the model. To meet the real-time requirements of online environments, an asynchronous parameter update mechanism is used, allowing the large model to receive new user cognitive signals while completing full-parameter optimization in the background.

[0097] Specifically, the parameter update process is decoupled from the user interaction process, such as Figure 4 , ensuring the timeliness of system response through background asynchronous updates, and maintaining consistency during the update process through a parameter version management mechanism. To reduce memory consumption and improve update efficiency, a gradient accumulation strategy is used:

[0098] ;

[0099] wherein is the accumulation step.

[0100] When the accumulation reaches the preset step, gradient clipping is performed to prevent gradient explosion:

[0101] ;

[0102] Then a full parameter update is performed:

[0103] ;

[0104] After the update, the is set to zero and the next round of accumulation is entered.

[0105] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the term "includes" and tautological expressions thereof, such as "including," "includes," "include," "contains," "containing," and so forth, mean "comprising."

[0106] It should be understood that, although the various steps in the flowcharts of the drawing figures are shown in a sequential order, these steps are not necessarily performed in the order shown. Unless explicitly stated, as can be apparent from the disclosure and the desired results, the execution of some steps can be concurrent, in some cases, the execution of steps can be in reverse order, depending on the particular application. Also, one of ordinary skill in the art will recognize that some steps can be combined with other steps, split into multiple steps, or omitted entirely, and the order of the steps can be changed while still staying within the scope of the present application. Further, at least some of the steps in the flowcharts of the drawing figures can include multiple steps or stages, which can be performed in a different order than shown, or in parallel, or in an interleaved manner, or in a different manner, while still staying within the scope of the present application.

[0107] In one embodiment, the present application further provides a computer system, which can be a server. The computer system comprises a processor, a memory and a network interface connected by a system bus. The processor of the computer system is configured to provide computing and control capabilities. The memory of the computer system comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer system is configured to store data used in the above method. The network interface of the computer system is configured to communicate with an external terminal through a network connection. The computer program is configured to be executed by the processor to implement the above method.

[0108] The technical features of the above embodiments can be combined in any manner. For brevity, not all possible combinations of the technical features in the above embodiments are described, but it should be understood that any combination of the technical features is within the scope of the present disclosure, as long as the combination does not result in a contradiction.

[0109] It will be obvious to a person skilled in the art that the application is not limited to the details of the foregoing exemplary embodiments and can be implemented in other concrete forms without departing from the spirit or essential characteristics of the application. The embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein and no

[0110] Furthermore, it should be understood that although the description is made on the basis of embodiments, not every embodiment contains only one independent technical solution, and the description is made in this way only for the sake of clarity, and a person skilled in the art should consider the description as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by a person skilled in the art.

Claims

1. A large-scale model value alignment method based on user dynamic cognitive incentives, characterized in that, include: The system captures cognitive signals generated during the user's interaction with the large model in real time and quantifies them into numerical or vector form; the cognitive signals include explicit feedback and implicit feedback. Based on weight vectors pre-trained offline for each user: Samples are taken from historical cognitive signals of user interactions with the large model, and expert evaluations are conducted to generate reward scores; using all historical cognitive signals of user interactions with the large model as input features and expert evaluation reward scores as supervision labels, the current user's weight vector is trained and solved through a supervised learning task. ; The quantized cognitive signals are weighted and fused to generate a scalar reward signal: During real-time interaction between the user and the large model, the corresponding quantized cognitive signals are constructed into feature vectors, and the scalar reward signal is generated. By providing the current user Pre-trained weight vector The dot product with the eigenvector yields: ; For transpose, Indicates user At any moment For the output of large models The immediate cognitive characteristics, Indicates user Deadline The state of historical cognition This represents a feature extraction function that maps immediate cognitive features and historical cognitive states into a unified feature vector. The index at time t; The following approach is used to improve the near-end policy optimization algorithm to achieve parameter updates for large models in online single-sample scenarios: scalar reward signals are allocated to each generation step of the large model, state values ​​are estimated using the time difference method, advantage functions are calculated by combining state values, and a value network is trained as a baseline. Then, the policy is optimized by shearing the objective function and adaptive KL divergence constraints. When updating parameters in a large model, a full parameter update is adopted, along with an asynchronous parameter update mechanism, so that the large model can complete the full parameter update in the background while receiving new user cognitive signals.

2. The large-scale model value alignment method based on user dynamic cognitive incentives according to claim 1, characterized in that, Explicit feedback refers to the user's direct interaction with the output of the large model, including numerical ratings of the output, binary feedback, and textual feedback. The textual feedback is converted into numerical evaluation scores by using a large language model with frozen parameters and a predefined rating scale.

3. The large-scale model value alignment method based on user dynamic cognitive incentives according to claim 1, characterized in that, Implicit feedback refers to the user's unconscious interaction with the output content of the large model, including the user's reading time, number of repeated views, subsequent inquiry behavior, and task completion rate in a specific scenario; the interaction time is logarithmically transformed, the subsequent inquiry behavior is processed into a binary symbol, and the task completion rate is converted into a percentage.

4. The large-scale model value alignment method based on user dynamic cognitive incentives according to claim 1, characterized in that, The process of allocating the scalar reward signal to each generation step of the large model, estimating the state value using the time-difference method, and calculating the advantage function based on the state value specifically includes: During the process of generating output from a large model, the previously generated large model will be used as the output. A sequence of tokens is defined as a state. , will generate The operation of each token is defined as an action; According to the set reward distribution rules, the rewards are broken down into a series of instant rewards arranged in time steps. For the state... The next generation Instant rewards after obtaining tokens The time difference method is used to estimate the state value; The formula for estimating state value is: ； in, Indicates the state The value of the state under the following conditions Represents the mathematical expectation operator. Discount factor; advantage function Defined as: 。 5. The large-scale model value alignment method based on user dynamic cognitive incentives according to claim 4, characterized in that, The optimization strategy involves shearing the objective function and applying adaptive KL divergence constraints. Policy gradient objective function of the improved proximal policy optimization algorithm for: ； in, The importance sampling ratio, For reference strategy, To truncate parameters; This represents the i-th training sample input to the large model. express The output generated by the corresponding large model, Represents the parameters of a large model. Indicates input training samples When Mathematical expectation on Indicates will Cut off in the interval ( The cut operation within ) Indicates by The defined large model policy distribution; To ensure training stability, KL divergence constraints are introduced. : ； in, The loss function ultimately used to train large models. These are the weighting coefficients for the KL divergence constraint.

6. The large-scale model value alignment method based on user dynamic cognitive incentives according to claim 4, characterized in that, Training a value network as a baseline specifically includes: Training Value Network loss function for: ； Value network The parameters.

7. The large-scale model value alignment method based on user dynamic cognitive incentives according to claim 5, characterized in that, When updating parameters in the large model, a full parameter update is adopted, along with an asynchronous parameter update mechanism. This allows the large model to complete the full parameter update in the background while receiving new user cognitive signals. Specifically, this includes: The asynchronous parameter update mechanism employs a gradient accumulation strategy: ； in, This represents the accumulated gradient vector obtained by the asynchronous parameter update mechanism. To accumulate steps; when the preset number of steps is reached, gradient clipping is performed first to prevent gradient explosion: ； This represents the gradient vector after gradient clipping. This represents the clipping operator. The threshold hyperparameter represents the gradient clipping parameter; Then the large model undergoes a full parameter update: ； This represents the parameters of the large model before the update. For the updated large model parameters, This represents the training learning rate, which will be updated after all parameters are updated. Set to zero and proceed to the next round of accumulation.

8. A computer system comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

Images