System and method for learning dense reward language model based on predicted user gaze

Abstract

FIELD: training language models.SUBSTANCE: training language models using reinforcement learning based on human feedback. System which contains a language model module, a reward model module, a user gaze parameter prediction module, a token importance distribution module, a token reward distribution module, and a reinforcement learning module. The method includes receiving a request, generating a response, forming a scalar reward, predicting gaze parameters from the response, forming a normalized distribution of token importance, distributing the scalar reward across tokens, and updating the parameters of the language model using a reinforcement learning algorithm.EFFECT: accelerating the convergence of training while maintaining or improving the quality of responses.10 cl, 2 dwg

Description

[0001] Technical field

[0002] The invention relates to the field of computer technology and artificial intelligence, namely to methods and systems for training models for processing and generating natural language using machine learning and reinforcement learning based on human feedback (RLHF) methods.

[0003] The invention can be used in the development of dialogue systems, virtual assistants, decision support systems and other software systems in which language models are customized to user preferences.

[0004] State of the art

[0005] Language model training systems based on reinforcement learning using human feedback are known. In these systems, a reward model is formed that evaluates the model's responses based on pairs of "preferred / unpreferred" responses, and then the model's policy is optimized, for example, using proximal policy optimization (PPO). In such systems, the reward is typically specified as a single scalar value for the entire generated sequence, which leads to signal sparseness and the need for numerous optimization steps to achieve the required quality.

[0006] For example, US Patent No. 11,586,830 discloses a system and method for controlling natural language generation based on reinforcement learning, in which a reward model is formed for selecting text response options for an agent. However, the reward is formed at the sequence level and is not distributed among tokens based on user attention data. International application WO 2021 247231 describes a system for supervised natural language generation using reinforcement learning methods and a reward model, where quality assessment is also performed at the level of entire text responses without taking into account the user's attention distribution during reading. EP 3,772,012 discloses a method for processing natural language using trainable models and user feedback, but it does not use the predicted distribution of reading time to generate dense token-level rewards.

[0007] There are known approaches to reward denseness in which a sequential scalar signal is distributed across tokens using the reward model's internal attention maps, allowing for token-level importance estimates and faster learning. However, such approaches do not rely on real or predicted data about the user's attention distribution while reading text. There are also known models for predicting user gaze parameters trained on corpora with eye tracking data, which estimate, in particular, fixation duration and total token reading time. However, these models are applied separately from RLHF processes and are not used to generate dense rewards when training language models.

[0008] At the same time, the problem of increasing the efficiency of training language models within the RLHF framework by forming a token-level dense reward consistent with the predicted distribution of the user's attention when reading the text remains unsolved, which would accelerate the convergence of training without degrading the final quality of the model.

[0009] A drawback of known solutions based on reinforcement learning using human feedback is the sparseness of the reward signal, which is generated as a single scalar value for the entire generated sequence. This complicates the precise determination of the contributions of individual tokens to the final score and leads to an increased number of training steps to achieve the required quality of the language model. An additional drawback is the lack of information about the user's attention distribution while reading text: known methods either completely ignore gaze data or generate token-level scores based solely on the models' internal attention mechanisms, which does not ensure consistency between the reward signal and actual human perception of the text.

[0010] Technical task

[0011] The technical objective of the invention is to increase the efficiency of training a language model within the framework of reinforcement learning using human feedback by generating a token-level dense reward signal consistent with the predicted distribution of the user's attention when reading a text.

[0012] Essence of the invention

[0013] To solve the specified technical problem, a system and method for training a language model with dense reward based on the predicted user's gaze are proposed, in which:

[0014] A text sequence generated by the language model in response to a user request is fed to the system input;

[0015] the pre-trained model for predicting user gaze parameters calculates, for the tokens of the sequence, values ​​that characterize the user's attention when reading, at least the total reading time of the tokens;

[0016] Based on the obtained attention values, a normalized distribution of token importance is formed, for example, by means of exponential normalization (softmax);

[0017] The scalar reward obtained from the reward model or based on the user's evaluation is distributed among the tokens according to the specified importance distribution to form token-level reward values;

[0018] The language model parameters are updated using a reinforcement learning algorithm such as PPO or GRPO based on token-level dense reward.

[0019] The system may comprise at least a gaze parameter prediction module, a token importance distribution generation module, a token scalar reward distribution module, and a reinforcement language model training module.

[0020] Technical result

[0021] The technical result of the invention is to increase the efficiency of language model training, resulting in accelerated convergence of the reinforcement learning process using human feedback while maintaining or improving the quality of generated text responses by generating a token-level, dense reward signal based on the predicted distribution of the user's attention while reading the text. An additional technical result is a reduction in the computational costs of training the language model by reducing the number of necessary optimization steps to achieve a given level of response quality.

[0022] Brief description of drawings

[0023] Fig. 1 shows the structural diagram of the dense reward language model learning system based on the user's predicted gaze.

[0024] Fig. 2 shows a flowchart of a method for training a dense reward language model based on a user's predicted gaze.

[0025] List of designations:

[0026] The following symbols are present in Fig. 1:

[0027] 101 - user query input module;

[0028] 102 - language model module;

[0029] 103 - reward model module;

[0030] 104 - user gaze parameter prediction module;

[0031] 105 - module for generating distribution of token importance;

[0032] 106 - token reward distribution module;

[0033] 107 - Reinforcement Learning Module;

[0034] 108 - memory / training data database;

[0035] The following symbols are present in Fig. 2:

[0036] 201 - "Start" block;

[0037] 202 - block “Getting a user’s text request”;

[0038] 203 - block "Generating a response by a language model";

[0039] 204 - block "Formation of scalar reward";

[0040] 205 - block "Prediction of gaze parameters by response";

[0041] 206 - block “Formation of token importance distribution”;

[0042] 207 - block "Distribution of rewards by tokens";

[0043] 208 - block "Updating the parameters of the language model using the reinforcement learning algorithm";

[0044] 209 - "End" block.

[0045] Detailed description of the invention

[0046] The invention is explained using an example of implementation that does not limit the scope of legal protection.

[0047] The system for training a language model with dense reward based on the predicted user gaze comprises at least a user query input module 101, a language model module 102, a reward model module 103, a user gaze parameter prediction module 104, a token importance distribution generation module 105, a token reward distribution module 106, a reinforcement learning module 107, and memory 108, implemented, for example, in the form of a database of training data and model parameters. These modules can be implemented software-based on one or more computing devices comprising at least one processor and a memory device, or in the form of a distributed computing system.

[0048] The user query input module 101 is configured to receive a text query from the user and transmit it to the language model module 102. The language model module 102 is configured to generate a text response based on the text query received from the module 101. The language model in the module 102 can be a large-scale transformer language model, pre-trained on a text corpus and further trained using a supervised learning (SFT) scheme.

[0049] Reward model module 103 is configured to receive a "query-generated response" pair and generate a scalar reward value characterizing the preference of the corresponding response in terms of human preferences. In one embodiment, module 103 comprises a neural network reward model trained on paired response preference data and uses the normalized output values ​​of this model as the scalar reward.

[0050] The user gaze parameter prediction module 104 is configured to receive a text sequence, including at least the generated response of the language model 102 and, if necessary, the user's query, and to calculate for each token of said sequence the values ​​of parameters characterizing the user's attention during reading, such as the total reading time of the token. In one embodiment, the module 104 comprises a neural network model pre-trained on text corpora with annotated eye tracking data and is used in frozen form during the training stage of the language model 102.

[0051] Token importance distribution generation module 105 is configured to receive the attention parameter values ​​predicted by module 104 and calculate a normalized token importance distribution. Specifically, module 105 may be configured to calculate token weights by applying exponential normalization (softmax), in which each value of the total token reading time is assigned a weight proportional to the exponent of the specified value, normalized by the sum of the exponents across all tokens in the sequence.

[0052] The token reward distribution module 106 is configured to receive a scalar reward value generated by the reward model module 103 or obtained based on the user's assessment, and distribute said value among the tokens of the generated sequence in accordance with the token importance distribution calculated by the module 105. As a result, the module 106 generates token-level reward values, the sum of which is equivalent to the original scalar reward, while tokens with greater predicted importance receive higher reward values.

[0053] The reinforcement learning module 107 is configured to update the parameters of the language model 102 based on the token-level reward signal generated by the module 106. In one embodiment, the module 107 uses the proximal policy optimization (PPO) algorithm or the gradient relative policy optimization (GRPO) algorithm to update the parameters of the language model 102, wherein the token-level reward values ​​are used to calculate the advantage values ​​and the learning objective function. The memory 108 is configured to store the training data, the parameters of the language model 102, the reward model 103, and the gaze parameter prediction model 104.

[0054] The method for training a dense reward language model based on a predicted user gaze is implemented, for example, in accordance with the flowchart in Fig. 2.

[0055] At step 201 (“Start”), the system modules and the initial parameters of the language model 102 are initialized. At step 202, a user’s text query is received using the user’s query input module 101. At step 203, the language model module 102 generates at least one text response based on the received query.

[0056] At step 204, reward model module 103 generates a scalar reward value for the generated response, characterizing the response's compliance with the user's preferences. At step 205, user gaze parameter prediction module 104 receives the generated response and calculates attention parameter values ​​for the response tokens, at least the total reading time of the tokens.

[0057] At step 206, the token importance distribution generation module 105 generates a normalized token importance distribution based on the attention parameter values, wherein in one embodiment, exponential normalization (softmax) is applied, ensuring that the token weights are positive and sum to one. At step 207, the token reward distribution module 106 distributes the scalar reward value obtained in step 204 among the tokens of the generated response in proportion to the corresponding token importance weights generated in step 206, forming a token-level dense reward signal.

[0058] At step 208, the reinforcement learning module 107 updates the parameters of the language model 102 in accordance with the selected reinforcement learning algorithm based on the token-level reward signal generated in step 207, wherein tokens with greater predicted importance contribute more to the updating of the model parameters. The described sequence of steps 202-208 can be repeated multiple times for a plurality of requests and responses until the training termination criterion is met, after which, at step 209 ("End"), the training of the language model 102 is terminated.

[0059] In particular embodiments, the user gaze parameter prediction module 104 may additionally calculate other attention metrics, such as the duration of the first fixation or the number of fixations, and the token importance distribution generation module 105 may use a weighted combination of these metrics to generate the importance distribution. In other embodiments, the reinforcement learning module 107 may utilize various RL algorithms, and the language model module 102 may utilize various language model architectures, while the overall sequence of steps for generating a token-level dense reward based on the predicted user gaze is maintained, which does not depart from the essence of the invention.

[0060] Examples of implementation

[0061] In one embodiment, language model module 102 is implemented based on a pre-trained transformer model with approximately 7 billion parameters, further trained using a supervised learning scheme on a corpus of instructions and responses. Reward model module 103 is implemented as a neural network model trained on a dataset of paired human response preferences. The reward model receives a text query and a generated response as input, and produces a scalar reward value as output.

[0062] The user gaze parameter prediction module 104 is implemented based on a neural network model pre-trained on text corpora with annotated eye tracking data and predicting the total reading time (Total Reading Time) for each token of the generated response. At step 205, the predicted total reading time is calculated for each token of the generated response. The token importance distribution generation module 105 then transforms the resulting value vector into a normalized token importance distribution using exponential normalization, in which the token weight is defined as the exponential of the predicted total reading time, normalized by the sum of the exponentials across all tokens of the response.

[0063] The token reward distribution module 106 at step 207 receives the scalar reward value calculated by the reward model module 103 and distributes it among the tokens of the generated response in proportion to the token importance weights generated by module 105, resulting in the formation of a token-level dense reward signal. The reinforcement learning module 107 updates the parameters of the language model 102 using the proximal policy optimization (PPO) algorithm, using the formed token-level reward signal to calculate the benefits and the learning objective function.

[0064] The application of the described method in a series of experiments leads to a reduction in the number of training steps required to achieve a given level of response quality, compared to the variant that uses only sequential scalar reward, while maintaining comparable values ​​of quality assessments on an independent reward model.

[0065] In another embodiment, user gaze parameter prediction module 104 additionally predicts the first fixation duration and number of fixations for each token, and token importance distribution generation module 105 generates an integral token importance metric as a weighted combination of total reading time, first fixation duration, and number of fixations. In this case, the normalized token importance distribution is calculated based on this integral metric, allowing for consideration of various aspects of the user's attention while reading text.

[0066] Reinforcement learning module 107 in this example is implemented using the GRPO algorithm. The token-level reward signal generated by module 106 is used to construct the learning objective function, and the parameters of language model 102 are updated taking into account the relative advantages of tokens receiving higher reward values. Experiments demonstrate accelerated convergence of the learning process compared to the baseline without taking into account the user's predicted view, confirming the achievement of the stated technical result.

[0067] The claimed set of essential features enables the generation of a token-level dense reward signal consistent with the predicted distribution of user attention while reading text, and its use in updating the language model parameters using reinforcement learning algorithms. This, in turn, accelerates the convergence of the language model training process while maintaining or improving the quality of the generated text responses, compared to solutions that use only sequential scalar rewards or token-level assessments that are not based on user attention data.

Claims

1. A method for training a dense reward language model based on a user's predicted gaze, comprising the steps of receive the user's text request, using a language model, generate a text response to a specified text query, generate a scalar reward value for the generated text response using the reward model, characterized by the fact that additionally using a pre-trained model for predicting the user's gaze parameters, for each token of at least the generated text response, the values ​​of the user's attention parameters during reading, at least the total reading time of the corresponding token, are calculated, based on the specified values ​​of the attention parameters, a normalized distribution of the importance of tokens is formed, distribute the specified scalar reward value among the tokens of the generated text response in accordance with a normalized distribution of token importance to form token-level reward values, update the parameters of the language model using a reinforcement learning algorithm based on the generated token-level reward values.

2. The method according to paragraph 1, characterized in that the normalized distribution of the importance of tokens is formed by applying exponential normalization, in which each value of the total reading time of a token is assigned a weight proportional to the exponent of the specified value, normalized by the sum of the exponents for all tokens of the generated text response.

3. The method according to claim 1 or 2, characterized in that the reinforcement learning algorithm comprises a proximal policy optimization (PPO) algorithm and / or a gradient relative policy optimization (GRPO) algorithm.

4. The method according to any one of paragraphs 1-3, characterized in that the pre-trained model for predicting user gaze parameters is trained on a corpus of texts with annotated eye tracking data, including measurements of the total reading time of tokens.

5. The method according to any one of paragraphs 1-4, characterized in that the pre-trained model for predicting the user's gaze parameters additionally calculates for each token the duration of the first fixation and / or the number of fixations, wherein the normalized distribution of the importance of tokens is formed on the basis of a weighted combination of the total reading time, the duration of the first fixation, and the number of fixations.

6. A system for learning a dense reward language model based on a user's predicted gaze, comprising a user query input module configured to receive a user text query, a language model module capable of generating a text response based on a user's text query, a reward model module configured to generate a scalar reward value for the generated text response, characterized in that the system additionally contains a module for predicting user gaze parameters, configured to calculate for each token of at least the generated text response the values ​​of the user's attention parameters when reading, at least the total reading time of the corresponding token, a module for generating a distribution of token importance, configured to generate a normalized distribution of token importance based on the specified values ​​of attention parameters, a token reward distribution module configured to distribute the specified scalar reward value between the tokens of the generated text response in accordance with a normalized distribution of the importance of the tokens to form token-level reward values, a reinforcement learning module capable of updating the parameters of the language model module based on the generated token-level reward values.

7. The system according to claim 6, characterized in that the module for generating the distribution of token importance is designed with the possibility of applying exponential normalization to the values ​​of the total token reading time.

8. The system according to claim 6 or 7, characterized in that the reinforcement learning module is configured to implement a proximal policy optimization (PPO) algorithm and / or a gradient relative policy optimization (GRPO) algorithm.

9. The system according to any one of paragraphs 6-8, characterized in that the module for predicting the user's gaze parameters contains a neural network model trained on a corpus of texts with annotated eye tracking data.

10. The system according to any one of paragraphs 6-9, characterized in that the system additionally comprises a memory configured to store training data and parameters of the language model modules, the reward model, and the model for predicting the parameters of the user's gaze.

Images