Difficulty-aware learning based large language model personalization alignment method and device

Abstract

The application provides a large language model individualization alignment method and device based on difficulty perception learning, and belongs to the technical field of model training. The method comprises the following steps: obtaining multiple groups of training samples; each group of training samples comprises a user question, a user portrait, and a target reply corresponding to the user question under the user portrait; calculating the perplexity of the target reply in each group of training samples; dividing the multiple groups of training samples into high-perplexity training samples and low-perplexity training samples according to a preset perplexity threshold and the perplexity of the target reply in each group of training samples; performing first training on a preset large language model based on the low-perplexity training samples and a likelihood loss function, performing second training on the preset large language model after the first training based on the high-perplexity training samples and reinforcement learning, and obtaining a target large language model, so that the target large language model completes individualization alignment. The application can improve the individualization alignment effect of the model.

Description

Technical Field

[0001] This application belongs to the field of model training technology, and more specifically, relates to a personalized alignment method and device for large language models based on difficulty-aware learning. Background Technology

[0002] In the field of AI-generated content, personalized generation aims to provide standardized, unbiased, and factually accurate information through general alignment preferences. However, personalized generation requires that large language models no longer be blank slates during the generation process, but rather capable of presenting unique language styles, value orientations, and cognitive patterns based on specific user profiles. Therefore, the essence of personalized generation is to deeply integrate specific user constraints into the generated responses while maintaining the ability to follow instructions. This requires models to possess extremely strong role-playing and style transfer capabilities, rather than simple knowledge retrieval.

[0003] Traditional personalized training is often limited to supervised fine-tuning, which is essentially a behavior cloning based on maximum likelihood estimation. However, relying solely on maximum likelihood estimation can only allow the model to imitate specific sentence structures, and it is often difficult to capture deep logic. Furthermore, it can easily lead to style drift or catastrophic forgetting when the model is faced with complex and long text generation, resulting in mediocre output performance after training. Summary of the Invention

[0004] The purpose of this application is to provide a personalized alignment method and apparatus for large language models based on difficulty-aware learning, so as to solve at least one of the above-mentioned technical problems.

[0005] A first aspect of this application provides a personalized alignment method for large language models based on difficulty-aware learning, comprising: Obtain multiple sets of training samples; each set of training samples includes: user questions, user profiles, and target responses to user questions under the user profiles; multiple sets of training samples contain multiple user profiles; Calculate the perplexity of the target response in each training sample set. The greater the perplexity, the more difficult it is for the large language model to learn from that set of training samples. Based on the preset perplexity threshold and the perplexity of the target response in each training sample, multiple training samples are divided into high-perplexity training samples and low-perplexity training samples. The preset large language model is trained for the first time using low-perplexity training samples and the likelihood loss function. The preset large language model after the first training is then trained for the second time using high-perplexity training samples and reinforcement learning to obtain the target large language model, so that the target large language model can complete personalized alignment.

[0006] A second aspect of this application provides a personalized alignment device for a large language model based on difficulty-aware learning, comprising: The sample acquisition module is used to acquire multiple sets of training samples; each set of training samples includes: user questions, user profiles, and target responses to user questions under the user profiles; multiple sets of training samples contain multiple user profiles; The perplexity determination module is used to calculate the perplexity of the target response in each set of training samples. The greater the perplexity, the greater the difficulty for the large language model to learn from that set of training samples. The sample partitioning module is used to divide multiple training samples into high-perplexity training samples and low-perplexity training samples according to a preset perplexity threshold and the perplexity of the target response in each training sample. The model training module is used to perform the first training of the preset large language model based on low perplexity training samples and the likelihood loss function, and to perform the second training of the preset large language model after the first training based on high perplexity training samples and reinforcement learning, so as to obtain the target large language model and enable the target large language model to complete personalized alignment.

[0007] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the above-described personalized alignment method for a large language model based on difficulty-aware learning.

[0008] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described personalized alignment method for a large language model based on difficulty-aware learning.

[0009] The beneficial effects of the personalized alignment method and apparatus for large language models based on difficulty-aware learning provided in this application are as follows: This application embodiment quantifies the learning difficulty by calculating the perplexity of the target response and classifies samples into high and low perplexity samples according to a threshold. This avoids the problem in traditional undifferentiated mixed training where high-difficulty samples interfere with the model's ability to learn basic instruction compliance in the early stages of training. The pre-set large language model first consolidates its basic personalized generation logic on low-perplexity samples before gradually addressing high-difficulty samples, effectively preventing catastrophic forgetting and improving training stability. Secondly, this application embodiment first completes the first training with low-perplexity samples and likelihood loss, enabling the model to quickly grasp the basic generation rules under the constraints of the user profile. Then, a second training is conducted using high-perplexity samples and reinforcement learning, overcoming the deficiency of traditional supervised fine-tuning using only maximum likelihood estimation, which cannot uncover deep user preference logic and improving the depth of personalized generation. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating a personalized alignment method for a large language model based on difficulty-aware learning, provided as an embodiment of this application; Figure 2 A structural block diagram of a personalized alignment device for a large language model based on difficulty-aware learning provided in an embodiment of this application; Figure 3 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0014] Please refer to Figure 1 , Figure 1 The flowchart of a personalized alignment method for a large language model based on difficulty-aware learning provided in an embodiment of this application can be executed by an electronic device. The method may include: S101-S104.

[0015] S101: Obtain multiple sets of training samples.

[0016] In this embodiment, each training sample includes: user questions, user profiles, and target responses to user questions under user profiles; multiple training samples contain multiple user profiles.

[0017] In this embodiment, a user question refers to a query command initiated by a user, representing the user's information needs or interaction intent. A user profile is a structured description of the personalized characteristics of a specific user, including but not limited to language style (e.g., formal / colloquial, concise / verbal), value orientation (e.g., conservative / radical, optimistic / pessimistic), or cognitive pattern (e.g., logically rigorous / emotionally divergent). A target response refers to a standard response that conforms to the characteristics of a given user question and corresponding user profile. It serves as a supervisory signal for model training, guiding the model to learn the mapping relationship for generating responses that match preferences under the constraints of a specific user profile.

[0018] S102: Calculate the perplexity of the target response in each training sample group.

[0019] In this embodiment, perplexity is an indicator used to quantify the uncertainty of a large language model in predicting the target response sequence. Its value is positively correlated with the difficulty of the model learning the sample. The larger the perplexity, the more uncertain the model is in predicting the sequence pattern of the target response, the more complex the personalized features contained in the training samples, and the higher the difficulty of the model learning the training samples. Conversely, the smaller the perplexity, the more basic the personalized patterns of the training samples, and the lower the difficulty of the model learning.

[0020] In one embodiment, the perplexity of the target response in each training sample can be calculated based on the following:

[0021] in, The perplexity level is represented by T, and the length of the target response is represented by T. This represents the probability that the large language model predicts for the t-th word. This represents the t-th word in the target response. This represents the first (t-1) tokens in the target response.

[0022] S103: According to the preset perplexity threshold and the perplexity of the target response in each training sample, divide multiple training samples into high-perplexity training samples and low-perplexity training samples.

[0023] In this embodiment, the perplexity of the target response in high-perplexity training samples is greater than a preset perplexity threshold, while the perplexity of the target response in low-perplexity training samples is less than or equal to the preset perplexity threshold. The perplexity threshold can be set based on experience. In this embodiment, the perplexity threshold is preferably a value that can divide the high-perplexity training samples and low-perplexity training samples into a 1:9 ratio.

[0024] S104: The preset large language model is trained for the first time based on low perplexity training samples and likelihood loss function. The preset large language model after the first training is trained for the second time based on high perplexity training samples and reinforcement learning to obtain the target large language model, so that the target large language model can complete personalized alignment.

[0025] In this embodiment, the base model of the preset large language model is a decoder large language model based on the Transformer architecture. This application does not limit the specific model structure, but in specific training, open source models with strong instruction compliance capabilities, such as Qwen2.5-7B-Instruct, can be used as the base model.

[0026] In this embodiment, the likelihood loss function refers to the supervised learning loss function constructed based on maximum likelihood estimation. Specifically, in this embodiment, it can be the negative log-likelihood loss function. Its core function is to measure the probability difference between the model-generated response and the target response. By minimizing this loss function, the preset model is guided to maximize the probability of generating the target response under the given user question and user profile, so that the model learns the basic personalized generation logic.

[0027] In this embodiment, the first training refers to the supervised fine-tuning stage based on low-perplexity training samples and the likelihood loss function. The core objective of this stage is to enable the model to quickly master the ability to generate basic personalized responses based on user profile constraints in low-interference, low-difficulty training data, while consolidating the model's ability to follow general instructions, laying the foundation for subsequent training on high-difficulty samples, and avoiding damage to the model's basic capabilities in the early stages of training by high-difficulty samples.

[0028] In this embodiment, the second training refers to the policy optimization stage based on high-perplexity training samples and reinforcement learning. The core objective of the second training is to enable the model to gradually learn complex personalized features based on the training results of the first stage, improve the ability to capture and transfer deep user preferences, and at the same time, avoid the model from deviating from the basic capabilities learned in the first training through constraint mechanisms (such as KL divergence regularization), so as to achieve the dual goals of consolidating basic capabilities and improving high-order personalized capabilities.

[0029] In this embodiment, the target large language model refers to the model obtained after the second training. The target large language model has the ability to generate personalized responses that fit the characteristics of the user profile based on the input user question and user profile, while maintaining the ability to follow general instructions.

[0030] As can be seen from the above, the embodiments of this application quantify the learning difficulty by calculating the perplexity of the target response and dividing it into high and low perplexity samples according to a threshold. This avoids the problem in traditional undifferentiated mixed training where high-difficulty samples interfere with the model's ability to learn basic instruction compliance in the early stages of training. The pre-set large language model first consolidates the basic personalized generation logic on low-perplexity samples and then gradually deals with high-difficulty samples, effectively preventing catastrophic forgetting and improving training stability. Secondly, the embodiments of this application first complete the first training with low-perplexity samples and likelihood loss, enabling the model to quickly grasp the basic generation rules under the constraints of the profile; then, a second training is conducted with high-perplexity samples and reinforcement learning, making up for the deficiency of traditional supervised fine-tuning using only maximum likelihood estimation, which cannot uncover deep user preference logic and improving the depth of personalized generation.

[0031] Considering that real training data is often sparsely distributed and covers fewer scenarios, in one embodiment of this application, multiple training samples are divided into multiple real training samples and multiple synthetic training samples; wherein synthetic training samples can expand personalized scenarios and supplement user types and interaction scenarios not covered by real training samples.

[0032] In one embodiment, each set of real training samples is obtained based on the following method: Obtain a set of historical dialogue data, which contains a user query and the corresponding answer to the user query; input the historical dialogue data of each set belonging to the same user into a large language model to generate a user profile for that user; use the user query contained in the set of historical dialogue data as the user question, and the corresponding answer to the user query as the target response, and form a set of real training samples based on the user profile corresponding to the user in the set of historical dialogue data.

[0033] In this embodiment, historical dialogue data refers to the interaction records between the user and the large language model. Each set of data includes the user's query and corresponding answer, representing the user's interaction intent and preference expression in a real-world scenario. However, since it is often difficult to obtain user profiles from real-world data, in this embodiment, historical dialogue data belonging to the same user can be input into the large language model. The large language model's summarizing capabilities are then used to automatically abstract and output structured descriptive text, which is the user profile corresponding to that user.

[0034] In one embodiment, each group forming the training samples is obtained based on the following method: Generate a random question; input the random question into a large language model containing preset user profiles to obtain the answer to the random question; use the random question as the user question and the answer to the random question as the target response, and form a set of synthetic training samples based on the preset user profiles; wherein, the multiple sets of training samples correspond to at least two preset user profiles.

[0035] In this embodiment, randomized problems are used to simulate diverse interaction scenarios, avoiding homogenization of synthetic data scenarios and ensuring the diversity of scenarios and task coverage of synthetic samples. Pre-defined user profiles refer to user profiles with clearly defined personalized characteristics, such as an experienced teacher. The number of pre-defined user profile types corresponding to the synthetic training samples is no less than two. This is to avoid synthetic data covering only a single user type, ensuring that the model encounters diverse personalized constraints during the training phase and improving its generalization ability to unknown user types.

[0036] As can be seen from the above, the embodiments of this application divide the training samples into two categories: real and synthetic. This solves the inherent defects of sparse distribution and insufficient scene coverage of real training data. The two types of samples complement each other, making the training data both realistic and diverse, laying a high-quality data foundation for subsequent personalized alignment training. Secondly, in the embodiments of this application, the synthetic training samples simulate diverse interaction scenarios with random questions to avoid scenario homogenization. At the same time, at least two different user profiles are preset, allowing the preset large language model to be exposed to diverse personalized constraints, thereby improving the generalization ability of the preset large language model to unknown user types.

[0037] In one embodiment of this application, the first training of a preset large language model based on low perplexity training samples and a likelihood loss function includes: For each set of low-perplexity training samples, the user questions and user profiles in the set of low-perplexity training samples are concatenated to obtain the perceptual sequence corresponding to the set of low-perplexity training samples. Input the perceptual sequence corresponding to each group of low-perplexity training samples into a pre-defined large language model to obtain the initial response corresponding to each group of low-perplexity training samples. The pre-defined large language model is trained for the first time based on the initial response corresponding to each set of low-perplexity training samples, the target response in each set of low-perplexity training samples, and the likelihood loss function.

[0038] In this embodiment, user profiles and user questions in low-perplexity training samples can be continuously combined at the text level according to a preset structured text format (such as "[Persona]:{User Profile Content}[Query]:{User Question Content}") to obtain a perceptual sequence. The core purpose is to explicitly integrate the personalized constraint of user profiles into the model input space, so that the preset large language model can capture task requirements and personalized constraints simultaneously during the encoding stage, which is different from the pure question input paradigm without profile splicing.

[0039] In this embodiment, the initial response refers to the unoptimized response generated by the preset large language model based on its initial parameters after receiving the perceptual sequence input. It is the response result of the preset large language model to the perceptual sequence input under the current parameter state. The target response is equivalent to the standard response. Therefore, in this embodiment, the initial response and target response of each group of low perplexity training samples can be used as input to calculate the likelihood loss function value. The loss gradient is then passed to the parameters of each layer of the preset large language model through the backpropagation algorithm. The model parameters are iteratively updated using the gradient descent method until the preset convergence conditions are met (such as the loss value converging to a preset threshold range, or completing a preset number of training rounds), thus completing the first training.

[0040] In this embodiment, a likelihood loss function is provided:

[0041] in, Represents the likelihood loss function. This represents training samples with low perplexity. This represents the expected loss of low-perplexity training samples. This indicates a user issue. Representing user profiles, Indicates the target response. Indicates the length of the target response. This indicates a pre-defined large language model. This represents the t-th word in the target response. This represents the first (t-1) tokens in the target response.

[0042] As can be seen from the above, in the first training iteration of this application, a perceptual sequence is obtained by structurally splicing user questions and profiles, explicitly integrating personalized constraints into the model input space, enabling the model to capture task requirements and personalized preferences during the encoding stage. Secondly, after inputting the perceptual sequence into the model to obtain the initial response, this application calculates the likelihood loss based on the initial and target responses. The model parameters are iteratively updated through backpropagation and gradient descent. The negative log-likelihood loss accurately measures the probability difference between the generated result and the target response, guiding the model to maximize the probability of generating a response that matches the profile constraints.

[0043] In one embodiment of this application, a second training process is performed on a pre-defined large language model that has undergone the first training, based on high-perplexity training samples and reinforcement learning, including: For each group of high-perplexity training samples: By concatenating the user questions and user profiles in this set of high-perplexity training samples, the perceptual sequence corresponding to this set of high-perplexity training samples is obtained. Input the perceptual sequence corresponding to the set of high-perplexity training samples into the preset large language model after the first training, and obtain multiple candidate responses for the set of high-perplexity training samples; Determine the advantage value for each candidate response in the set of high-perplexity training samples; The pre-set large language model, after the first training, is trained a second time based on the advantage values ​​of multiple candidate responses and the objective function.

[0044] In this embodiment, the concatenation logic for high-perplexity samples maintains the same format as that for low-perplexity samples. Multiple candidate responses refer to the differentiated responses generated by the pre-defined large language model after the first training iteration, upon receiving the perceptual sequence of the same high-perplexity sample, through an uncertain sampling strategy. This is not a single deterministic output from the supervised fine-tuning stage (the first training process). The core purpose of generating multiple candidate responses is to construct a comparison space for strategy optimization, providing diverse generation strategy samples for subsequent advantage value calculation, thereby uncovering deep-seated personalized generation patterns.

[0045] In this embodiment, the advantage value is a numerical index that quantifies the superiority or inferiority of a single candidate response relative to other candidate responses in the same group of samples. It is used to transform the qualitative judgment of whether a candidate response meets the deep personalization constraints into a quantitative signal that can be used for policy gradient optimization, and characterizes the improvement space of the generation strategy corresponding to the candidate response under the current sample.

[0046] In this embodiment, the objective function can be the objective function of Group Relative Policy Optimization (GRPO). In one embodiment, the objective function can be:

[0047] in, Describe the objective function. This represents the expectation of the objective function value for high-perplexity training samples. This represents training samples with high perplexity. This represents the number of candidate responses for each group of high-perplexity training samples. This represents the advantage value of the i-th candidate response. This represents the pre-set large language model in the current training process. This refers to the model before the current parameter update. This represents the i-th candidate response. This represents the modulus of the i-th candidate response, i.e., its length; Indicates the penalty coefficient. This represents the KL divergence. This represents the magnitude of the probability change between the current preset large language model and the preset large language model in the previous training process when predicting the current word, and is used as the weight for importance sampling to correct the distribution bias caused when updating the current model using historical sampling data. Used to The value of is restricted to and between, This represents the clipping factor, which can be set based on experience.

[0048] The specific expression is:

[0049] The specific expression for KL divergence is:

[0050] in It is a reference model, that is, a pre-set large language model that has been trained for the first time. It is used to limit the deviation of the target large language model relative to the pre-set large language model trained for the first time, so as to ensure the stability of training.

[0051] In this embodiment, the candidate response advantage value of each high perplexity sample is used as the core input, and the policy gradient is calculated by substituting it into the objective function. The parameters of the preset large language model after the first training are iteratively updated by the policy gradient ascent method until the convergence condition is met, such as reaching the preset number of iterations.

[0052] In one embodiment, the advantage value of each candidate response is calculated based on the following method: Multiple candidate responses are scored based on a reward function to obtain a score for each candidate response; the reward function contains multiple scoring indicators; different scoring indicators are used to evaluate each candidate response from different dimensions; the advantage value of each candidate response is determined based on the score corresponding to each candidate response.

[0053] In this embodiment, the reward function is used to quantify the fit between candidate responses and the ideal response constrained by the user profile during reinforcement learning. The reward function can contain two scoring metrics, such as ROUGE-1 and ROUGE-L. ROUGE-1 is a recall metric based on tuple matching, primarily representing the lexical overlap between the candidate and target responses. Specifically, it calculates the proportion of individual words in the candidate response that overlap with the target response relative to the total number of words in the target response. The evaluation dimension of ROUGE-1 is the keyword hit rate at the text surface level, reflecting the candidate response's ability to reproduce the core semantic information of the target response. ROUGE-L is a recall metric based on the longest common subsequence, representing the consistency of sentence structure and semantic sequence between the candidate and target responses. It does not require consecutive matching words, only focusing on the logical order of words. The evaluation dimension of ROUGE-L is the deep structural fluency and logical coherence of the text, reflecting the degree of fit between the candidate response and the target response in terms of sentence structure and textual logic.

[0054] In this embodiment, the reward function can be defined as the weighted calculation result of ROUGE-1 and ROUGE-L, where the weighted calculation weight of ROUGE-1 is 0.4 and the weighted calculation weight of ROUGE-L is 0.6.

[0055] In this embodiment, the advantage value can be calculated as follows: Based on the comprehensive score of all candidate responses to the same high perplexity sample, a baseline score (usually the mean or median of the scores within the group) is calculated. This baseline value represents the average quality level of the candidate responses within the group. Next, the baseline score within the group is subtracted from the comprehensive score of a single candidate response to obtain the original advantage value of that candidate response. A positive original advantage value indicates that the response is better than the average level within the group, while a negative value indicates that it is worse than the average level within the group. Finally, all original advantage values ​​within the group are normalized (e.g., standardized or scaled) to ensure that the advantage values ​​fall within a preset numerical range, thereby improving the stability of the objective function optimization. Ultimately, the advantage values ​​of the candidate responses that can be directly input into the objective function are obtained.

[0056] As can be seen from the above, in the second training of this embodiment, the perceptual sequence splicing format of high-perplexity samples is consistent with that of low-perplexity samples, ensuring the continuity of the model's input encoding logic and avoiding the failure of basic capabilities due to format changes, thus laying a solid input foundation for learning complex personalized features. Secondly, this embodiment generates multiple candidate responses through uncertainty sampling, constructing a policy optimization comparison space, breaking through the limitation of traditional supervised fine-tuning's single output being unable to uncover deep personalized patterns. The advantage value calculation relies on a multi-dimensional reward function containing ROUGE-1 and ROUGE-L, scoring and weighting the responses from the dimensions of word overlap and structural logic, respectively, transforming the qualitative judgment of whether candidate responses meet deep personalized constraints into quantitative scores; after intra-group benchmark normalization and normalization processing, an advantage value that can be directly used for optimization is obtained, ensuring the comprehensiveness and numerical stability of the scoring. Finally, this embodiment also calculates the policy gradient and iteratively updates the parameters based on the GRPO objective function, using the advantage value as the core, and uses KL divergence regularization constraints to prevent the model from deviating from its basic capabilities, solving the problems of style drift and difficulty in capturing deep logic in traditional training, improving the ability to capture and transfer deep user preferences, and optimizing the personalized generation effect.

[0057] Corresponding to the personalized alignment method for large language models based on difficulty-aware learning in the above embodiment, Figure 2 This is a structural block diagram of a personalized alignment device for a large language model based on difficulty-aware learning, provided as an embodiment of this application. For ease of explanation, only the parts relevant to the embodiment of this application are shown. References Figure 2 The personalized alignment device 20 for a large language model based on difficulty-aware learning includes: a sample acquisition module 21, a perplexity determination module 22, a sample segmentation module 23, and a model training module 24.

[0058] The sample acquisition module 21 is used to acquire multiple sets of training samples; each set of training samples includes: user questions, user profiles, and target responses to user questions under the user profiles; the multiple sets of training samples contain multiple user profiles; The perplexity determination module 22 is used to calculate the perplexity of the target response in each group of training samples. The greater the perplexity, the greater the difficulty for the large language model to learn from the group of training samples. The sample partitioning module 23 is used to partition multiple training samples into high-perplexity training samples and low-perplexity training samples according to a preset perplexity threshold and the perplexity of the target response in each training sample. The model training module 24 is used to perform the first training of the preset large language model based on low perplexity training samples and the likelihood loss function, and to perform the second training of the preset large language model after the first training based on high perplexity training samples and reinforcement learning, so as to obtain the target large language model and enable the target large language model to complete personalized alignment.

[0059] In one embodiment of this application, multiple sets of training samples are divided into multiple sets of real training samples and multiple sets of composite training samples; The sample acquisition module 21 is specifically used to acquire a set of historical dialogue data, which contains a user query and the corresponding answer to the user query; input the historical dialogue data of each set belonging to the same user into the large language model to generate the user profile of the user; take the user query contained in the set of historical dialogue data as the user question, and the answer corresponding to the user query as the target response, and form a set of real training samples based on the user profile corresponding to the user in the set of historical dialogue data. Generate a random question; input the random question into a large language model containing preset user profiles to obtain the answer to the random question; use the random question as the user question and the answer to the random question as the target response, and form a set of synthetic training samples based on the preset user profiles; wherein, the multiple sets of training samples correspond to at least two preset user profiles.

[0060] In one embodiment of this application, the model training module 24 is specifically used to concatenate the user questions and user profiles in each group of low-perplexity training samples to obtain the perceptual sequence corresponding to the group of low-perplexity training samples. Input the perceptual sequence corresponding to each group of low-perplexity training samples into a pre-defined large language model to obtain the initial response corresponding to each group of low-perplexity training samples. The pre-defined large language model is trained for the first time based on the initial response corresponding to each set of low-perplexity training samples, the target response in each set of low-perplexity training samples, and the likelihood loss function.

[0061] In one embodiment of this application, the likelihood loss function is:

[0062] in, Represents the likelihood loss function. This represents training samples with low perplexity. This represents the expected loss of low-perplexity training samples. This indicates a user issue. Representing user profiles, Indicates the target response. Indicates the length of the target response. This indicates a pre-defined large language model. This represents the t-th word in the target response. This represents the first (t-1) tokens in the target response.

[0063] In one embodiment of this application, the model training module 24 is specifically used for training samples with high perplexity for each group: By concatenating the user questions and user profiles in this set of high-perplexity training samples, the perceptual sequence corresponding to this set of high-perplexity training samples is obtained. Input the perceptual sequence corresponding to the set of high-perplexity training samples into the preset large language model after the first training, and obtain multiple candidate responses for the set of high-perplexity training samples; Determine the advantage value for each candidate response in the set of high-perplexity training samples; The pre-set large language model, after the first training, is trained a second time based on the advantage values ​​of multiple candidate responses and the objective function.

[0064] In one embodiment of this application, the model training module 24 is specifically used to score multiple candidate responses based on a reward function to obtain a score corresponding to each candidate response; the reward function contains multiple scoring indicators; different scoring indicators are used to evaluate each candidate response from different dimensions; The advantage value of each candidate response is determined based on the score corresponding to each candidate response.

[0065] In one embodiment of this application, the perplexity determination module 22 is specifically used to calculate the perplexity of the target response in each group of training samples based on the following formula:

[0066] in, The perplexity level is represented by T, and the length of the target response is represented by T. This represents the probability that the large language model predicts for the t-th word. This represents the t-th word in the target response. This represents the first (t-1) tokens in the target response.

[0067] See Figure 3 , Figure 3 This is a schematic block diagram of an electronic device provided according to an embodiment of this application. Figure 3 The electronic device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of each module / unit in the above-described device embodiments, for example... Figure 2The functions of the sample acquisition module 21, the perplexity determination module 22, the sample partitioning module 23, and the model training module 24 are shown.

[0068] It should be understood that, in the embodiments of this application, the processor 301 may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0069] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0070] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store device type information.

[0071] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this application can execute the implementation method described in the personalized alignment method of the large language model based on difficulty-aware learning provided in the embodiments of this application, or they can execute the implementation method of the electronic device described in the embodiments of this application, which will not be repeated here.

[0072] In another embodiment of this application, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0073] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0074] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0075] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0076] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.

[0077] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0078] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0079] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0080] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A personalized alignment method for large language models based on difficulty-aware learning, characterized in that, include: Obtain multiple sets of training samples; Each training sample set includes: a user question, a user profile, and a target response to the user question under the user profile; the multiple training samples contain multiple user profiles; Calculate the perplexity of the target response in each training sample set; the greater the perplexity, the greater the difficulty for the large language model to learn from that training sample set. The multiple training samples are divided into high-perplexity training samples and low-perplexity training samples according to the preset perplexity threshold and the perplexity of the target response in each training sample group. The preset large language model is trained for the first time based on the low perplexity training samples and the likelihood loss function. The preset large language model after the first training is trained for the second time based on the high perplexity training samples and reinforcement learning to obtain the target large language model, so that the target large language model can complete personalized alignment.

2. The personalized alignment method for large language models based on difficulty-aware learning as described in claim 1, characterized in that, The multiple training samples are divided into multiple real training samples and multiple composite training samples; Each set of real training samples was obtained based on the following method: Obtain a set of historical dialogue data, which contains a user query and the corresponding answer; input the historical dialogue data belonging to the same user into the large language model to generate a user profile for that user; The user queries contained in this set of historical dialogue data are taken as user questions, and the answers corresponding to the user queries are taken as target responses. A set of real training samples are formed based on the user profiles corresponding to the users in this set of historical dialogue data. Each group of training samples is obtained based on the following method: A random question is generated; the random question is input into a large language model containing preset user profiles to obtain the answer corresponding to the random question; the random question is used as the user question, and the answer corresponding to the random question is used as the target response, and a set of synthetic training samples is formed based on the preset user profiles; wherein the multiple sets of synthetic training samples correspond to at least two preset user profiles.

3. The personalized alignment method for large language models based on difficulty-aware learning as described in claim 1, characterized in that, The first training of the preset large language model based on the low perplexity training samples and the likelihood loss function includes: For each set of low-perplexity training samples, the user questions and user profiles in the set of low-perplexity training samples are concatenated to obtain the perceptual sequence corresponding to the set of low-perplexity training samples. Input the perceptual sequence corresponding to each group of low-perplexity training samples into the preset large language model to obtain the initial response corresponding to each group of low-perplexity training samples; The first training of the preset large language model is performed based on the initial response corresponding to each group of low-perplexity training samples, the target response in each group of low-perplexity training samples, and the likelihood loss function.

4. The personalized alignment method for large language models based on difficulty-aware learning as described in claim 1, characterized in that, The likelihood loss function is: in, Represents the likelihood loss function. This refers to the low-perplexity training samples. This represents the expected loss of low-perplexity training samples. This indicates a user issue. Representing user profiles, Indicates the target response. Indicates the length of the target response. This refers to the preset large language model. This represents the t-th word in the target response. This represents the first (t-1) tokens in the target response.

5. The personalized alignment method for large language models based on difficulty-aware learning as described in claim 1, characterized in that, The second training of the pre-set large language model after the first training, based on the high-perplexity training samples and reinforcement learning, includes: For each group of high-perplexity training samples: By concatenating the user questions and user profiles in this set of high-perplexity training samples, the perceptual sequence corresponding to this set of high-perplexity training samples is obtained. Input the perceptual sequence corresponding to the set of high-perplexity training samples into the preset large language model after the first training, and obtain multiple candidate responses for the set of high-perplexity training samples; Determine the advantage value of each candidate response for the set of high-perplexity training samples; Based on the advantage values ​​of the multiple candidate responses and the objective function, the preset large language model after the first training is trained a second time.

6. The personalized alignment method for large language models based on difficulty-aware learning as described in claim 5, characterized in that, The advantage value of each candidate response is calculated based on the following method: The multiple candidate responses are scored based on a reward function to obtain a score for each candidate response; the reward function contains multiple scoring indicators; different scoring indicators are used to evaluate each candidate response from different dimensions; The advantage value of each candidate response is determined based on the score corresponding to each candidate response.

7. The personalized alignment method for large language models based on difficulty-aware learning as described in claim 1, characterized in that, The perplexity of the target response in each training sample is calculated in the following way: in, The perplexity of the target response is represented by T, and the length of the target response is represented by T. This represents the probability that the large language model predicts for the t-th word. This represents the t-th word in the target response. This represents the first (t-1) tokens in the target response.

8. A personalized alignment device for a large language model based on difficulty-aware learning, characterized in that, include: The sample acquisition module is used to acquire multiple sets of training samples; Each training sample set includes: a user question, a user profile, and a target response to the user question under the user profile; the multiple training samples contain multiple user profiles; The perplexity determination module is used to calculate the perplexity of the target response in each group of training samples. The greater the perplexity, the greater the difficulty for the large language model to learn from the group of training samples. The sample partitioning module is used to partition the multiple training samples into high-perplexity training samples and low-perplexity training samples according to a preset perplexity threshold and the perplexity of the target response in each training sample. The model training module is used to perform a first training on the preset large language model based on the low perplexity training samples and the likelihood loss function, and to perform a second training on the preset large language model after the first training based on the high perplexity training samples and reinforcement learning, so as to obtain the target large language model and enable the target large language model to complete personalized alignment.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images