A large language model personalization preference automatic perception and alignment method

Abstract

This invention discloses a method for automatic perception and alignment of personalized preferences in large language models, relating to the field of natural language processing. The method constructs a dual-head architecture model containing a shared backbone network, an adaptive preference perception module, and a conditional generation module by collecting a triplet dataset of user question text, target answer text, and multidimensional true preference score vectors. A total loss function is constructed based on weighted multidimensional mean squared error loss and sequence-level cross-entropy loss, and the model parameters are jointly optimized end-to-end. During inference, only the user needs to input a natural language question; the model automatically predicts the preference score vector and generates a personalized response. This invention solves the problems of inconsistent training and inference, high user interaction burden, and limited inference efficiency in traditional methods, achieving accurate alignment and high-quality personalized generation of multi-target preferences while ensuring low resource consumption.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method for automatic perception and alignment of personalized preferences in large language models. Background Technology

[0002] Large Language Models (LLMs) have demonstrated powerful capabilities in complex reasoning, creative generation, and cross-domain task solving in the field of natural language processing, providing efficient technical support for various text processing and interaction scenarios. However, how to combine large language models with human values ​​to ensure that the model outputs are beneficial, safe, honest, and context-aware remains a core challenge that urgently needs to be addressed in this field.

[0003] Traditional model alignment techniques, typically represented by human feedback-based reinforcement learning (RLHF), have achieved some success in improving model usefulness and safety. However, these techniques suffer from fundamental design limitations. Their core assumption is that human preferences can be approximated by a single scalar reward function. This design implicitly simulates the preferences of "average users" or "mass users," completely ignoring the diversity and heterogeneity of human values. In practical applications, human preferences involve multi-dimensional and conflicting needs, such as balancing information authenticity with creativity, or weighing usefulness against harmlessness. Compressing these multi-dimensional preference needs into a single reward score not only loses the fine-grained information within the preference structure but also results in models that only cater to the needs of "average users," failing to meet the personalized needs of different groups and application scenarios. Ultimately, this leads to models whose outputs in specific scenarios are neither professional nor appropriate.

[0004] To address these issues, research focus in academia and industry has gradually shifted from single-objective scalar reward optimization to multi-objective preference alignment. Existing multi-objective preference alignment methods can be broadly categorized into three types: The first category is model fusion technology, represented by Rewarded Soups (RS). This technology fine-tunes the training of corresponding "expert models" for each independent objective such as safety and usefulness. During the inference stage, a new model is obtained by linearly interpolating the weights of each expert model, thereby achieving a trade-off between different objectives.

[0005] The second category is multi-objective optimization training techniques. These methods aim to directly optimize multiple objectives during the training phase, seeking Pareto optimal solutions by improving the loss function or gradient update strategy. Among them, GAPO treats alignment as an explicit multi-objective optimization problem and proposes a mechanism for dynamically adjusting gradients to ensure that each update of the model simultaneously improves or at least does not harm all objectives, achieving the Pareto front only by training a single model. AMoPO proposes a method that eliminates the need for reward and reference models, further reducing resource consumption and enabling adaptive balancing of multiple preference dimensions to achieve Pareto optimal solutions.

[0006] The third category is inference-time guided and conditional training techniques. This type of method consists of two parts: one part optimizes the decoding part of the model's inference to meet diverse user needs; the other part performs conditional training on the model's own capabilities, explicitly training the model to understand diverse user needs. Among them, MOD proposes a decoding-time alignment algorithm that does not require retraining. In the inference stage, it achieves dynamic and precise control over diverse alignment objectives by linearly weighting and combining the logits (predicted probabilities) output by multiple single-objective models. RiC and CPO directly input the reward value as part of the input question into the model, enabling the model to learn to generate responses based on the given reward expectation, achieving dynamic preference adjustment with only a single training iteration.

[0007] Although existing multi-objective preference alignment methods have improved the model's personalized adaptation ability to some extent, they still have the following significant drawbacks: First, there is a training-inference inconsistency: In most conditional training methods and some multi-objective optimization methods, the model relies on explicit preference scores provided by an external reward model during the training phase, concatenating these scores with the original input to guide the model's learning. However, during the inference phase, the model needs to receive preference scores manually set by the user to control the generation process. This difference in the source of preference signals between the training and inference phases means that the model internalizes the preference distribution induced by the reward model during training, but responds to the user-defined preference scores during inference. The two types of scores differ significantly in their distribution characteristics and implicit semantic information. This distribution misalignment makes it difficult for the model to accurately align with the user's true intent during inference, ultimately leading to generation results that deviate from the optimal solution and exhibit suboptimal performance.

[0008] Second, the user interaction burden is high: Existing methods generally require users to explicitly and manually set preference scores or weight combinations during the inference phase to control model behavior. This operation places high cognitive and operational demands on users. Users not only need to understand the semantics of preference scores within the model and their impact on the generated results, but also need to ensure that the set scores accurately reflect the true intent and are consistent with the preference distribution learned during model training; otherwise, the model response will deviate from expectations. In real-world human-computer interaction scenarios, users typically interact with large models using natural language and do not actively provide structured preference signals or numerical constraints. Therefore, these methods significantly increase the user's interaction burden, weakening the usability and universality of the technology in practical applications.

[0009] Third, inference efficiency is limited: Model fusion methods and multi-objective optimization methods require dynamic adjustment or switching of linear interpolation strategies during the inference phase based on different user needs to achieve fine-grained behavior control. This online adjustment mechanism introduces additional computational overhead and scheduling costs, increasing overall inference latency. Meanwhile, existing inference-time alignment methods generally rely on one or more auxiliary models to supervise, evaluate, or constrain the output of the base model. The multi-model collaborative inference paradigm inevitably leads to higher inference latency and significant consumption of GPU memory and computational resources, limiting the practical usability of these methods in low-latency or resource-constrained scenarios. Summary of the Invention

[0010] Therefore, this study aims to address the technical problems of training-inference inconsistency, high user interaction burden, and limited inference efficiency in existing multi-objective alignment methods for large language models, and to achieve adaptive perception and accurate alignment of the model with users' multi-dimensional personalized preferences.

[0011] To address the aforementioned technical problems, this invention provides a method for automatic perception and alignment of personalized preferences in large language models, comprising the following steps: Step S1: Collect a triplet dataset consisting of user question text, target answer text, and multidimensional true preference score vector; Step S2: Construct a preference-aware alignment model with a shared representation space, including: constructing a shared backbone network based on a pre-trained large language model, and configuring an adaptive preference-aware module and a condition generation module in parallel on the upper layer of the shared backbone network. The adaptive preference-aware module adopts a lightweight multilayer perceptron to map the semantic features of the user question text into a predicted preference score vector. The condition generation module adopts the language model head natively configured in the pre-trained large language model to receive the enhanced input composed of the user question text and the true preference score vector, and generate a response text that meets the requirements. Step S3: Input the user question text in the triplet dataset into the shared backbone network to extract semantic features, output the predicted preference score vector through the adaptive preference perception module, and calculate the weighted multidimensional mean squared error loss as the preference perception loss by combining the preset dimension importance weight vector and the true preference score vector. Step S4: Concatenate the user question text with the true preference score vector to form an enhanced input sequence, input it into the shared backbone network and the conditional generation module, calculate the generation probability of each word, and then obtain the sequence-level cross-entropy loss as the conditional generation loss; wherein, the word is the smallest semantic unit obtained after the response text generated by the model is processed by word segmentation. Step S5: Based on the preference-aware loss and the conditional generation loss, obtain the total loss function. Based on the total loss function, perform end-to-end joint optimization on all parameters of the shared backbone network, adaptive preference-aware module, and conditional generation module of the preference-aware alignment model, so that the model learns the mapping relationship from question to preference score and the ability to generate answers under preference guidance, until the model converges and the trained preference-aware alignment model is obtained. Step S6: Input the natural language question input by the user into the trained preference-aware alignment model. The preference-aware module automatically predicts the preference score vector and concatenates it with the embedding features of the original question to form an enhanced input. This enhanced input is then fed into the condition generation module to generate an answer text that meets the user's personalized needs.

[0012] In one embodiment of the present invention, the method for constructing the preference-aware loss is as follows: Given user question text Predict a continuous ideal preference score vector Used to quantify user activity The potential demand intensity across different preference dimensions; during the training phase, the true preference score vector obtained for each training sample is denoted as... ; To learn from the user question text To the ideal preference score vector The mapping relationship for a given problem Time-based true preference score vector The conditional distribution is modeled, and its distribution follows the predicted preference score vector output by the model. The mean and covariance matrix are... multivariate Gaussian distribution ; because Each preference dimension in a given problem Under the condition of mutual independence, the covariance matrix is... Simplified to an isotropic diagonal matrix ;in, The hyperparameters that control the uncertainty of the distribution are represented. for identity matrix; To maximize the true preference score vector Log-likelihood under the multivariate Gaussian distribution The training objective of the model is equivalent to minimizing the vector of true preference scores. and predicted preference score vector The loss term is composed of the weighted sum of squares of the errors between them; Based on the aforementioned dimension importance weight vector We calculate the perceived preference loss by weighting the loss terms for all preference dimensions.

[0013] In one embodiment of the present invention, the preference-perceived loss The calculation formula is as follows: , in, and These represent the components of the true preference score vector and the model predicted mean vector in the kth dimension, respectively; Represents the dimension importance weight vector The Middle The weights of each preference dimension.

[0014] In one embodiment of the present invention, the dimension importance weight vector In, corresponding to any number Weights of each preference dimension satisfy and The conditions for obtaining these conditions are as follows: For any number Each preference dimension samples unnormalized weighting factors from a Gaussian distribution. : ,in and For the prior hyperparameters of this dimension; The unnormalized weight factors for all dimensions are normalized using the following formula to obtain the normalized weights. : .

[0015] In one embodiment of the present invention, the method for constructing the conditional generation loss is as follows: Based on the triplet dataset ,in For user question text, For a multidimensional true preference score vector, The response text consists of multiple word units; User question text and the true preference score vector Perform natural language concatenation to obtain an enhanced input sequence; The enhanced input sequence is input into the shared backbone network, and through multi-head attention and feedforward neural network encoding, a hidden state sequence that integrates problem semantic features and preference constraint information is obtained; The hidden state sequence is input into the condition generation module, and the response text is calculated word-by-word. The generation probability of all target words in the text; Based on the probabilistic chain rule, the joint probability of the response text y is calculated based on the generation probability of all target words, and the negative logarithm is obtained by taking the negative log-likelihood. The sequence-level cross-entropy loss, i.e., the conditional generation loss, is obtained by minimizing the negative log-likelihood as the optimization objective.

[0016] In one embodiment of the present invention, the hidden state sequence is input into the condition generation module, and the response text is calculated word-by-word. The method for determining the generation probability of all target word elements is as follows: Extract the final hidden state vector of the hidden state sequence. The vector is input into the condition generation module as the initial context state for its generation process; wherein, Represents the enhanced input sequence The length of the word sequence obtained after word segmentation; For the t-th time step, the condition generation module is based on the historical generation sequence prior to the t-th word. Based on the initial context state, the unnormalized logarithmic probability vector for the current time step is output through linear transformation and feature mapping. ; The Softmax function is used to process the unnormalized log probability vector. Normalization is performed to convert it into a word-level generation probability distribution: ,in, Represents the vector at time step t. In the middle, target word The corresponding logarithmic probability value; This represents the total number of unique tokens that the model can generate; Index variable representing the word sequence list; Represents the vector at time step t. In , the word sequence list No. Logarithmic probability values ​​corresponding to each word element; Repeat the above steps to continuously generate the probability distribution of subsequent word units until the model generates a preset terminator or the length of the generated word units reaches the length of the response text. The maximum length T of the tokens in the dictionary is used to obtain the response text. The generation probability of all target words in the text: .

[0017] In one embodiment of the present invention, the formula for calculating the joint probability of the response text y based on the generation probabilities of all target words is as follows: .

[0018] In one embodiment of the invention, the conditional generation loss The calculation formula is as follows: , in, For model parameters The conditional probability distribution at time t, representing the conditional probability distribution in the input problem. and preference vector Under the constraint, the probability that the model generates the response text y; Represents the target text sequence The number of lexical units included; Indicates the time step index of sequence generation; The model represents the first time. The unnormalized log probability vector output at each time step, with dimension [ ]. Shared backbone network Based on historical context ,question and preference vector Calculated; Represents the vector at time step t. In the middle, target word The corresponding logarithmic probability value; This represents the total number of unique tokens that the model can generate; Index variable representing the word sequence list; Represents the vector at time step t. In , the word sequence list No. The logarithmic probability value corresponding to each lexical unit.

[0019] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the large language model personalized preference automatic perception and alignment method.

[0020] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method for automatic perception and alignment of personalized preferences of a large language model.

[0021] Compared with the prior art, the above-described technical solution of the present invention has the following advantages: This invention, through constructing a dual-head architecture with a shared representation space and implementing end-to-end joint optimization, possesses four core advantages: First, it completely solves the training-inference inconsistency problem of traditional methods. Relying on an adaptive preference-aware module, the model automatically predicts preference vectors during the inference phase, ensuring a high degree of consistency between the inference distribution and the training distribution, significantly improving the alignment accuracy and robustness of responses. Second, it achieves semantically driven, seamless interaction. Users do not need to manually set preference scores or possess professional knowledge; they can obtain personalized responses simply through natural language input, greatly reducing the interaction threshold and cognitive burden. Third, it breaks through the efficiency bottleneck of multi-objective inference. Only a lightweight multilayer perceptron module needs to be added, without loading additional auxiliary models. It can achieve Pareto optimal trade-offs in the multi-objective optimization space with the same inference speed and memory usage as the basic model, adapting to low-latency, resource-constrained industrial scenarios. Fourth, it enhances the model's ability to internalize complex preferences. Through joint optimization of composite loss functions, it enables the model to keenly capture subtle differences in user preferences, finding the optimal balance between conflicting goals such as safety and usefulness, authenticity and creativity, effectively avoiding the "average user fallacy" and generating more professional and appropriate responses. Attached Figure Description

[0022] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0023] Figure 1 This is a flowchart illustrating an automatic perception and alignment method for personalized preferences in a large language model provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the model training and inference process proposed in this invention; Figure 3 This is a comparison of the macro average win rate of the method described in this invention and other mainstream preference alignment methods on multiple datasets; Figure 4 This is a standard benchmark performance comparison of the method described in this invention with other mainstream preference alignment methods in a general dialogue scenario; Figure 5 It is a comparison of the macro average win rates of various methods under different preference-guided strategies; Figure 6 This refers to the impact of the weights of the preference-perceived loss on model performance; Figure 7This is the additional delay introduced by the adaptive preference-aware module during the inference phase. Detailed Implementation

[0024] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0025] Reference Figure 1 and Figure 2 As shown, this invention provides a method for automatic perception and alignment of personalized preferences in large language models, comprising the following steps: Step S1: Collect user question text Target Response Text and multidimensional true preference score vector Triple dataset ;in Indicates the number of preference dimensions. This represents the sample index, where N is the number of samples; Step S2: Construct a preference-aware alignment model with a shared representation space, including: constructing a shared backbone network based on a pre-trained large language model, and configuring an adaptive preference-aware module and a condition generation module in parallel on the upper layer of the shared backbone network. The adaptive preference-aware module adopts a lightweight multilayer perceptron to map the semantic features of the user question text into a predicted preference score vector. The condition generation module adopts a language model head natively configured with the pre-trained large language model to receive the enhanced input composed of the user question text and the true preference score vector, and generate a response text that meets the requirements. Step S3: Input the user question text in the triplet dataset into the shared backbone network to extract semantic features, output the predicted preference score vector through the adaptive preference perception module, and calculate the weighted multidimensional mean squared error loss as the preference perception loss by combining the preset dimension importance weight vector and the true preference score vector. Step S4: Concatenate the user question text with the true preference score vector to form an enhanced input sequence, input it into the shared backbone network and the conditional generation module, calculate the generation probability of each token, and then obtain the sequence-level cross-entropy loss as the conditional generation loss; wherein, the token is the smallest semantic unit obtained after word segmentation of the response text generated by the model. Step S5: Based on the preference-aware loss and the conditional generation loss, obtain the total loss function. Based on the total loss function, perform end-to-end joint optimization on all parameters of the shared backbone network, adaptive preference-aware module, and conditional generation module of the preference-aware alignment model, so that the model learns the mapping relationship from question to preference score and the ability to generate answers under preference guidance, until the model converges and the trained preference-aware alignment model is obtained. Step S6: Input the natural language question input by the user into the trained preference-aware alignment model. The preference-aware module automatically predicts the preference score vector and concatenates it with the embedding features of the original question to form an enhanced input. This enhanced input is then fed into the condition generation module to generate an answer text that meets the user's personalized needs.

[0026] Furthermore, in this embodiment, the construction goal of the adaptive preference-aware module is: based on user question text... Based on the semantic features, the potential ideal preference score vector is inferred. Used to quantify user activity The potential demand intensity across different preference dimensions; during the training phase, using the true preference score vector... To monitor signals, learn "user question text" →Ideal Preference Score Vector The mapping relationship between "" and "" is essentially about the conditional distribution. Modeling is then performed. The specific construction process is as follows: Given the true preference score vector Let d be a continuous vector, and let it follow the predicted preference score vector output by the adaptive preference perception module. The mean and covariance matrix The probability expression for a multivariate Gaussian distribution that measures variance is: .

[0027] To simplify calculations and combine various preference dimensions in a given user question text The property of mutual independence under the condition will affect the covariance matrix. Let it be an isotropic diagonal matrix ,in, The hyperparameters that control the uncertainty of the distribution are represented. for Identity matrix.

[0028] The isotropic diagonal matrix Substituting the probability density function of the multivariate Gaussian distribution, we get: ; Taking the logarithm of the probability density function and expanding the derivation, we obtain the log-likelihood function: .

[0029] During model training and optimization and The constant term is independent of the model parameters and has no impact on model training; therefore, maximizing the log-likelihood function is the optimal approach. This is equivalent to maximizing the last term in the above formula. .

[0030] Let the true preference score vector be... Predicting preference score vector Vector norm Expanded into a scalar summation of each dimension's components: ,in The true preference score vector The Components in each dimension Represents the predicted preference score vector The Components in each dimension.

[0031] Considering the varying importance of different preference dimensions in the multi-objective alignment task of large language models, a dimension importance weight vector is introduced. This vector is used to adjust the contribution weights of different dimensions, and it is generated as follows: For any i Each preference dimension samples unnormalized weighting factors from a Gaussian distribution. The expression is ,in and These are the prior hyperparameters corresponding to this dimension; Perform a Softmax normalization operation on the unnormalized weight factors obtained from the sampling to obtain the result that satisfies and The normalization formula for the legal weight vector of the constraints is: .

[0032] Combining the aforementioned dimensional importance weight vector The loss function of the adaptive preference-aware module is constructed by using the scalar expansion of the vector norm. Its expression is in the form of weighted multidimensional mean square error: .

[0033] Furthermore, the construction goal of the condition generation module proposed in this embodiment is to approximate a conditional probability distribution. That is, in the given user question text Compared with the true preference vector Under the dual guidance, response texts that meet the diverse and personalized needs of users are generated. The specific construction process is as follows: Define reply text A sequence consisting of multiple lexical units is denoted as... ,in For the first in the sequence The number of lexical units at each position, where T represents the number of lexical units obtained after the model generates the response text and processes it into words.

[0034] User question text and the true preference score vector Perform natural language concatenation to obtain the enhanced input sequence. ; The enhanced input sequence The input to the shared backbone network is processed through multi-head attention and a feedforward neural network to obtain a hidden state sequence that integrates problem semantic features and preference constraint information. ,in, Indicates the augmented input sequence The length of the word sequence obtained after word segmentation. The hidden state sequence. In, each element ( Corresponding enhanced input sequence After word segmentation, the first A high-dimensional semantic feature vector of each token; The hidden state sequence is input into the condition generation module, and all target words are calculated per word based on the autoregressive decoding mechanism. The generation probability is as follows: Extract the hidden state vector of the last token in the hidden state sequence. As the final hidden state vector, this vector is input into the condition generation module as the initial context state for its generation process; Define the size of the native vocabulary of the pre-trained large language model as: Specifically, this refers to the total number of unique tokens in the vocabulary set corresponding to the condition generation module (the native language model head of the pre-trained large language model). This vocabulary is the native configuration vocabulary of the pre-trained large language model, containing Chinese characters, words, punctuation marks, numbers, English words, and special markers (such as separators, end marks, etc.) in natural language scenarios. The value of is consistent with the size of the original vocabulary of the pre-trained large language model (such as the Llama series and Qwen series), and is the complete set of word types that the model can generate; For the t-th time step ( The condition generation module is based on the historical generation sequence before the t-th word. Based on the initial context state, the unnormalized logarithmic probability vector (Logits vector) for the current time step is output through linear transformation and feature mapping. The dimension of this vector is related to the vocabulary size. Consistent, each dimension corresponds to the unnormalized log probability of a unique word in the vocabulary; For the Logits vector Perform Softmax normalization to obtain the target word at time step t. The generation probability is expressed as: ,in, Represents the Logits vector target word The corresponding logarithmic probability value; Represents the Logits vector Chinese word sequence list Logarithmic probability values ​​corresponding to each word element; Repeat the above steps to continuously generate the probability distribution of subsequent word units until the model generates a preset terminator or the length of the generated word units reaches the length of the response text. The maximum length T of the tokens in the dictionary is used to obtain the response text. All target words The generation probability: .

[0035] According to the probability chain rule, the response text sequence conditional generation probability Decomposed into a product of word-by-word autoregressive generation probabilities, the expression is: , In the formula, This represents the historical sequence of generated lexical units before the t-th lexical unit. This decomposition ensures that the model follows the autoregressive generation logic, generating the current lexical unit based on the historical context and input conditions.

[0036] To reduce computational complexity and improve optimization stability, the logarithm of the conditional probability in the above product form is taken, transforming the product operation into a summation operation, resulting in the log-likelihood function: .

[0037] The goal of model training is to maximize the log-likelihood function, which is equivalent to minimizing its negative value, i.e., the negative log-likelihood loss.

[0038] The loss function of the conditional generation module is constructed with minimizing the negative log-likelihood as the optimization objective. This loss is essentially the sequence-level cross-entropy loss, expressed as: This loss function-driven model learns to generate output content that highly matches the response text, given user questions and preference constraints, ensuring the accuracy, fluency, and personalization of the response.

[0039] By combining the loss derivations of the adaptive preference-aware module and the condition generation module, and by negating the joint log-likelihood function, a total loss function for end-to-end model training is constructed. Its expression is as follows: .

[0040] In the actual training process of the model, hyperparameters are introduced. The weights of the preference-perceived loss term are adjusted. This invention aims to minimize the total loss function. As the core optimization objective, this objective mathematically endows the model with two core learning capabilities: One is to drive the model to learn accurately from user question text. The mapping relationship to the potential preference vector enables the model to accurately infer a multidimensional preference distribution that conforms to human expectations. Secondly, it enables the model to acquire the ability to generate response text that matches the user's personalized needs under the dual constraints of preference vector s and question text x, thereby optimizing the conditional generation distribution. .

[0041] The design of this total loss function replaces the external heuristic rules relied upon by traditional methods with an intrinsically adaptive preference-aware mechanism, fundamentally bridging the distributional differences between the training and inference phases and ensuring consistency in the generation process. Simultaneously, it eliminates the reliance on manually set preference scores, effectively addressing the technical pain point of excessive user interaction burden. The total loss function is used to jointly optimize all parameters of the shared backbone network, preference-aware module, and conditional generation module using gradient descent until the model converges.

[0042] To verify the effectiveness of the self-guided alignment method (SGA) proposed in this invention in multi-objective preference alignment tasks, three mainstream preference alignment datasets—HH-RLHF, UltraFeedback, and HelpSteer2—were selected as evaluation benchmarks. Contextual reward injection (RiC), controllable preference optimization (CPO), and supervised fine-tuning (SFT) were used as comparison methods. Comparative experiments were conducted on six base models with different parameter levels: the Llama series of large language models (Llama-3.2-1B-Instruct, Llama-3.2-3B-Instruct, Llama-3.1-8B-Instruct) and the Qwen series of large language models (Qwen3-4B, Qwen2.5-7B-Instruct, Qwen3-8B).

[0043] The experiment uses macro-average win rate as the core evaluation metric, which is defined as the arithmetic average of the win rates across different preference dimensions (such as safety, usefulness, and creativity) in each dataset. This metric comprehensively reflects the overall alignment performance of the model in multi-objective scenarios, effectively avoiding the bias of single-dimensional evaluation. Experimental results are as follows: Figure 3 As shown.

[0044] Depend on Figure 3 Experimental data show that the SGA method proposed in this invention exhibits significant performance advantages in all evaluation datasets and base model combination scenarios. Specifically, the data is as follows: On the HH-RLHF dataset, the average win rate of the SGA method reaches 52.6%, which is higher than the comparison methods RiC (48.1%), CPO (50.9%), and SFT (44.8%). On the UltraFeedback dataset, the SGA method, with an average win rate of 38.1%, significantly outperforms RiC (32.4%), CPO (33.0%), and SFT (31.4%). On the HelpSteer2 dataset, the advantage of the SGA method is even more prominent, with an average win rate of 55.9%, significantly leading RiC (43.0%), CPO (44.3%), and SFT (36.9%).

[0045] From the perspective of base model adaptability, models with different parameter levels all achieve significant performance improvements when paired with the SGA method: Llama-3.1-8B-Instruct paired with the SGA method achieves the highest win rate of 66.9% on the HelpSteer2 dataset, while Qwen3-8B paired with the SGA method achieves excellent performance of 53.6% and 44.5% on the HH-RLHF and UltraFeedback datasets, respectively. These experimental results fully verify that the SGA method possesses the following core advantages: first, cross-dataset stability, maintaining optimal performance in datasets with diverse preference distributions; second, cross-model generalization, adaptable to base models with different architectures and parameter levels; and third, superior core performance, achieving a significant win rate improvement compared to existing mainstream technologies, powerfully demonstrating the effectiveness and advancement of this invention's technical solution in improving the multi-objective alignment performance of large language models.

[0046] To further verify the comprehensive effectiveness of the SGA method proposed in this invention in general dialogue scenarios, two industry-recognized standard benchmark evaluation sets were selected for verification experiments. At the same time, mainstream technical solutions were selected as the comparison group, including supervised fine-tuning method (SFT), context reward injection (RiC), and controllable preference optimization (CPO).

[0047] The evaluation process employs two standard benchmark sets with differentiated core metrics: the Arena-Hard benchmark set focuses on performance evaluation in high-difficulty dialogue scenarios, with the core metric being the pairwise win rate, defined as the probability that the tested model and the benchmark model GPT-4 are judged as superior after generating results for the same task input, directly reflecting the model's competitiveness in complex scenarios; the MT-Bench benchmark set covers multi-turn dialogue benchmarks across eight task categories, including writing, reasoning, mathematics, and programming, using a 1-10 scoring system, with the final score being the arithmetic mean of the scores from two rounds of dialogue, quantifying the model's multi-turn dialogue coherence, instruction understanding accuracy, and task execution capability.

[0048] Experimental results are as follows Figure 4 As shown, the SGA method exhibits significant performance advantages across all test models and evaluation metric combinations, as detailed below: Regarding Arena-Hard win rate performance, the SGA method achieved an average win rate of 31.9% across various base models, significantly higher than SFT (11.0%), RiC (18.6%), and CPO (14.6%). The Llama-3.1-8B-Instruct+SGA combination achieved the highest win rate at 48.2%, a 10.3 percentage point improvement over the second-best CPO method, demonstrating its potential to approach GPT-4 performance in challenging dialogue scenarios. Regarding MT-Bench scores, the SGA method achieved an average score of 7.6, far exceeding SFT (6.5), RiC (6.5), and CPO (6.1). In core task dimensions such as logical reasoning and professional writing, the SGA method consistently scored above 7.6, showcasing excellent task adaptability and output quality.

[0049] The experimental results fully validate the core technical advantages of the SGA method: First, through the joint optimization of adaptive preference awareness and condition generation, it not only achieves multi-objective preference alignment but also simultaneously improves the model's general dialogue capabilities; Second, in high-difficulty reasoning and multi-turn interaction scenarios, the SGA method demonstrates superior performance stability and scenario adaptability compared to existing mainstream technologies; Third, its consistently excellent performance across models and parameter scales proves that the method has good generalization ability and can be widely adapted to base models with different architectures.

[0050] To evaluate the effectiveness of the adaptive preference-aware module, targeted ablation experiments were designed. Using the SGA method of this invention as the baseline, two variant models were constructed as the control group, adjusting the preference input strategy only during the inference phase. The condition generation module and other training configurations were kept completely identical, and the single-variable principle was followed to ensure experimental rigor. The benchmark group employs a complete technical solution, dynamically inferring a multi-dimensional preference vector s from the input question x through an adaptive preference-aware module, providing precise guidance signals for the generation module. Comparison group 1 uses a fixed-score strategy, removing the adaptive preference inference logic and using a preset fixed preference score combination as input, specifically set to {0.2 / 0.25, 0.5, 1.0}. Here, "0.2 / 0.25" represents a preference score of 0.2 on the HelpSteer2 dataset and a preference score of 0.25 on the HH-RLHF and UltraFeedback datasets, used to evaluate the model's sensitivity to the fixed guidance signal and the mapping relationship between the control signal and the generation quality. Comparison group 2 uses an oracle score strategy, directly using the true preference scores labeled on the test set as input. This strategy provides a theoretically optimal upper bound for preference guidance, used to verify whether the inference performance of the adaptive preference-aware module can approach or surpass the true labels.

[0051] The experiment is based on the Llama series and Qwen series multi-parameter scale base models, and the evaluation is carried out on three major preference alignment datasets: HH-RLHF, UltraFeedback, and HelpSteer2. The core indicator is still the macro average win rate to ensure the comparability of the results.

[0052] Experimental results are as follows Figure 5 As shown, the performance of the three schemes is compared and analyzed in detail below: (1) Module effectiveness verification: The SGA method showed a significantly higher macro average win rate than the fixed score strategy in all datasets and base model combinations. Specifically, on the HelpSteer2 dataset, the win rate of the Llama-3.2-8B-Instruct+SGA combination reached 58.3%, which is 3.5 percentage points higher than the fixed score strategy (the highest value is 54.8%). On the HH-RLHF dataset, the average win rate of the SGA method was 0.4 percentage points higher than the best performance of the fixed score strategy (54.1%), proving that the adaptive preference awareness module can provide more accurate guidance signals and is a key component for performance improvement.

[0053] (2) Model signal sensitivity verification: The fixed score strategy showed significant performance differences under different preference score settings, and the overall win rate was lower than that of the SGA method. This indicates that the model has successfully established a stable mapping relationship between the control signal (preference score) and the generation quality. It can adjust the generation logic according to the guidance signal and has the dual ability of "understanding the generation goal" and "optimizing the generation process", which provides a foundation for adaptive guidance.

[0054] (3) Performance Validation of Preference Inference: The macro average win rate of the SGA method outperforms the Oracle score strategy on most datasets. This result shows that the adaptive preference perception module does not simply fit the real preference score, but can dynamically weigh the weight distribution of multi-dimensional preferences based on the semantic features of the input question, and infer the optimal preference combination that better fits the actual generation needs, achieving a breakthrough from "passively using real labels" to "actively optimizing guiding signals".

[0055] This ablation experiment, through a "benchmark-variant" comparative design, clarified that the adaptive preference perception module not only enables the model to have strong sensitivity to guidance signals and stable mapping capabilities, but also achieves guidance effects that surpass the real labels through a dynamically optimized preference inference mechanism. This effectively supports multi-target preference alignment and personalized response generation, further verifying the scientific nature and superiority of the technical solution of this invention.

[0056] To verify the key hyperparameters in the total loss function of this invention To investigate the regulation mechanism and performance impact of hyperparameters, a targeted hyperparameter sensitivity analysis experiment was designed. The variance of the hyperparameter λ and the preference perception loss term was also analyzed. There is a fixed mapping relationship, and its core function is to adjust the weight ratio of preference perception loss in the total loss function, thereby affecting the model's fitting accuracy to the preference distribution.

[0057] The experiment selected two typical parameter scale base models, Llama-3.2-3B-Instruct and Qwen3-4B, keeping the training data, optimizer configuration (AdamW, weight decay of 0.01), number of training epochs, and other conditions completely consistent, only changing the value of the hyperparameter λ. The test range of λ was set from 1 to 1000, covering low-weight, medium-weight, and high-weight intervals to comprehensively capture the impact trend of parameter changes on model performance. The experimental evaluation metric continued the macro-average win rate mentioned above, ensuring the comparability of the results with the previous experiments. The experimental results are as follows: Figure 6 As shown.

[0058] Depend on Figure 6 The experimental data show that the value of the hyperparameter λ is significantly positively correlated with the model performance. The specific rules and mechanisms are analyzed as follows: As the hyperparameter λ gradually increases from 1 to 1000, the macro-average win rates of both the Llama-3.2-3B-Instruct and Qwen3-4B models show a continuous and significant upward trend. Specifically, the win rate of the Llama-3.2-3B-Instruct model increases from 42.3% when λ=1 to 58.7% when λ=1000, and the win rate of the Qwen3-4B model increases from 41.8% to 57.9%, verifying the effectiveness of λ in regulating model performance.

[0059] Hyperparameter λ and variance The mathematical mapping relationship determines its physical meaning; when λ=1000, the corresponding variance... At this point, the model introduces a low-entropy prior constraint on the preference-aware loss term. This constraint, by enhancing the weight of the preference-aware loss, forces the model to more accurately fit the subtle distribution of preferences in the real dataset during training, reducing the impact of different training data perturbations on the preference inference results, thereby improving the stability and accuracy of preference alignment.

[0060] The experimental results are entirely consistent with the theoretical analysis of this invention. A larger λ value improves the model's fitting accuracy to the true preference distribution through low-entropy prior constraints, while a smaller λ value results in insufficient weighting of the preference-perceived loss, making it difficult for the model to fully learn the preference mapping relationship. This finding verifies the scientific validity of the λ parameter design in the total loss function and provides a clear basis for parameter tuning in subsequent model deployments.

[0061] Figure 7 The demonstration shows the additional latency introduced by the adaptive preference-aware module during the inference phase. Compared to the overall generation time (typically tens of seconds), this millisecond-level overhead is negligible in practical use. This indicates that the inference overhead of the adaptive preference-aware module of this invention is extremely low and accounts for a very small proportion of the overall latency, having no substantial impact on model service efficiency. While achieving the core functions of multi-objective preference alignment and personalized response generation, this module balances inference efficiency and user experience, providing crucial efficiency support for the industrialization of the technical solution and further validating the comprehensive advantages of this invention in terms of functionality, performance, and efficiency.

[0062] In addition, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the large language model personalized preference automatic perception and alignment method.

[0063] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method for automatic perception and alignment of personalized preferences of a large language model.

[0064] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0065] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0066] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0067] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0068] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for automatic perception and alignment of personalized preferences in a large language model, characterized in that, include: Step S1: Collect a triplet dataset consisting of user question text, target answer text, and multidimensional true preference score vector; Step S2: Construct a preference-aware alignment model with a shared representation space, including: constructing a shared backbone network based on a pre-trained large language model, and configuring an adaptive preference-aware module and a condition generation module in parallel on the upper layer of the shared backbone network. The adaptive preference-aware module adopts a lightweight multilayer perceptron to map the semantic features of the user question text into a predicted preference score vector. The condition generation module adopts the language model head natively configured in the pre-trained large language model to receive the enhanced input composed of the user question text and the true preference score vector, and generate a response text that meets the requirements. Step S3: Input the user question text from the triplet dataset into the shared backbone network to extract semantic features. The adaptive preference-aware module outputs a predicted preference score vector. Combined with a preset dimensional importance weight vector and the true preference score vector, a weighted multidimensional mean squared error loss is calculated as the preference-aware loss. The calculation formula is as follows: ， in, and These represent the components of the true preference score vector and the model predicted mean vector in the kth dimension, respectively; Represents the dimension importance weight vector The Middle The weights of each preference dimension Hyperparameters representing the control of distribution uncertainty; Step S4: Concatenate the user question text with the true preference score vector to form an enhanced input sequence, input it into the shared backbone network and the conditional generation module, calculate the generation probability of each word, and then obtain the sequence-level cross-entropy loss as the conditional generation loss; wherein, the word is the smallest semantic unit obtained after the response text generated by the model is processed by word segmentation. Step S5: Based on the preference-aware loss and the conditional generation loss, obtain the total loss function. Based on the total loss function, perform end-to-end joint optimization on all parameters of the shared backbone network, adaptive preference-aware module, and conditional generation module of the preference-aware alignment model, so that the model learns the mapping relationship from question to preference score and the ability to generate answers under preference guidance, until the model converges and the trained preference-aware alignment model is obtained. Step S6: Input the natural language question input by the user into the trained preference-aware alignment model. The preference-aware module automatically predicts the preference score vector and concatenates it with the embedding features of the original question to form an enhanced input. This enhanced input is then fed into the condition generation module to generate an answer text that meets the user's personalized needs.

2. The method of claim 1, wherein the method further comprises: The method for constructing the preference-aware loss is as follows: Given user question text Predict a continuous ideal preference score vector Used to quantify user activity Potential demand intensity across different preference dimensions; In the training phase, the resulting true preference score vector for each training sample is evaluated, denoted as ; To learn from the user's question text To the ideal preference score vector The mapping relationship for a given problem Time-based true preference score vector The conditional distribution is modeled, and its distribution follows the predicted preference score vector output by the model. The mean and covariance matrix are... multivariate Gaussian distribution ; because Each preference dimension in a given problem Under the condition of mutual independence, the covariance matrix is... Simplified to an isotropic diagonal matrix ;in, for identity matrix; To maximize the true preference score vector Log-likelihood under the multivariate Gaussian distribution The training objective of the model is equivalent to minimizing the vector of true preference scores. and predicted preference score vector The loss term is composed of the weighted sum of squares of the errors between them; Based on the aforementioned dimension importance weight vector We calculate the perceived preference loss by weighting the loss terms for all preference dimensions.

3. The method for automatic perception and alignment of personalized preferences in a large language model according to claim 2, characterized in that, The dimension importance weight vector In, corresponding to any number Weights of each preference dimension satisfy and The conditions for obtaining these conditions are as follows: For any number Each preference dimension samples unnormalized weighting factors from a Gaussian distribution. : ,in and For the prior hyperparameters of this dimension; The unnormalized weight factors of all dimensions are normalized by using the following formula to obtain the normalized weights : 。 4. The method of claim 1, wherein the method further comprises: The method for constructing the conditional generation loss is as follows: Based on the triplet dataset ,in For user question text, For a multidimensional true preference score vector, The response text consists of multiple word units; concatenating the user question text and the real preference score vector to obtain an enhanced input sequence; The enhanced input sequence is input into the shared backbone network, and through multi-head attention and feedforward neural network encoding, a hidden state sequence that integrates problem semantic features and preference constraint information is obtained; inputting the sequence of hidden states into the conditional generation module, computing the reply text word by word generation probabilities of all target word units in the middle Based on the probability chain rule, and based on the generation probability of all target words, the following is calculated: The joint probability is calculated, and its logarithm is taken as the negative number to obtain the negative log-likelihood. The sequence-level cross-entropy loss, i.e. the conditional generation loss, is obtained by minimizing the negative log-likelihood as the optimization objective.

5. The method for automatic perception and alignment of personalized preferences in a large language model according to claim 4, characterized in that, include: inputting the hidden state sequence into the conditional generation module, calculating the reply text word by word The method for generating the probability of all target word units is as follows: Extract the final hidden state vector of the hidden state sequence. The vector is input into the condition generation module as the initial context state for its generation process; wherein, Represents the enhanced input sequence The length of the word sequence obtained after word segmentation; for the t-th time step, the conditional generation module generates a sequence based on history before the t-th token and the initial context state, by linear transformation and feature mapping outputs an unnormalized log probability vector of the current time step ; normalizing the vector of unnormalized log probabilities using a Softmax function normalizing the vector of unnormalized log probabilities using a Softmax function ， in, Represents the vector at time step t. In the middle, target word The corresponding logarithmic probability value; This represents the total number of unique tokens that the model can generate; Index variable representing the word sequence list; Represents the vector at time step t. , the token sequence list is Logarithmic probability values ​​corresponding to each word element; Repeat the above steps to continuously generate the probability distribution of subsequent word units until the model generates a preset terminator or the length of the generated word units reaches the length of the response text. The maximum length T of the tokens in the dictionary is used to obtain the response text. The generation probability of all target words in the text: .

6. The method for automatic perception and alignment of personalized preferences in a large language model according to claim 5, characterized in that, include: Based on the generation probability of all the target words, the response text is calculated. The formula for the joint probability is as follows: 。 7. The method for automatic perception and alignment of personalized preferences in a large language model according to claim 6, characterized in that, The conditional generation loss The calculation formula is as follows: ， in, For model parameters The conditional probability distribution at time t, representing the conditional probability distribution in the input problem. and preference vector Under the constraints, the model generates response text. The probability of; Represents the target text sequence The number of lexical units included; Indicates the time step index of sequence generation; The model represents the first time. The unnormalized log probability vector output at each time step, with dimension [ ]. Shared backbone network Based on historical context ,question and preference vector Calculated; Represents the vector at time step t. In the middle, target word The corresponding logarithmic probability value; This represents the total number of unique tokens that the model can generate; Index variable representing the word sequence list; Represents the vector at time step t. , the token sequence list is The logarithmic probability value corresponding to each lexical unit.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the automatic perception and alignment method for personalized preferences of a large language model as described in any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for automatic perception and alignment of personalized preferences in a large language model as described in any one of claims 1 to 7.

Images