Method and device for generating personalized response of large language model agent

Abstract

The invention discloses a big language model agent personalized response generation method and device, and relates to the technical field of personalized recommendation. Comprising the steps of obtaining a preference set from historical interaction behaviors, and constructing a group of preference chains for each preference based on a thinking chain; rewriting the interaction behavior at the current moment based on the generation function to obtain a plurality of semantic equivalence versions and an enhanced interaction set; determining the fine-grained alignment score of each group of preference chains and enhanced interaction sets and the relative strength of each preference based on a scoring function; taking the interaction behavior at the current moment and the response prefix with the initial state being null as basic input, and constructing a prompt template for each group of preference chains; and carrying out weighted summation on the probability distribution of the next token according to the relative preference intensity to obtain personalized target distribution, carrying out word-by-word sampling according to the personalized target distribution to generate the next token, splicing the tokens into a response prefix, updating the response prefix, and outputting a complete personalized response until the response is finished.

Description

Technical Field

[0001] The present invention relates to the field of personalized recommendation technology, and more specifically to a method and device for generating personalized responses of a large language model intelligent agent. Background Art

[0002] The rise of Large Language Models (LLMs) has driven the development of autonomous intelligent agents. These agents typically incorporate modules such as role-playing, memory, planning, and action. LLMs enable complex task reasoning, dynamic decision-making, and self-optimization through techniques such as Chain of Thought (CoT), Tree of Thoughts (ToT), and Reflexion. These agents have been widely used in fields such as marketing and software development. With increasing user demand for personalized interactions, customized agent behavior has become a key research direction. In personalized tasks involving LLMs, a user's current intent is influenced by their preference set, which can be extracted from historical interactions, and each preference has a specific strength. Traditional approaches (such as collaborative filtering) lack semantic understanding and rich interaction capabilities, and are gradually being replaced by language-driven LLMs personalization methods.

[0003] Current LLMs personalization methods fall into two main categories: fine-tuning methods, which use user data to adjust model parameters to suit preferences. However, these methods require significant computational resources, and static models are difficult to adapt to real-time changes in user needs. Training-free personalization methods include real-time data retrieval (RAG) and prompt engineering. The former injects context by retrieving historical user information, while the latter guides model output through prompts. Both methods achieve lightweight personalization without modifying model parameters. However, existing methods generally suffer from shortcomings: weak modeling of preference strength and difficulty controlling model generation when user feedback is sparse or inconsistent, resulting in insufficient precision and flexibility in personalized responses.

[0004] In summary, the existing technologies have the following shortcomings: the fine-tuning method relies on a large amount of high-quality user data, which is costly and has poor scalability. The model cannot dynamically adapt to changes in preferences, and its practicality is limited. The prompt engineering relies on the model's understanding of the prompt, which is not stable enough. The model is prone to ignoring prompts or producing misunderstandings, and it cannot quantify and adjust the strength of preferences, and its effect decays in long conversations. The RAG method relies on statically stored historical preferences and cannot reflect the user's real-time new preferences. It also lacks a mechanism to balance conflicting preferences and makes it difficult to dynamically adjust preference weights. These defects result in the inability of the intelligent agent to continuously update preferences, quantify the current preference strength and act on the generation process, ultimately resulting in personalized responses that are "too much or too little" or being at a loss when preferences conflict, and failing to meet the diverse and dynamically changing needs of users. Summary of the Invention

[0005] Embodiments of the present invention provide a method and device for generating personalized responses for a large language model intelligent agent. Under conditions where preferences change dynamically and explicit feedback is scarce, the method and device can effectively and accurately estimate the current strength of a user's preferences without additional user feedback, and accordingly adjust and control the personalized responses generated by the intelligent agent in real time and in a fine manner, thereby improving the accuracy, flexibility and adaptability of the LLM intelligent agent in serving users.

[0006] An embodiment of the present invention provides a method for generating personalized responses from a large language model agent, including:

[0007] Obtain the historical interaction behaviors between the user and the agent, the current interaction behaviors, and the user's current intentions;

[0008] Obtaining a preference set from the historical interaction behavior, and constructing a set of preference chains for each preference included in the preference set based on the thought chain; rewriting the interaction behavior at the current moment based on the generating function to obtain multiple semantically equivalent versions and an enhanced interaction set; determining a fine-grained alignment score of each set of preference chains and the enhanced interaction set and the relative strength of each preference based on the scoring function;

[0009] Taking the current interaction behavior and the response prefix with an initial empty state as basic input, constructing a prompt template for each set of the preference chains, wherein the prompt template obtains the next token probability distribution under each preference condition based on the large language model;

[0010] The next token probability distribution is weighted and summed according to the relative strength to obtain a personalized target distribution, and the next token is sampled word by word according to the personalized target distribution. The next token is spliced ​​into the response prefix and the response prefix is ​​updated until the response is completed and a complete personalized response is output.

[0011] An embodiment of the present invention provides a large language model agent personalized response generation device, comprising:

[0012] An acquisition unit, used to obtain the historical interaction behaviors between the user and the agent, the current interaction behaviors, and the user's current intention;

[0013] a determination unit configured to obtain a preference set from the historical interaction behavior, construct a set of preference chains for each preference included in the preference set based on the thought chain; rewrite the interaction behavior at the current moment based on a generating function to obtain multiple semantically equivalent versions and an enhanced interaction set; and determine a fine-grained alignment score between each set of preference chains and the enhanced interaction set and a relative strength of each preference based on a scoring function;

[0014] an obtaining unit, configured to take the interaction behavior at the current moment and the response prefix whose initial state is empty as basic input, and construct a prompt template for each set of the preference chains, wherein the prompt template obtains the next token probability distribution under each preference condition based on the large language model;

[0015] An output unit is configured to perform a weighted summation of the next token probability distribution according to the relative strength to obtain a personalized target distribution, generate the next token by sampling word by word according to the personalized target distribution, splice the next token into the response prefix and update the response prefix until the response is completed and a complete personalized response is output.

[0016] An embodiment of the present invention provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes any one of the above-mentioned large language model intelligent agent personalized response generation methods.

[0017] An embodiment of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes any one of the above-mentioned methods for generating personalized responses of a large language model agent.

[0018] Embodiments of the present invention provide a method and apparatus for generating personalized responses for a large language model (LLM) agent. This method proposes a preference strength estimation mechanism based on semantic alignment of the LLM, addressing the common reliance on user feedback or static labels in existing techniques. This mechanism leverages the semantic understanding capabilities of the language model, eliminating the need for user-labeled data and relying on training signals, making it more suitable for real-world deployments. Furthermore, the method introduces a "preference arithmetic" mechanism to formally fuse the LLM output distributions under various preference conditions during generation. This addresses the shortcomings of existing prompt-based methods: the inability to quantitatively control the strengths of multiple preferences, resulting in responses that tend to favor a single preference or be ambiguous, and lacking adjustability. Furthermore, the method employs a bilateral enhancement strategy. Unlike traditional methods that directly use raw preferences and conversational input, this enhancement significantly improves the accuracy of preference estimation and the stability of response generation. Under conditions where preferences are dynamically changing and explicit feedback is scarce, this method effectively and accurately estimates the user's current preference strength without requiring additional user feedback. Based on this information, the agent's generated personalized responses can be finely tuned and controlled in real time, thereby improving the accuracy, flexibility, and adaptability of the LLM agent's user services. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A schematic diagram of a process flow for a method for generating personalized responses using a large language model agent according to an embodiment of the present invention;

[0021] Figure 2 A schematic structural diagram of a large language model agent personalized response generation device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0023] Large language models are widely used in various personalized agents to provide customized services and responses based on user preferences. By understanding and adapting to preferences, these agents can provide more relevant and satisfying user interactions. However, achieving high-quality personalized responses still faces many challenges, including:

[0024] 1) Dynamic Preferences: Users' interests and preferences are not static; they may evolve over time, context, or mood. For example, users may want the agent to adjust its tone or content focus at different stages of a conversation. Traditional static personalization methods struggle to capture these dynamic preference changes.

[0025] 2) Scarce User Feedback: In real-world applications, users rarely provide explicit feedback such as ratings and clicks to continuously inform the system of their preferences. Agents can often only implicitly infer preference signals from limited interactions, making it difficult to accurately model preference strength.

[0026] 3) Low accuracy of natural language guidance: Current LLMs typically rely on natural language instructions to adjust the output style. However, relying solely on users' verbal descriptions of preferences to guide the model is often suboptimal. In long conversations, even with the use of methods like prompt engineering and retrieval enhancement, the model's adherence to preferences can significantly decrease. This means that relying solely on verbal cues in conversations cannot guarantee that the model will always respond in the style / preferences expected by the user.

[0027] Existing personalized LLM applications struggle to consistently and precisely meet users' evolving preferences. Against this backdrop, it's crucial to propose a technology that can automatically adapt to changes in preference strength and control response generation.

[0028] In recent years, the rise of large language models has significantly boosted the development of autonomous agents, enabling them to leverage powerful reasoning and knowledge comprehension capabilities to perform tasks with human-like intelligence. Agents based on large language models have been widely used in fields such as marketing and software development. As technological capabilities improve, users' expectations for agents are shifting towards personalized interactive experiences, making the customization of agent behavior a key research topic.

[0029] In real-world applications, such as personalized web interactions and conversational recommendations, models must be flexible and adaptable to evolving user needs. Traditional approaches, while widely used, lack semantic understanding and rich interaction capabilities. Personalization methods based on large language models are gradually evolving towards language-driven intelligent agents, enabling a more personalized and responsive user experience.

[0030] Currently, there are two main approaches to large language model personalization: fine-tuning and training-free personalization. Fine-tuning adjusts model parameters using user-specific data to achieve better preference adaptation, but requires significant computing resources. Training-free personalization methods, such as Real-time Data Retrieval (RAG) and Prompt Engineering, achieve lightweight personalized output by retrieving external data or adjusting prompts in real time without modifying model parameters.

[0031] However, these current methods still have shortcomings in modeling preference strength and effectively controlling model generation when user feedback is sparse or inconsistent. The following solutions are proposed in the prior art to address these issues:

[0032] 1) Fine-tuning-based personalization: This approach fine-tunes models based on user data or trains dedicated small models to reflect preferences. This approach requires a large amount of high-quality preference data, is costly, and difficult to scale. Furthermore, the fine-tuned models are static and cannot adapt to real-time changes in user preferences. Any shift in user interests requires retraining the model, limiting its practicality.

[0033] 2) Prompt Engineering Personalization: Prompts are used to insert preference descriptions or examples into conversations, guiding the LLM to respond in a specific style or preference. This method has a low implementation barrier, but relies on the model's compliance with prompts, making it unstable. The model sometimes ignores prompts or misinterprets them, resulting in insufficient personalization. Furthermore, the impact of prompts on output is difficult to precisely control, and there is no quantitative method for adjusting preference strength. In extended conversations, relying solely on prompts will gradually weaken the effectiveness of the model's compliance with preferences.

[0034] 3) Retrieval Augmentation (RAG)-based Personalization: This approach retrieves user historical preferences or profiles in the form of documents and injects them into the model context, thereby personalizing responses. This enriches the model's user context to a certain extent, but it still has shortcomings. First, the retrieved content is often pre-stored static preferences and cannot reflect the user's new preferences. Second, the model lacks a mechanism for balancing multiple retrieved preferences during generation. In particular, when different preferences conflict, traditional RAG methods cannot dynamically adjust preference weights to smoothly control the output.

[0035] In summary, existing methods either require additional supervisory signals or expensive training, or lack fine-grained control over preference strength. Specifically, these methods exhibit the following challenges in modeling dynamic preferences: they cannot continuously update preferences based on conversations, cannot determine the strength of a user's current preferences without additional user feedback, and lack mechanisms to quantify preference strength and apply it to the generation process. These limitations often result in inadequate personalized responses: either a single preference is applied across the board (potentially overdoing it), or the model becomes unpredictable when preferences conflict, lacking flexibility and failing to meet the diverse and evolving needs of users.

[0036] In real-world scenarios, users often have multiple coexisting preferences, such as liking both healthy meals and junk food. However, the impact of each preference on user decision-making varies and fluctuates dynamically over time. This influence is defined here as preference strength.

[0037] To overcome these technical challenges, embodiments of the present invention propose a dynamic preference strength modeling and application framework based on adaptive preference arithmetic, namely AdaPA-Agent. This framework comprises two key components: First, to address the challenge of accurately estimating preference strength without explicit user feedback, an alignment-based strength estimation method is proposed. This method uses dual-side augmentation combined with a large language model alignment scorer to assess the degree of fit between each preference and the current user-agent interaction context, thereby accurately inferring preference strength. Second, to effectively utilize the estimated preference strength to guide content generation, a controllable personalized generation method is proposed. This component linearly combines the next-token probability distributions conditioned on each preference and uses the estimated preference strength as a weight to modulate the overall probability distribution of the LLM output, thereby achieving personalized response generation.

[0038] The method provided by this embodiment of the present invention validates AdaPA-Agent on two tasks: conversational recommendation and personalized web page interaction. Experimental results demonstrate that, compared to existing advanced methods, AdaPA-Agent significantly improves the quality of personalized generation, enabling the LLM agent to generate personalized responses that better align with the user's dynamically changing intentions and preferences.

[0039] Figure 1 A flow chart of a method for generating personalized responses of a large language model agent provided by an embodiment of the present invention; Figure 1 Taking the example of FIG. 1 as an example, the method for generating personalized responses of a large language model agent provided by an embodiment of the present invention is described in detail. Figure 1 As shown, the method includes the following steps:

[0040] Step 101, obtaining the user's current intention based on the historical interaction behaviors and the current interaction behaviors between the user and the agent;

[0041] Step 102: Obtain a preference set from the historical interaction behavior, construct a set of preference chains for each preference included in the preference set based on the thought chain; rewrite the interaction behavior at the current moment based on the generating function to obtain multiple semantically equivalent versions and an enhanced interaction set; determine the fine-grained alignment score of each set of preference chains and the enhanced interaction set and the relative strength of each preference based on the scoring function;

[0042] Step 103: Taking the current interaction behavior and the response prefix with an initial empty state as basic input, a prompt template is constructed for each set of the preference chains. The prompt template obtains the next token probability distribution under each preference condition based on the large language model;

[0043] Step 104: Perform weighted summation on the next token probability distribution according to the relative strength to obtain a personalized target distribution, generate the next token by sampling word by word according to the personalized target distribution, splice the next token into the response prefix and update the response prefix until the response is completed and a complete personalized response is output.

[0044] Step 101 is the process of user interaction and input acquisition. Specifically, the user's interaction data is collected, that is, the historical interaction behavior between the user and the agent, the user's current interaction behavior, and the determination of the user's current intention are recorded.

[0045] In practical applications, historical interaction behavior can be expressed by the following formula:

[0046] (1)

[0047] in, Represents the user, Indicates that the user A single interactive behavior data in an interaction, The total number of preferences expressed, Represents a user The historical interaction behavior of Indicates the round number of historical interactions, ranging from 1 to , Represents a user The total number of historical interaction rounds.

[0048] For example, in the movie recommendation scenario, the user's historical conversations (such as "I have always liked watching Director A's early movies") and current input (such as "I feel a little tired today and want to watch something relaxing") are collected.

[0049] In step 102 , a preference set is obtained from historical interaction behaviors, and a set of preference chains is constructed for each preference included in the preference set based on the thought chain.

[0050] In practical applications, the preference chain is a three-stage preference chain, which can be expressed as:

[0051] (2)

[0052] The preference set can be expressed as:

[0053] (3)

[0054] in, Indicates the A preference chain of preferences, Indicates the Preferences, represents the original preference, which can be directly obtained from each preference Extracted from Representing context-based preference reconstruction can enhance semantic clarity, To list specific things or examples of behavior to make it concrete and The meaning of Indicates the chain of thought, represents a set of preferences, The total number of preferences expressed.

[0055] In the embodiment of the present invention, the three-stage preference chain provided in the above method has a three-layer structure from coarse to fine, which provides rich preference information for the subsequent alignment scoring.

[0056] For example, if the preference set includes three preferences, namely , then for each preference we can get three sets of preference chains as shown below:

[0057] (Prefer works by literary and artistic directors), ;

[0058] (prefer drama films), ;

[0059] (prefer light-hearted and humorous movies), .

[0060] Furthermore, the large language model is called as a generation function to rewrite the interaction behavior at the current moment to obtain multiple semantically equivalent versions, and the multiple semantically equivalent versions are merged with the interaction behavior at the current moment to obtain an enhanced interaction set.

[0061] Specifically, increasing the interaction set can be expressed as:

[0062] (4)

[0063] in, represents the enhanced interaction set, represents the generating function, Indicates the Interactive behavior at all times, Represents a set union operation.

[0064] In practice, multiple semantically equivalent versions are , each Keep the same turn number as the current interaction behavior, but rewrite the content lexically and syntactically to expand the expression space of user intention.

[0065] For example, if the current interactive behavior is "I feel a little tired today and want to watch something relaxing.", then the large language model generation function is called ,Will Rewritten as a semantically equivalent version: {"Let's have some mindless comedy movies.", "I just want to laugh now, nothing too serious.", "Recommend some funny movies!"}. Build an enhanced interaction collection : {"I'm a bit tired today and want to watch something lighthearted.", "A little mindless comedy.", "I just want a laugh right now, nothing too serious.", "Recommend some funny movies!"}.

[0066] Compared to static embeddings or keyword matching, large language models offer advantages in semantic understanding, enabling context-aware and refined alignment analysis. Furthermore, using a large language model, a scoring function is used to calculate a fine-grained alignment score for each preference chain and interaction sample in the enhanced interaction set. This fine-grained alignment score is then normalized across all preferences to the relative strength of each preference, enabling preference strength inference without explicit feedback.

[0067] Among them, the fine-grained alignment score and the relative strength of each preference can be expressed as:

[0068] (5)

[0069] (6)

[0070] in, Indicates the A preferred fine-grained alignment score, represents the enhanced interaction set, Indicates the A preference chain of preferences, Represents an enhanced interaction set The number of samples in express For each interaction sample, Indicates the The relative strength of a preference, Represents the scoring function of the large language model, outputting a fine-grained alignment score of 1-10. express The sum of the preferred fine-grained alignment scores, The total number of preferences expressed, Represents the enhanced interaction set All interaction samples in Perform a sum operation.

[0071] For example, in the movie recommendation example, the fine-grained alignment scores are: ; The normalized preference intensity distributions are: .

[0072] Through the above process, the method provided by the embodiment of the present invention can adaptively estimate and update the relative strength of each preference without requiring additional feedback from the user, thereby providing accurate support for personalized content generation.

[0073] After obtaining the relative strength of each preference, the core issue is how to use these weights to guide the large language model to generate responses that continuously reflect the user's current intent. In practical applications, due to the inherent ambiguity of natural language prompts, directly embedding numerical weights into text prompts is often unstable. Recent research has shown that by arithmetically combining the next token distributions generated by the large language model under different style conditions, complex text styles can be effectively generated. Inspired by this, an embodiment of the present invention proposes a preference arithmetic mechanism (Preference Arithmetic), which is to perform a weighted combination of the next token probability distributions under multiple preference conditions according to a formula and use the estimated relative strength of each preference as the weight. This mechanism can achieve fine-grained control over the influence of each user's preference on the final generated response.

[0074] Specifically, at each step of generating a response prefix, the large language model outputs a probability distribution of the next token. This is called the personalized target distribution, which is affected by the user's current intention. The personalized target distribution is modeled as A weighted combination of preference conditional distributions, where each personalized target distribution is generated based on the next token probability distribution.

[0075] In step 103, the context is initialized, and the current interaction behavior and the response prefix with an initial empty state are used as basic inputs to construct a prompt template for each set of preference chains; specifically, the prompt template includes "historical preference chain, current conversation, and generated response fragment."

[0076] It should be noted that the “basic input” here refers to the most core and basic original information when building a generation task, which is the starting point and basis for all subsequent generation logic. “Basic input” includes the current interactive behavior, which refers to the user’s current round (the The response prefix refers to the part of the response fragment that has been generated during the current round generation process (i.e., the first token).

[0077] In actual applications, the generation task needs to clarify both "what the user currently wants" (interaction behavior at the current moment) and "what has been said" (response prefix) to ensure that the generated content not only meets user needs but also avoids repetition, contradiction or disconnection.

[0078] For example, suppose a user in a food recommendation scenario currently interacts with the request "Recommend a restaurant suitable for a date, not too noisy" and the generated response prefix is ​​"Recommend two quiet restaurants suitable for a date:." In this case, the "base input" is a combination of these two components: "user requirement (date, not too noisy)" + the generated prefix (recommend two quiet restaurants). Subsequent steps build on this foundation, incorporating user preferences (e.g., "prefer Japanese food" and "mid-range budget") to generate specific restaurant names and justifications.

[0079] Furthermore, the constructed prompt template is input into the large language model to generate the next token probability distribution under each preference condition. Specifically, the next token probability distribution under each preference condition can be expressed as:

[0080] (7)

[0081] in, Indicates the Preferences is the next token probability distribution conditional on, represents the kth token to be generated, Indicates the generated response prefix, Indicates the Interactive behavior at all times, Indicates the Preferences, represents the generative function of the large language model, Indicates the A preference chain of preferences.

[0082] In step 104, the next token probability distribution determined in step 103 is weighted and summed according to the relative strength to obtain the personalized target distribution; that is, the next token probability distribution under the preference condition is by linking the preference Interact with current behavior and generated response prefix Together they serve as input prompts and are fed into the large language model to generate the personalized target. Specifically, the personalized target distribution is as follows:

[0083] (8)

[0084] in, represents the personalized target distribution, Indicates the user's current intention. Indicates the The relative strength of a preference, represents the kth token to be generated, Indicates the generated response prefix, Indicates the Interactive behavior at all times, Indicates the A preference.

[0085] Furthermore, the next token is generated word by word based on the personalized target distribution. This token is then concatenated into the response prefix and the response prefix is ​​updated until the response ends and a complete personalized response is output. That is, in each round, the next token is sampled based on the personalized target distribution and then concatenated into the current response prefix, updating the current response prefix. This sampling and output continues until the termination token is obtained and a complete personalized response is output.

[0086] For example, initialize the context: the current interaction behavior and generated response prefix as input.

[0087] Construct a prompt template (with wheel as an example):

[0088] For each preference chain , construct the following prompt: [historical preference chain] + current conversation: "I want to relax today Recommend some funny movies. "+The response snippet has been generated;

[0089] Large language model generates preference conditional probability distribution:

[0090] ;

[0091] ;

[0092] ;

[0093] Calculate the weighted combination distribution:

[0094] ;

[0095] Sampling generates new tokens , spliced ​​into the response prefix. Repeat the steps until the generation encounters EOS (End of Sequence).

[0096] To more clearly describe the method for generating personalized responses using a large language model agent provided by an embodiment of the present invention, the method is further described below using the application of a movie recommendation intelligent assistant as an example:

[0097] 1. Scene and Character Settings

[0098] In an intelligent movie recommendation system, users interact with the intelligent agent through multiple rounds of dialogue, expressing their preferences for different types of movies (such as art films, dramas, and light comedies). The intelligent agent needs to dynamically estimate the strength of the preference and generate personalized recommendations based on the AdaPA-Agent framework.

[0099] 2. Process and specific formula application

[0100] Step 201: Collect user interaction data:

[0101] Record user historical interaction behaviors , the current interactive behavior , clarify the user's current intention .

[0102] Among them, user historical interaction behavior Here are a few examples:

[0103] "I always like watching Director A's early films."

[0104] : "Drama works are more likely to impress me."

[0105] "I like to watch some light-hearted movies on weekends to relax."

[0106] Interaction behavior at the current moment : "I'm a little tired today and want to watch something relaxing."

[0107] Step 202, based on the fine-grained alignment and the relative strength estimation of each preference:

[0108] 202-1, from Extract the preference set from: , building a three-stage preference chain through the thinking chain .

[0109] Among them, the three preferences included in the preference set and the three corresponding preference chains are:

[0110] (Prefer works by literary and artistic directors), ;

[0111] (prefer drama films), ;

[0112] (prefer light-hearted and humorous movies), .

[0113] 202-2, call the large language model generation function ,Will Rewrite into a semantically equivalent version and build an enhanced interaction set .

[0114] Specifically: {"Let's watch some mindless comedy movies.", "I just want to laugh right now, nothing too serious.", "Recommend some funny movies!"}. {"I'm a bit tired today and want to watch something lighthearted.", "A little mindless comedy.", "I just want a laugh right now, nothing too serious.", "Recommend some funny movies!"}.

[0115] 202-3, LLM-based alignment score and strength calculation: For each preference chain and Interaction samples, through the LLM scoring function Calculate the alignment score, which is normalized to the relative strength of each preference.

[0116] Specifically, assuming the LLM scoring results (example values):

[0117] Fine-grained alignment scores for the first set of preference chains: , , , ,but: .

[0118] Fine-grained alignment scores for the second set of preference chains: , , , ,but: .

[0119] Fine-grained alignment scores for the third set of preference chains: , , , ,but: .

[0120] The relative strength of each preference: , , .

[0121] Step 203: Controllable personalized generation:

[0122] 203-1, construct the generated input and prompt template, initialize the context, and set the generated response prefix (Initially empty, meaning nothing has been generated yet), for each preference chain Construct a prompt template as follows:

[0123] The first set of preference chains The prompt template: .

[0124] The second set of preference chains The prompt template: .

[0125] The third preference chain The prompt template: .

[0126] 203-2, generate preference condition distribution, prompt to enter LLM, generate the next token probability distribution under each preference condition , the above three prompt templates are input into the large language model respectively to generate the "next token probability distribution" under each preference condition (that is, the next possible word predicted by the model and its probability), as follows:

[0127] The first set of preference chains The probability distribution of the next token under the condition (preference for works by literary and artistic directors): The next word may be "A literary and artistic film" (probability 25%), "Director A" (probability 20%), "art film" (probability 15%), or "recommendation" (probability 10%).

[0128] The second set of preference chains Conditional probability distribution of the next token (preference for drama films): The next word may be "A drama movie" (probability 22%), "drama film" (probability 18%), "recommended" (probability 12%), "emotionally delicate" (probability 10%).

[0129] The third preference chain The probability distribution of the next token under the condition (light-hearted humor preference): The next word may be "A comedy movie" (probability 30%), "B comedy movie" (probability 25%), "recommended" (probability 15%), "funny" (probability 10%).

[0130] 203-3, Weighted Combination Distribution Formula: By Preference Intensity Weighted combination distribution to obtain personalized target distribution : The relative strength of each preference calculated according to step 202: (Art Director), (Drama), (Lighthearted and humorous.) A weighted combination of the three distributions above yields a “personalized target distribution.”

[0131] The following word is "recommended" as an example:

[0132] Preference Chain The probability of "recommendation" under these conditions is 10%, and the contribution is: 0.17×10%=1.7%.

[0133] Preference Chain The probability of "recommendation" under these conditions is 12%, and the contribution is: 0.21×12%=2.52%.

[0134] Preference Chain The probability of "recommendation" under these conditions is 15%, and the contribution is: 0.62×15%=9.3%.

[0135] The total probability of "recommendation" after combination: 1.7%+2.52%+9.3%≈13.52%.

[0136] Similarly, the probabilities of other words (such as A comedy movie, A drama movie, etc.) are also calculated in this way, and finally the probability of words related to "lighthearted humor" is significantly higher.

[0137] 203-4, Autoregressive sampling generation: Each round of sampling tokens , and update the response prefix , and continue until EOS is generated.

[0138] The model generates responses by sampling word by word according to the "personalized target distribution". The process is as follows:

[0139] First round of sampling: Based on the combined distribution, the word "recommendation" has the highest probability (about 13.52%) and is selected as the first token. At this time, the response prefix is ​​generated for the first time and updated to .

[0140] Second round of sampling: Using the new prefix "recommend" as the context, regenerate the conditional distribution of the three preferences (the "generated response fragment" in the prompt template now becomes "recommend"), and then combine them according to the strength of preference. Assuming that the combination probability of the word "you" is the highest, it is selected, and the response prefix is ​​updated to .

[0141] The third round of sampling: Generate a new conditional distribution with the context "recommend you". After combination, the word "watch" has the highest probability and is selected. The prefix is ​​updated to .

[0142] Subsequent rounds: Continue to repeat the sampling process and generate "A comedy movie", "B comedy movie", "these", "funny", "movie", "light rhythm", "relaxing", "suitable for relaxing body and mind" in turn. . Until the model generates EOS.

[0143] The final complete response is: "I recommend you to watch comedy movie A or comedy movie B tonight. These funny movies have a light pace and a light plot, which are suitable for relaxing."

[0144] In summary, embodiments of the present invention provide a method and apparatus for generating personalized responses for a large language model (LLM) agent. This method proposes a preference strength estimation mechanism based on semantic alignment of the LLM, addressing the common reliance on user feedback or static labels in existing techniques. This mechanism leverages the semantic understanding capabilities of the language model, eliminating the need for user-labeled data and relying on training signals, making it more suitable for real-world deployments. Furthermore, this method introduces a "preference arithmetic" mechanism to formally fuse the LLM output distributions under various preference conditions during generation. This addresses the shortcomings of existing prompt-based methods: the inability to quantitatively control the strengths of multiple preferences, resulting in responses that tend to favor a single preference or be ambiguous, and lacking adjustability. Furthermore, this method employs a bilateral enhancement strategy. Unlike traditional methods that directly use raw preferences and conversational input, this enhancement significantly improves the accuracy of preference estimation and the stability of response generation. Under conditions where preferences are dynamically changing and explicit feedback is scarce, this method effectively and accurately estimates the user's current preference strength without requiring additional user feedback. Based on this information, the agent's generated personalized responses can be finely tuned and controlled in real time, thereby improving the accuracy, flexibility, and adaptability of the LLM agent's user services.

[0145] Based on the same inventive concept, an embodiment of the present invention provides a device for generating personalized responses of a large language model intelligent agent. Since the principle of solving technical problems by this device is similar to the method for generating personalized responses of a large language model intelligent agent, the implementation of this device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0146] Figure 2 A schematic diagram of the structure of a large language model agent personalized response generation device provided by an embodiment of the present invention, such as Figure 2 As shown, the device includes an acquiring unit 21 , a determining unit 22 , a obtaining unit 23 and an output unit 24 .

[0147] An acquisition unit 21 is used to acquire historical interaction behaviors between the user and the agent, current interaction behaviors, and the user's current intention;

[0148] A determination unit 22 is configured to obtain a preference set from the historical interaction behavior, construct a set of preference chains for each preference included in the preference set based on the thought chain; rewrite the interaction behavior at the current moment based on a generating function to obtain multiple semantically equivalent versions and an enhanced interaction set; and determine a fine-grained alignment score between each set of preference chains and the enhanced interaction set and a relative strength of each preference based on a scoring function;

[0149] an obtaining unit 23 for taking the interaction behavior at the current moment and the response prefix whose initial state is empty as basic input, and constructing a prompt template for each set of the preference chains, wherein the prompt template obtains a next token probability distribution under each preference condition based on a large language model;

[0150] The output unit 24 is configured to perform a weighted summation on the next token probability distribution according to the relative strength to obtain a personalized target distribution, generate the next token by sampling word by word according to the personalized target distribution, splice the next token into the response prefix and update the response prefix until the response is completed and a complete personalized response is output.

[0151] Preferably, the preference chain is determined by the following formula:

[0152]

[0153] The enhanced interaction set is determined by the following formula:

[0154]

[0155] in, Indicates the A preference chain of preferences, Indicates the Preferences, represents the original preference, represents context-based preference reconstruction, To list specific things or examples of behavior. Indicates the Interactive behavior at all times, represents the generating function, represents the enhanced interaction set, represents the set union operation, Indicates a chain of thought.

[0156] Preferably, the fine-grained alignment score and the relative strength of each preference are determined by the following formula:

[0157]

[0158]

[0159] in, Indicates the A preferred fine-grained alignment score, represents the enhanced interaction set, Indicates the A preference chain of preferences, Represents an enhanced interaction set The number of samples in express For each interaction sample, Indicates the The relative strength of a preference, represents the scoring function of the large language model, The total number of preferences expressed, express The sum of the fine-grained alignment scores of the preferences.

[0160] Preferably, the prompt template includes "historical preference chain, current conversation, and generated response fragment";

[0161] The next token probability distribution is determined by the following formula:

[0162]

[0163] The personalized target distribution is determined by the following formula:

[0164]

[0165] in, Indicates the Preferences is the next token probability distribution conditional on, represents the kth token to be generated, Indicates the generated response prefix, Indicates the Interactive behavior at all times, Indicates the Preferences, represents the generative function of the large language model, Indicates the A preference chain of preferences, represents the personalized target distribution, Indicates the user's current intention. Indicates the The relative strength of a preference.

[0166] It should be understood that the units included in the above-described large language model agent personalized response generation device are merely logical divisions based on the functions implemented by the device. In actual applications, these units can be combined or separated. Furthermore, the functions implemented by the large language model agent personalized response generation device provided in this embodiment correspond exactly to the large language model agent personalized response generation method provided in the above-described embodiment. The more detailed processing flow implemented by this device has been described in detail in the above-described method embodiment 1 and will not be described in detail here.

[0167] Another embodiment of the present invention also provides a computer device, which includes: a processor and a memory; the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes the various steps of the large language model intelligent agent personalized response generation method described in the above method embodiment.

[0168] Another embodiment of the present invention also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a computer device, the computer device executes each step of the method for generating personalized responses of a large language model intelligent agent described in the above method embodiment.

[0169] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0170] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for generating personalized responses from a large language model agent, characterized in that: include: Obtain the historical interaction behaviors between the user and the agent, the current interaction behaviors, and the user's current intentions; Obtaining a preference set from the historical interaction behavior, and constructing a set of preference chains for each preference included in the preference set based on the thought chain; Rewriting the interaction behavior at the current moment based on the generating function to obtain multiple semantically equivalent versions and an enhanced interaction set; determining a fine-grained alignment score between each set of preference chains and the enhanced interaction set and a relative strength of each preference based on a scoring function; Taking the current interaction behavior and the response prefix with an initial empty state as basic input, constructing a prompt template for each set of the preference chains, wherein the prompt template obtains the next token probability distribution under each preference condition based on the large language model; The next token probability distribution is weighted and summed according to the relative strength to obtain a personalized target distribution, and the next token is sampled word by word according to the personalized target distribution. The next token is spliced ​​into the response prefix and the response prefix is ​​updated until the response is completed and a complete personalized response is output.

2. The method according to claim 1, wherein The preference chain is determined by the following formula: The enhanced interaction set is determined by the following formula: in, Indicates the A preference chain of preferences, Indicates the Preferences, represents the original preference, represents context-based preference reconstruction, To list specific things or examples of behavior. Indicates the Interactive behavior at all times, represents the generating function, represents the enhanced interaction set, Indicates the chain of thought, Represents a set union operation.

3. The method according to claim 1, wherein The fine-grained alignment score and the relative strength of each preference are determined by the following formula: in, Indicates the A preferred fine-grained alignment score, represents the enhanced interaction set, Indicates the A preference chain of preferences, Represents an enhanced interaction set The number of samples in express For each interaction sample, Indicates the The relative strength of a preference, represents the scoring function of the large language model, The total number of preferences expressed, Represents the enhanced interaction set All interaction samples in Perform sum operation, express The sum of the fine-grained alignment scores of the preferences.

4. The method according to claim 1, wherein The prompt template includes "historical preference chain, current conversation, and generated response fragment"; The next token probability distribution is determined by the following formula: The personalized target distribution is determined by the following formula: in, Indicates the Preferences is the next token probability distribution conditional on, represents the kth token to be generated, Indicates the generated response prefix, Indicates the Interactive behavior at all times, Indicates the Preferences, represents the generative function of the large language model, Indicates the A preference chain of preferences, represents the personalized target distribution, Indicates the user's current intention. Indicates the The relative strength of a preference.

5. A large language model agent personalized response generation device, characterized in that: include: An acquisition unit, used to obtain the historical interaction behaviors between the user and the agent, the current interaction behaviors, and the user's current intention; a determining unit, configured to obtain a preference set from the historical interaction behavior, and construct a set of preference chains for each preference included in the preference set based on the thought chain; Rewriting the interaction behavior at the current moment based on the generating function to obtain multiple semantically equivalent versions and an enhanced interaction set; determining a fine-grained alignment score between each set of preference chains and the enhanced interaction set and a relative strength of each preference based on a scoring function; an obtaining unit, configured to take the interaction behavior at the current moment and the response prefix whose initial state is empty as basic input, and construct a prompt template for each set of the preference chains, wherein the prompt template obtains the next token probability distribution under each preference condition based on the large language model; An output unit is configured to perform a weighted summation of the next token probability distribution according to the relative strength to obtain a personalized target distribution, generate the next token by sampling word by word according to the personalized target distribution, splice the next token into the response prefix and update the response prefix until the response is completed and a complete personalized response is output.

6. The device according to claim 5, characterized in that The preference chain is determined by the following formula: The enhanced interaction set is determined by the following formula: in, Indicates the A preference chain of preferences, Indicates the Preferences, represents the original preference, represents context-based preference reconstruction, To list specific things or examples of behavior. Indicates the Interactive behavior at all times, represents the generating function, represents the enhanced interaction set, represents the set union operation, Indicates a chain of thought.

7. The device according to claim 5, characterized in that The fine-grained alignment score and the relative strength of each preference are determined by the following formula: in, Indicates the A preferred fine-grained alignment score, represents the enhanced interaction set, Indicates the A preference chain of preferences, Represents an enhanced interaction set The number of samples in express For each interaction sample, Indicates the The relative strength of a preference, represents the scoring function of the large language model, The total number of preferences expressed, Represents the enhanced interaction set All interaction samples in Perform sum operation, express The sum of the fine-grained alignment scores of the preferences.

8. The device according to claim 5, wherein The prompt template includes "historical preference chain, current conversation, and generated response fragment"; The next token probability distribution is determined by the following formula: The personalized target distribution is determined by the following formula: in, Indicates the Preferences is the next token probability distribution conditional on, represents the kth token to be generated, Indicates the generated response prefix, Indicates the Interactive behavior at all times, Indicates the Preferences, represents the generative function of the large language model, Indicates the A preference chain of preferences, represents the personalized target distribution, Indicates the user's current intention. Indicates the The relative strength of a preference.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the large language model agent personalized response generation method as described in any one of claims 1-4.

10. A computer-readable storage medium, characterized in that A computer program is stored, and when the computer program is executed by a processor, the processor executes the large language model agent personalized response generation method as described in any one of claims 1-4.

Images