Method and apparatus for personalized response generation of large language model agent

Abstract

The application discloses a large language model intelligent agent personalized response generation method and device, and relates to the technical field of personalized recommendation. The method comprises the following steps: obtaining a preference set from historical interaction behaviors, and constructing a group of preference chains for each preference based on a thinking chain; rewriting the current moment interaction behavior based on a generation function to obtain a plurality of semantic equivalent versions and an enhanced interaction set; determining the fine-grained alignment score of each group of preference chains and the enhanced interaction set and the relative strength of each preference based on a scoring function; taking the current moment interaction behavior and the response prefix with an initial state of null as a basic input, and constructing a prompt template for each group of preference chains; performing weighted summation on the next token probability distribution according to the relative preference strength to obtain a personalized target distribution, word by word sampling to generate the next token according to the personalized target distribution, splicing the token into the response prefix and updating the response prefix until the complete personalized response is output.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of personalized recommendation, and more particularly to a method and device for generating personalized responses of a large language model agent. BACKGROUND

[0002] The rise of large language models (LLMs) has driven the development of autonomous agents, which typically include role modeling, memory, planning, and action modules. These agents can implement complex task reasoning, dynamic decision-making, and self-optimization through technologies such as Chain-of-Thought (CoT), Tree of Thoughts (ToT), and Reflexion, and have been widely applied in fields such as marketing and software development. As users demand more personalized interactions, customization of agent behavior has become an important research direction. In the personalized tasks of LLM agents, the user's current intent is influenced by their preference set, which can be extracted from historical interactions, and each preference corresponds to a specific intensity. Traditional methods, such as collaborative filtering, lack semantic understanding and rich interaction capabilities, and are gradually being replaced by language-driven LLMs for personalization.

[0003] Current LLM personalization methods can be divided into two categories: one is the fine-tuning method, which adjusts model parameters to adapt to preferences through user data, but requires a large amount of computing resources, and the static model cannot adapt to real-time changes in user needs. The second is the no-training personalization method, including real-time data retrieval (RAG) and prompt engineering. The former injects context by retrieving user historical information, and the latter guides the model output through prompts. Both methods do not require modification of model parameters, achieving lightweight personalization. However, existing methods generally have the following shortcomings: weak modeling ability for preference intensity, difficulty in effectively controlling model generation when user feedback is sparse or inconsistent, resulting in insufficient precision and flexibility of personalized responses.

[0004] In summary, the existing technology has the following shortcomings: fine-tuning methods rely on a large amount of high-quality user data, have high costs and poor scalability, and the model cannot dynamically adapt to changes in preferences, limiting its practicality; prompt engineering relies on the model's understanding of prompts, and has poor stability, as the model can easily ignore prompts or produce understanding biases, and cannot quantitatively adjust preference intensity, resulting in performance degradation in long conversations; RAG methods rely on static stored historical preferences and cannot reflect real-time new preferences of users, and lack a mechanism to balance conflicting preferences, making it difficult to dynamically adjust preference weights; these defects result in the inability of the agent to continuously update preferences, quantify the intensity of the current preference, and apply it to the generation process, ultimately causing the personalized response to be "overkill" or to be unable to adapt when there is a conflict in preferences, and failing to meet the diverse and dynamic needs of users. SUMMARY

[0005] The embodiment of the application provides a large language model intelligent agent personalized response generation method and device, which can effectively and accurately estimate the current preference intensity of a user without additional user feedback under the condition that the preference is dynamically changed and explicit feedback is scarce, and accordingly, real-time and fine adjustment and control of the personalized response generated by the intelligent agent are realized, thereby improving the accuracy, flexibility and adaptability of the LLM intelligent agent in serving the user.

[0006] The embodiment of the application provides a large language model intelligent agent personalized response generation method, which comprises:

[0007] obtaining historical interaction behaviors between a user and an intelligent agent, current moment interaction behaviors and the user's current moment intention;

[0008] obtaining a preference set from the historical interaction behaviors, constructing a group of preference chains for each preference included in the preference set based on a thought chain, rewriting the current moment interaction behaviors based on a generation function to obtain a plurality of semantically equivalent versions and an enhanced interaction set, and determining a fine-grained alignment score of each group of preference chains and the enhanced interaction set and a relative intensity of each preference based on a scoring function;

[0009] taking the current moment interaction behaviors and an initial state empty response prefix as a basic input, constructing a prompt template for each group of the preference chains, and obtaining a next token probability distribution under each preference condition based on a large language model;

[0010] weighting and summing the next token probability distribution according to the relative intensity to obtain a personalized target distribution, word-by-word sampling to generate a next token according to the personalized target distribution, splicing the next token into the response prefix and updating the response prefix until a complete personalized response is output.

[0011] The embodiment of the application provides a large language model intelligent agent personalized response generation device, which comprises:

[0012] An obtaining unit is configured to obtain historical interaction behaviors between a user and an intelligent agent, current moment interaction behaviors and the user's current moment intention;

[0013] A determining unit is configured to obtain a preference set from the historical interaction behaviors, construct a group of preference chains for each preference included in the preference set based on a thought chain, rewrite the current moment interaction behaviors based on a generation function to obtain a plurality of semantically equivalent versions and an enhanced interaction set, and determine a fine-grained alignment score of each group of preference chains and the enhanced interaction set and a relative intensity of each preference based on a scoring function;

[0014] The obtaining unit obtains, as basic input, the interaction behavior at the current time and a response prefix with an initial state of null, and constructs a prompt template for each of the preference chains, the prompt template being based on a large language model to obtain a next token probability distribution under each preference condition;

[0015] The output unit performs weighted summation on the next token probability distribution according to the relative strengths, obtains a personalized target distribution, generates a next token by word-by-word sampling according to the personalized target distribution, and concatenates the next token to the response prefix and updates the response prefix until a complete personalized response is output.

[0016] The embodiment of the application provides a computer device, which comprises a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to enable the processor to execute the large language model agent personalized response generation method.

[0017] The embodiment of the application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to enable the processor to execute the large language model agent personalized response generation method.

[0018] The embodiment of the application provides a large language model agent personalized response generation method and device, which proposes a preference strength estimation mechanism based on semantic alignment of a large language model, solves the problem that existing technologies generally rely on user feedback or static labels, and utilizes the semantic understanding ability of a language model without user annotation data and without relying on training signals, so that the method is more suitable for actual deployment environment. Furthermore, the method introduces a "preference arithmetic" mechanism to formulaically fuse the large language model output distribution under each preference condition during generation, solves the defects of existing prompt methods, that is, the multiple preference strengths cannot be quantitatively controlled, the generated response tends to be single preference or ambiguous, and the method lacks adjustability. Further, the method adopts a double-sided enhancement strategy, which is different from the traditional method of directly using original preferences and dialogue input. The enhancement process significantly improves the accuracy of preference estimation and the stability of response generation. Under the condition that the preference dynamically changes and explicit feedback is scarce, the method can effectively and accurately estimate the current preference strength of the user without additional user feedback, and accordingly, the personalized response generated by the agent is adjusted and controlled in real time and finely, thereby improving the accuracy, flexibility and adaptability of the LLM agent in serving users. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0020] Figure 1 The large language model agent personalized response generation method flowchart provided by the embodiment of the present application;

[0021] Figure 2 The large language model agent personalized response generation device structure schematic diagram provided by the embodiment of the present application. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0023] Large language models are widely used in various personalized agents to provide customized services and responses according to user preferences. By understanding and adapting to preferences, these agents can provide more relevant and satisfactory user experiences. However, there are still many challenges to achieve high-quality personalized responses, including:

[0024] 1) Dynamic changes in preferences: Users' interests and preferences are not constant and may evolve over time, context, or emotion. For example, users may want agents to adjust tone or content emphasis at different stages of the conversation. Traditional static personalization methods are difficult to capture such dynamic changes in preferences in a timely manner.

[0025] 2) User feedback is scarce: In practical applications, users rarely provide explicit feedback such as ratings, clicks, etc. to continuously inform the system of their preferences. Agents often have to infer preference signals from limited interactions, making it difficult to accurately model preference strength.

[0026] 3) Low accuracy of natural language guidance: Current LLMs usually rely on natural language instructions to adjust output style, but relying solely on user language descriptions of preferences to guide the model often does not achieve satisfactory results. In long conversations, even with methods such as prompt engineering or retrieval enhancement, the degree of compliance with preferences by the model will still decrease significantly. This means that relying solely on language prompts in conversations makes it difficult to ensure that the model always responds in the style / preference expected by the user.

[0027] Existing personalized LLM applications have difficulty in continuously and finely meeting the changing preference needs of users. In this context, it is of great significance to propose a technical solution that can automatically adapt to changes in preference intensity and control response generation.

[0028] In recent years, the rise of large language models has significantly driven the development of autonomous agents, enabling them to utilize powerful reasoning and knowledge understanding capabilities to perform tasks with human-like intelligence. Agents based on large language models have been widely applied in marketing, software development, and other fields. With the improvement of technical capabilities, users' expectations for agents have gradually shifted to personalized interactive experiences, making the customization of agent behavior an important research direction.

[0029] In practical applications, such as personalized web interactions and conversational recommendations, models need to have flexible capabilities to adapt to users' changing needs. Although traditional methods are widely used, they lack the ability of semantic understanding and rich interaction. Personalization methods based on large language models are gradually developing towards language-driven agents, which can provide more personalized and responsive user experiences.

[0030] Currently, there are two main approaches to personalizing large language models: fine-tuning and training-free personalization. Fine-tuning methods adjust model parameters through user-specific data to achieve better preference adaptation, but require a large amount of computing resources; training-free personalization methods, such as real-time data retrieval (RAG) and prompt engineering, achieve lightweight personalized output by retrieving external data or adjusting prompts in real time without modifying model parameters.

[0031] However, these methods currently have deficiencies in modeling preference intensity and effectively controlling model generation when user feedback is sparse or inconsistent. In existing technology, the following solutions are proposed to address these issues:

[0032] 1) Fine-tuning-based personalization: Adjusting the model or training a dedicated small model on user data to reflect preferences. This method requires a large amount of high-quality preference data, is costly and difficult to scale. In addition, the fine-tuned model is static and cannot be adjusted in real time according to user preferences, and once the user's interest changes, the model needs to be retrained, limiting its practicality.

[0033] 2) Prompt engineering personalization: using prompts to add preference descriptions or examples in the dialogue to guide the LLM to respond in a specific style / preference. This method has a low implementation threshold, but relies on the model's cooperation with the prompt, which is unstable. The model sometimes ignores the prompt or has understanding bias, resulting in insufficient personalization. Moreover, the strength of the prompt words on the output is difficult to accurately control, and there is a lack of quantitative preference strength adjustment methods. In the context of long dialogue, simply relying on prompts, the model's compliance with preferences will gradually weaken.

[0034] 3) RAG-based personalization: retrieve and inject user history preferences or personal information in the form of documents into the model context to personalize the response. This enriches the model's user background information to some extent, but still has shortcomings: on the one hand, the retrieved content is usually pre-stored static preferences, which cannot reflect the user's current new preference tendency; on the other hand, there is a lack of mechanism for the model to balance multiple retrieved preference information in generation, especially when there is a conflict between different preferences, traditional RAG methods cannot dynamically adjust the preference weight to smoothly control the output.

[0035] In summary, existing methods either require additional supervision signals or expensive training, or lack fine-grained control of preference intensity. In particular, these methods have the following problems in dynamic preference modeling: they cannot continuously update preferences according to the dialogue, they cannot judge how strong the user's current preference is without additional user feedback, and they lack a mechanism to quantify preference intensity and apply it to the generation process. These limitations result in the following problems in model personalized response: either applying a certain preference one-sidedly (which may be overkill), or being at a loss when there is a conflict of preferences, lacking flexibility, and failing to meet the user's diverse and changing needs.

[0036] In real-world application scenarios, users often have multiple coexisting preferences, such as liking both healthy food and junk food. However, the influence of each preference on user decision-making is not the same and fluctuates dynamically over time. This influence is defined as preference intensity.

[0037] To overcome the above technical challenges, the embodiment of the application proposes a dynamic preference strength modeling and application framework based on adaptive preference arithmetic (Adaptive Preference Arithmetic), namely AdaPA-Agent, which includes two key components: first, to address the challenge of accurately estimating preference strength without explicit user feedback, an alignment-based strength estimation method is proposed. This method combines a large language model alignment scorer through dual-side augmentation to evaluate the degree of fit between each preference and the current user-agent interaction context, thereby achieving accurate inference of preference strength. Second, to effectively utilize the estimated preference strength to guide content generation, a controllable personalized generation method is proposed. This component linearly combines the next-token probability distribution conditioned on each preference, and uses the estimated preference strength as the weight to regulate the overall probability distribution of LLM output, thereby achieving personalized response generation.

[0038] The method provided by the embodiment of the application verifies AdaPA-Agent on the tasks of conversational recommendation and personalized web interaction. Experimental results show that, compared with existing advanced methods, AdaPA-Agent can significantly improve the quality of personalized generation, prompting the LLM agent to generate personalized responses that better meet the user's dynamically changing intentions and preferences.

[0039] Figure 1 The method flowchart of the large language model agent personalized response generation method provided by the embodiment of the application is shown in the figure. The following will take Figure 1 as an example to introduce in detail the large language model agent personalized response generation method provided by the embodiment of the application. As shown in Figure 1 , the method includes the following steps:

[0040] Step 101, based on the historical interaction behavior between the user and the agent, the current time interaction behavior, the user's current time intention is obtained;

[0041] Step 102, obtain a set of preferences from the historical interaction behavior, and construct a set of preference chains for each preference included in the set of preferences based on the thought chain. Rewrite the current time interaction behavior based on the generation function to obtain a plurality of semantic equivalent versions and an enhanced interaction set. Determine the fine-grained alignment score of each set of preference chains and the enhanced interaction set and the relative strength of each preference based on the scoring function;

[0042] Step 103: Using the current interaction behavior and the initial empty response prefix as basic input, construct a prompt template for each set of preference chains. The prompt template is based on a large language model to obtain the probability distribution of the next token under each preference condition.

[0043] Step 104: Perform a weighted summation of the probability distribution of the next token based on the relative strength to obtain a personalized target distribution. Generate the next token by sampling word by word based on the personalized target distribution. Concatenate the next token into the response prefix and update the response prefix until the response ends and a complete personalized response is output.

[0044] Step 101 is the process of user interaction and input acquisition. Specifically, it involves collecting user interaction data, namely, recording the historical interaction behavior between the user and the intelligent agent, the user's current interaction behavior, and determining the user's current intent.

[0045] In practical applications, historical interaction behavior can be represented by the following formula:

[0046] (1)

[0047] in, Indicates user, Indicates the user's position in the first month. Single interaction behavior data in a single interaction. This represents the total number of preferences. Indicates user Historical interaction behavior, This indicates the sequence number of the historical interaction rounds, ranging from 1 to... , Indicates user The total number of historical interactions.

[0048] For example, in a movie recommendation scenario, the system collects the user's historical conversations (such as "I've always enjoyed watching director A's early films") and current inputs (such as "I'm a little tired today and want to watch something lighthearted").

[0049] In step 102, a preference set is obtained from historical interaction behavior, and a set of preference chains is constructed for each preference included in the preference set based on the thought chain.

[0050] In practical applications, the preference chain is a three-stage preference chain, which can be represented as:

[0051] (2)

[0052] The preference set can be represented as:

[0053] (3)

[0054] wherein, represents the th preference chain, represents the th preference, represents the original preference, which can be directly extracted from each preference , represents the context-based preference reconstruction, which can enhance semantic clarity, represents the enumeration of specific examples of things or behaviors to concretize the meaning of , represents the chain of thoughts, represents the set of preferences,

[0055] In embodiments of the present application, the three-stage preference chain provided in the above method has a three-layer structure from coarse to fine, which provides rich preference information for subsequent alignment scoring.

[0056] For example, if the preference set includes three preferences, i.e. , then for each preference, three groups of preference chains can be obtained as shown below:

[0057] (prefer to watch works of art directors), ;

[0058] (prefer to watch works of art directors), ;

[0059] (prefer to watch works of art directors), .

[0060] Further, a large language model is called as a generation function to rewrite the interactive behavior at the current time to obtain multiple semantically equivalent versions. The multiple semantically equivalent versions and the interactive behavior at the current time are merged to obtain an enhanced interaction set.

[0061] Specifically, the enhanced interaction set can be represented as:

[0062] (4)

[0063] wherein, represents the enhanced interaction set, represents the generation function, represents the th moment of interactive behavior, represents the set union operation.

[0064] In practical applications, the multiple semantically equivalent versions are each Maintain the same number of rounds as the current interaction, but rewrite the content lexically and syntactically to expand the expressive space of user intent.

[0065] For example, if the current interaction behavior is "I'm a little tired today, I want to watch something lighthearted," then the large language model generation function will be invoked. ,Will Rewritten as a semantically equivalent version: {"Looking for some lighthearted comedy movies." "I just want to laugh, nothing too serious." "Recommend some funny films!"} Building Enhanced Interaction Collections : "I'm a bit tired today, I want to watch something lighthearted." "Some comedy movies that don't require much brainpower." "I just want to laugh, nothing too serious." "Recommend some funny movies!"

[0066] Compared to static embedding or keyword matching, large language models have a greater advantage in semantic understanding, enabling context-aware and refined alignment analysis. Furthermore, using a large language model, for each preference chain and interaction samples in the enhanced interaction set, a fine-grained alignment score is calculated using a scoring function. The fine-grained alignment scores of all preferences are then normalized to the relative strength of each preference, achieving preference strength inference without explicit feedback.

[0067] The fine-grained alignment score and the relative strength of each preference can be expressed as:

[0068] (5)

[0069] (6)

[0070] in, Indicates the first Fine-grained alignment score for each preference. Represents an enhanced interaction set. Indicates the first A preference chain of preferences, Represents an enhanced set of interactions The number of samples in the middle express Each interaction sample in the middle, Indicates the first The relative strength of each preference This represents the scoring function of a large language model, outputting a fine-grained alignment score from 1 to 10. express The sum of fine-grained alignment scores for each preference. This represents the total number of preferences. representing the enhanced interaction set all interaction samples in the set performing a summation operation.

[0071] For example, in the movie recommendation example, the fine-grained alignment scores are respectively: ; and the normalized preference intensity distributions are respectively: .

[0072] Through the above process, the method provided by the embodiment of the application can adaptively estimate and update the relative intensity of each preference without additional user feedback, thereby providing accurate support for personalized content generation.

[0073] After obtaining the relative intensity of each preference, the core problem is how to use these weights to guide the large language model to generate responses that continuously reflect the current intention of the user. In practical applications, due to the inherent ambiguity of natural language prompts, directly embedding numerical weights into text prompts usually has unstable effects. Recent research shows that by arithmetically combining the next token distribution generated by the large language model under different style conditions, complex text styles can be effectively generated. Inspired by this, the embodiment of the application proposes a preference arithmetic mechanism, that is, the next token probability distribution under multiple preference conditions is combined according to the formula, and the estimated relative intensity of each preference is used as the weight. This mechanism can achieve fine-grained control of the influence of each user's preference in the final generated response.

[0074] Specifically, at each step of generating the response prefix, the large language model outputs a probability distribution of the next token. Here, it is denoted as the personalized target distribution, which is influenced by the user's current intention at the moment. The personalized target distribution is modeled as a weighted combination of individual preference condition distributions, and each personalized target distribution is generated based on the next token probability distribution.

[0075] In step 103, initialize the context, take the interaction behavior at the current moment and the initial state of the empty response prefix as the basic input, and construct a prompt template for each group of preference chains; specifically, the prompt template includes “historical preference chain, current dialogue, generated response fragment”.

[0076] It should be noted that the “basic input” here refers to the most core and basic original information when constructing the generation task, which is the starting point and basis for all subsequent generation logic. The “basic input” includes the interaction behavior at the current moment, which refers to the user's action in the current turn (the first turn in the first round of dialogue in the example of the movie recommendation), and the initial state of the empty response prefix. The input information for each round (e.g., user questions, descriptions of needs, or dialogue content); the response prefix refers to the partial response fragments already generated in the current round (i.e., the first...). (Content preceding each token).

[0077] In practical applications, the generated task needs to clearly define both "what the user currently wants" (the current interaction behavior) and "what has already been said" (response prefix) in order to ensure that the generated content not only meets the user's needs but also avoids repetition, contradiction, or disconnect.

[0078] For example, suppose a user in a food recommendation scenario currently interacts with the message: "Recommend a restaurant suitable for couples' dates, not too noisy"; and the generated response prefix is: "Recommend two quiet restaurants suitable for couples' dates:". In this case, the "basic input" is the combination of these two parts: "user needs (couples' date, not noisy) + the generated prefix (recommend two quiet restaurants)". Subsequent steps will build upon this foundation, combining the user's preferences (such as "prefers Japanese food" and "moderate budget") to generate specific restaurant names and reasons.

[0079] Furthermore, the constructed prompt template is input into the large language model to generate the next token probability distribution under each preference condition. Specifically, the next token probability distribution under each preference condition can be represented as:

[0080] (7)

[0081] in, Indicates the first Individual preferences Given the probability distribution of the next token, [the following is a list of conditions]. This represents the k-th token to be generated. Indicates the generated response prefix. Indicates the first Real-time interactive behavior, Indicates the first A preference, Represents the generation function of a large language model. Indicates the first A preference chain of preferences.

[0082] In step 104, the next token probability distribution determined in step 103 is weighted and summed according to the relative strength to obtain the personalized target distribution; that is, the next token probability distribution under the preference condition. It is by using preference chains With current interaction behavior and the generated response prefix Together, these are used as input prompts and fed into the large language model for generation. Specifically, the personalized target distribution is shown below:

[0083] (8)

[0084] wherein, denotes the personalized target distribution, denotes the user's current intention, denotes the relative strength of the kth preference, denotes the kth generated token, denotes the generated response prefix, denotes the kth time instant interaction behavior, denotes the kth preference.

[0085] Further, the next token is generated by word-by-word sampling according to the personalized target distribution, the token is spliced into the response prefix, and the response prefix is updated until the complete personalized response is output. That is, the next token is generated according to the personalized target distribution in each round, and then the obtained token is spliced into the current response prefix, that is, the current response prefix is updated, and the sampling output is continued until the terminal token is obtained, and the complete personalized response is output.

[0086] For example, the context is initialized: the current time instant interaction behavior and the generated response prefix are taken as inputs.

[0087] The prompt template is constructed (taking the kth round as an example):

[0088] For each preference chain , the following prompt is constructed: [historical preference chain]+current dialogue: "I want to relax today and recommend a funny movie." + current generated response fragment;

[0089] The large language model generates a preference conditional probability distribution:

[0090] ;

[0091] ;

[0092] ;

[0093] The weighted combination distribution is calculated:

[0094] ;

[0095] The new token is generated by sampling ​​​​, which is concatenated into the response prefix. Repeat the steps until the generation encounters EOS (End of Sequence).

[0096] To more clearly introduce the large language model agent personalized response generation method provided by the embodiments of the application, the following takes the application in the movie recommendation intelligent assistant scene as an example to introduce the method again:

[0097] I. Scene and role setting

[0098] In a certain intelligent movie recommendation system, a user interacts with an agent through multiple rounds of dialogue to express preferences for different types of movies (such as art films, drama films, and light comedies), and the agent needs to dynamically estimate the preference intensity based on the AdaPA-Agent framework and generate personalized recommendations.

[0099] II. Process and specific formula application

[0100] Step 201, collect user interaction data:

[0101] Record the user's historical interaction behavior , the current time's interaction behavior , and the user's current intention .

[0102] Among them, the user's historical interaction behavior includes the following examples:

[0103] "I always like to watch the early films of Director A."

[0104] "The plot-based works are more likely to move me."

[0105] "I like to watch some light movies on weekends to relax my mind."

[0106] The current time's interaction behavior : "I'm a bit tired today, I want to watch something light."

[0107] Step 202, estimate based on fine-grained alignment and the relative intensity of each preference:

[0108] 202-1, extract the preference set from : through the thought chain to build a three-stage preference chain .

[0109] Among them, the preference set includes three preferences and their respective three groups of preference chains:

[0110] ​ (prefer works of art directors), ;

[0111] (prefer drama), ;

[0112] (prefer light-hearted and humorous films), .

[0113] 202-2, call large language model generation function , rewrite into a semantically equivalent version, build an enhanced interaction set .

[0114] Specifically: {"Come a comedy movie without thinking.", "I just want to laugh now, don't be too heavy.", "Recommend a funny movie! "} {"Today is a little tired, want to see something light." "Come a comedy movie without thinking." "I just want to laugh now, don't be too heavy." "Recommend a funny movie! "}

[0115] 202-3, alignment score and strength calculation based on LLM: for each preference chain and interaction samples, calculate the alignment score through the LLM scoring function , normalize the alignment score to the relative strength of each preference.

[0116] Specifically, assuming the LLM scoring result (example value):

[0117] The fine-grained alignment score of the first group of preference chains: , , , , then: .

[0118] The fine-grained alignment score of the second group of preference chains: , , , , then: .

[0119] The fine-grained alignment score of the third group of preference chains: , , , , then: .

[0120] The relative strength of each preference: , , .

[0121] Step 203, controllable personalized generation:

[0122] 203-1, construct input and prompt templates, initialize context, set generated response prefix (initially empty, i.e. no content has been generated yet), for each preference chain Construct a prompt template, as follows:

[0123] The prompt template for the first group of preference chains: . .

[0124] The prompt template for the second group of preference chains: . .

[0125] The prompt template for the third group of preference chains: . .

[0126] 203-2, generate preference condition distribution, input prompt into LLM, generate next token probability distribution under each preference condition Input the above three prompt templates into the large language model respectively to generate the "next token probability distribution" (i.e. the next possible word and its probability predicted by the model) under each preference condition, as follows:

[0127] The next token probability distribution under the first group of preference chains (literary director work preference): the next word may be "A literary film" (probability 25%), "Director A" (probability 20%), "literary film" (probability 15%), "recommend" (probability 10%). The next token probability distribution under the second group of preference chains (drama film preference): the next word may be "A drama film" (probability 22%), "drama film" (probability 18%), "recommend" (probability 12%), "emotional subtlety" (probability 10%).

[0128] The next token probability distribution under the third group of preference chains (light and humorous preference): the next word may be "A comedy film" (probability 30%), "B comedy film" (probability 25%), "recommend" (probability 15%), "funny" (probability 10%).

[0129] 203-3, weighted combination distribution formula: according to preference intensity Weighted combination distribution, get personalized target distribution

[0130] 203-3, weighted combination distribution formula: according to preference intensity Weighted combination distribution, get personalized target distribution ​The relative strength of each preference calculated in step 202: (Action Director), (Drama), (Light-hearted Humor). The three distributions are combined with weights to obtain the "personalized target distribution".

[0131] Take the word "recommend" as an example:

[0132] Preference chain The probability of "recommend" under the condition is 10%, and the contribution is 0.17 x 10% = 1.7%.

[0133] Preference chain The probability of "recommend" under the condition is 12%, and the contribution is 0.21 x 12% = 2.52%.

[0134] Preference chain The probability of "recommend" under the condition is 15%, and the contribution is 0.62 x 15% = 9.3%.

[0135] The total probability of "recommend" after combination: 1.7% + 2.52% + 9.3% ≈ 13.52%.

[0136] Similarly, the probabilities of other words (such as A comedy movie, A drama movie, etc.) are calculated in the same way, and finally the probability of "light-hearted humor" related words is significantly higher.

[0137] 203-4, Autoregressive sampling generation: in each round of sampling, the token is generated, and the response prefix is updated, until the EOS is generated.

[0138] The model generates responses word by word according to the "personalized target distribution", the process is as follows:

[0139] First round of sampling: based on the combined distribution, the probability of the word "recommend" is the highest (about 13.52%), so it is selected as the first token. At this time, the response prefix is generated for the first time, and updated to .

[0140] Second round of sampling: with the new prefix "recommend" as the context, three conditional distributions of preferences are generated again (at this time, the "generated response fragment" in the prompt template becomes "recommend"), and then combined according to the preference intensity. Suppose the combined probability of the word "you" is the highest, it is selected, and the response prefix is updated to .

[0141] Third round of sampling: generate a new conditional distribution based on the context "recommend you", and the word "watch" has the highest probability after combination, so it is selected, and the prefix is updated to .

[0142] Subsequent rounds: Continue to repeat the sampling process to generate "A comedy movie", "B comedy movie", "these", "funny", "movie", "fast-paced", "light-hearted", "suitable for relaxing body and mind . Until the model generates EOS.

[0143] The final complete response is: "I recommend you to watch A comedy movie or B comedy movie tonight, these funny movies are fast-paced and light-hearted, suitable for relaxing body and mind."

[0144] To sum up, the embodiment of the application provides a large language model agent personalized response generation method and device, which proposes a preference intensity estimation mechanism based on large language model semantic alignment, solving the problem that the prior art generally relies on user feedback or static labels; The mechanism uses the semantic understanding ability of the language model, without user labeled data or reliance on training signals, so it is more suitable for actual deployment environment; Furthermore, the method introduces a "preference arithmetic" mechanism, which formulates and fuses the large language model output distribution under each preference condition during generation, solving the defects of the existing prompt method: unable to realize quantitative control of multiple preference intensities, the generated response tends to be single preference or ambiguous, and lacks adjustability; Further, the method uses a bilateral enhancement strategy, unlike the traditional method of directly using the original preference and dialogue input, this enhancement process significantly improves the accuracy of preference estimation and the stability of response generation. Under the condition that the preference changes dynamically and explicit feedback is scarce, the method can effectively and accurately estimate the user's current preference intensity without additional user feedback, and accordingly adjust and control the personalized response generated by the agent in real time and in detail, thereby improving the accuracy, flexibility and adaptability of the LLM agent in serving users.

[0145] Based on the same inventive concept, the embodiments of the application provide a large language model agent personalized response generation device. Since the principle of the device solving the technical problem is similar to that of the large language model agent personalized response generation method, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described again.

[0146] Figure 2 The large language model agent personalized response generation device provided by the embodiment of the application is shown in the structure diagram as shown in Figure 2 The device comprises an acquisition unit 21, a determination unit 22, an obtaining unit 23 and an output unit 24.

[0147] The acquisition unit 21 is configured to acquire the historical interaction behavior between the user and the agent, the current time interaction behavior and the user's current time intention.

[0148] The determining unit 22 is configured to obtain a preference set from the historical interaction behaviors, construct a set of preference chains for each preference included in the preference set based on a thinking chain, rewrite the interaction behavior at the current moment based on a generation function to obtain a plurality of semantically equivalent versions and an enhanced interaction set, and determine a fine-grained alignment score of each set of preference chains and the enhanced interaction set and a relative strength of each preference based on a scoring function.

[0149] The obtaining unit 23 is configured to take the interaction behavior at the current moment and an initial state empty response prefix as a basic input, construct a prompt template for each set of the preference chains, and obtain a next token probability distribution under each preference condition based on a large language model.

[0150] The output unit 24 is configured to weight and sum the next token probability distribution according to the relative strength to obtain a personalized target distribution, generate a next token by word-by-word sampling according to the personalized target distribution, splice the next token into the response prefix and update the response prefix, and output a complete personalized response until the response ends.

[0151] Preferably, the preference chain is determined by the following formula:

[0152]

[0153] The enhanced interaction set is determined by the following formula:

[0154]

[0155] wherein, represents a preference chain of the i-th preference, represents the i-th preference, represents an original preference, represents a context-based preference reconstruction, represents an enumeration of specific examples or behavior examples, represents an interaction behavior at the i-th moment, represents a generation function, represents an enhanced interaction set, represents a set union operation, represents a thinking chain. Preferably, the fine-grained alignment score and the relative strength of each preference are determined by the following formula:

[0156]

[0157]

[0158] ​​​

[0159] wherein, denotes the fine-grained alignment score of the kth preference, denotes the enhanced interaction set, denotes the preference chain of the kth preference, denotes the number of samples in the enhanced interaction set denotes each interaction sample in the enhanced interaction set, denotes the relative strength of the kth preference, denotes the scoring function of the large language model, denotes the total number of preferences, denotes the sum of the fine-grained alignment scores of the kth preference.

[0160] Preferably, the prompt template comprises “historical preference chain, current dialogue, generated response fragment”;

[0161] The next token probability distribution is determined by the following formula:

[0162]

[0163] The personalized target distribution is determined by the following formula:

[0164]

[0165] wherein, denotes the next token probability distribution conditioned on the kth preference denotes the kth token to be generated, denotes the generated response prefix, denotes the interaction behavior at the tth time, denotes the kth preference, denotes the generation function of the large language model, denotes the preference chain of the kth preference, denotes the personalized target distribution, denotes the user’s current intention, denotes the relative strength of the kth preference.

[0166] ​​​​​​​​​​It should be understood that the units included in the above large language model agent personalized response generation apparatus are only logical divisions according to the functions implemented by the apparatus, and in actual application, the units can be stacked or split. And the functions implemented by the large language model agent personalized response generation apparatus provided in the embodiment correspond one-to-one to the large language model agent personalized response generation method provided in the above embodiment. For more detailed processing flow of the apparatus, the above method embodiment has been described in detail, and will not be described in detail here.

[0167] Another embodiment of the present application also provides a computer device, which comprises a processor and a memory; the memory is used to store computer program code, the computer program code comprises computer instructions; when the processor executes the computer instructions, the electronic device executes each step of the large language model agent personalized response generation method provided in the above method embodiment.

[0168] Another embodiment of the present application also provides a computer readable storage medium, which stores computer instructions, when the computer instructions run on a computer device, the computer device executes each step of the large language model agent personalized response generation method provided in the above method embodiment.

[0169] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to these embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0170] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A method for generating personalized responses of large language model agents, characterized in that, The method comprises the following steps: acquiring historical interaction behaviors between a user and an agent, current moment interaction behaviors and the user's current moment intention; acquiring a preference set from the historical interaction behaviors, and constructing a set of preference chains for each preference included in the preference set based on a thinking chain; rewriting the current moment interaction behaviors based on a generation function to obtain a plurality of semantic equivalent versions and an enhanced interaction set; determining a fine-grained alignment score of each set of preference chains and the enhanced interaction set and a relative strength of each preference based on a scoring function; constructing a prompt template for each set of the preference chains by taking the current moment interaction behaviors and an initial state empty response prefix as a basic input, wherein the prompt template is based on a large language model to obtain a next token probability distribution under each preference condition; performing weighted summation on the next token probability distribution according to the relative strength to obtain an individualized target distribution, and generating a next token by word-by-word sampling according to the individualized target distribution, splicing the next token into the response prefix and updating the response prefix until a complete individualized response is output.

2. The method of claim 1, wherein, The preference chain is determined by the following formula: The enhanced interaction set is determined by the following formula: wherein, represents a preference chain of the th preference, represents the th preference, represents the original preference, represents a context-based preference reconstruction, represents listing specific things or examples of behavior, represents the th moment of interaction behavior, represents a generating function, represents an augmented interaction set, represents a chain of thought, represents a set union operation.

3. The method of claim 1, wherein, The fine-grained alignment score and the relative strength of each preference are determined by the following formula: wherein, represents the fine-grained alignment score of the th preference, represents the enhanced interaction set, represents the preference chain of the th preference, represents the number of samples in the enhanced interaction set represents each interaction sample in the th preference, represents the scoring function of the large language model, represents the total number of preferences, represents the summation operation on all interaction samples in the enhanced interaction set represents the sum of the fine-grained alignment scores of theth preference.​​​ 4. The method of claim 1, wherein, The prompt template comprises "historical preference chain, current dialogue, generated response fragment"; The next token probability distribution is determined by the following formula: The individualized target distribution is determined by the following formula: in, Indicates the first Individual preferences Given the probability distribution of the next token, [the following is a list of conditions]. This represents the k-th token to be generated. Indicates the generated response prefix. Indicates the first Real-time interactive behavior, Indicates the first A preference, Represents the generation function of a large language model. Indicates the first A preference chain of preferences, Represents personalized target distribution. Indicates the user's current intent. Indicates the first The relative strength of each preference.

5. The apparatus for personalized response generation of large language model agent, characterized in that, The method comprises the following steps: an acquisition unit is configured to acquire historical interaction behaviors between a user and an agent, current moment interaction behaviors and the user's current moment intention; a determination unit is configured to acquire a preference set from the historical interaction behaviors, and construct a set of preference chains for each preference included in the preference set based on a thinking chain; rewriting the current moment interaction behaviors based on a generation function to obtain a plurality of semantic equivalent versions and an enhanced interaction set; determining a fine-grained alignment score of each set of preference chains and the enhanced interaction set and a relative strength of each preference based on a scoring function; an obtaining unit is configured to construct a prompt template for each set of the preference chains by taking the current moment interaction behaviors and an initial state empty response prefix as a basic input, wherein the prompt template is based on a large language model to obtain a next token probability distribution under each preference condition; an output unit is configured to perform weighted summation on the next token probability distribution according to the relative strength to obtain an individualized target distribution, and generate a next token by word-by-word sampling according to the individualized target distribution, splice the next token into the response prefix and update the response prefix until a complete individualized response is output.

6. The apparatus of claim 5, wherein, The preference chain is determined by the following formula: The enhanced interaction set is determined by the following formula: wherein, represents a preference chain of the first preference, represents the first preference, represents the original preference, represents a context-based preference reconstruction, represents listing specific things or examples of behavior, represents the moment of interaction behavior, represents a generating function, represents an enhanced interaction set, represents a set union operation, represents a thought chain.

7. The apparatus of claim 5, wherein, The fine-grained alignment score and the relative strength of each preference are determined by the following formula: wherein, represents a fine-grained alignment score of the th preference, represents an enhanced interaction set, represents a preference chain of the th preference, represents a number of samples in the enhanced interaction set represents each interaction sample in the th preference, represents a relative strength of the th preference, represents a scoring function of the large language model, represents a total number of preferences, represents a summation operation on all interaction samples in the enhanced interaction set represents a sum of fine-grained alignment scores of the preferences.​​ 8. The apparatus of claim 5, wherein, The prompt template comprises "historical preference chain, current dialogue, generated response fragment"; The next token probability distribution is determined by the following formula: The individualized target distribution is determined by the following formula: The method comprises the following steps: wherein, denotes the next token probability distribution conditioned on the kth preference, denotes the kth preference, denotes the kth token to be generated, denotes the generated response prefix, denotes the kth preference, denotes the kth preference, denotes the kth preference, denotes the kth preference, denotes the generation function of the large language model, denotes the preference chain of the kth preference, denotes the preference chain of the kth preference, denotes the personalized target distribution, denotes the user’s current intent, denotes the kth preference, denotes the relative strength of the kth preference.

9. A computer device, comprising: The computer device comprises a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to enable the processor to execute the large language model agent personalized response generation method according to any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The computer program is stored and executed by the processor to enable the processor to execute the large language model agent personalized response generation method according to any one of claims 1-4.

Images