Decoding stage role-playing agent persona following method and system

Abstract

The application relates to the technical field of artificial intelligence, and discloses a character setting following method and system for a role-playing intelligent agent in a decoding stage. The method comprises the following steps: for an input prompt word of a large language model, which comprises a context situation, a character setting attribute set and a user query, evaluating an importance score of each character setting attribute in the character setting attribute set under the context situation; based on the importance score, constructing a reward function for guiding the large language model to respond to a target character setting; and in the decoding reasoning stage of the large language model, using the reward function to dynamically adjust the probability distribution of the large language model in generating a word element at each time step, so that the generated response conforms to the target character setting under the context situation. Through real-time estimation of the importance of the character attribute and dynamic adjustment of the generation distribution in the reasoning process, the application can adaptively adjust the role performance according to different contexts and tasks, so as to generate an output that is more in line with the target character setting.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a method and system for role-playing intelligent agent personality following in the decoding stage. BACKGROUND

[0002] With the development of large language models, role-playing intelligent agents have been widely applied in dialogue systems, social science research, and virtual interactions. By setting a specific character profile, intelligent agents can simulate the personality traits, behavior patterns, and values of the character in dialogue, thereby enhancing the authenticity and consistency of the interaction. However, existing technologies still have obvious deficiencies in adapting and maintaining the consistency of the character profile in dynamic scenarios, i.e., they cannot achieve personality following.

[0003] Existing methods mainly include non-parametric methods based on prompt words and parameterized methods based on training. The former usually relies on adding character attribute descriptions and other prompt engineering methods to the input to guide the model to generate content that meets the set requirements. However, this method only processes the prompt at the shallow semantic level and lacks deep modeling of character attributes and situational adaptability, leading to generated content that deviates from the role setting. In contrast, the parameterized method uses large-scale corpora to enhance the role consistency of the model through fine-tuning, but it often requires high computational resources and a large amount of manually annotated data, especially in complex tasks involving multiple roles and scenarios, the data collection and modeling costs are very high. In addition, existing methods generally lack dynamic adaptability. Psychological research shows that humans activate different personality traits in different situations, while current intelligent agents usually treat character settings as static constraints and cannot adjust attribute weights with changing situations, thereby reducing the authenticity and situational relevance of generated content. Therefore, there is an urgent need for a technology with dynamic adaptability to improve the personality consistency and performance of role-playing intelligent agents in multiple scenarios and multiple roles. SUMMARY

[0004] To solve the above technical problems, the present application provides a method and system for role-playing intelligent agent personality following in the decoding stage. The present application can directly adjust the dynamic attribute weights in the decoding stage without introducing additional computational or storage overhead. The present application can maintain the adaptability and consistency of the character profile in multiple scenarios by estimating the importance of the character attributes in real time during the inference process and dynamically adjusting the generation distribution, which can adaptively adjust the role performance according to different contexts and tasks, thereby generating output that better meets the target character setting. The present application can be embedded in any language model framework without modifying the model structure to achieve personality consistency control.

[0005] To solve the above technical problems, the present application adopts the following technical solutions:

[0006] In a first aspect, the present application provides a method for persona following of a decoding stage role-playing agent, comprising:

[0007] For an input prompt word of a large language model containing a context scenario, a persona attribute set and a user query, an importance score of each persona attribute in the persona attribute set under the context scenario is evaluated, and the importance score is used to quantify the influence degree of the corresponding persona attribute on the response generated by the large language model;

[0008] Based on the importance score, a reward function for guiding the response of the large language model to align with the target persona is constructed, and the reward function integrates the weighted contribution of each persona attribute based on the corresponding importance score;

[0009] In the decoding inference stage of the large language model, the reward function is used to dynamically adjust the probability distribution of the large language model for generating word units at each time step, so that the generated response conforms to the target persona under the context scenario.

[0010] In one embodiment, the evaluation of the importance score of each persona attribute in the persona attribute set under the context scenario specifically includes:

[0011] The importance score of the persona attribute is calculated as:

[0012] ;

[0013] wherein, represents the input prompt word, represents the input prompt word after removing the persona attribute from the input prompt word, represents the response generated by the large language model based on the input prompt word represents the conditional probability, represents the "defined as".

[0014] In one embodiment, the construction of the reward function for guiding the response of the large language model to align with the target persona based on the importance score specifically includes:

[0015] ;

[0016] is a preliminary reward function, represents the input prompt word, represents the response generated by the large language model, represents the importance score of the i-th persona attribute ​​​​The total number of character attributes. for Corresponding step-by-step character development rewards;

[0017] The initial reward function is normalized to obtain the final reward function. :

[0018] .

[0019] In one embodiment, At time step Corresponding step-by-step character design rewards for:

[0020] ;

[0021] in, For time step index, For large language models at time steps The generated lexical units, For large language models at time steps The generated response, This indicates that the large language model incorporates persona attributes into the input prompts. Given the distribution of predicted output for the next word, This indicates that the large language model does not include persona attributes in the input prompts. In the case of time step The predicted output distribution of the word units. Indicates from Remove character attributes The input prompt word after that.

[0022] In one embodiment, dynamically adjusting the probability distribution of word units generated by the large language model at each time step using the reward function, so that the generated response conforms to the target persona in the context, specifically includes:

[0023] Dynamically adjust the lexical units generated at time step t of the large language model probability distribution :

[0024] ;

[0025] in, Indicates input prompt words, Indicates at time step The generated word units, Indicates at time step The generated response, Representing the large language model at time steps a predicted output distribution of the token, denotes a regularization parameter, is a reward function, is a normalization factor, ensuring is a valid probability distribution:

[0026] ;

[0027] is a reward function, denotes a predicted output distribution of the token generated by the large language model at time step , is a response generated by the large language model at time step , denotes a predicted output distribution of the token generated by the large language model at time step , under the condition that the input prompt contains a persona attribute.

[0028] In a second aspect, the present application provides a persona following system for decoding stage role-playing agents, comprising:

[0029] a persona importance evaluation module: for an input prompt of a large language model containing a context scenario, a set of persona attributes, and a user query, evaluating an importance score of each persona attribute in the set of persona attributes under the context scenario, the importance score being used to quantify the influence degree of the corresponding persona attribute on the response generated by the large language model;

[0030] a reward function construction module: based on the importance score, constructing a reward function for guiding the response of the large language model to align with the target persona, the reward function fusing the weighted contributions of each persona attribute based on the corresponding importance score;

[0031] an inference module: in the decoding inference stage of the large language model, using the reward function to dynamically adjust the probability distribution of the large language model to generate tokens at each time step, so that the generated response conforms to the target persona under the context scenario.

[0032] In one embodiment, the evaluation of the importance score of each persona attribute in the set of persona attributes under the context scenario specifically includes:

[0033] calculating the importance score of the persona attribute :

[0034] ;

[0035] wherein, denotes an input prompt, ​denotes the input prompt word, after removing the persona attribute from the input prompt word, denotes the response generated by the large language model based on the input prompt word, denotes the conditional probability, denotes the “defined as”.

[0036] In one embodiment, the constructing, based on the importance score, a reward function for guiding the large language model response to align with the target persona, specifically includes:

[0037] ;

[0038] for the preliminary reward function, denotes the input prompt word, denotes the response generated by the large language model, denotes the importance score of the i-th persona attribute, denotes the total number of persona attributes, denotes the corresponding step-by-step persona reward; normalizing the preliminary reward function to obtain the final reward function :

[0039]

[0040] .

[0041] In one embodiment, the corresponding step-by-step persona reward at time step is:

[0042] ;

[0043] wherein, denotes the index of time step , denotes the word token generated by the large language model at time step , denotes the response generated by the large language model at time step , denotes the predicted output distribution of the next word token by the large language model when the input prompt word contains the persona attribute , denotes the predicted output distribution of the word token at time step by the large language model when the input prompt word does not contain the persona attribute , denotes the corresponding step-by-step persona reward; denotes the input prompt word, ​​​​​​​Removing human attribute after input prompt words.

[0044] In one of the embodiments, the dynamic adjustment of the probability distribution of the generated token by the large language model at each time step by using the reward function, so that the generated response meets the target persona under the context, specifically includes:

[0045] Dynamic adjustment of the probability distribution of the generated token by the large language model at time step t :

[0046] ;

[0047] Wherein, represents the input prompt word, represents the generated token at time step , represents the generated response at time step , represents the predicted output distribution of the large language model for the token at time step , represents the regularization parameter, is the reward function, is a normalization factor, ensuring is a valid probability distribution: ;

[0048] ;

[0049] is the reward function, represents the generated token by the large language model at time step , represents the generated response by the large language model at time step , represents the predicted output distribution of the large language model for the token at time step , under the condition that the input prompt word contains the persona attribute .

[0050] The system and method in the present application correspond to the specific technical solutions of the method, and are also applicable to the system.

[0051] Compared with the prior art, the beneficial technical effects of the present application are:

[0052] ​​​​​(1) No additional training or parameter overhead: The method directly estimates the importance of the character attributes and dynamically aligns them in the inference stage without retraining the language model or introducing additional parameterized modules. (2) Dynamic character attribute control: The importance estimation of the character attributes is introduced in the inference stage, which dynamically adjusts the weights of different character attributes according to the current dialogue context, accurately controls the performance of the role in different contexts, and makes the generated text more consistent with the target character setting. (3) Good universality and scalability: The method does not depend on specific model structure or training strategy, and can be widely adapted to mainstream large language model architectures, and can be extended to multi-role and multi-context character alignment tasks. This method has wide application potential and practical value in social simulation, virtual interaction, educational companionship, and entertainment games. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 The method flowchart in the embodiment of the present application is shown in the following.

[0054] Figure 2 The overall architecture diagram in the embodiment of the present application is shown in the following. DETAILED DESCRIPTION

[0055] A preferred embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0056] As shown in the following, Figure 1 The person setting following method of the decoding stage role-playing intelligent agent in the present application includes the following steps:

[0057] S1, for the input prompt word of the large language model containing the context situation, the person setting attribute set and the user query, the importance score of each person setting attribute in the person setting attribute set under the context situation is evaluated, and the importance score is used to quantify the influence degree of the corresponding person setting attribute on the response generated by the large language model;

[0058] S2, based on the importance score, a reward function for guiding the large language model response to align with the target person setting is constructed, and the reward function integrates the weighted contribution of each person setting attribute based on the corresponding importance score;

[0059] S3, in the decoding inference stage of the large language model, the reward function is used to dynamically adjust the probability distribution of the word element generated by the large language model at each time step, so that the generated response conforms to the target person setting under the context situation.

[0060] Role-playing intelligent agent technology is generally defined as: an artificial intelligence system used to simulate a specified role, which uses the advanced capabilities of large language models to simulate human characteristics and perform lively role-playing performances.

[0061] The process proposed in the present application is shown in the followingFigure 2 As shown, it includes the following three parts: (1) personae importance assessment; (2) personae-driven reward function construction; (3) inference stage alignment. The three parts work together in the inference stage to adjust the output distribution of the large language model in the form of dynamic rewards, realizing real-time alignment of the personae and the context.

[0062] 1. Personae importance assessment.

[0063] According to psychological theories, the influence of personae attributes on behavior is not static, but dynamic with the context. Therefore, the present application introduces a personae importance assessment module, which can assess the importance of each personae attribute in different contexts without the need for labeled data.

[0064] Specifically, given a complete input prompt , the probability of the large language model generating an output can be represented as:

[0065] ;

[0066] This represents the joint probability of the large language model generating the entire answer given the complete input prompt. Wherein represents the context, represents the set of personae attributes containing personal attributes such as style, personality, and professional personality test results (MBTI), represents the user query. According to information theory, the influence of a certain personae attribute on the output can be measured by conditional mutual information (CMI):

[0067] ;

[0068] Wherein, represents the input prompt after removing the personae attribute. represents the conditional mutual information, represents the conditional entropy. This mutual information represents the information gain brought by the personae attribute to the output , the greater the value, the more significant the attribute's influence on the output of the large language model.

[0069] Due to the large output space of the large language model, direct entropy calculation is computationally expensive, so the present application uses a single representative output sample to approximate the overall distribution. Assuming that there is a standard output that meets the requirements of the character portrait, the conditional mutual information can be rewritten as: ​

[0070] ;

[0071] ;

[0072] Thus, the character attributes are obtained. Importance score :

[0073] ;

[0074] “ " means "defined as".

[0075] Due to the real standard response in role-playing tasks with multiple roles and multiple scenarios Often unavailable, this makes directly calculating the importance of character attributes challenging. This invention proposes a self-supervised approximation method that utilizes responses generated by a large language model itself under complete prompts. As a proxy, assume the distribution generated by the large language model and If there is a positive correlation, then Can be used as A reliable proxy. This correlation assumption is reasonable because the training objective of large language models is to maximize the probability of generating a realistic response given a cue. This indicates the response generated based on the complete prompt. The importance score obtained Indicates a response based on true standards. The importance score obtained.

[0076] Ultimately, in the absence of real-world labeled responses, this invention provides a theoretically sound and practically reliable method for quantifying the importance of personas, adaptable to the current context:

[0077] .

[0078] 2. Construction of a persona-driven reward function.

[0079] This section is used to integrate the importance score estimated by the persona importance assessment module into the reward function, so as to adjust the output of the large language model during the inference phase to keep it consistent with the target persona.

[0080] First, define the multi-target alignment problem. Constrained reinforcement learning objectives are achieved by adjusting the behavior of large language models to conform to specific properties through a reward function:

[0081] .

[0082] in, is the base model to be aligned, is the aligned model distribution, denotes the mathematical expectation, is the reward function, which quantifies the preference level of the given complete input prompt and response , denotes divergence, is a regularization parameter. The objective aims to achieve output alignment through reward optimization without changing the parameters of the large language model.

[0083] The objective of the present application is to align the response of the large language model with the preset set of persona attributes , which is equivalent to maximizing the divergence between the unconstrained model strategy and the constrained strategy. Therefore, for each persona attribute , the term can be expressed as the expected log ratio predicted by the large language model at the th time step:

[0084] .

[0085] Based on this formula, the step-by-step persona reward for each attribute can be defined as:

[0086] .

[0087] where and represent the predicted output distribution of the large language model for the next word unit with and without the persona attribute . This step-by-step reward effectively captures the local influence of each persona attribute during the generation of each word unit, allowing real-time adjustment of generation probabilities through reward guidance.

[0088] To align multiple persona attributes in the set of persona attributes of the role, a multi-objective strategy alignment framework is proposed. Based on the traditional multi-objective alignment method and considering the priority of different persona attributes in the context, a dynamic weighting mechanism is introduced. Specifically, a persona importance score is assigned to each persona attribute , and these importance scores are used to construct a weighted reward function :

[0089] .

[0090] However, this weighting scheme, while enabling multi-objective alignment, has a key limitation: unconstrained optimization can lead to all Maximizing simultaneously, blurring the priority hierarchy between objectives, hinders the generation of personalized, preference-aware Pareto-optimal solutions.

[0091] Therefore, the present application proposes a normalized reward function to encourage the desired persona ordering, i.e. while preserving the expected priority of aligned objectives:

[0092] ;

[0093] where, represents the reward vector. In maximizing the reward , it will prompt the individual reward to maintain the same ordering as its corresponding importance score , thus explicitly preserving the hierarchy of persona attributes in the alignment process; represents the F2 norm.

[0094] 3. Reasoning phase alignment.

[0095] This part can dynamically adjust the output probability distribution of the large language model without fine-tuning, so that it aligns with the target persona. Specifically, by integrating into the decoding process of the large language model, the generation probability at each time step is dynamically adjusted.

[0096] By using the Lagrange multiplier method to construct the function, the closed-form solution of the probability output can be obtained by derivation. At each time step , the alignment strategy of the large language model can be calculated by the following formula:

[0097] ;

[0098] where, is the normalization factor, ensuring is a valid probability distribution:

[0099] ;

[0100] In this way, the large language model will dynamically adjust the generation probability at each time step according to the current context and persona importance, so as to generate responses consistent with the target persona.

[0101] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the terms "comprises", "comprising", "includes", "including" and the like are specifically intended to be open-ended and to mean that other steps, components, features, and / or objects not specifically recited are also contemplated.

[0102] It should be understood that although the steps in the flowcharts of the drawings are shown in a sequential order, these steps are not necessarily performed in the order shown. Unless explicitly stated, as long as there is no conflict, the order of the steps can be changed. Furthermore, at least one of the steps in the flowcharts of the drawings can include multiple steps or multiple stages, which are not necessarily performed at the same time, but can be performed at different times, and the order of the steps or stages can not necessarily be sequential, but can be performed in a round-robin or alternating manner with other steps or steps or stages in other steps.

[0103] Based on the description of the method embodiments, the application further provides a system. The system can be a software (application), module, component, server, client, etc. using the method described in the embodiments of the present application and combined with the necessary implementation hardware. Based on the same innovative concept, the system in one or more embodiments provided by the embodiments of the present application is described in the following embodiments. Since the implementation scheme of the system solving the problem is similar to the method, the implementation of the specific system in the embodiments of the present application can be referred to the implementation of the foregoing method, and the repeated parts will not be described herein. The term "module" or "module" used below is a combination of software and / or hardware that can realize a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware is also possible and is conceived.

[0104] A character role-playing intelligent agent character following system, comprising:

[0105] A character importance evaluation module: for an input prompt word of a large language model containing a context situation, a character attribute set and a user query, evaluating an importance score of each character attribute in the character attribute set under the context situation, the importance score being used to quantify the influence degree of the corresponding character attribute on the response generated by the large language model;

[0106] A reward function construction module: based on the importance score, constructing a reward function for guiding the large language model response to align with the target character, the reward function fusing the weighted contribution of each character attribute based on the corresponding importance score;

[0107] The reasoning module: in the decoding reasoning stage of the large language model, the reward function is used to dynamically adjust the probability distribution of generating word units at each time step of the large language model, so that the generated response meets the target persona under the context situation.

[0108] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not contradict, they should be considered within the scope of the present disclosure.

[0109] It is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and therefore all changes falling within the meaning and scope of the equivalent elements of the claims should be included in the present application, and any reference signs in the claims should not be considered as limiting the claims.

[0110] Furthermore, it should be understood that although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the description manner of the specification is only for the sake of clarity, and those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can be combined appropriately to form other embodiments that those skilled in the art can understand.

Claims

1. A method for character following of a decoding phase role-playing agent, the method comprising: The method comprises the following steps: For the input prompt word of the large language model containing the context situation, the set of persona attributes and the user query, evaluating an importance score of each persona attribute in the set of persona attributes under the context situation, the importance score being used to quantify the influence degree of the corresponding persona attribute on the response generated by the large language model; Based on the importance score, a reward function is constructed for guiding the response of the large language model to align with the target persona, and the reward function integrates the weighted contribution of each persona attribute based on the corresponding importance score; In the decoding inference stage of the large language model, the probability distribution of the word units generated by the large language model at each time step is dynamically adjusted using the reward function, so that the generated response conforms to the target persona under the context, specifically including: dynamically adjusting the probability distribution of the word units generated by the large language model at time step t is the input prompt word, is the generated word unit at time step , is the generated response at time step , represents the predicted output distribution of the large language model for the word unit at time step , is a regularization parameter, is a reward function, is a normalization factor, ensuring is a valid probability distribution: is a reward function, is the word unit generated by the large language model at time step , is the generated response by the large language model at time step , is the predicted output distribution of the large language model for the word unit at time step when the input prompt word contains a persona attribute .​​​​ 2. The method of claim 1, wherein, The evaluation of the importance score of each persona attribute in the set of persona attributes under the context situation specifically comprises: Computing a personae attribute of importance score : ； in, Indicates input prompt words, Indicates from Remove character attributes The input prompt word after that, The large language model is based on input prompts. The generated response, Represents conditional probability. It means "defined as".

3. The method of claim 1, wherein, The construction of the reward function for guiding the response of the large language model to align with the target persona based on the importance score specifically comprises: ； is a preliminary reward function, denotes an input prompt, denotes a response generated by a large language model, denotes an importance score of the i-th persona attribute, denotes a total number of persona attributes, is a preliminary reward function, is a corresponding step-by-step persona reward; The preliminary reward function is normalized to obtain a final reward function : 。 4. The method of claim 3, wherein, At time step Corresponding step-wise human reward For: ； wherein, is an index of a time step, is a wordpiece generated by the large language model at time step is a response generated by the large language model at time step represents a predicted output distribution of the large language model for the next wordpiece in the case that the input prompt word contains a person-specifying attribute, represents a predicted output distribution of the large language model for the wordpiece at time step in the case that the input prompt word does not contain a person-specifying attribute, represents the input prompt word after removing the person-specifying attribute from ​​​​​​​ 5. A persona-following system for decoding a stage role-playing intelligent agent, characterized in that, The method comprises the following steps: The persona importance evaluation module: for the input prompt word of the large language model containing the context situation, the set of persona attributes and the user query, evaluating an importance score of each persona attribute in the set of persona attributes under the context situation, the importance score being used to quantify the influence degree of the corresponding persona attribute on the response generated by the large language model; The reward function construction module: based on the importance score, a reward function is constructed for guiding the response of the large language model to align with the target persona, and the reward function integrates the weighted contribution of each persona attribute based on the corresponding importance score; The reasoning module: in the decoding reasoning stage of the large language model, the reward function is used to dynamically adjust the probability distribution of the word units generated by the large language model at each time step, so that the generated response conforms to the target persona under the context situation, specifically comprising: dynamically adjusting the probability distribution of the word units generated by the large language model at time step t : ; is the input prompt word, is the generated word unit at time step is the generated response at time step is the predicted output distribution of the large language model for the word unit at time step is a regularization parameter, is a reward function, is a normalization factor, ensuring is a valid probability distribution: ; is a reward function, is the word unit generated by the large language model at time step is the generated response by the large language model at time step is the predicted output distribution of the large language model for the word unit at time step when the input prompt word contains a persona attribute .​​​​​​ 6. The persona-following system for decoding phase role-playing agents according to claim 5, wherein, The evaluation of the importance score of each persona attribute in the set of persona attributes under the context situation specifically comprises: Computing a personae attribute of importance score : ； in, Indicates input prompt words, Indicates from Remove character attributes The input prompt word after that, The large language model is based on input prompts. The generated response, Represents conditional probability. It means "defined as".

7. The persona following system for decoding phase role-playing intelligent agents of claim 5, wherein, The construction of the reward function for guiding the response of the large language model to align with the target persona based on the importance score specifically comprises: ； is a preliminary reward function, denotes an input prompt word, denotes a response generated by a large language model, denotes an importance score of the i-th persona attribute, is a total number of persona attributes, is a total number of persona attributes, is a total number of persona attributes, is a corresponding step-by-step persona reward; The preliminary reward function is normalized to obtain a final reward function : 。 8. The persona following system for decoding stage role-playing agents according to claim 7, wherein, At time step Corresponding step-wise human reward For: ； wherein, is an index of a time step, is a wordpiece generated by the large language model at time step is a response generated by the large language model at time step represents a predicted output distribution of the large language model for the next wordpiece given that the input prompt word contains a person-specifying attribute, represents a predicted output distribution of the large language model for the wordpiece at time step given that the input prompt word does not contain a person-specifying attribute, represents the input prompt word after removing the person-specifying attribute from ​​​​​​​

Images