Method for aligning and tuning personality consistency of large reinforcement learning role model and related product

Abstract

The invention relates to the technical field of natural language processing, and discloses a reinforcement learning role large model personality consistency aligning and tuning method and a related product. The method comprises the following steps: acquiring target personality setting information and performing personality setting on a large role model; after a response text is generated in each round of dialogue, performing real-time evaluation in combination with target personality setting information and dialogue context to obtain a personality consistency score of the response text; when the personality consistency score is lower than a score threshold value, performing grading processing according to the deviation degree: when the deviation degree is relatively small, injecting implicit personality correction prompt information, and when the deviation degree is relatively large, constructing a personality alignment training sample and adding the personality alignment training sample into a training queue; and when a triggering condition is met, a personality integrating degree reward value output by the personality reward model is taken as a reward signal, the model is finely adjusted by adopting a reinforcement learning algorithm, and a KL divergence punishment limitation strategy drift is added, so that a'detection-deviation correction-tuning 'closed-loop mechanism is formed, the personality drift phenomenon in a long dialogue is effectively inhibited, and the cross-wheel personality consistency is remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, and particularly relates to a method for optimizing the personality consistency alignment of a large reinforcement learning role model, a device for optimizing the personality consistency alignment of a large reinforcement learning role model, a system for optimizing the personality consistency alignment of a large reinforcement learning role model, a computer device, and a computer-readable storage medium. Background Technology

[0002] Large pre-trained language models (LLMs) have been widely used in conversational AI assistants, virtual digital humans, and other fields. However, in current technology, these models are prone to personality drift and inconsistency during long, multi-turn conversations. For example, a model may begin the conversation as humble and mild-mannered, but after several rounds of interaction, its tone may become assertive, arrogant, or contradictory. This inconsistency in character not only weakens the user experience and reduces user trust in the AI ​​assistant, but may also raise ethical and safety concerns.

[0003] Current methods primarily rely on statically preset personality cues (e.g., adding fixed persona descriptions before model input) or one-time, concentrated fine-tuning training to shape the model's personality. However, these static methods struggle to ensure the stability of the model's personality throughout a long conversation. Once the conversational context or user situation changes, the model often deviates from the expected role setting. In particular, traditional instruction-based training or one-time fine-tuning cannot prevent the model from occasionally deviating from its role, lacking a real-time correction mechanism. Furthermore, previous technologies lack quantitative evaluation metrics for personality consistency, making it difficult for developers to promptly detect whether the model's personality has deviated. For these reasons, existing technologies lack an effective solution that can monitor and correct personality deviations in real time during conversations and maintain personality consistency through continuous learning.

[0004] In recent years, some studies have begun to focus on the assessment and long-term consistency maintenance of personality in large-scale models. Examples include constructing personality-related assessment data to more comprehensively evaluate the personality consistency performance of dialogue agents, and introducing long-term memory and knowledge retention mechanisms to improve consistency in long-term dialogues. These studies provide references for personality consistency detection and assessment, but have not yet formed a closed-loop solution capable of real-time monitoring, immediate correction, and offline optimization during dialogue. In short, how to ensure that the model "always communicates with the user like the same person" in long conversations remains a pressing problem that needs to be solved in current technologies. Summary of the Invention

[0005] This invention provides a reinforcement learning role model personality consistency alignment optimization method, a reinforcement learning role model personality consistency alignment optimization device, a computer device, and a computer-readable storage medium, which can monitor and correct personality biases online and significantly improve the consistency of role model personality expression in long dialogues.

[0006] To facilitate understanding of the present invention, the relevant technical terms involved in the present invention will be briefly explained below.

[0007] Target personality setting information: A collection of information used to define the personality attributes and behavioral style that the character model should exhibit in dialogue, including personality traits, tone of voice, background setting, value preferences, and prohibited expressions.

[0008] Personality trait information: Personality facts or preference entries that are extracted from historical conversations and written into the personality knowledge base during the dialogue process and can be retrieved and accessed, including the source round and timestamp.

[0009] Personality consistency: The degree to which the response text matches the target personality profile and the context of the conversation.

[0010] Personality consistency score: a scalar value used to quantify the strength of personality consistency, with the preferred value range being [0,1].

[0011] Personality consistency score threshold: A threshold used to determine whether to trigger correction or optimization. When the personality consistency score is greater than or equal to the personality consistency score threshold, the response text is considered to meet the personality consistency requirements; when the personality consistency score is less than the personality consistency score threshold, personality deviation is considered to exist.

[0012] Deviation difference and deviation difference threshold: The deviation difference is defined as the difference between the personality consistency score and the personality consistency score threshold. When the deviation difference is less than the deviation difference threshold, it is judged as a minor deviation; when the deviation difference is greater than or equal to the deviation difference threshold, it is judged as a serious deviation.

[0013] Implicit personality correction prompts: Correction information that is not visible to the user and is injected into the model input context as a system message / hidden prompt. Implicit personality correction prompts are generated based on the personality points and deviation types in the target personality setting information.

[0014] Insertion frequency: The frequency at which implicit personality correction prompts are injected in consecutive dialogue rounds, used to balance the timeliness of correction and the naturalness of dialogue, and can be dynamically adjusted as the deviation persists.

[0015] Round threshold: When the number of rounds in the same session where there are slight deviations reaches the round threshold, the system will be upgraded from "prompt correction" to "model fine-tuning".

[0016] Similarity threshold: A threshold used to constrain whether personality feature information can be injected. When the similarity between the input text vector and the personality feature information vector is greater than or equal to the similarity threshold, the personality feature information is considered to be relevant to the current round of dialogue and can be injected; otherwise, it is not injected to avoid false detection.

[0017] Reference strategy and penalty coefficient: During the model fine-tuning process triggered by personality bias, the reference strategy can be set as the strategy after the initial fine-tuning of the character's large model; the penalty coefficient is used to limit the degree of deviation between the updated strategy and the reference strategy.

[0018] In a first aspect, the reinforcement learning role large model personality consistency alignment optimization method provided by this invention includes: Obtain target personality setting information, and set the personality of the large character model based on the target personality setting information; In each dialogue round, input text is received, and response text corresponding to the input text is generated through the role model; Based on the dialogue context of the input text and the target personality setting information, the personality consistency of the response text is evaluated to obtain a personality consistency score; If the personality consistency score does not reach the scoring threshold, and the difference between the two does not reach the difference threshold, then implicit personality correction prompts are generated based on the response text for subsequent dialogue rounds to inject into the character big model; if the personality consistency score does not reach the scoring threshold, and the difference between the two reaches the difference threshold, then personality alignment training samples are constructed based on relevant dialogue data in the current dialogue round and added to the training queue. After publishing the response text, the next dialogue round begins. When the preset triggering conditions are met, based on the personality alignment training samples in the training queue, the pre-trained personality reward evaluation model is called to obtain the personality fit reward value of the response text. The personality fit reward value is used as the reward signal for reinforcement learning to drive the reinforcement learning algorithm to update the strategy of the large character model, thereby completing the fine-tuning training of the large character model.

[0019] Secondly, the reinforcement learning role large model personality consistency alignment optimization device provided by the present invention includes: The personality setting module is used to obtain target personality setting information and set the personality of the large character model based on the target personality setting information; The language model interaction module is used to receive input text in each dialogue round and generate response text corresponding to the input text through the role big model; The consistency detection module is used to evaluate the personality consistency of the response text based on the dialogue context of the input text and the target personality setting information, and obtain a personality consistency score. The personality alignment optimization module is used to generate implicit personality correction prompts for injecting into the character model in subsequent dialogue rounds when the personality consistency score does not reach the scoring threshold and the difference between the two does not reach the difference threshold; and to construct personality alignment training samples based on relevant dialogue data in the current dialogue round and add them to the training queue when the personality consistency score does not reach the scoring threshold and the difference between the two reaches the difference threshold. The language model interaction module is also used to proceed to the next dialogue round after the response text is published; The model training module is used to perform fine-tuning training on the large character model when preset trigger conditions are met. This includes: based on personality alignment training samples in the training queue, calling a pre-trained personality reward evaluation model to obtain personality fit reward values, and using the personality fit reward values ​​as a reward signal for reinforcement learning to drive the reinforcement learning algorithm to update the strategy of the large character model, thereby completing the fine-tuning training of the large character model.

[0020] Optionally, in one embodiment, the personality alignment optimization module is further configured to introduce the Kullback-Leibler divergence (KL divergence) between the updated policy and the reference policy as a penalty term into the loss function during the policy update process, so as to limit the policy update magnitude; wherein, the reinforcement learning algorithm is a reinforcement learning algorithm suitable for model policy update, and the loss function includes the truncation target term and penalty term of the corresponding reinforcement learning algorithm.

[0021] Optionally, in one embodiment, the consistency detection module is used to calculate the semantic similarity between the response text and the target personality setting information to obtain a personality matching score; calculate the semantic coherence between the response text and the dialogue context to obtain a context coherence score; and fuse the personality matching score and the context coherence score to obtain a personality consistency score.

[0022] Optionally, in one embodiment, the reinforcement learning role large model personality consistency alignment optimization device provided by the present invention further includes a dialogue record and memory module, used to retrieve first personality feature information that matches the input text with keywords, the personality feature information being extracted based on historical dialogue content; the first personality feature information is injected into the dialogue generation process of the role large model in the form of hidden hints or decoding constraints.

[0023] Optionally, in one embodiment, the dialogue recording and memory module is further configured to inject second personality feature information, whose vector similarity to the input text reaches a similarity threshold, into the dialogue generation process of the large character model in the form of hidden hints or decoding constraints when the retrieval of first personality feature information fails.

[0024] Optionally, in one embodiment, the personality alignment optimization module is used to obtain the number of dialogue rounds in the same dialogue where the cumulative personality consistency score has not reached the scoring threshold when the personality consistency score has not reached the scoring threshold and the difference between the two has not reached the difference threshold; when the number of dialogue rounds has not reached the round threshold, it generates implicit personality correction prompts for injection into subsequent dialogue rounds based on the response text; the model training module is also used to perform fine-tuning training on the large character model when the number of dialogue rounds reaches the round threshold.

[0025] The personality alignment scheme for a large role model provided by this invention obtains target personality setting information and sets the personality of the large role model based on the target personality setting information; in each dialogue round, it receives input text and generates response text corresponding to the input text through the large role model; based on the dialogue context of the input text and the target personality setting information, it evaluates the personality consistency of the response text and obtains a personality consistency score; if the personality consistency score does not reach the scoring threshold and the difference between the two does not reach the difference threshold, it generates implicit personality correction prompts for subsequent dialogue rounds to be injected into the large role model based on the response text; if If the personality consistency score does not reach the scoring threshold, but the difference between the two scores reaches the difference threshold, then a personality alignment training sample is constructed based on the relevant dialogue data in the current dialogue round. After providing the response text to the user, the next dialogue round begins. When a preset trigger condition is met, fine-tuning training of the large role model is performed, including: based on the constructed personality alignment training sample, calling a pre-trained personality reward evaluation model to obtain the personality fit reward value of the response text; using the personality fit reward value as a reward signal for reinforcement learning to drive the reinforcement learning algorithm to update the strategy of the large role model, thereby completing the fine-tuning training of the large role model. This achieves a closed-loop mechanism for dynamic perception, real-time calibration, and progressive optimization of the personality performance of the large role model. It can monitor personality deviations online and automatically select correction prompts or model fine-tuning strategies according to the degree of deviation, significantly improving the consistency of personality expression in long dialogues. Compared with the prior art, this invention has at least the following beneficial effects and technical advantages: Closed-loop dynamic alignment mechanism: Personality control points are set in the model input, reasoning and output stages to form a closed-loop control from consistency detection and evaluation to feedback adjustment. It supports online monitoring and timely correction of personality deviations, so that personality settings and output behaviors are dynamically aligned.

[0026] Multi-level correction combined with reinforcement learning: This approach combines online personality correction prompts with offline reinforcement learning optimization. In the short term, it achieves rapid correction through implicit personality correction prompts. In the long term, it influences the model through a personality reward assessment model and reinforcement learning algorithms, gradually guiding the model towards the predetermined personality.

[0027] Personality memory and knowledge integration: Construct a personality knowledge base and dialogue memory mechanism to store the personality characteristic information accumulated during the dialogue process in a structured manner, and inject it back as a generation constraint in the generation process by means of hidden prompts or decoding constraints to ensure the continuity and consistency of personality settings in multiple rounds of dialogue.

[0028] User feedback-driven personalized optimization: It supports updating the target personality setting information based on user feedback and reflects it in subsequent fine-tuning training, so that the personality expression of the large character model can be adjusted in a controllable and gradual manner, taking into account both stability and plasticity.

[0029] Quantitative assessment and automatic optimization: Measurable scoring feedback is generated through quantitative indicators such as personality matching degree and contextual consistency. The personality reward assessment model then outputs reward signals to drive model fine-tuning, transforming personality consistency control from subjective experience into a measurable and optimizable technical process.

[0030] Furthermore, the technical effects of the present invention can be verified by reproducible quantitative indicators: Under the same basic model, the same dialogue script and the same context length constraints, compared with the baseline scheme (1) which only provides a static personality setting prompt once at the beginning of the dialogue and does not take any corrective or optimization measures thereafter, the baseline scheme (2) which adds a complete static personality setting prompt when generating response text in each round, and the baseline scheme (3) which performs a centralized personality data fine-tuning training on the large character model and does not perform online learning updates, the present invention significantly reduces the frequency of personality drift and inconsistencies by online detection and hierarchical control of personality consistency scores; at the same time, by injecting implicit personality correction prompts only in necessary rounds, it avoids the continuous occupation of the context window by repeated long prompts in each round, thereby maintaining the naturalness of the dialogue and stable personality consistency without significantly increasing input overhead. See the relevant control experiments in the specific implementation. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a schematic diagram of the architecture of the reinforcement learning role large model personality consistency alignment optimization system provided in this embodiment of the invention; Figure 2 This is a flowchart illustrating the reinforcement learning role large model personality consistency alignment optimization method provided in this embodiment of the invention; Figure 3This is a schematic diagram illustrating the personality consistency score obtained through personality consistency assessment in an embodiment of the present invention; Figure 4 This is a flowchart illustrating the process of fine-tuning the large character model in an embodiment of the present invention; Figure 5 This is a schematic diagram of the reinforcement learning role large model personality consistency alignment optimization device provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0033] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0034] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0035] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0036] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [the described condition or event] is detected," or "in response to detection of [the described condition or event]."

[0037] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0038] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0039] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0040] Please refer to Figure 1 This invention provides a reinforcement learning role large model personality consistency alignment optimization system. This system can serve as a supervisory control layer on a dialogue system. Its architecture includes the following main functional modules, which work together to form a closed-loop feedback mechanism: The Personality Knowledge Base Module (101) stores and provides target personality setting information, as well as dynamically accumulated personality characteristic information related to the personality traits of the role model (such as the role model's self-described preferences and experiences). The target personality setting information can be pre-configured, and personality characteristic information can be continuously extracted during the dialogue. The Personality Knowledge Base Module (101) can store data in a structured form, such as a knowledge graph or key-value database, recording each piece of information as a subject (role model), attribute (target personality setting information or personality characteristic information)-value pair, along with the source (which round of dialogue the information comes from). The Personality Knowledge Base Module (101) is equivalent to the role model's self-awareness model, providing a basis for subsequent consistency detection and response text generation. When personality settings are adjusted or added during the dialogue, the Personality Knowledge Base Module (101) will also be updated accordingly to maintain the latest accuracy of the role model's personality profile. It should be noted that the information in the personality knowledge base module (101) can be stored for a long time across the conversation cycle to support the continuous personality consistency of the role model in different conversations and on different days. Even if the conversation starts again the next day, the role model still remembers the personality facts established before, thus maintaining the continuity of the personality image.

[0041] The dialogue recording and memory module (102) is used to save all historical dialogue content in real time (including user input text and response text generated by the role model) and extract personality-related clues (i.e., personality characteristic information) from it and store them in the personality knowledge base module (101). During long dialogues, the dialogue recording and memory module (102) can appropriately summarize and compress the historical dialogue content and extract personality characteristics such as personality traits and preferences shown by the role model in past interactions. For example, it can count the frequency of polite language used by the role model in the most recent rounds of dialogue and record the viewpoints and positions expressed by the role model. When the role model generates new response text, the dialogue recording and memory module (102) provides relevant personality characteristic information for reference to ensure that the role model does not contradict itself when generating response text. For example, if the role model previously claimed to "like classical music", the dialogue recording and memory module (102) will provide this information to the role model as a constraint to avoid conflict with the generated response text (such as incorrectly answering "I never listen to music"). By improving the long-term memory of dialogues, the system prevents the character model from forgetting its previous personality traits, thus enhancing consistency in multiple dialogues. It should be noted that the content recorded by the dialogue recording and memory module (102) is not only used for maintaining consistency within a single conversation, but can also be retained for a certain period after the conversation ends, depending on the system design: for example, the personality characteristic information of a user in multiple conversations with the character model can be persistently stored. When the user interacts with the character model again in the future, the dialogue recording and memory module (102) can retrieve the stored personality characteristic information to maintain personality continuity. This cross-conversation memory retention strategy ensures that the personality expression of the character model remains stable over a long period of time and multiple conversations. If a conflict is detected between newly acquired personality characteristic information and existing records in the personality knowledge base module (101), the dialogue recording and memory module (102) can perform an update according to a predetermined strategy: for example, it can retain personality characteristic information with a more recent timestamp, or send a prompt to the technician to manually confirm the conflict and adjust the character settings, thereby resolving the contradictions between memory entries and maintaining a unified personality profile.

[0042] The consistency detection module (103) is used to automatically detect and evaluate the personality consistency of the response text after each round of the role model generates the response text. The inputs to the consistency detection module (103) include the response text of the current role model, the target personality setting information, and the dialogue context provided by the dialogue record and memory module (102). The consistency detection module (103) can combine rule strategies and machine learning models to determine whether the response text meets the personality requirements: on the one hand, it detects the personality matching degree between the response text and the target personality setting information; on the other hand, it detects the coherence between the response text and the dialogue context. The detection result can be a simple pass / fail sign, or it can include scores for each sub-indicator (such as personality matching score, context coherence score, etc.). The consistency detection module (103) is equivalent to a quality monitor for the output of the role model, used to detect personality deviations in the response text in a timely manner. Once the detection result shows that the response text fails the personality verification, the module will output a signal indicating "inconsistency" and the possible deviation type, and submit it to the subsequent personality alignment and optimization module (104) for processing.

[0043] The personality alignment optimization module (104) adjusts the subsequent behavior of the character model according to the consistency detection results. The personality alignment optimization module (104) contains two main sub-functions: prompt correction and model training, which are used for real-time correction and offline optimization, respectively. The prompt correction function is used to handle minor deviations. Implicit personality correction prompts are inserted before the generation of subsequent dialogue rounds to guide the character model to correct its tone or wording. The model training function is used to handle serious deviations or frequent inconsistencies. When a major conflict is detected between the response text generated by the character model and the personality setting, corresponding dialogue data is collected to construct personality alignment training samples and trigger model fine-tuning. Specifically: For minor personality deviations (such as a slightly stiff reply tone that does not violate the core character setting), the personality alignment optimization module (104) will automatically generate an implicit personality correction prompt and insert it into the model input when generating response text in subsequent dialogue rounds. The implicit personality correction prompt is typically generated based on the target personality setting information in the personality knowledge base module (101) and the detected deviation types. For example, the generated implicit personality correction prompt might be: "(System prompt: Please maintain a humble and polite tone when communicating with the user.)". This allows the large character model to refer to this prompt when generating response text, thereby adjusting the output style and making the generated response text more closely match the preset personality setting. The generation of implicit personality correction prompts can use predefined templates or rules, such as filling in corresponding personality trait reminder statements according to the deviation category; or it can be dynamically written by a small language model based on the target personality setting information and deviation situation. To ensure that the prompt is efficient and does not interfere with normal dialogue, its content is usually compressed, retaining only the key personality correction information, and the word length is limited (e.g., no more than one sentence or a certain number of lexical units). The frequency of prompt insertion is dynamically controlled according to actual needs: if it is only an occasional deviation, it can be chosen not to insert or to insert once and then observe the effect; if the deviation tendency continues to appear, prompts can be inserted continuously or multiple times in subsequent rounds, and even the tone of the prompts can be strengthened, until the detection indicators show that the personality consistency has recovered to above the threshold. Once the personality expression of the large role model returns to stability and the consistency detection module (103) has not alarmed for a long time, the personality alignment optimization module (104) will automatically stop inserting prompts to avoid excessive intervention affecting the naturalness of the dialogue. Through this on-demand implicit prompt correction mechanism, slight personality deviations in the large role model can be quickly reversed, while minimizing the impact visible to the user.

[0044] When severe personality deviations occur or personality inconsistencies are frequently detected within the same conversation, prompts alone are insufficient. In such cases, the personality alignment optimization module (104) initiates a model fine-tuning process for in-depth adjustments. Specifically, the personality alignment optimization module (104) packages the dialogue data showing deviations in the current conversation (including input text, response text generated by the large character model, and the expected correct personality response text) into personality alignment training samples. In cases of severe deviations, the system can also trigger an alarm mechanism to alert technical personnel to potential personality control issues. Without interrupting the current conversation, the collected personality alignment training samples are sent to the background training queue. The model training unit (108) can fine-tune the large-scale role model in two ways: one is small-scale supervised fine-tuning, where the expected correct personality response text is generated manually or by rules as a reference answer, and a training pair is formed with the actual response text generated by the large-scale role model that has a deviation, and supervised parameter updates are performed on the large-scale role model; the other preferred method is reinforcement learning training, where the personality reward evaluation model (107) is used to score the response text of the large-scale role model in these situations, and the obtained reward is used as a signal to update the model parameters through the policy gradient algorithm. Under the reinforcement learning scheme, the model training unit (108) will periodically (e.g., whenever a certain number of samples are accumulated or during dialogue idle periods) trigger model fine-tuning: the large-scale role model regenerates the response text in the dialogue situation corresponding to the personality alignment training samples, calculates its personality reward, and adjusts the model policy according to the reward level. During the training process, various stability constraints are also set, such as monitoring the distribution difference between the new policy and the reference policy to avoid drastic changes. After several rounds of iterative training, the personality consistency ability of the large-scale role model will be significantly enhanced. The adjusted model parameters will be loaded and take effect in subsequent dialogues, ensuring that the character model does not repeat previous errors in similar situations. By combining prompting correction with model fine-tuning, a comprehensive personality alignment strategy, from immediate correction to continuous learning, is achieved, continuously improving the stability of the character model's performance in the personality dimension.

[0045] Evaluation Feedback Module (105): This module collects and synthesizes data on the effectiveness of personality consistency alignment and receives external feedback to further improve the system. On one hand, the Evaluation Feedback Module (105) can calculate the overall personality consistency index of each conversation and generate a report after each conversation, such as: "The average personality consistency score of this conversation is 9 / 10, with 2 minor deviations (both of which have been automatically corrected)". This summary information can be viewed by system maintainers to understand the stability trend of the personality performance of the role model. On the other hand, the Evaluation Feedback Module (105) receives subjective feedback from users or administrators, such as user comments that "the role model's tone was a bit stiff" or operations personnel pointing out that a certain response does not meet the brand personality requirements. The Evaluation Feedback Module (105) will store this feedback information in the Personality Knowledge Base Module (101) as a supplement and correction to the personality profile of the role model; at the same time, or as needed, the feedback will be passed to the Personality Alignment Optimization Module (104) to participate in decision-making, so that the system can refer to human feedback when updating personality settings or model strategies in the future, and realize a human-centered personality optimization closed loop. For example, if multiple users request a more humorous tone from the main character model, the system can appropriately adjust the personality settings to increase the weight of "humor," and accordingly add reward samples containing humorous elements during subsequent reinforcement learning training. By incorporating human feedback, the personality of the main character model evolves in the direction desired by the users.

[0046] Language Model Interaction Module (106): The core unit of the system, namely the large-scale character model ontology, is used to generate natural language response text based on the input text and dialogue context. This module can be composed of a large-scale pre-trained language model (such as a model with billions of parameters based on the Transformer architecture) and undergo preliminary supervised fine-tuning before deployment to give it a basic preset personality speaking style. In actual dialogue, the Language Model Interaction Module (106) receives the input text and the context information provided by the system (including implicit personality correction prompts, retrieved personality feature information, etc.), and generates the next response text by combining these conditions. Under the architecture of this invention, the Language Model Interaction Module (106) is jointly supervised by the consistency detection module (103) and the personality alignment optimization module (104): the former monitors whether the response deviates from the personality requirements from the output end, and the latter exerts a corrective influence on it from the input end and the model parameter end through prompts or training. Once the personality alignment optimization module (104) triggers model fine-tuning (e.g., offline reinforcement learning optimization is completed), the language model interaction module (106) will load the updated parameters and generate responses using the adjusted strategy in subsequent dialogues. This architecture ensures that the large role model can generate coherent and useful response text based on the user's input text, while always being constrained by the personality control layer and will not deviate from the preset personality norms.

[0047] The personality reward assessment model (107) is a dedicated assessment network trained offline, used to score the personality fit of the response text of the large character model, thereby providing a reward signal during the reinforcement learning phase. The input to the personality reward assessment model includes the response text generated by the large character model, target personality setting information, and necessary dialogue context, and outputs a personality fit reward value that characterizes the degree of consistency. The higher the value, the more the response text matches the expected personality and context requirements. For example, the personality reward assessment model (107) can be constructed using a deep neural network (such as the Transformer architecture). It should be noted that the personality reward assessment model (107) in this invention is not only used for feedback during the training phase, but can also be integrated with dialogue history information to assist the consistency detection module (103) in judging the global consistency of long dialogues (e.g., by identifying potential drift trends through the overall scoring of multi-turn dialogues).

[0048] The modules described above have clear division of labor and cooperate with each other to form the personality consistency alignment optimization system of the present invention. In practical applications, they can achieve data interaction through software call relationships or hardware communication connections: the consistency detection module (103) listens to the output of the language model interaction module (106) and generates a detection report; the personality alignment optimization module (104) decides whether to call the prompt correction sub-function or the model training sub-function based on the report content; when the model is trained, the personality reward evaluation model (107) participates in the calculation of the quantified reward signal for the optimization algorithm; the dialogue record and memory module (102) provides contextual support for the input side of the language model interaction module (106) and provides historical basis for detection and reward evaluation. The whole system is like adding a layer of personality monitoring and adjustment control loop to the dialogue agent: real-time monitoring of the personality performance of the large role model, immediate correction if deviation occurs, and subsequent fine-tuning and updating of the model to improve it. This control layer ensures that the large role model always communicates with the user with the expected personality traits, significantly improving the consistency and stability of the personality performance of the large role model in long dialogue scenarios.

[0049] Please refer to Figure 2 This is a flowchart illustrating a method for optimizing the personality consistency alignment of a large reinforcement learning role model, as disclosed in an embodiment of the present invention. Figure 2 As shown, the process of this reinforcement learning role large model personality consistency alignment optimization method can be as follows: In S101, target personality setting information is obtained, and personality settings are made for the large character model based on the target personality setting information.

[0050] A role model is a large language model that plays a specific role in a conversation (such as a customer service representative, teacher, or psychological consultant).

[0051] Target personality setting information refers to the set of information used to define the personality attributes and behavioral style that the character model should exhibit in dialogue, including but not limited to personality traits, tone of voice, background setting, value preferences, etc. For example, the character model can be set as "humble, polite, and patient customer service representative". This description information constitutes its target personality setting information, which is used to anchor the style boundaries and behavioral logic output by the character model.

[0052] For example, a large-scale pre-trained language model is first selected as the base model (e.g., a mainstream pre-trained language model architecture with billions of parameters, such as an attention-based architecture). This large-scale pre-trained language model is trained on a massive general corpus and possesses rich language generation capabilities. Subsequently, dialogue data related to the target personality type is collected or constructed, and the large-scale pre-trained language model is further trained accordingly. The dialogue data can include a large number of question-and-answer dialogues reflecting the target personality type; for example, dialogue data for a customer service personality could come from customer service chat logs, dialogue data for an educator personality could come from teacher-student dialogue texts, dialogue data for a psychological support personality could come from psychological counseling dialogue cases, and so on. In these dialogue data, the responses of the large-scale pre-trained language model all conform to the expected target personality type. By minimizing the loss function between the model response and the expected response, the model parameters of the large-scale pre-trained language model are iteratively adjusted, enabling it to gradually learn to adopt expressions consistent with the target personality type in similar contexts. For personality types with insufficient dialogue data, data augmentation or transfer learning techniques can be used to generate high-quality dialogue data. For example, by leveraging the generative capabilities of large-scale pre-trained language models, inputting target personality information allows the models to generate simulated dialogues that align with that personality, thus expanding the dialogue corpus. If necessary, a small number of manually written personality descriptions can be introduced as training signals to guide the large-scale pre-trained language models towards a specific personality, ultimately resulting in a large-scale character model with that specific personality.

[0053] To facilitate the deployment of multiple personalities, the training of the large character model can also employ lightweight fine-tuning techniques such as low-rank adaptation or prompt tuning. These techniques preserve the capabilities of the basic model while requiring only minor parameter updates to adapt to different personality characters. In practice, a small set of adaptation parameters is trained separately for each target personality setting, and saved as a personality parameter plugin. During deployment, the corresponding plugin is loaded as needed to quickly complete the personality setting, achieving flexible switching between "one model, multiple personalities".

[0054] In addition, prompt engineering can be used to inject target personality information into input prompts in the form of structured instructions without modifying the model parameters of large-scale pre-trained language models, thus guiding the large-scale pre-trained language models to exhibit the preset personality. For example, the input prompt could be "You are a professional customer service representative with a friendly yet formal tone who can empathize with the user's emotions."

[0055] Furthermore, it should be noted that a rigorous personality consistency assessment is necessary before deploying the role model in practical applications. For example, a series of pre-set dialogue scenario scripts can be prepared, and multiple rounds of dialogue can be conducted with the role model to evaluate its personality consistency. For instance, the role model can be allowed to assume the target role and engage in hundreds of dialogues with the script user, with manual checks to ensure that its behavior consistently aligns with the target role's established personality. If the role model is found to deviate from its intended style on certain issues, targeted samples can be added to fine-tune the role model. This "factory calibration" process minimizes the risk of personality drift during actual operation, ensuring that the role model possesses basic personality consistency from the outset.

[0056] The following section uses the server as the execution entity to elaborate on the reinforcement learning role large model personality consistency alignment optimization method provided by this invention.

[0057] In S102, in each dialogue round, input text is received, and response text corresponding to the input text is generated through the role big model.

[0058] After a real user initiates a conversation through a terminal device, the server receives the input text sent by the user through the terminal device in real time during each round of the conversation. This input text can be a natural language question, a short instruction, and so on.

[0059] The server then sends the input text to the character model, which generates corresponding response text. In addition to the input text, the server can also input the target personality setting information into the character model to further ensure that the character model strictly adheres to the preset personality when generating response text.

[0060] In S103, based on the dialogue context of the input text and the target personality setting information, the personality consistency of the response text is evaluated to obtain a personality consistency score.

[0061] Please refer to Figure 3After the large character model generates the response text, the server then uses an encoder to encode the response text, the dialogue context corresponding to the input text, and the target personality setting information into vector representations. The three vector representations are then concatenated into a joint vector representation. A preset scoring function maps this vector representation into a personality consistency score within the range of [0,1]. The higher the score, the higher the personality consistency of the response text.

[0062] For example, let , , These represent the encoders used for target personality setting information, dialogue context, and response text, respectively (the three can be the same encoder or different encoders).

[0063] The vector representation of target personality setting information is as follows: ; The vector representation of the dialogue context is as follows: ; The vector representation of the response text is as follows: ; in, This indicates information about the target personality type. Indicates the context of the dialogue. This represents the response text.

[0064] The three vectors are concatenated to obtain a joint vector representation: ; in This indicates vector concatenation.

[0065] In addition, attention fusion, gating fusion, or weighted fusion can be used to fuse the three vector representations to obtain a joint vector representation.

[0066] Subsequently, the joint vector representation is mapped to a scalar personality consistency score using a scoring function: ; in, The scoring function, in practice, can be implemented using a regression structure (such as a feedforward neural network) and can be expressed as: ; in, This represents the activation function. This represents a trainable weight vector.

[0067] Optionally, in one embodiment, based on the dialogue context of the input text and the target personality setting information, the personality consistency of the response text is evaluated to obtain a personality consistency score, including: Calculate the semantic similarity between the response text and the target personality setting information to obtain a personality matching score; Calculate the semantic coherence between the response text and the dialogue context to obtain a context coherence score; The personality fit score and the contextual coherence score are combined to obtain the personality consistency score.

[0068] In this embodiment of the invention, the scoring function is broken down into two main dimensions for quantitative evaluation: personality compatibility score. Contextual coherence score .

[0069] Personality compatibility score Used to measure response text Information related to target personality settings The scoring function for the semantic matching degree, specifically the personality matching degree dimension, can be expressed as: ; in: This is the output of a semantic embedding function or encoder; Represents the dot product of vectors; Represents the vector norm; (The interval can be kept consistent through linear transformation or truncation), and the larger the value, the higher the personality matching degree of the response text.

[0070] In other embodiments, It can also be obtained through keyword coverage, style classification probability, or rule matching score, all of which should meet the requirements that the scores are comparable and normalizable.

[0071] Contextual coherence score Used to measure response text The degree of consistency with the dialogue context in terms of semantic coherence and continuity of personality characterization. Let the dialogue history set be: ; For each historical statement Calculate the local consistency score: ; The minimum value is taken as the context coherence score. This aims to achieve the effect of "strict punishment for even a single instance of obvious contradiction": ; in It can be constructed based on semantic similarity or logical consistency. To enhance implementability, the following two optional implementation methods are provided (either one or a combination thereof): Implementation based on contradiction probability: A logical consistency discriminator is used to output the "contradiction probability". And mapped to a consistency score: ; but ; This form is equivalent to "minimum consistency" and is convenient for implementing engineering threshold control.

[0072] Implementation based on linguistic coherence: Calculate the coherence index of responses within the context of the dialogue (such as perplexity or its monotonicity), and then use a mapping function... Normalize it to Score: ; in It is a monotonically decreasing mapping, so that the lower the perplexity, the higher the consistency.

[0073] In any of the above implementations, it should be guaranteed that And the score can be compared with Combine them.

[0074] As shown above, after obtaining the personality matching score and the contextual coherence score, the server merges the scores from the two dimensions to obtain the personality consistency score. There are no specific restrictions on the fusion method, including but not limited to taking the minimum value, the maximum value, the arithmetic mean, the weighted sum, etc.

[0075] For example, taking the weighted sum directly as a personality consistency score, it can be expressed as: ; in: This represents a weighting coefficient used to adjust the contribution ratio of the personality fit score and the contextual coherence score. The value of can be flexibly controlled to allow the character model to focus more on the consistency of personality style or the coherence of dialogue content. Taking a smaller value is suitable for dialogue scenarios where personality image is strongly constrained, while when... Taking a larger value is suitable for dialogue scenarios that emphasize coherence across multiple turns and avoid contradictions. It should be noted that... The value can be determined by those skilled in the art based on parameter tuning of the verification set, or it can be dynamically adjusted during operation according to a strategy (e.g., temporarily increasing when frequent contextual contradictions are detected). (The value of ).

[0076] In other embodiments, to avoid dimension variation The changes make it difficult to standardize the scoring thresholds, so the weighted sum can be further normalized: ; Thus, the value is located at Personality consistency score within the interval.

[0077] In S104, if the personality consistency score does not reach the scoring threshold and the difference between the two does not reach the difference threshold, then implicit personality correction prompts are generated based on the response text for injecting the character big model in subsequent dialogue rounds; if the personality consistency score does not reach the scoring threshold and the difference between the two reaches the difference threshold, then personality alignment training samples are constructed based on the relevant dialogue data in the current dialogue round and added to the training queue.

[0078] It should be noted that the embodiments of the present invention also include a scoring threshold. This scoring threshold is used to constrain whether the response text generated by the large character model meets the standard. When the personality consistency score S reaches the scoring threshold, the response text is considered to meet the standard; when the personality consistency score does not reach the scoring threshold, the response text is considered to fail. The scoring threshold can be set according to the actual application requirements. For example, for scenarios with strict requirements, a higher threshold can be selected to pursue higher consistency.

[0079] Furthermore, a difference threshold is also provided in this embodiment of the invention. The difference threshold is used to measure the significance of the deviation between the personality consistency score and the scoring threshold. This difference threshold can be flexibly adjusted according to the actual application scenario. For example, a smaller difference threshold can be set in a highly sensitive dialogue scenario to trigger more timely fine-tuning training. The difference threshold and the scoring threshold work together to form a dual criterion for personality consistency.

[0080] Specifically, when the personality consistency score does not reach the scoring threshold and the difference between the two is ( The difference threshold was not reached. When the error message indicates that the deviation is minor, the server initiates a lightweight correction mechanism. Based on the response text, it generates implicit personality correction prompts for injecting into the character model in subsequent dialogue rounds, thereby achieving online correction of the character model, preventing small deviations from accumulating and expanding, and ensuring the continuity of the dialogue experience.

[0081] In practice, the implicit personality correction prompts concisely and clearly indicate in which aspects of the character model need adjustment, and reiterate the relevant requirements of the preset personality. The server can generate implicit personality correction prompts using one or a combination of the following rules: Template-based generation rules: The server pre-establishes a mapping table of "deviation type → prompt template". Accordingly, in addition to assessing the personality consistency of the response text, the server also identifies the specific deviation types in the response text from the personality settings (such as "insufficient politeness", "overly harsh tone", "attitude does not conform to the settings", etc.), and extracts corresponding personality keywords from the target personality setting information, filling them into the corresponding template to generate implicit personality correction prompts. For example, if the deviation type is "insufficient politeness", and the target personality setting information includes "polite treatment"; then the implicit personality correction prompt generated by the server could be: "System prompt: Please maintain a polite and friendly tone." If the deviation type is "overly harsh tone", and the target personality setting information includes "gentle and kind", then the prompt would be: "System prompt: Please respond with a gentle and kind tone." Key Point Extraction and Generation Rules: The system extracts key personality traits, such as trait words, behavioral guidelines, and prohibited expressions, from the target personality profile. These are then combined with actual deviations in the response text to dynamically generate targeted implicit personality correction prompts. For example, when it detects that a user's address is omitted, honorifics are missing, or the sentence structure is command-like, the server automatically extracts key points such as "respectful titles," "use more 'please' and 'thank you'," and "avoid command-like expressions," combining them into an implicit personality correction prompt: "System prompt: Please remember to use appropriate honorifics and include 'please' in your requests to show respect."

[0082] To prevent implicit personality correction prompts from becoming too long and occupying the context window, the server truncates or compresses their length after generation, ensuring that their length does not exceed a preset number of characters or a preset number of tokens. For example, if the generated implicit personality correction prompt exceeds the preset number of characters, the server removes redundant modifiers, retaining only the key information directly related to the current deviation type.

[0083] It should be noted that, in order to improve the naturalness of the correction and reduce the risk of over-intervention, the insertion frequency of implicit personality correction prompts can be dynamically controlled according to the degree of deviation: for very slight deviations (personality consistency score does not reach the scoring threshold, the difference between the two does not reach the second difference threshold, and the second difference threshold is less than the difference threshold), insertion can be temporarily suspended to avoid frequent intervention leading to stiff or repetitive responses; for slight deviations (personality consistency score does not reach the scoring threshold, the difference between the two does not reach the difference threshold, but reaches the second difference threshold), implicit personality correction prompts can be repeatedly inserted in multiple rounds of dialogue until the personality consistency score of the response text generated by the character model reaches the scoring threshold; at the same time, the server will record the correction history for subsequent optimization of personality settings and prompt strategies.

[0084] Understandably, once the implicit personality correction prompts are inserted, they exist in the model input as system messages. When generating response text, the role model incorporates these prompts along with the input text and the dialogue context into its reasoning. Because the implicit personality correction prompts explicitly define personality requirements, the generation distribution of the role model will shift towards expressions that align with that personality. For example, it will reduce the probability of impolite expressions and increase the probability of polite or target-style language, allowing the role model to complete the correction while maintaining natural expression.

[0085] In addition, when the personality consistency score does not reach the scoring threshold and the difference between the two reaches the difference threshold ( When a critical deviation is detected, it indicates a severe deviation. At this point, the server initiates an offline tuning mechanism. Based on relevant dialogue data from the current dialogue round, it constructs personality alignment training samples. These samples are then used to fine-tune the large-scale character model when preset trigger conditions are met, systematically correcting personality deviations in the large-scale character model. This mechanism balances real-time response with long-term optimization, ensuring dialogue fluency while continuously improving the consistency of personality expression in the large-scale character model. When constructing personality alignment training samples, the server includes not only relevant dialogue data containing the input and response texts but also the expected correct personality response text corresponding to the input text. The input text, response text, and correct personality response text are then combined to form a personality alignment training sample and added to the training queue. The method for obtaining the correct personality response text is not specifically limited; it can be obtained through manual annotation, generation by a rule engine, or by calling the output of a high-confidence expert model.

[0086] In S105, after publishing the response text, the next dialogue round begins. When the preset triggering conditions are met, fine-tuning training of the character model is performed, including: based on the personality alignment training samples in the training queue, calling the pre-trained personality reward evaluation model to obtain the personality fit reward value, and using the personality fit reward value as the reward signal for reinforcement learning to drive the reinforcement learning algorithm to update the strategy of the character model, thereby completing the fine-tuning training of the character model.

[0087] In this embodiment of the invention, the server outputs the response text generated by the large role model to the recipient of the current dialogue round, i.e., the terminal device used by the user to join the dialogue, thus realizing the publication of the response text. The server can either directly output the response text to the terminal device, or it can perform security filtering on the response text (e.g., deleting obviously inappropriate content from the response text) before transmitting it to the terminal device.

[0088] In other embodiments, the server may also replace the response text with a preset apology text or rejection text (such as "I cannot fulfill your request at the moment" or "I need more time to think") before sending it to the terminal device, ensuring both user experience and content security.

[0089] In addition, the server monitors the dialogue status in real time and fine-tunes the large character model based on personality-aligned training samples in the training queue when preset trigger conditions are met. The configuration of the preset trigger conditions is not specifically limited here; for example, it could be configured as the current dialogue ending, or the response text showing significant deviations in N consecutive dialogue rounds, etc.

[0090] The personality reward assessment model is a pre-trained specialized model used to quantify the personality fit between the response text generated by the large role model and the target personality setting information. Its inputs include the target personality setting information, the dialogue context, and the response text. The output is a scalar score representing the degree of matching between the response text and the target personality setting information, i.e., the personality fit reward value.

[0091] In practical implementation, a deep neural network (such as the Transformer architecture) can be used to build the personality reward assessment model. The scoring functions of the two dimensions of personality matching degree and contextual coherence in the above embodiment, as well as the weighted sum calculation process, can be learned through a training process containing rich example dialogues. The personality alignment training samples can come from manual annotation (marking which responses are consistent and which are inconsistent), or the pre-trained language model can be used to automatically judge and score a large number of generated samples to obtain pseudo-labels.

[0092] To improve correction efficiency, the server can reuse the personality reward assessment model to generate personality consistency assessment and personality fit reward values. Specifically, the server can use a weighted sum of the initial personality fit and contextual coherence scores output by the personality reward assessment model. ) is used as a reward value for personality compatibility, while the normalized enhancement ( It is used as a personality consistency score.

[0093] After obtaining the personality fit reward value, the server uses this reward value as a reward signal for reinforcement learning, driving the reinforcement learning algorithm to update the policy of the large character model. The choice of reinforcement learning algorithm is not limited here, and may include, but is not limited to, PPO (Proximal Policy Optimization), SAC (Soft Actor-Critic), or A2C (Advantageous Actor-Critic), etc.

[0094] For details, please refer to the following: Figure 4 The reinforcement learning process can be formalized as a Markov decision process: State: Dialogue context (such as recent input text, response text, etc.) Action: The response text generated by the large character model in the given state; Reward: The reward value for personality compatibility output by the personality reward assessment model; Strategy: The current parameter configuration of the character's large model determines the probability distribution of its action selection under a given state; Let the strategy be The model strategy is The optimization objective is to maximize the expected cumulative reward. To implement policy updates, an advantage function is introduced. (e.g., obtained from baseline estimation of the value function), and the parameters are iteratively updated using the policy gradient method.

[0095] The formula for unbiased estimation of policy gradient can be expressed as: .

[0096] in: This indicates the direction of the gradient of the loss function with respect to the parameters; Let be the logarithmic probability of the policy on the action; This is the advantage function, which reflects the degree of improvement of the action relative to the baseline (positive advantage indicates better than the baseline, and negative advantage indicates worse than the baseline). expect It can be approximated by the trajectory data obtained from sampling.

[0097] Proximity-based policy optimization (PPO) is used to limit the magnitude of a single update. The old policy parameters are defined as follows: The probability ratio between the old and new strategies is: .

[0098] PPO uses a clipping loss function to constrain the range of r(θ), and its loss function can be expressed as: ; in: To truncate the threshold hyperparameter, the policy update magnitude is limited, with a typical value range of 0.1 to 0.2; Trim r(θ) to interval; This means taking the smaller value between the undone and pruned items, so that when the policy update is too large, the pruned target is used to suppress unstable updates.

[0099] By minimizing That is to make Convergence can improve the reward for personality consistency while maintaining the stability of strategy updates.

[0100] After making the above fine-tuning to the character model, the parameters related to personality expression in the character model have been optimized, which can better serve the next user dialogue.

[0101] In other embodiments, supervised fine-tuning can also be used, in which the parameters of the large character model are updated based on the input text as a condition and the correct personality response text as a label.

[0102] In other embodiments, if the personality consistency score reaches the scoring threshold, no correction is needed, and the response text can be published directly to proceed to the next dialogue round.

[0103] Optionally, in one embodiment, to prevent the new strategy from deviating excessively from the reference strategy, the reinforcement learning role large model personality consistency alignment optimization method provided by the present invention further includes: During the policy update process, the distribution difference between the updated policy and the reference policy is introduced into the loss function as a penalty term.

[0104] The distribution differences can be quantified using KL divergence or JS divergence, and the reference strategy can be fixed as the model parameters after initial fine-tuning.

[0105] For example, the current policy Reference Strategy The difference in distributions between them is measured by the KL divergence, and a penalty term is added to the loss function: ; in: KL divergence is used to measure the difference in distributions between two strategies; The penalty coefficient controls the strength of the constraint. A larger penalty coefficient emphasizes keeping the policy close to the reference policy, thus improving output stability; a smaller penalty coefficient makes the policy more likely to move quickly toward higher rewards, but may introduce the risk of language style drift.

[0106] In one embodiment, the server can dynamically determine the value of the penalty coefficient. For example, when the KL divergence exceeds the divergence threshold, the penalty coefficient can be increased, and when the KL divergence is lower than the divergence threshold, the penalty coefficient can be decreased, so that the policy difference is kept within a controllable range.

[0107] For example, a moderate penalty coefficient, such as 0.1, is initially assigned. Then, the KL divergence between the large character model and the reference policy is monitored after each policy update. If the KL divergence continues to rise and exceeds the divergence threshold (indicating excessive changes in output distribution), the penalty coefficient is increased, such as by multiplying it by 1.5. Conversely, if the KL divergence is below the divergence threshold, the penalty coefficient is appropriately reduced, such as by dividing it by 1.5. Through this feedback control, the observed KL divergence is maintained within a reasonable range (e.g., 0.5–1.5), thereby balancing the magnitude of personalized adjustments to the large character model's output.

[0108] Optionally, in one embodiment, the reinforcement learning role large model personality consistency alignment optimization method provided by the present invention further includes: The system retrieves first personality trait information that matches keywords in the input text. This personality trait information is extracted based on historical dialogue content. The primary personality traits are injected into the dialogue generation process of the character model through hidden hints or decoding constraints.

[0109] It should be noted that, in this embodiment of the invention, the server also maintains a personality knowledge base. This knowledge base stores target personality setting information for the character model, as well as dynamically accumulated personality characteristic information (such as experiences, preferences, viewpoints, etc. mentioned by the character model in historical dialogue content). This personality characteristic information is continuously extracted by the server from historical dialogue content. The personality characteristic information can be stored in a structured form, such as organized in the form of a knowledge graph or key-value pairs, to ensure the retrieval and scalability of the personality characteristic information.

[0110] In this embodiment of the invention, after receiving input text, the server first retrieves personality feature information from the personality knowledge base that matches keywords (such as who you are, your hobbies, what you like, your experiences, etc.) in the input text. If the retrieval is successful, the personality feature information that matches the keywords in the input text is recorded as the first personality feature information. Subsequently, this first personality feature information is injected into the dialogue generation process of the character model corresponding to the input text in the form of hidden hints or decoding constraints. For example, if the character model claims to "like classical music" in historical dialogue content, this information will be stored as personality feature information in the personality knowledge base. When the user's input text contains "What music do you usually listen to?", the server will retrieve this personality feature information and inject it into the dialogue generation process in the form of hidden hints or decoding constraints, ensuring that the model naturally continues the personality setting of "liking classical music" in the response.

[0111] Hidden hints refer to embedding personality trait information into the model input in a way that is not visible to the user (such as hidden paragraphs, system prompts, etc.), so that the character model explicitly refers to this personality trait information when generating response text. Decoding constraints, on the other hand, force the character model to prioritize words consistent with personality trait information in key generation steps by adjusting the probability distribution of lexical units. For example, if the first personality trait information "hobby: music" is retrieved, when "your hobby" is mentioned in the input text, the server injects the feature "hobby: music" into the decoding constraint, so that the character model increases the probability of words related to "music" during the decoding process, ensuring that the final response mentions "music", thus conforming to the previous persona.

[0112] Optionally, in one embodiment, retrieving first personality feature information that matches keywords in the input text further includes: When the retrieval of the first personality feature information fails, the second personality feature information, which has a vector similarity to the input text that reaches a similarity threshold, is injected into the dialogue generation process of the large character model in the form of hidden hints or decoding constraints.

[0113] In this embodiment of the invention, the server also encodes personality feature information from the personality knowledge base into vector representations and constructs a personality feature vector index to support efficient semantic retrieval. The encoding method for personality feature information is not specifically limited here; for example, the server can use pre-trained language models such as BERT or Sentence-BERT to embed and encode personality feature text, generating a high-dimensional, dense vector representation.

[0114] Correspondingly, when the input text does not match any explicit keywords, the server encodes the input text into the same vector space as the personality feature information, obtaining a vector representation of the input text. It then retrieves personality feature vector indices from the personality feature vector index that have a cosine similarity reaching a similarity threshold, thereby obtaining the corresponding personality feature information as the second personality feature information. When multiple personality feature vector indices are retrieved, the server can select the top-k indices from these as the second personality feature information. The value of k can be dynamically adjusted according to actual needs; for example, k=3 can balance retrieval accuracy and computational efficiency.

[0115] It should be noted that when there are semantically conflicting personality trait information in the Top-k search results, the server can introduce a conflict resolution mechanism to prioritize retaining personality trait information that is frequently consistent with the user's recent interactions and has the most recent timestamp, etc.

[0116] Optionally, in one embodiment, before generating implicit personality correction prompts for injection into subsequent dialogue rounds based on the response text, the method further includes... If the personality consistency score does not reach the scoring threshold, and the difference between the two does not reach the difference threshold, then obtain the cumulative number of dialogue rounds in the same dialogue where the personality consistency score does not reach the scoring threshold. If the number of dialogue rounds does not reach the threshold, implicit personality correction prompts are generated based on the response text to be injected into subsequent dialogue rounds. If the number of dialogue rounds reaches the round threshold, the process will proceed to fine-tune the training of the large character model.

[0117] In this embodiment of the invention, the server also accumulates statistics on the number of dialogue rounds in which the personality consistency score does not reach the scoring threshold. For example, a counter can be used to record in real time the number of dialogue rounds in the current dialogue in which the personality consistency score does not reach the scoring threshold.

[0118] It should be noted that the embodiments of the present invention include a round threshold, which is used to constrain the upper limit of tolerance for personality consistency deviation. It can be dynamically set according to the complexity of the actual dialogue scenario and the stability expected by the user. For example, it can be set to 2 in the financial customer service scenario and relaxed to 5 in the creative writing assistance scenario.

[0119] In the current dialogue round, if the personality consistency score does not reach the scoring threshold and the difference between the two does not reach the difference threshold, the server does not immediately generate implicit personality correction prompts for subsequent dialogue rounds. Instead, it first obtains the number of dialogue rounds in the current conversation where the accumulated personality consistency score has not reached the scoring threshold. Subsequently, the server compares this number of dialogue rounds with a preset round threshold. If the number of dialogue rounds does not reach the round threshold, it is determined that the personality deviation is still within a controllable range and only requires dynamic calibration through lightweight correction prompts. If the number of dialogue rounds reaches the round threshold, it indicates that the personality deviation has accumulated to the point where intervention is required. At this time, the model fine-tuning process is triggered, and the large role model is fine-tuned and trained based on the personality alignment training samples in the training queue. For details, please refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0120] In other embodiments, if the server collects user feedback on personality during or after the dialogue (e.g., the user wants adjustments to a certain aspect of the character model's personality), it will update the personality knowledge base or adjust the weight parameters related to personality expression in the character model during the fine-tuning phase, integrating this feedback into the subsequent evolution of the character model. This forms a cyclical mechanism of online monitoring, real-time correction, offline reinforcement, and updated deployment, continuously improving the consistency and stability of the character model's personality expression. Regardless of how many rounds of dialogue the user engages with the character model, the character model can consistently interact with the user with the expected personality traits; even if it needs to evolve gradually based on user preferences, the changes are controlled and gradual, without suddenly producing inconsistent behavior that is unacceptable to the user.

[0121] To verify the effectiveness of the invention, a simulated dialogue experiment and effect evaluation were conducted.

[0122] Experimental Scenario: Several dialogue scripts with different topic shifts and emotional changes were selected, and the role models engaged in multiple rounds of dialogue under different alignment strategies. The personality presets of the role models were "patient, friendly, and polite in tone." The dialogue lengths ranged from dozens to hundreds of rounds, covering various scenarios such as daily consultations and opinion discussions.

[0123] Comparison scheme: The method of the present invention is compared with three baseline schemes: (1) No aligned baseline - Only a static personality setting prompt is provided at the beginning of the conversation, and no corrective or adjustment measures are taken thereafter; (2) Continuous baseline reminder - When generating response text in each round, a complete static personality setting reminder is attached to continuously strengthen the personality reference of the character model; (3) One-time fine-tuning of the baseline—a concentrated training of personality data is performed on the large character model, and then no online learning updates are performed in the dialogue.

[0124] The above scheme (2) simulates the practice of repeated prompts in each round in the existing technology, while scheme (3) simulates the practice of shaping personality by offline fine-tuning alone.

[0125] Evaluation metrics: The personality consistency assessment strategy provided in the above embodiments of this invention is used to evaluate the personality consistency score of the response text generated by the role model in each dialogue round, and the number of personality deviations occurring during the dialogue is recorded. Simultaneously, the differences in content quality and diversity of the role model's response text under different schemes are considered to ensure fair comparison.

[0126] Experimental results show that the role model using scheme (1) gradually exhibits personality drift in long dialogues: around the 20th round of dialogue, the role model begins to occasionally deviate from polite tone (such as using harsh or impatient language); by the 50th round, there are multiple instances of inconsistent personality, requiring manual intervention to correct. Quantitative statistics show that in 100 rounds of dialogue, the average personality consistency score of the role model using scheme (1) is only about 0.7, with more than 15 instances of obvious personality inconsistencies in the later stages of the dialogue. In contrast, the role model using the alignment optimization scheme of this invention achieves an average personality consistency score of 0.90 (out of 1.0) per round even after hundreds of rounds of dialogue, with more than 95% of the response text fully conforming to the preset personality, and only a very small number (5 rounds) of slight deviations being detected by the system and corrected in the next round through prompts. In another long-term test involving 1,000 rounds of dialogue, the average personality consistency score of the role model using the alignment optimization scheme of the present invention remained at 0.9, with only a small number of dialogue rounds requiring correction and all of them being automatically corrected; while the role model using scheme (1) not only had an average score of less than 8 / 10, but also had a much higher number of deviations than the scheme of the present invention.

[0127] For scheme (2), the experiment found that repeating complete personality prompts in each round mitigated personality drift in the big role model to some extent: the big role model maintained good politeness consistency in the early stages of the dialogue, and the average personality consistency score was improved compared to the no scheme (1) (from about 0.7 to about 0.8). However, due to the addition of long prompts in each round occupying a lot of context, the length of the big role model's response was squeezed, and the output of the big role model was slightly mechanical and rigid due to the repetition of prompts. Although continuous prompts reduced some deviations (about 10 obvious deviations in 100 rounds of dialogue), it still could not completely avoid the occurrence of personality inconsistencies, and users rated the dialogue naturalness of the big role model under this path as slightly decreased.

[0128] For scheme (3), the character model, after being fine-tuned with offline personality data, exhibits a style that is relatively consistent with the character's persona in the initial stage of the dialogue. However, as the number of dialogue rounds increases, the disadvantage of this scheme lacking an online correction mechanism gradually becomes apparent: once a new situation not covered in the fine-tuned data is encountered, the character model will still experience personality deviations, and because there is no continuous learning, the deviations that appear early cannot be self-corrected and can only accumulate. In the actual test, the consistency of the character model in the first 10 rounds of dialogue was acceptable (the score was close to 0.85), but after 50 rounds, multiple deviations appeared (such as the tone gradually becoming stiff), and the average personality consistency score dropped to about 0.78, with the number of deviations falling between that of scheme (1) and the scheme of this invention.

[0129] Statistical analysis shows that the proposed solution has a significant advantage over the baseline solutions mentioned above in terms of personality consistency index. Based on data from 100 rounds of dialogue trials, compared to solution (1), the proposed solution increased the mean personality consistency score by approximately 28% and reduced the frequency of deviations by nearly 70%; compared to solutions (2) and (3), the mean personality consistency score also increased by more than 10%. A t-test was used for significance analysis, and the results were statistically significant at a 95% confidence level, proving that the improvement of the proposed solution was not a random fluctuation but statistically significant. Furthermore, it was observed that introducing the proposed solution did not reduce the rationality and diversity of the generated response text from the role-based large model. There were no statistically significant differences in indicators such as dialogue task completion rate in the comparative experiment, indicating that the proposed solution successfully controlled the output style within the expected range while maintaining the usefulness of the response text.

[0130] Furthermore, long-term stability across conversations was tested. Specifically, the large-scale character model engaged in one conversation with the user each day for several consecutive days, and the changes in the personality consistency score curve of each daily conversation were tracked. For the large-scale character model not using the present invention, a static personality setting prompt was re-entered at the start of each new conversation, but similar drifting patterns recurred, and the personality consistency score fluctuated significantly across conversations. For example, the large-scale character model not using the present invention saw its consistency score drop below 0.6 in the latter part of the first day's conversation, and then fell below 0.65 again on the second day after a similar number of rounds, requiring manual supervision and adjustment for each conversation, indicating poor stability in personality performance. In contrast, the large-scale character model using the present invention, due to continuous learning and optimization after each conversation, tended to start the next conversation at a higher consistency level and remained stable. Based on the results of 7 days of multi-round conversations, it can be concluded that the average personality consistency score of the large-scale character model under the present invention remained consistently around 0.9, with minimal fluctuations between days (score standard deviation of approximately 0.02), and no significant degradation trend was observed. This means that the character model continuously adjusts its strategy with each interaction, becoming more robust over time and avoiding the phenomenon of "restarting the dialogue and the personality becoming distorted again." This is especially important for applications that provide continuous companionship: even if users interact with the character model the next day, they can still feel a consistent personality performance.

[0131] The above fully demonstrates the effectiveness of this invention in improving personality consistency: whether within a single long conversation or in long-term use across conversations, this solution is significantly superior to traditional static prompts or one-time fine-tuning solutions. It can maintain the stability and consistency of the personality of the large role model with a high degree of automation, while keeping the response content natural and interesting.

[0132] This invention's method can be widely applied to various dialogue scenarios requiring large character models to stably assume specific roles over a long period, and is particularly significant in fields that emphasize interactive trust relationships and image consistency. Typical application scenarios include, but are not limited to: Virtual Customer Service and Intelligent Assistants: Enterprise customer service robots or personal assistants need to consistently communicate with users in a uniform style and tone. For example, brand customer service is required to maintain a professional and polite attitude at all times. This invention ensures that the persona model maintains consistency in its personality performance throughout long-term, multi-batch customer interactions, avoiding sudden shifts in tone from rigid to playful, thereby improving the consistency of service image and user trust in the brand.

[0133] Medical Consultation and Psychological Counseling Role Models: In medical consultation or psychological counseling scenarios, the personality stability of the role model is particularly important. Such role models need to consistently demonstrate patience, empathy, and professionalism. The personality alignment mechanism of this invention prevents the model from shifting attitudes or responding inappropriately due to contextual changes, ensuring that the role model provides counseling in a gentle and caring manner throughout the conversation, giving the user a consistent sense of reassurance.

[0134] Long-term companionship digital partner: For long-term companionship scenarios such as chatting and caregiving, users often prefer to interact with a stable and familiar "partner." If the character model's personality is inconsistent, users will feel it's unrealistic. This invention allows the personality of the companionship character model to remain consistent over the long term and to evolve slowly based on user feedback. This ensures that personality changes align with user expectations and are gradual and natural, preventing users from suddenly feeling that the "personality has changed." This makes the digital partner more realistic and credible.

[0135] Digital Avatars and Virtual Spokespersons: When a digital persona serves as a digital avatar of a real person (such as a virtual immortalized image of a deceased relative or a virtual spokesperson for a celebrity), maintaining consistency between the persona's behavior and the real person's personality is crucial. This invention can continuously correct the output of the digital persona, ensuring its style and tone remain consistent with the real person's past behavior. For example, a digitally immortalized grandfather persona can be continuously adjusted based on family feedback, gradually aligning its language style with the user's memory of the target person's characteristics. This makes the digital avatar more convincingly reproduce the real person's personality. In such applications requiring high levels of personality credibility, this invention provides indispensable technical assurance.

[0136] In addition to the typical scenarios mentioned above, this invention is also applicable to various large-scale dialogue character models that require stable character development, such as educational assistants, game NPCs, and dialogue content creation. By adjusting the personality profile and reward evaluation criteria according to specific fields, the method of this invention can be seamlessly adapted to different applications. For example, in educational scenarios, the personality dimension of patience and rigor can be emphasized, while in game NPCs, a certain range of personality variations can be allowed while maintaining the core settings. The alignment framework provided by this invention is universal and configurable, and can meet the personality consistency requirements of large-scale character models in diverse scenarios.

[0137] In summary, compared with the prior art, the present invention has significant advantages in several aspects: Consistent and consistent personality performance: Traditional solutions often only shape the model's personality through preset prompts at the beginning of a conversation, failing to guarantee stability after prolonged interaction. This invention, through detection and feedback in each round, ensures that the large-scale character model maintains consistency between its words and actions and the preset personality regardless of the duration of the conversation, significantly enhancing user immersion and trust.

[0138] Automated alignment optimization: Previously, manually adjusting model parameters or adding rules was required to correct personality biases, which was costly and incomplete. This invention utilizes the reward-punishment mechanism of reinforcement learning, allowing the large-scale character model to self-correct erroneous behaviors during interactions, gradually approaching the preset personality. This automates the fine-grained control of the character model's personality expression, reducing the workload of manual intervention.

[0139] Solving the multi-turn drift problem: Personality drift in long dialogue scenarios is a recognized challenge in the industry, and effective solutions have been lacking in the past. This invention significantly improves the role-playing stability of the large role model throughout the entire dialogue by adding a personality consistency reward constraint. The introduction of a personality knowledge base also ensures that the large role model does not forget what it has said, fundamentally eliminating inconsistencies.

[0140] Quantitative assessment enhances reliability: Traditional personality shaping and assessment often rely on intuition. This solution explicitly defines personality consistency as a measurable indicator and forms a closed-loop scoring feedback system. This makes the quality of the character model's personality expression verifiable, facilitating continuous improvement. Simultaneously, quantitative assessment acts as a safety net, enabling timely detection and correction of abnormal personality deviations, increasing the system's reliability and controllability.

[0141] Flexible Adaptability to User Needs: This invention supports continuous adjustment of the character model's personality settings based on actual user interactions. If certain preset personality elements are found to be unsuitable, the server can iteratively optimize the personality settings themselves, making the character model's personality expression more in line with user expectations. Compared to fixed and rigid personality settings, this invention gives the character model the ability to "evolve," making it more natural and believable when accompanying users over the long term.

[0142] Through the above improvements, this invention endows the large-scale character model with a stable, human-like personality. Whether in applications such as virtual customer service, medical consultation, or long-term companionship, the large-scale character model can consistently and effectively fulfill its specific role or personality, something that existing technologies cannot fully achieve.

[0143] To facilitate better implementation of the above-described reinforcement learning role large model personality consistency alignment optimization method, this invention also provides a corresponding reinforcement learning role large model personality consistency alignment optimization device. The meanings of the terms used are the same as in the above-described reinforcement learning role large model personality consistency alignment optimization method; for specific implementation details, please refer to the descriptions in the above method embodiments.

[0144] Please refer to Figure 5 The reinforcement learning role model personality consistency alignment optimization device also includes a personality setting module 210, a language model interaction module 220, a consistency detection module 230, a personality alignment optimization module 240, and a model training module 250. Detailed descriptions of each functional module are as follows: The personality setting module 210 is used to obtain target personality setting information and set the personality of the large character model based on the target personality setting information; The language model interaction module 220 is used to receive input text in each dialogue round and generate response text corresponding to the input text through the role big model; The consistency detection module 230 is used to evaluate the personality consistency of the response text based on the dialogue context of the input text and the target personality setting information, and obtain a personality consistency score. The personality alignment optimization module 240 is used to generate implicit personality correction prompts for injecting the character model into subsequent dialogue rounds based on the response text when the personality consistency score does not reach the scoring threshold and the difference between the two does not reach the difference threshold; and to construct personality alignment training samples based on relevant dialogue data in the current dialogue round and add them to the training queue when the personality consistency score does not reach the scoring threshold and the difference between the two reaches the difference threshold. The language model interaction module 220 is also used to enter the next dialogue round after publishing the response text; The model training module 250 is used to perform fine-tuning training on the large character model when preset trigger conditions are met. This includes: based on personality alignment training samples in the training queue, calling a pre-trained personality reward evaluation model to obtain the personality fit reward value of the response text, and using the personality fit reward value as a reward signal for reinforcement learning to drive the reinforcement learning algorithm to update the strategy of the large character model, thereby completing the fine-tuning training of the large character model.

[0145] Optionally, in one embodiment, the personality alignment optimization module 240 is further configured to introduce the Kullback-Leibler divergence (KL divergence) between the updated policy and the reference policy as a penalty term into the loss function during the policy update process, so as to limit the policy update magnitude; wherein, the reinforcement learning algorithm is a reinforcement learning algorithm suitable for model policy update, and the loss function includes the truncation target term and penalty term of the corresponding reinforcement learning algorithm.

[0146] Optionally, in one embodiment, the consistency detection module 230 is used to calculate the semantic similarity between the response text and the target personality setting information to obtain a personality matching score; calculate the semantic coherence between the response text and the dialogue context to obtain a context coherence score; and fuse the personality matching score and the context coherence score to obtain a personality consistency score.

[0147] Optionally, in one embodiment, the reinforcement learning role large model personality consistency alignment optimization device provided by the present invention further includes a dialogue record and memory module, used to retrieve first personality feature information that matches the input text with keywords, the personality feature information being extracted based on historical dialogue content; the first personality feature information is injected into the dialogue generation process of the role large model in the form of hidden hints or decoding constraints.

[0148] Optionally, in one embodiment, the dialogue recording and memory module is further configured to inject second personality feature information, whose vector similarity to the input text reaches a similarity threshold, into the dialogue generation process of the large character model in the form of hidden hints or decoding constraints when the retrieval of first personality feature information fails.

[0149] Optionally, in one embodiment, the personality alignment optimization module 240 is used to obtain the number of dialogue rounds in the same dialogue where the cumulative personality consistency score has not reached the scoring threshold when the personality consistency score has not reached the scoring threshold and the difference between the two has not reached the difference threshold; when the number of dialogue rounds has not reached the round threshold, it generates implicit personality correction prompt information for injection into subsequent dialogue rounds based on the response text; the model training module is also used to perform fine-tuning training on the large character model when the number of dialogue rounds reaches the round threshold.

[0150] Specific limitations regarding the reinforcement learning role large model personality consistency alignment optimization device can be found in the limitations of the reinforcement learning role large model personality consistency alignment optimization method mentioned above, and will not be repeated here. Each module in the aforementioned reinforcement learning role large model personality consistency alignment optimization device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0151] In one embodiment, a computer device is provided, the internal structure of which can be shown as follows: Figure 6 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface connects to external wireless clients, providing wireless network access services to the connected clients. When the computer device is configured as a server, the computer program is executed by the processor to implement the reinforcement learning role large model personality consistency alignment optimization method provided by this invention, for example: Obtain target personality setting information, and set the personality of the large character model based on the target personality setting information; In each dialogue round, input text is received, and response text corresponding to the input text is generated through the role model; Based on the dialogue context of the input text and the target personality setting information, the personality consistency of the response text is evaluated to obtain a personality consistency score; If the personality consistency score does not reach the scoring threshold, and the difference between the two does not reach the difference threshold, then implicit personality correction prompts are generated based on the response text for subsequent dialogue rounds to inject into the character big model; if the personality consistency score does not reach the scoring threshold, and the difference between the two reaches the difference threshold, then personality alignment training samples are constructed based on relevant dialogue data in the current dialogue round and added to the training queue. After publishing the response text, the next dialogue round begins. When the preset trigger conditions are met, fine-tuning training of the character model is performed, including: based on the personality alignment training samples in the training queue, calling the pre-trained personality reward evaluation model to obtain the personality fit reward value, and using the personality fit reward value as the reward signal for reinforcement learning to drive the reinforcement learning algorithm to update the strategy of the character model, thereby completing the fine-tuning training of the character model.

[0152] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the reinforcement learning role large model personality consistency alignment optimization method described in the above embodiment.

[0153] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0154] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0155] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for optimizing the personality consistency alignment of a large reinforcement learning role model, characterized in that, include: Obtain target personality setting information, and set the personality of the large character model based on the target personality setting information; In each dialogue round, input text is received, and response text corresponding to the input text is generated through the character model. Based on the dialogue context of the input text and the target personality setting information, the personality consistency of the response text is evaluated to obtain a personality consistency score; If the personality consistency score does not reach the scoring threshold and the difference between the two does not reach the difference threshold, then implicit personality correction prompts are generated based on the response text for subsequent dialogue rounds to inject into the character model; if the personality consistency score does not reach the scoring threshold and the difference between the two reaches the difference threshold, then personality alignment training samples are constructed based on relevant dialogue data in the current dialogue round and added to the training queue. After publishing the response text, the next dialogue round begins. When the preset triggering conditions are met, fine-tuning training of the character model is performed, including: based on the personality alignment training samples in the training queue, calling the pre-trained personality reward evaluation model to obtain the personality fit reward value, and using the personality fit reward value as a reward signal for reinforcement learning to drive the reinforcement learning algorithm to update the strategy of the character model, thereby completing the fine-tuning training of the character model.

2. The reinforcement learning role-based large model personality consistency alignment optimization method according to claim 1, characterized in that, Also includes: During the policy update process, the Kullback-Leibler divergence (KL divergence) between the updated policy and the reference policy is introduced as a penalty term into the loss function to limit the policy update magnitude; wherein, the reinforcement learning algorithm is a reinforcement learning algorithm suitable for model policy update, and the loss function includes a truncation target term corresponding to the reinforcement learning algorithm and the penalty term.

3. The reinforcement learning role-based large model personality consistency alignment optimization method according to claim 1, characterized in that, The process of evaluating the personality consistency of the response text based on the dialogue context of the input text and the target personality setting information to obtain a personality consistency score includes: Calculate the semantic similarity between the response text and the target personality setting information to obtain a personality matching score; Calculate the semantic coherence between the response text and the dialogue context to obtain a context coherence score; The personality fit score and the contextual coherence score are combined to obtain the personality consistency score.

4. The reinforcement learning role large model personality consistency alignment optimization method according to claim 1, characterized in that, Also includes: Retrieve first personality feature information that matches keywords in the input text; the personality feature information is extracted based on historical dialogue content. The first personality trait information is injected into the dialogue generation process of the character model in the form of hidden hints or decoding constraints.

5. The reinforcement learning role-based large model personality consistency alignment optimization method according to claim 4, characterized in that, The retrieval of first personality feature information that matches keywords in the input text also includes: When the retrieval of the first personality feature information fails, the second personality feature information, which has a vector similarity to the input text that reaches a similarity threshold, is injected into the dialogue generation process of the character model in the form of hidden hints or decoding constraints.

6. The reinforcement learning role large model personality consistency alignment optimization method according to claim 1, characterized in that, Before generating implicit personality correction prompts for subsequent dialogue rounds based on the response text, the process also includes... If the personality consistency score does not reach the scoring threshold and the difference between the two does not reach the difference threshold, then obtain the number of dialogue rounds in the same dialogue where the personality consistency score does not reach the scoring threshold. If the number of dialogue rounds does not reach the round threshold, implicit personality correction prompts are generated based on the response text to be injected into subsequent dialogue rounds. If the number of dialogue rounds reaches the round threshold, then proceed to the step of fine-tuning the large model of the character.

7. A device for optimizing the personality consistency alignment of a large reinforcement learning role model, characterized in that, include: The personality setting module is used to acquire target personality setting information and set the personality of the large character model based on the target personality setting information; The language model interaction module is used to receive input text in each dialogue round and generate response text corresponding to the input text through the role big model; The consistency detection module is used to evaluate the personality consistency of the response text based on the dialogue context of the input text and the target personality setting information, and obtain a personality consistency score. The personality alignment optimization module is used to generate implicit personality correction prompts for injecting into the character model in subsequent dialogue rounds based on the response text when the personality consistency score does not reach the scoring threshold and the difference between the two does not reach the difference threshold; and to construct personality alignment training samples based on relevant dialogue data in the current dialogue round and add them to the training queue when the personality consistency score does not reach the scoring threshold and the difference between the two reaches the difference threshold. The language model interaction module is also used to enter the next dialogue round after publishing the response text; The model training module is used to perform fine-tuning training on the large character model when a preset trigger condition is met. This includes: based on the personality alignment training samples in the training queue, calling a pre-trained personality reward evaluation model to obtain a personality fit reward value, and using the personality fit reward value as a reward signal for reinforcement learning to drive the reinforcement learning algorithm to update the strategy of the large character model, thereby completing the fine-tuning training of the large character model.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the reinforcement learning role large model personality consistency alignment optimization method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the reinforcement learning role large model personality consistency alignment optimization method according to any one of claims 1 to 6.

10. A reinforcement learning role-based large-scale model personality consistency alignment optimization system, characterized in that, include: The personality knowledge base module is used to store target personality setting information and personality characteristic information that is dynamically updated during the dialogue process; The dialogue recording and memory module is used to save historical dialogue content and extract personality characteristic information related to personality traits from it and write it into the personality knowledge base module for use in the dialogue process. The consistency detection module is used to evaluate the personality consistency of the response text and output a consistency score after the large role model generates the response text, based on the target personality setting information in the personality knowledge base module and the dialogue context provided by the dialogue record and memory module. The personality alignment and optimization module is used to receive the consistency score from the consistency detection module and perform corresponding adjustment operations when it is determined that there is a personality deviation based on the consistency score: when the deviation is slight, it generates implicit personality correction prompt information for injection into subsequent dialogue rounds. When the degree of deviation is severe, relevant dialogue data is collected to construct personality alignment training samples and added to the training queue; A personality reward assessment model is used to perform personality fit scoring based on response text in personality alignment training samples in the training queue and output personality fit reward value. The personality reward assessment model scores the personality fit based on the personality matching degree between the response text and the target personality setting information and the coherence with the dialogue context during the personality fit scoring process. The model training unit is used to use the personality fit reward value output by the personality reward evaluation model as a reward signal to drive the reinforcement learning algorithm to update the strategy of the character model when the preset triggering conditions are met, thereby completing the fine-tuning training of the character model. as well as The language model interaction module, including the large character model ontology, is used to receive input text and dialogue context provided by the dialogue record and memory module and generate response text. After the model training unit completes the policy update, the updated policy is loaded for subsequent response text generation.

Images