Role playing model training method, role playing method, device and equipment

Abstract

The application provides a role-playing model training method, a role-playing method, a device and equipment, wherein the method comprises: determining the score corresponding to each sample pair in a sample pair set, training an initial reward model based on a first ranking result of all scores and a second ranking result of labels of all sample pairs to obtain a reward reference model, and performing language style reinforcement training on an initial role-playing model based on model parameters of the reward reference model and sample dialogue data to obtain a role-playing model. In the method, the reward reference model can accurately distinguish between candidate answers related to a role and candidate answers not related to the role through score ranking training. Then, based on the model parameters of the reward reference model and the sample dialogue data, the role-playing model is trained, so that the initial role-playing model has the same discrimination ability as the reward reference model, thereby realizing the high-fidelity, high-consistency and reusable role-playing ability of the role-playing model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a training method, role-playing method, apparatus, and equipment for a role-playing model. Background Technology

[0002] With the rapid development of Large Language Models (LLMs), their applications in content generation, knowledge-based question answering, and human-computer dialogue are becoming increasingly widespread. Among these, virtual role-playing (or "digital humans" or "virtual companions") based on Large Language Models has become an important commercialization direction, demonstrating enormous potential in scenarios such as gaming, entertainment, marketing, and psychological support.

[0003] A successful role-playing model not only needs to mimic the character's common language style, but also needs to accurately and consistently present the character's background knowledge, personality traits, and values. Especially for well-known IP (Intellectual Property) characters with a large world view and a deep fan base (such as characters in anime, movies, and novels), users have extremely high requirements for the "fidelity" of their words and actions.

[0004] However, existing large language models face significant bottlenecks when performing deep IP role-playing. Although the model may have learned a great deal of knowledge about a specific IP during the pre-training phase, this knowledge is often in a "lazy" state when generating dialogue, making it difficult to activate effectively. The model often "forgets" the character settings, resulting in inconsistencies, factual errors, or "out-of-character" (OOC) errors, severely impacting the user experience. This phenomenon is known as the "knowledge-behavior separation" problem in large language models. Summary of the Invention

[0005] This invention provides a training method, role-playing method, apparatus, and device for a role-playing model, which addresses the deficiency of "knowledge-behavior separation" in existing large language models for IP role-playing. This enables the large language model to not only "know" the knowledge of the role, but also to "become" the role, seamlessly and naturally integrating deep knowledge into its dialogue behavior, thereby achieving high-fidelity, high-consistency, and reusable role-playing capabilities.

[0006] This invention provides a method for training a role-playing model, comprising the following steps: Obtain an initial role-playing model and a set of sample pairs; the set of sample pairs includes positive sample pairs, negative sample pairs, and positive-negative sample pairs, wherein the two positive samples in the positive sample pair are created from candidate answers related to the role, and the two negative samples in the negative sample pair are created from candidate answers unrelated to the role; each positive-negative sample pair consists of one positive sample and one negative sample. Determine the score corresponding to each sample pair in the sample pair set, and train the initial reward model based on the first ranking result of all the scores and the second ranking result of the labels of all the sample pairs to obtain the reward reference model; Based on the model parameters of the reward reference model and the sample dialogue data, the initial role-playing model is subjected to language style enhancement training to obtain the role-playing model.

[0007] According to a training method for a role-playing model provided by the present invention, the initial role-playing model is subjected to language style enhancement training based on the model parameters of the reward reference model and sample dialogue data to obtain the role-playing model, comprising: The model parameters of the reward module in the initial role-playing model are initialized using the model parameters of the reward reference model to obtain the role-playing model to be trained; Based on the sample dialogue data, the role-playing model to be trained is subjected to language style enhancement training to obtain the role-playing model.

[0008] According to a training method for a role-playing model provided by the present invention, the step of performing language style enhancement training on the role-playing model to be trained based on the sample dialogue data to obtain the role-playing model includes: The sample dialogue data is input into the role-playing model to be trained, and the recommended answer style text corresponding to the sample dialogue data is output by the role-playing model to be trained. Based on the reward value corresponding to the recommended answer style text, the role-playing model to be trained is subjected to language style enhancement training to obtain the role-playing model.

[0009] According to a training method for a role-playing model provided by the present invention, the initial reward model is trained based on a first ranking result of all the scores and a second ranking result of the labels of all the sample pairs to obtain a reward reference model, comprising: The ranking loss is determined based on the first ranking result of all the scores and the second ranking result of the labels of all the sample pairs; Based on all the scores and the label scores of all the sample pairs, determine the score loss; The target loss is determined based on the ranking loss and the score loss, and the initial reward model is adjusted based on the target loss to obtain the reward reference model.

[0010] According to a training method for a role-playing model provided by the present invention, the sample pairs are extracted from the role knowledge archive based on the language style paradigm; The steps for constructing the character knowledge profile include: Execute the following loop until the preset condition is met; Perform requirement parsing on the proxy task sent by the client to obtain the requirement parsing results; Based on the results of the requirement analysis, the proxy task is divided into multiple sub-tasks; Information from the information sources corresponding to each sub-task is integrated, and the role knowledge archive is determined based on the integrated information from the information sources corresponding to each sub-task. The preset conditions include the character knowledge profile reaching preset depth and preset breadth standards.

[0011] According to a method for training a role-playing model provided by the present invention, the training steps of the initial role-playing model include: The target role and user profile are input into a large language model to obtain multi-turn dialogue training data output by the large language model; the multi-turn dialogue training data is dialogue content that conforms to the role setting of the target role and the user profile. Based on the multi-turn dialogue training data, the original role-playing model is fine-tuned under supervision to obtain the initial role-playing model.

[0012] According to a training method for a role-playing model provided by the present invention, the step of inputting the target role and user profile into a large language model to obtain multi-turn dialogue training data output by the large language model includes: The target role and the user profile are input into the large language model to obtain candidate multi-turn dialogue training data output by the large language model; The candidate multi-turn dialogue training data is subjected to dialogue quality detection to obtain the multi-turn dialogue training data. The dialogue quality detection includes testing at least one of the following: logical coherence, language style, accuracy of character portrayal, richness of content, and interest.

[0013] The present invention also provides a role-playing method, comprising the following steps: Obtain the role to be played and dialogue data; The role to be played and the dialogue data are input into the role-playing model to obtain the recommended response text of the role to be played, which is output by the role-playing model. The role-playing model is obtained by performing the training method described above.

[0014] The present invention also provides a training device for a role-playing model, comprising the following units: The first acquisition unit acquires an initial role-playing model and a set of sample pairs; the set of sample pairs includes positive sample pairs, negative sample pairs, and positive-negative sample pairs, wherein the two positive samples in the positive sample pair are created from candidate answers related to the role, and the two negative samples in the negative sample pair are created from candidate answers unrelated to the role; the positive-negative sample pair consists of one positive sample and one negative sample. The training unit is used to determine the score corresponding to each sample pair in the sample pair set, and to train the initial reward model based on the first ranking result of all the scores and the second ranking result of the labels of all the sample pairs to obtain the reward reference model. The language style training unit is used to perform language style enhancement training on the initial role-playing model based on the model parameters of the reward reference model and sample dialogue data, so as to obtain the role-playing model.

[0015] The present invention also provides a role-playing device, comprising the following units: The second acquisition unit is used to acquire the role to be played and the question text; The input unit is used to input the role to be played and the question text into the role-playing model to obtain the role text of the role to be played output by the role-playing model; The role-playing model is obtained by performing the training method described above.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a training method for a role-playing model as described above, or to implement the role-playing method.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method of the role-playing model as described above, or implements the role-playing method.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements a training method for a role-playing model as described above, or implements the role-playing method.

[0019] This invention provides a training method, role-playing method, apparatus, and device for a role-playing model. The method determines the score corresponding to each sample pair in a sample pair set. Based on a first ranking of all scores and a second ranking of all sample pair labels, an initial reward model is trained to obtain a reward reference model. Based on the model parameters of the reward reference model and sample dialogue data, the initial role-playing model undergoes language style enhancement training to obtain a role-playing model. In this method, the reward reference model, through score ranking training, can accurately distinguish between candidate answers relevant to the role and those irrelevant to the role. Then, based on the model parameters of the reward reference model and sample dialogue data, the initial role-playing model undergoes language style enhancement training to obtain a role-playing model. This gives the initial role-playing model the same discriminative ability as the reward reference model. Therefore, during language style enhancement training, the role-playing model can continuously adjust its generation strategy according to the reward signal, making the generated answers more consistent with the character's language style. In this way, the generation strategy of the role-playing model gradually aligns with a high-fidelity speech style, thereby achieving a high-fidelity, high-consistency, and reusable role-playing capability. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is one of the flowcharts illustrating the training method for the role-playing model provided by this invention.

[0022] Figure 2 This is the second flowchart illustrating the training method for the role-playing model provided by this invention.

[0023] Figure 3 This is a flowchart illustrating the role-playing method provided by the present invention.

[0024] Figure 4 This is a schematic diagram of the structure of the training device for the role-playing model provided by the present invention.

[0025] Figure 5 This is a schematic diagram of the role-playing device provided by the present invention.

[0026] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] The terms "first," "second," etc., used in this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and that the objects distinguished by "first," "second," etc., are generally of the same class.

[0029] In related technologies, role-playing mainly adopts the following solutions: Prompt engineering: This method guides the model to role-play by providing detailed character descriptions, backstories, and dialogue rules at the input. While simple and easy to implement, its effectiveness is inconsistent. In long dialogues, the model is prone to deviating from the initial instructions and fails to internalize deep knowledge of the character.

[0030] Limited-sample context learning: Several high-quality role-playing dialogue examples are added to the prompts for the model to reference and imitate. Compared with prompts that rely solely on instructions, this method shows a slight improvement, but it is limited by the length of the model's context window, making it difficult to achieve long-term, stable role memory.

[0031] Supervised fine-tuning based on general corpora: This method fine-tunes the model using general role-playing dialogue data or data collected from the internet. While this helps the model grasp certain role-playing styles, the wide variety and inconsistent quality of training data, coupled with its lack of specificity, makes it difficult to ensure the accuracy of knowledge about specific IP characters.

[0032] Enhanced Retrieval Generation: By connecting to an external database containing knowledge related to intellectual property roles, when a user asks a question, the system first retrieves relevant information from the database and inputs it into the model along with the user's question to help generate a more accurate answer.

[0033] However, the above-mentioned existing technical solutions have the following obvious drawbacks: The limitations of engineering and contextual learning are highlighted: they are "temporary" solutions that cannot fundamentally change the model's inherent knowledge structure and behavioral patterns, and are prone to failure in complex, multi-turn dialogues.

[0034] The knowledge ambiguity of general SFT (Supervised Fine-Tuning): The training data often fails to cover the core knowledge points and unique speech patterns of the IP character, resulting in the model being "similar in form but not in spirit" and unable to answer questions about the character's deeper background.

[0035] The "illusion" and "fragmentation" problems of RAG (Retrieval-Augmented Generation): Theoretically, the knowledge fragments retrieved by the RAG system may be incomplete or ambiguous, and the model may still produce incorrect or inconsistent answers when integrating this information.

[0036] Disjointed dialogue: The model may simply mechanically "recite" retrieved knowledge instead of "internalizing" and expressing it in the character's tone and personality, resulting in stiff dialogue and a lack of immersion. For example, the model's response may sound like "reciting encyclopedia" rather than the character "recalling their own experiences."

[0037] To address the aforementioned problems, this invention provides a method for training a role-playing model. Figure 1 This is one of the flowcharts illustrating the training method for the role-playing model provided by this invention, such as... Figure 1 As shown, the method includes steps 110, 120 and 130.

[0038] Step 110: Obtain the initial role-playing model and sample pair set; the sample pair set includes positive sample pairs, negative sample pairs, and positive-negative sample pairs, wherein the two positive samples in the positive sample pair are created from candidate answers related to the role, and the two negative samples in the negative sample pair are created from candidate answers unrelated to the role; the positive-negative sample pair consists of one positive sample and one negative sample.

[0039] Specifically, first, an initial role-playing model and a set of sample pairs are obtained. The initial role-playing model refers to a role-playing model that has not yet undergone reward module parameter optimization and language style specialization training.

[0040] The parameters of the initial role-playing model can be pre-trained or randomly generated; this embodiment of the invention does not impose specific limitations on this. For example, the initial role-playing model can be a model fine-tuned by SFT supervision from a pre-trained model. The pre-trained model can be a large language model, such as the IFlytek Spark model, XLNet (extreme multi-label learning network), or the Robustly Optimized BERT approach model; this embodiment of the invention does not impose specific limitations on this.

[0041] Here, the sample pair set includes positive sample pairs, negative sample pairs, and positive-negative sample pairs. In a positive sample pair, both positive samples are created from candidate answers relevant to the character, and in a negative sample pair, both negative samples are created from candidate answers unrelated to the character. A positive-negative sample pair consists of one positive sample and one negative sample; that is, the pair is created from a candidate answer relevant to the character and a candidate answer unrelated to the character. For example, a positive sample could be a core line of dialogue, a catchphrase, or a candidate answer that highly matches the character's language style. A negative sample could be a candidate answer with a generic language style, out-of-character (OOC), or clearly inconsistent with the character's personality. Each sample pair in the sample pair set can be extracted from the character's knowledge archive based on a language style paradigm.

[0042] In one embodiment, the character is Sun Wukong. Positive sample pairs could be "Where are you running, monster! Take this from my old Sun staff!" and "Hehe, here comes my old Sun!". Positive and negative sample pairs could be "Where are you running, monster! Take this from my old Sun staff!" and "Haha, life is short, enjoy it while you can, let's drink!". Negative sample pairs could be "Haha, life is short, enjoy it while you can, let's drink!" and "Great! Let's drink until we're drunk!". This embodiment of the invention does not specifically limit these pairings.

[0043] It should be noted that the sample pair set in this embodiment of the invention includes three types: positive sample pairs, negative sample pairs, and positive-negative sample pairs, and is not limited to only these three sets of samples. For example, a positive sample may include 5 candidate answers related to a role, then the number of positive sample pairs is... The number of positive sample pairs is 10. The calculation method for the number of negative sample pairs and positive-negative sample pairs is the same, and will not be repeated here.

[0044] Step 120: Determine the score corresponding to each sample pair in the sample pair set, and train the initial reward model based on the first ranking result of all the scores and the second ranking result of the labels of all the sample pairs to obtain the reward reference model.

[0045] Specifically, after obtaining the set of sample pairs, a score can be determined for each sample pair. The score for each sample pair reflects the quality of the samples within that pair or the degree to which they conform to the role setting. The reward reference model is a reward model used in subsequent language style reinforcement training.

[0046] Here, each sample pair can be assigned a score according to the scoring criteria. For example, a candidate answer that perfectly matches the character's language style in a sample pair is given 5 points; a candidate answer that is somewhat consistent but not the character's original lines is given 4 points; a candidate answer that is generally consistent with the character's style but has minor flaws in its delivery is given 3 points; a candidate answer that contains a large number of discrepancies with the character's profile is given 2 points; and a candidate answer that is completely unresponsive to instructions and unusable is given 1 point.

[0047] After obtaining the score for each sample pair in the sample pair set, all scores can be sorted to obtain a first ranking result. For example, all scores can be sorted from highest to lowest to obtain the first ranking result.

[0048] After obtaining the first ranking of all scores, the initial reward model can be trained based on the first ranking of all scores and the second ranking of the labels of all sample pairs. The trained initial reward model is then used as the reward reference model (RM).

[0049] It is understandable that the greater the difference between the first and second sorting results, the greater the sorting loss; the smaller the difference between the first and second sorting results, the smaller the sorting loss.

[0050] It should be noted that the reward reference model, trained by ranking scores, can accurately distinguish between candidate answers that are relevant to the role and candidate answers that are not, and output corresponding reward values ​​to reflect the degree of conformity between the candidate answers and the role settings.

[0051] Step 130: Based on the model parameters of the reward reference model and the sample dialogue data, perform language style enhancement training on the initial role-playing model to obtain the role-playing model.

[0052] Specifically, after obtaining the reward reference model, the initial role-playing model can be trained to enhance language style based on the model parameters of the reward reference model and sample dialogue data to obtain the role-playing model.

[0053] Here, after training the reward reference model, the model parameters of the reward module in the initial role-playing model can be determined based on the model parameters of the reward reference model. This allows for the effective transfer of the model parameters from the reward reference model to the reward module of the initial role-playing model, providing stable reward signals for subsequent role-playing optimization.

[0054] Furthermore, the initial role-playing model, after its parameters have been determined, can be trained with language style enhancement based on sample dialogue data to obtain a new role-playing model. For example, in language style enhancement training, the responses generated by the initial role-playing model will be evaluated by a reward module. Responses that match the role's style will receive a reward, thereby guiding the initial role-playing model's generation strategy towards a higher fidelity speech style. This improves the role-playing model from "formal resemblance" to "spiritual resemblance," addressing subtle differences in speech style.

[0055] Understandably, the reward module in the initial role-playing model is a reward model used to evaluate the fidelity of the character's speech style; that is, the reward module is a dialogue reward model. The reward module scores the candidate responses in the input sample pairs. Candidate responses that highly match the character's speech style (positive samples) are given full or high scores; candidate responses that deviate significantly from the character's style (negative samples) are given low or negative scores. Therefore, by aligning fine-grained behaviors through the reward module, the gap between "knowing" and "doing" is effectively bridged.

[0056] Here, language style enhancement training can use PPO (Proximal Policy Optimization), TRPO (Trust Region Policy Optimization), DPO (Direct Preference Optimization), etc., and this embodiment of the invention does not specifically limit the specific methods used.

[0057] Here, the sample dialogue data is extracted from multi-turn dialogue training data, which can be obtained by inputting the target role and user profile into a large language model and outputting the data.

[0058] The method provided in this invention determines the score corresponding to each sample pair in the sample pair set, and trains an initial reward model based on a first ranking result of all scores and a second ranking result of all sample pair labels to obtain a reward reference model. Based on the model parameters of the reward reference model and sample dialogue data, the initial role-playing model undergoes language style enhancement training to obtain a role-playing model. In this method, the reward reference model, through score ranking training, can accurately distinguish between candidate answers relevant to the role and those irrelevant to the role. Then, based on the model parameters of the reward reference model and sample dialogue data, the initial role-playing model undergoes language style enhancement training to obtain a role-playing model. This gives the initial role-playing model the same discriminative ability as the reward reference model. Therefore, during language style enhancement training, the role-playing model can continuously adjust its generation strategy according to the reward signal, making the generated answers more consistent with the character's language style. In this way, the generation strategy of the role-playing model gradually aligns with a high-fidelity speech style, thereby achieving a high-fidelity, high-consistency, and reusable role-playing capability.

[0059] Based on the above embodiments, step 130 includes: Step 131: Initialize the model parameters of the reward module in the initial role-playing model using the model parameters of the reward reference model to obtain the role-playing model to be trained; Step 132: Perform language style enhancement training on the role-playing model to be trained based on the sample dialogue data to obtain the role-playing model.

[0060] Specifically, after training and obtaining the reward reference model, the model parameters of the reward module in the initial role-playing model can be initialized using the model parameters of the reward reference model to obtain the role-playing model to be trained.

[0061] Specifically, the model parameters of the trained reward reference model can be directly copied to the reward module of the initial role-playing model, enabling the reward module to have relatively accurate scoring capabilities from the initial stage. If there is some structural overlap between the reward module and the reward reference model, only the parameters of the non-shared parts can be copied to reduce redundancy. In addition, if the reward module structure is small or needs further compression, knowledge distillation can be used to transfer knowledge from the reward reference model to the reward module. After the model parameters are initialized, the parameters of the reward module of the initial role-playing model are frozen, so that it only serves as a provider of reward signals in subsequent reinforcement learning training and does not participate in the parameter updates of the policy model; alternatively, some parameters can be unfrozen and fine-tuned according to actual needs to better adapt to the output distribution of the policy model, which is not specifically limited in this embodiment of the invention.

[0062] Based on the above approach, the model parameters of the reward reference model can be effectively migrated to the reward module of the initial role-playing model, providing stable reward signal support for subsequent role-playing optimization.

[0063] Then, based on the sample dialogue data, the role-playing model to be trained is subjected to language style enhancement training to obtain the role-playing model. For example, in language style enhancement training, the answers generated by the role-playing model to be trained will be evaluated by the reward module. Answers that conform to the role style will be rewarded, thereby guiding the generation strategy of the role-playing model to be trained to align with a higher fidelity speech style, realizing the improvement of the role-playing model from "formal resemblance" to "spiritual resemblance", and solving the subtle differences in speech style.

[0064] Based on the above embodiments, step 132 includes: Step 1321: Input the sample dialogue data into the role-playing model to be trained to obtain the recommended answer style text corresponding to the sample dialogue data output by the role-playing model to be trained; Step 1322: Based on the reward value corresponding to the recommended answer style text, perform language style reinforcement training on the role-playing model to be trained to obtain the role-playing model.

[0065] Specifically, first, the sample dialogue data is input into the role-playing model to be trained, and the recommended response style text corresponding to the sample dialogue data output by the role-playing model is obtained. Here, the recommended response style text is the dialogue text that conforms to the role's style.

[0066] Subsequently, based on the reward value corresponding to the recommended answer style text, the role-playing model to be trained undergoes language style reinforcement training to obtain the role-playing model. For example, the reward value of the recommended answer style text can be calculated based on preset style evaluation indicators (such as language style consistency, accuracy of emotional expression, role feature matching degree, etc.). The reward value can be determined by manual annotation, automatic scoring model, or a combination of both; this embodiment of the invention does not specifically limit this. Next, the reward value is used as a feedback signal using a reinforcement learning algorithm (such as PPO) to update the parameters of the role-playing model to be trained, making it more inclined to output high-quality answer text that conforms to the target style in subsequent generation.

[0067] The method provided in this invention generates recommended response-style text by inputting sample dialogue data into a role-playing model to be trained, and then performs language style reinforcement training based on the reward value corresponding to the recommended response-style text. This helps improve the role-playing model's ability to understand and control language style. By introducing a reward mechanism, the role-playing model can gradually learn to generate responses that better match the character setting and style requirements, enhancing character consistency and naturalness of expression. Simultaneously, the reinforcement training method makes the model more robust and adaptable in style performance, suitable for various interaction scenarios. This method effectively improves the generation quality of the role-playing model and the user experience.

[0068] Based on the above embodiments, step 120, which involves training the initial reward model based on the first ranking result of all the scores and the second ranking result of the labels of all the sample pairs, to obtain a reward reference model, includes: Step 121: Determine the ranking loss based on the first ranking result of all the scores and the second ranking result of the labels of all the sample pairs; Step 122: Based on all the scores and the label scores of all the sample pairs, determine the score loss; Step 123: Determine the target loss based on the ranking loss and the score loss, and adjust the initial reward model based on the target loss to obtain the reward reference model.

[0069] Specifically, the ranking loss can be determined based on the first ranking result of all scores and the second ranking result of the labels of all sample pairs. The ranking loss measures the difference between the first and second ranking results in the initial reward model.

[0070] Here, the formula for the ranking loss is as follows: in, Represents the sorting loss. This indicates the initial reward model for a given cue. and candidate answers The output score, This indicates the selected candidate answer in the sample pair (the positive sample in the second ranking result). This represents the candidate answer that was rejected in the sample pair (the negative sample in the second ranking result). and The sorting information of the labels used to reflect all sample pairs. This represents the sigmoid function. and This represents the score for all sample pairs. The first ranking result is used to reflect the scores of all sample pairs.

[0071] It should be noted that the ranking loss here takes the positive and negative samples in the second ranking result as an example. The ranking loss includes the difference between the first ranking result of the scores corresponding to all sample pairs in the sample pair set and the second ranking result of the labels of all sample pairs.

[0072] Then, based on the scores corresponding to all sample pairs and the label scores of all sample pairs, the score loss is determined. The formula for the score loss is as follows: in, Indicates score loss, This represents the score for each sample pair. This represents the label score for all sample pairs.

[0073] Understandably, the greater the difference between the scores of all sample pairs and the label scores of all sample pairs, the greater the score loss; conversely, the smaller the difference between the scores of all sample pairs and the label scores of all sample pairs, the smaller the score loss.

[0074] Finally, the target loss is determined based on the ranking loss and the score loss, and the initial reward model is adjusted based on the target loss. The adjusted initial reward model is then used as the reward reference model.

[0075] Here, the target loss can be determined based on the sum of the ranking loss and the score loss, or based on the weighted sum of the ranking loss and the score loss. This embodiment of the invention does not specifically limit this.

[0076] Here, based on the weighted sum of ranking loss and scoring loss, the formula for determining the target loss is as follows: in, Indicates target loss. and This represents a hyperparameter used to weight the two losses. Represents the sorting loss. This indicates a score loss.

[0077] The method provided in this invention, on the one hand, determines a ranking loss based on a first ranking result of all scores and a second ranking result of the labels of all sample pairs, enabling the initial reward model to effectively distinguish the quality of candidate answers; on the other hand, a score loss is determined based on the difference between the scores of all sample pairs and the label scores of all sample pairs, guiding the initial reward model to understand the absolute quality of candidate answers. A target loss is determined based on the ranking loss and the score loss, and the initial reward model is adjusted based on the target loss to obtain a reward reference model. This reward reference model can learn both the order of role preferences and grasp the fine-grained differences in answer quality, thereby outputting a smoother and more continuous reward signal. This joint training method enhances the robustness and generalization ability of the reward model, providing a more reliable reward basis for subsequent reinforcement learning training, and helping the role-playing model achieve high-fidelity and high-consistency generation effects.

[0078] Based on the above embodiments, the sample pairs are extracted from the character knowledge archive based on the language style paradigm; The steps for constructing the character knowledge profile include: Execute the following loop until the preset condition is met; Step 210: Perform requirement parsing on the proxy task sent by the client to obtain the requirement parsing result; Step 220: Based on the requirements analysis results, the proxy task is divided into multiple sub-tasks; Step 230: Integrate the information sources corresponding to each sub-task, and determine the role knowledge archive based on the integrated information sources corresponding to each sub-task. The preset conditions include the character knowledge profile reaching preset depth and preset breadth standards.

[0079] Specifically, the sample pairs are extracted from the character knowledge archive based on the language style paradigm. The character knowledge archive refers to a structured set of information describing the background, personality, language style, knowledge system, and other characteristics of a specific character, constructed by parsing the requirements of the agent tasks sent by the client, breaking down the tasks into multiple sub-tasks, and synthesizing the information sources corresponding to each sub-task. The language style paradigm refers to a set of structured rules extracted and constructed from the character knowledge archive based on a specific character, used to standardize the output style of the language generation system. The language style paradigm includes, but is not limited to, linguistic elements such as classic lines, catchphrases, word preferences, tone of voice, sentence structure, rhetorical devices, and expression habits.

[0080] Here, the steps for building a character knowledge profile include: The process continues until preset conditions are met. These preset conditions include the character knowledge profile reaching preset depth and preset breadth standards. The preset depth standard refers to the level of detail in the character knowledge profile's portrayal of a single dimension or attribute; that is, the depth of exploration and description of a specific aspect of the character (such as personality, language style, backstory, etc.). The preset breadth standard refers to the comprehensiveness and breadth of the character attributes or dimensions covered by the character knowledge profile; that is, whether the scope of character portrayal is sufficiently broad.

[0081] The iterative process includes parsing the requirements of the proxy task sent by the client, obtaining the requirements parsing results, then breaking down the proxy task into multiple sub-tasks based on the requirements parsing results, and finally synthesizing the information sources corresponding to each sub-task, and determining the role knowledge profile based on the information sources corresponding to each sub-task after information synthesis.

[0082] Here, the construction of the character knowledge archive primarily relies on Agentic Search methods, such as DeepResearch or other similar frameworks. These methods use an LLM agent to autonomously plan search steps, perform web searches, read and synthesize information, and iteratively deepen the research until a complex knowledge construction task is completed.

[0083] Task Initialization and Planning (Planning Agent): The operator only needs to input a high-level research objective, such as: "Build a comprehensive knowledge profile for the IP character 'Sun Wukong'". Here, "Build a comprehensive knowledge profile for the IP character 'Sun Wukong'" is the result of the requirements analysis.

[0084] A planning agent based on a large language model receives this goal and autonomously breaks it down into a logical execution plan containing multiple subtasks. For example, the planning agent might generate a list of subtasks as follows: Task 1: Search and summarize the basic background of Sun Wukong (birth of the stone monkey, apprenticeship).

[0085] Task 2: Outline his key experiences and skill development path (Havoc in Heaven, being imprisoned under the Five Elements Mountain, and the journey to the West).

[0086] Task 3: Analyze his core personality traits and character arc (from unruly to the Victorious Fighting Buddha).

[0087] Task 4: Explore his social network and interaction patterns (with Tang Sanzang, Zhu Bajie, Buddha, etc.).

[0088] Task 5: Collect his classic lines and common catchphrases ("Here comes Old Sun!"). Execution and multi-source information retrieval: One or more "execution agents" will process the above subtasks in parallel or serially.

[0089] For each subtask, the execution agent autonomously generates precise search engine query terms and calls the web search API (Application Programming Interface). It retrieves information from multiple sources, not just a single encyclopedia website. These sources can include official encyclopedias, academic paper databases, original e-book texts, in-depth discussion posts on social media, etc., to ensure the diversity and richness of information sources.

[0090] Furthermore, information integration and cross-validation are performed: the execution agent reads and understands the original web page content or text fragments obtained from multiple sources.

[0091] A "comprehensive agent" is responsible for extracting key information and cross-validating it. For example, if there are discrepancies between a description on an encyclopedia website and an in-depth analysis post on a fan forum, the agent will flag the difference or make a ruling based on preset trust rules (such as "official sources take precedence"), thereby filtering out erroneous or low-quality information.

[0092] Iterative Deepening and Knowledge Completion: After completing a preliminary round of information integration, the planning agent will assess the completeness of the current knowledge base.

[0093] If a knowledge gap or insufficiently explored area is discovered (e.g., "insufficient details of Sun Wukong's psychological activities during his imprisonment under the Five Elements Mountain"), the agent will automatically generate new, more exploratory subtasks (e.g., "search for analytical articles about Sun Wukong's state of mind during his five hundred years under the Five Elements Mountain") and initiate a new round of search-synthesis loop.

[0094] This iterative process will continue until the knowledge archive built reaches the preset depth and breadth standards.

[0095] Ultimately, all verified and synthesized knowledge points are integrated into a structured, multi-dimensional character profile (character biography). The mined information is then integrated into a structured JSON, XML, or database file. This character knowledge profile includes at least background information, character arc, social networks, language style paradigms, and core knowledge base, but this embodiment of the invention does not impose specific limitations on these aspects.

[0096] Here, background information may include the character's birth, family, upbringing, key experiences, etc., and this embodiment of the invention does not impose specific limitations on this.

[0097] The character arc can include the evolution of core goals, values, and personality, etc., and the embodiments of the present invention do not specifically limit this.

[0098] Social networks can include relationships, stances, and typical interaction patterns with other key figures, while language style paradigms can include classic lines, catchphrases, word preferences, tone of voice, etc. The core knowledge base can include the character's professional skills and unique knowledge within the worldview; however, this embodiment of the invention does not specifically limit these aspects.

[0099] Understandably, a character knowledge profile is a comprehensive, accurate, and multi-dimensional profile of character knowledge.

[0100] The method provided in this invention extracts sample pairs from a role knowledge archive based on a language style paradigm. The steps for constructing the role knowledge archive include: executing a loop process until preset conditions are met; the loop process includes: parsing the requirements of the agent task sent by the client to obtain the requirements parsing result; based on the requirements parsing result, splitting the agent task into multiple sub-tasks; synthesizing the information sources corresponding to each sub-task, and determining the role knowledge archive based on the information sources corresponding to each sub-task after information synthesis; the preset conditions are that the role knowledge archive reaches preset depth and preset breadth standards. This method uses autonomous agents for deep network research, fundamentally solving the bottleneck of role knowledge construction and ensuring the quality of input data for subsequent reward model training.

[0101] Based on the above embodiments, the training steps of the initial role-playing model include: Step 31: Input the target role and user profile into the large language model to obtain the multi-turn dialogue training data output by the large language model; the multi-turn dialogue training data is the dialogue content that conforms to the role setting of the target role and the user profile. Step 32: Based on the multi-turn dialogue training data, the original role-playing model is subjected to supervised fine-tuning to obtain the initial role-playing model.

[0102] Specifically, to obtain a better initial role-playing model, the initial role-playing model can be trained based on the following steps: First, obtain the original role-playing model. Here, the original role-playing model can be a pre-trained model, which can be a large language model. The large language model can be a pre-trained language model such as the iFlytek Spark model, XLNet model, or ROBERTa model. This embodiment of the invention does not specifically limit this.

[0103] After obtaining the original role-playing model, the target role and user profile can be input into the large language model to obtain multi-turn dialogue training data output by the large language model. The multi-turn dialogue training data is dialogue content that conforms to the role setting of the target role and the user profile.

[0104] Here, the target persona and user profile are input into the large language model. This input can be in the form of instructions, such as: "Please play the role of [User Profile X] and engage in a deep dialogue with [Role A] for no less than 30 rounds. During the dialogue, [Role A] must strictly adhere to all the settings in its role profile, while [User Profile X] needs to continuously ask questions or initiate interactions based on its personality traits." Upon receiving the instructions, the large language model can synthesize the dialogue in a "simulator" mode to obtain multi-round dialogue training data.

[0105] The target role is a role selected from the role knowledge profile. To simulate diverse interaction methods in the real world and avoid data homogeneity, this embodiment of the invention introduces user profiles based on psychological models (such as the Big Five personality traits: openness, conscientiousness, extraversion, agreeableness, and neuroticism). Ten different user types are preset, such as "curious explorer," "provocative questioner," "friendly listener," and "help seeker," and a dialogue strategy and questioning style are defined for each type.

[0106] Understandably, the large language model outputs a set of high-quality, long-turn, and highly consistent multi-turn dialogue data. By generating dialogues for 10 different user profiles, the large language model can obtain a rich dataset covering multiple interaction scenarios.

[0107] After obtaining multi-turn dialogue training data, the original role-playing model can be supervised fine-tuned (SFT) based on the multi-turn dialogue training data to obtain the initial role-playing model.

[0108] Understandably, supervised fine-tuning of the original role-playing model aims to achieve knowledge internalization by "injecting" high-quality synthetic data (multi-turn dialogue training data) into the original role-playing model, enabling it to initially grasp the knowledge and behavioral patterns of the role.

[0109] Here, the multi-turn dialogue training data can be formatted. For example, the multi-turn dialogue training data can be organized into the standard format required for the original role-playing model to perform SFT. The standard format is, for example, {"input": "...", "target": "..."}.

[0110] Here, a basic role-playing model is fine-tuned under supervision using multi-turn dialogue training data. The original role-playing model will learn how to stably invoke role knowledge and imitate its basic speech and behavior in dialogue.

[0111] In this embodiment of the invention, after performing SFT on the original role-playing model, the resulting initial role-playing model has a significant improvement in the accuracy of answering questions about role knowledge, with an improvement of about 20% in knowledge accuracy.

[0112] Based on the above embodiments, step 32 includes: Step 321: Input the target role and the user profile into the large language model to obtain the candidate multi-turn dialogue training data output by the large language model; Step 322: Perform dialogue quality detection on the candidate multi-turn dialogue training data to obtain the multi-turn dialogue training data; The dialogue quality detection includes testing at least one of the following: logical coherence, language style, accuracy of character portrayal, richness of content, and interest.

[0113] Specifically, the target role and user profile can be input into the large language model to obtain candidate multi-turn dialogue training data output by the large language model.

[0114] After obtaining candidate multi-turn dialogue training data, dialogue quality testing can be performed on the candidate multi-turn dialogue training data to obtain multi-turn dialogue training data. Dialogue quality testing includes at least one of the following: logical coherence, language style, accuracy of character portrayal, richness of content, and engagingness.

[0115] To ensure the usability of multi-turn dialogue training data, this embodiment of the invention employs an additional method based on a large language model scoring system combined with majority voting. Here, five leading industry models are used as judges to score the multi-turn dialogues according to several dimensions: logical coherence, language style, accuracy of character portrayal, richness of content, and engagingness. Finally, a majority vote is conducted to filter out dialogue data that receives low scores from the large language model, thus ensuring data quality.

[0116] Here, logical coherence refers to whether the dialogue maintains semantic consistency and natural contextual flow across multiple rounds of interaction, and whether there are any contradictions, jumps, or breaks. Language style refers to whether the dialogue conforms to preset language style requirements, such as formal / informal, humorous / serious, friendly / cold, etc. Accuracy of character portrayal refers to whether the dialogue accurately reflects the target character's personality, background, knowledge level, language habits, and other characteristics. Content richness refers to whether the dialogue contains sufficient information, detail, and diversity, avoiding empty, repetitive, or templated expressions. Engagement refers to whether the dialogue is attractive, interesting, or emotionally resonant, and whether it can stimulate the user's willingness to continue participating.

[0117] The method provided in this invention generates candidate multi-turn dialogue training data by inputting the target role and user profile into a large language model, and performs multi-dimensional quality checks on the data, including logical coherence, language style, accuracy of character portrayal, richness of content, and interest, thereby effectively improving the overall quality and usability of the multi-turn dialogue training data.

[0118] Based on any of the above embodiments Figure 2 This is the second flowchart illustrating the training method for the role-playing model provided by this invention, as shown below. Figure 2 As shown, this method starts with an autonomous AI (Artificial Intelligence) agent search framework. First, the planning agent constructs a structured role knowledge archive based on task requirements, forming a general IP role graph. It then integrates domain-person relationship networks and external multi-source information to generate queries and call APIs to perform information retrieval. Next, an iterative completion phase begins. Through information synthesis and cross-validation, the agent performs multi-source information fusion and consistency verification on the retrieval results, gradually completing and correcting the role knowledge archive to ensure its accuracy and completeness. Then, through a multi-perspective dialogue automatic synthesis module, multi-round dialogue training data is constructed based on the structured role knowledge archive and diverse user profiles. Combined with language style paradigm extraction and algorithm optimization, a sample pair set is constructed, and the score corresponding to each sample pair in the set is determined. Based on the first ranking result of all scores and the second ranking result of all sample pairs, the initial reward model is adjusted to obtain a reward reference model. Finally, the model parameters of the reward module in the initial role-playing model are initialized based on the model parameters of the reward reference model to obtain the role-playing model to be trained. Language style enhancement training is then performed on the initialized role-playing model based on sample dialogue data to obtain the final role-playing model. In this process, a role-playing model is trained using a preference alignment algorithm (such as PPO) to evaluate the quality and style consistency of the generated content. The reward signal provided by the reward module in the role-playing model is then used to obtain the final alignment model, achieving high-fidelity and high-consistency role-playing capabilities.

[0119] Here, the initial role-playing model is obtained by supervised fine-tuning of the original role-playing model (pre-trained model) based on multi-turn dialogue training data.

[0120] It is understood that the embodiments of this invention are reusable methods for enhancing the realism of role-playing. The core idea is to deeply optimize a large language model through a multi-step data augmentation strategy involving two stages: "knowledge internalization" and "behavior alignment." Specifically, Supervised Fine-Tuning (SFT) is used to achieve knowledge internalization, and a reward model based on language style is designed to achieve preference alignment. The embodiments of this invention provide a standardized end-to-end process from automated knowledge construction to model optimization, which can be efficiently and cost-effectively reused for any IP role.

[0121] Furthermore, this invention addresses the knowledge-behavior separation problem by internalizing deep knowledge through high-quality SFT archives and aligning refined behaviors through RM, effectively bridging the gap between "knowing" and "doing." Moreover, this invention enhances the realism and immersion of dialogue, ensuring not only the accuracy of knowledge but also making characters more vivid and believable through meticulous refinement of speech style.

[0122] Furthermore, embodiments of the present invention reduce reliance on RAG and the risk of fantasy. Since the core knowledge has been internalized as part of the model, there is no need or only a slight reliance on external retrieval in the dialogue, thereby significantly reducing the risk of fantasy and dialogue disconnect.

[0123] The embodiments of this invention can be further extended to full-process automation, achieving end-to-end automated processing from input IP name to output of the final optimized model by building an integrated platform; it can also be extended to multimodal role-playing, combining voice cloning and facial expression generation technologies to create a virtual character experience with consistent voice and form; in addition, it supports dynamic character evolution, enabling the model to automatically learn new knowledge and adapt to character changes as the original content (such as serialized animation and TV series) is updated; it can also introduce more complex multi-character interaction logic to improve the expressiveness and adaptability of the role-playing model in multi-character collaborative scenarios, thereby significantly enhancing the intelligence, immersion and scalability of the role-playing system.

[0124] Considering that after training the role-playing model, it can be applied to role-playing.

[0125] Based on any of the above embodiments, the present invention provides a role-playing method. Figure 3 This is a flowchart illustrating the role-playing method provided by the present invention, such as... Figure 3 As shown, the method includes: Step 310: Obtain the role to be played and dialogue data; Step 320: Input the role to be played and the dialogue data into the role-playing model to obtain the recommended response text of the role to be played output by the role-playing model; The role-playing model is obtained by performing the training method described above.

[0126] Specifically, the role to be played and dialogue data can be obtained. The role to be played is the role that the role-playing model needs to play in the future. The role to be played can be Sun Wukong, or it can be a teacher, student, lawyer, judge, programmer, etc. The embodiments of the present invention do not make specific limitations on this.

[0127] Dialogue data refers to multi-turn or single-turn dialogue content used in role-playing tasks to provide contextual information, including user input, role responses, and dialogue history, to help the role-playing model understand the current dialogue context.

[0128] After obtaining the role to be played and the dialogue data, the role to be played and the dialogue data can be input into the role-playing model to obtain the recommended response text of the role to be played output by the role-playing model.

[0129] The training steps for the role-playing model are as follows: First, obtain the initial role-playing model and sample pair set. Then, determine the score corresponding to each sample pair in the sample pair set, and determine the ranking loss based on the first ranking result of all scores and the second ranking result of the labels of all sample pairs. Determine the score loss based on the scores corresponding to all sample pairs and the label scores of all sample pairs.

[0130] Furthermore, based on the ranking loss and the score loss, a target loss is determined, and the initial reward model is trained based on the target loss to obtain a reward reference model.

[0131] Finally, the parameters of the reward module in the initial role-playing model are initialized based on the model parameters of the reward reference model. The initial role-playing model after parameter initialization is then trained with language style enhancement based on sample dialogue data to obtain the role-playing model.

[0132] The parameters of the initial role-playing model can be pre-trained or randomly generated; this embodiment of the invention does not impose specific limitations on this. For example, the initial role-playing model can be a model fine-tuned by SFT supervision on a pre-trained model. The pre-trained model can be a large language model, such as the iFlytek Spark large model, XLNet model, or ROBERTa model; this embodiment of the invention does not impose specific limitations on this.

[0133] Here, the sample pair set includes positive sample pairs, negative sample pairs, and positive-negative sample pairs. In a positive sample pair, both positive samples are created from candidate answers relevant to the character, and in a negative sample pair, both negative samples are created from candidate answers unrelated to the character. A positive-negative sample pair consists of one positive sample and one negative sample; that is, the pair is created from a candidate answer relevant to the character and a candidate answer unrelated to the character. For example, a positive sample could be a core line of dialogue, a catchphrase, or a candidate answer that highly matches the character's language style. A negative sample could be a candidate answer with a generic language style, out-of-character (OOC), or clearly inconsistent with the character's personality. Each sample pair in the sample pair set can be extracted from the character's knowledge archive based on a language style paradigm.

[0134] After training and obtaining the reward reference model, the model parameters of the reward module in the initial role-playing model can be initialized using the model parameters of the reward reference model to obtain the role-playing model to be trained.

[0135] Specifically, the model parameters of the trained reward reference model can be directly copied to the reward module of the initial role-playing model, enabling the reward module to have relatively accurate scoring capabilities from the initial stage. If there is some structural overlap between the reward module and the reward reference model, only the parameters of the non-shared parts can be copied to reduce redundancy. In addition, if the reward module structure is small or needs further compression, knowledge distillation can be used to transfer knowledge from the reward reference model to the reward module. After the model parameters are initialized, the parameters of the reward module of the initial role-playing model are frozen, so that it only serves as a provider of reward signals in subsequent reinforcement learning training and does not participate in the parameter updates of the policy model; alternatively, some parameters can be unfrozen and fine-tuned according to actual needs to better adapt to the output distribution of the policy model, which is not specifically limited in this embodiment of the invention.

[0136] The method provided in this embodiment of the invention obtains the role to be played and dialogue data, and then inputs the role to be played and dialogue data into a role-playing model to obtain the recommended response text of the role to be played output by the role-playing model. The role-playing model is obtained by performing the training method of the above-mentioned role-playing model, thereby improving the high fidelity and high consistency of the recommended response text output by the role-playing model.

[0137] The training apparatus for the role-playing model provided by the present invention will be described below. The training apparatus for the role-playing model described below can be referred to in correspondence with the training method for the role-playing model described above.

[0138] Based on any of the above embodiments, the present invention provides a training device for a role-playing model. Figure 4 This is a schematic diagram of the structure of the training device for the role-playing model provided by the present invention, as shown below. Figure 4 As shown, the device includes: The first acquisition unit 410 acquires an initial role-playing model and a set of sample pairs; the set of sample pairs includes positive sample pairs, negative sample pairs, and positive-negative sample pairs, wherein the two positive samples in the positive sample pair are created from candidate answers related to the role, and the two negative samples in the negative sample pair are created from candidate answers unrelated to the role; the positive-negative sample pair consists of one positive sample and one negative sample. Training unit 420 is used to determine the score corresponding to each sample pair in the sample pair set, and to train the initial reward model based on the first ranking result of all the scores and the second ranking result of the labels of all the sample pairs to obtain a reward reference model. The language style training unit 430 is used to perform language style enhancement training on the initial role-playing model based on the model parameters of the reward reference model and sample dialogue data to obtain the role-playing model.

[0139] The apparatus provided in this invention determines the score corresponding to each sample pair in the sample pair set, and trains an initial reward model based on a first ranking result of all scores and a second ranking result of all sample pair labels to obtain a reward reference model. Based on the model parameters of the reward reference model and sample dialogue data, the initial role-playing model undergoes language style enhancement training to obtain a role-playing model. In this method, the reward reference model, through score ranking training, can accurately distinguish between candidate answers relevant to the role and those irrelevant to the role. Then, based on the model parameters of the reward reference model and sample dialogue data, the initial role-playing model undergoes language style enhancement training to obtain a role-playing model. This gives the initial role-playing model the same discriminative ability as the reward reference model. Therefore, during language style enhancement training, the role-playing model can continuously adjust its generation strategy according to the reward signal, making the generated answers more consistent with the character's language style. In this way, the generation strategy of the role-playing model gradually aligns with a high-fidelity speech style, thereby achieving a high-fidelity, high-consistency, and reusable role-playing capability.

[0140] Based on any of the above embodiments, the language style training unit 430 specifically includes: An initialization unit is used to initialize the model parameters of the reward module in the initial role-playing model using the model parameters of the reward reference model, so as to obtain the role-playing model to be trained. The language style enhancement training unit is used to perform language style enhancement training on the role-playing model to be trained based on the sample dialogue data, so as to obtain the role-playing model.

[0141] Based on any of the above embodiments, the language style enhancement training unit is specifically used for: The sample dialogue data is input into the role-playing model to be trained, and the recommended answer style text corresponding to the sample dialogue data is output by the role-playing model to be trained. Based on the reward value corresponding to the recommended answer style text, the role-playing model to be trained is subjected to language style enhancement training to obtain the role-playing model.

[0142] Based on any of the above embodiments, the training unit 420 is specifically used for: The ranking loss is determined based on the first ranking result of all the scores and the second ranking result of the labels of all the sample pairs; Based on all the scores and the label scores of all the sample pairs, determine the score loss; The target loss is determined based on the ranking loss and the score loss, and the initial reward model is adjusted based on the target loss to obtain the reward reference model.

[0143] Based on any of the above embodiments, the sample pairs are extracted from the character knowledge archive based on the language style paradigm; It also includes a building unit, which is specifically used for: Execute the following loop until the preset condition is met; Perform requirement parsing on the proxy task sent by the client to obtain the requirement parsing results; Based on the results of the requirement analysis, the proxy task is divided into multiple sub-tasks; Information from the information sources corresponding to each sub-task is integrated, and the role knowledge archive is determined based on the integrated information from the information sources corresponding to each sub-task. The preset conditions include the character knowledge profile reaching preset depth and preset breadth standards.

[0144] Based on any of the above embodiments, an initial role-playing model training unit is further included, wherein the initial role-playing model training unit specifically includes: A multi-turn dialogue unit is defined to input the target role and user profile into a large language model, thereby obtaining multi-turn dialogue training data output by the large language model; the multi-turn dialogue training data is dialogue content that conforms to the role setting of the target role and the user profile. The supervised fine-tuning unit is used to perform supervised fine-tuning on the original role-playing model based on the multi-turn dialogue training data to obtain the initial role-playing model.

[0145] Based on any of the above embodiments, the determination of the multi-turn dialogue unit is specifically used for: The target role and the user profile are input into the large language model to obtain candidate multi-turn dialogue training data output by the large language model; The candidate multi-turn dialogue training data is subjected to dialogue quality detection to obtain the multi-turn dialogue training data. The dialogue quality detection includes testing at least one of the following: logical coherence, language style, accuracy of character portrayal, richness of content, and interest.

[0146] The role-playing device provided by the present invention is described below. The role-playing device described below and the role-playing method described above can be referred to in correspondence.

[0147] Based on any of the above embodiments, the present invention provides a role-playing device. Figure 5 This is a structural schematic diagram of the role-playing device provided by the present invention, such as... Figure 5 As shown, the device includes: The second acquisition unit 510 is used to acquire the role to be played and the question text; The input unit 520 is used to input the role to be played and the question text into the role-playing model to obtain the role text of the role to be played output by the role-playing model; The role-playing model is obtained by performing the training method described above.

[0148] The apparatus provided in this embodiment of the invention acquires the role to be played and dialogue data, and then inputs the role to be played and dialogue data into a role-playing model to obtain the recommended response text of the role to be played output by the role-playing model. The role-playing model is obtained based on the training method of the above-mentioned role-playing model, thereby improving the high fidelity and high consistency of the recommended response text output by the role-playing model.

[0149] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communications bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other through the communications bus 640. The processor 610 can call logical instructions in the memory 630 to execute a training method for a role-playing model. This method includes: acquiring an initial role-playing model and a set of sample pairs; the set of sample pairs includes positive sample pairs, negative sample pairs, and positive-negative sample pairs, wherein the two positive samples in each positive sample pair are created from candidate answers related to the role, and the two negative samples in each negative sample pair are created from candidate answers unrelated to the role; each positive-negative sample pair consists of one positive sample and one negative sample; determining the score corresponding to each sample pair in the set of sample pairs, and training the initial reward model based on a first ranking result of all the scores and a second ranking result of the labels of all the sample pairs to obtain a reward reference model; and performing language style reinforcement training on the initial role-playing model based on the model parameters of the reward reference model and sample dialogue data to obtain the role-playing model.

[0150] The processor 610 can also call logical instructions in the memory 630 to execute a role-playing method, which includes: acquiring a role to be played and dialogue data; inputting the role to be played and the dialogue data into a role-playing model to obtain a recommended response text of the role to be played output by the role-playing model; the role-playing model is obtained by executing the above-mentioned role-playing model training method.

[0151] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0152] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the training method for the role-playing model provided by the above methods. The method includes: obtaining an initial role-playing model and a set of sample pairs; the set of sample pairs includes positive sample pairs, negative sample pairs, and positive-negative sample pairs, wherein the two positive samples in the positive sample pairs are created from candidate answers related to the role, and the two negative samples in the negative sample pairs are created from candidate answers unrelated to the role; the positive-negative sample pairs consist of one positive sample and one negative sample; determining the score corresponding to each sample pair in the set of sample pairs, and training the initial reward model based on a first ranking result of all the scores and a second ranking result of the labels of all the sample pairs to obtain a reward reference model; and performing language style reinforcement training on the initial role-playing model based on the model parameters of the reward reference model and sample dialogue data to obtain the role-playing model.

[0153] When the computer program is executed by the processor, the computer is able to execute the role-playing method provided by the above methods, the method including: acquiring the role to be played and dialogue data; inputting the role to be played and the dialogue data into the role-playing model to obtain the recommended answer text of the role to be played output by the role-playing model; the role-playing model is obtained by executing the above-mentioned role-playing model training method.

[0154] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for training a role-playing model provided by the methods described above. The method includes: acquiring an initial role-playing model and a set of sample pairs; the set of sample pairs includes positive sample pairs, negative sample pairs, and positive-negative sample pairs, wherein the two positive samples in the positive sample pairs are created from candidate answers related to the role, and the two negative samples in the negative sample pairs are created from candidate answers unrelated to the role; each positive-negative sample pair consists of one positive sample and one negative sample; determining the score corresponding to each sample pair in the set of sample pairs, and training the initial reward model based on a first ranking result of all the scores and a second ranking result of the labels of all the sample pairs to obtain a reward reference model; and performing language style reinforcement training on the initial role-playing model based on the model parameters of the reward reference model and sample dialogue data to obtain the role-playing model.

[0155] When executed by a processor, the computer program implements the role-playing method provided by the above methods, the method comprising: acquiring a role to be played and dialogue data; inputting the role to be played and the dialogue data into a role-playing model to obtain a recommended response text of the role to be played output by the role-playing model; wherein the role-playing model is obtained by executing the above-described role-playing model training method.

[0156] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0157] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a role-playing model, characterized in that, include: Obtain an initial role-playing model and a set of sample pairs; the set of sample pairs includes positive sample pairs, negative sample pairs, and positive-negative sample pairs, wherein the two positive samples in the positive sample pairs are created from candidate answers related to the role, and the two negative samples in the negative sample pairs are created from candidate answers unrelated to the role; The positive and negative sample pair consists of one positive sample and one negative sample; Determine the score corresponding to each sample pair in the sample pair set, and train the initial reward model based on the first ranking result of all the scores and the second ranking result of the labels of all the sample pairs to obtain the reward reference model; The model parameters of the reward module in the initial role-playing model are initialized using the model parameters of the reward reference model to obtain the role-playing model to be trained; The sample dialogue data is input into the role-playing model to be trained, and the recommended answer style text corresponding to the sample dialogue data is output by the role-playing model to be trained. Based on the reward value corresponding to the recommended answer style text, the role-playing model to be trained is subjected to language style enhancement training to obtain the role-playing model.

2. The training method for the role-playing model according to claim 1, characterized in that, The initial reward model is trained based on the first ranking result of all the scores and the second ranking result of the labels of all the sample pairs to obtain a reward reference model, including: The ranking loss is determined based on the first ranking result of all the scores and the second ranking result of the labels of all the sample pairs; Based on all the scores and the label scores of all the sample pairs, determine the score loss; The target loss is determined based on the ranking loss and the score loss, and the initial reward model is adjusted based on the target loss to obtain the reward reference model.

3. The training method for the role-playing model according to claim 2, characterized in that, The sample pairs were extracted from the character knowledge archive based on the language style paradigm. The steps for constructing the character knowledge profile include: Execute the following loop until the preset condition is met; Perform requirement parsing on the proxy task sent by the client to obtain the requirement parsing results; Based on the results of the requirement analysis, the proxy task is divided into multiple sub-tasks; Information from the information sources corresponding to each sub-task is integrated, and the role knowledge archive is determined based on the integrated information from the information sources corresponding to each sub-task. The preset conditions include the character knowledge profile reaching preset depth and preset breadth standards.

4. The training method for the role-playing model according to claim 1, characterized in that, The training steps for the initial role-playing model include: The target role and user profile are input into a large language model to obtain multi-turn dialogue training data output by the large language model; the multi-turn dialogue training data is dialogue content that conforms to the role setting of the target role and the user profile. Based on the multi-turn dialogue training data, the original role-playing model is fine-tuned under supervision to obtain the initial role-playing model.

5. The training method for the role-playing model according to claim 4, characterized in that, The process of inputting the target role and user profile into a large language model to obtain multi-turn dialogue training data output by the large language model includes: The target role and the user profile are input into the large language model to obtain candidate multi-turn dialogue training data output by the large language model; The candidate multi-turn dialogue training data is subjected to dialogue quality detection to obtain the multi-turn dialogue training data. The dialogue quality detection includes testing at least one of the following: logical coherence, language style, accuracy of character portrayal, richness of content, and interest.

6. A role-playing method, characterized in that, include: Obtain the role to be played and dialogue data; The role to be played and the dialogue data are input into the role-playing model to obtain the recommended response text of the role to be played, which is output by the role-playing model. The role-playing model is obtained by performing the training method for the role-playing model as described in any one of claims 1 to 5.

7. A training device for a role-playing model, characterized in that, include: The first acquisition unit acquires an initial role-playing model and a set of sample pairs; the set of sample pairs includes positive sample pairs, negative sample pairs, and positive-negative sample pairs, wherein the two positive samples in the positive sample pairs are created from candidate answers related to the role, and the two negative samples in the negative sample pairs are created from candidate answers unrelated to the role; The positive and negative sample pair consists of one positive sample and one negative sample; The training unit is used to determine the score corresponding to each sample pair in the sample pair set, and to train the initial reward model based on the first ranking result of all the scores and the second ranking result of the labels of all the sample pairs to obtain the reward reference model. An initialization unit is used to initialize the model parameters of the reward module in the initial role-playing model using the model parameters of the reward reference model, so as to obtain the role-playing model to be trained. The input unit is used to input sample dialogue data into the role-playing model to be trained, and to obtain the recommended answer style text corresponding to the sample dialogue data output by the role-playing model to be trained; The language style training unit is used to perform language style enhancement training on the role-playing model to be trained based on the reward value corresponding to the recommended answer style text, so as to obtain the role-playing model.

8. A role-playing device, characterized in that, include: The second acquisition unit is used to acquire the role to be played and the question text; The input unit is used to input the role to be played and the question text into the role-playing model to obtain the role text of the role to be played output by the role-playing model; The role-playing model is obtained by performing the training method for the role-playing model as described in any one of claims 1 to 5.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the training method for the role-playing model as described in any one of claims 1 to 5, or the role-playing method as described in claim 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method of the role-playing model as described in any one of claims 1 to 5, or the role-playing method as described in claim 6.

Images