Role playing model training method and device

Abstract

The embodiment of the invention provides a role playing model training method, and the method comprises the steps: obtaining a dialogue data set and a role setting set of a plurality of roles, and constructing a plurality of groups of training data based on the dialogue data set and the role setting set of the plurality of roles; on the basis of the multiple groups of training data, performing supervision and fine tuning on a pre-trained base model to obtain an initial role playing model; and obtaining a preference data set, and performing preference optimization on the initial role playing model based on the preference data set to obtain a target role playing model. The training method provided by the embodiment of the invention is carried out in stages, and different training samples are constructed in each stage to optimize the base model, so that the model can be aligned with role playing tasks, human preferences and value views, and finally a role playing model with excellent performance in the aspects of dialogue ability, role consistency, role playing attraction and the like is obtained.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of language model technology, and in particular, to a method, apparatus, computer device, computer-readable storage medium, and computer program product for training a role-playing model. Background Art

[0002] A language model is a machine learning model based on a corpus that can be used to understand and generate natural language text. A language model can handle a variety of natural language tasks, such as question-answering, dialogue, translation, sentiment analysis, etc.

[0003] However, the current language models still have obvious shortcomings in the field of role-playing, such as inaccurate understanding of the role, difficulty in maintaining the character's speaking style, etc., resulting in poor performance of the language model in many aspects such as conversation ability, role consistency and role-playing attractiveness.

[0004] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the invention

[0005] The embodiments of the present application provide a training method, apparatus, computer device, computer-readable storage medium, and computer program product for a role-playing model to solve or alleviate one or more of the technical problems raised above.

[0006] One aspect of an embodiment of the present application provides a method for training a role-playing model, the method comprising: Acquire dialogue data sets and role setting sets of multiple characters, and construct multiple sets of training data based on the dialogue data sets and role setting sets of the multiple characters; Based on the multiple sets of training data, supervised fine-tuning is performed on the pre-trained base model to obtain an initial role-playing model; Obtain a preference data set, and perform preference optimization on the initial role-playing model based on the preference data set to obtain a target role-playing model; wherein the preference data set is used to define preferred dialogues and non-preferred dialogues; and the target role-playing model is used to play the multiple roles or new roles for dialogue.

[0007] Optionally, the acquiring of dialogue data sets and role setting sets of multiple characters, and constructing multiple sets of training data based on the dialogue data sets and role setting sets of multiple characters, includes: Based on a preset role knowledge base, obtaining a dialogue data set and a role setting set of the multiple roles, wherein the dialogue data set includes multiple rounds of dialogues sorted by time; The following operations are performed for each round of dialogue of each character to construct the multiple sets of training data: Based on the generation time and the content of the current dialogue, a reference dialogue is obtained from the dialogue data set of the current character, where the generation time of the reference dialogue is earlier than the generation time of the current dialogue; When the reference dialogue is obtained, the current dialogue, the reference dialogue and the character setting set of the current character are concatenated to obtain training data; In the case where the reference dialogue is not obtained from the dialogue data set of the current character: based on a preset reference dialogue set, the reference dialogue is obtained; the current dialogue, the reference dialogue and the role setting set of the current character are spliced ​​to obtain training data.

[0008] Optionally, the role knowledge base is obtained by the following operations: Analyze the preset media content and determine multiple roles; Extracting initial dialogue data and role setting sets of the plurality of roles from the media content; Inputting the initial dialogue data of the multiple characters into a pre-trained dialogue screening model to obtain dialogue data sets of the multiple characters through the dialogue screening model; The role knowledge base is constructed based on the dialogue data sets and role setting sets of the multiple roles.

[0009] Optionally, obtaining the preference data set includes: matching a first reference model with a second reference model based on the performance of the initial role-playing model; Inputting a preset question text into the first reference model to obtain a first reply text through the first reference model; Inputting the question text into the second reference model to obtain a second reply text through the second reference model; Determining a preference value of the first reply text and a preference value of the second reply text based on a pre-trained quality inspection and screening model; The preference data set is constructed based on the question text, the preference value of the first reply text, and the preference value of the second reply text.

[0010] Optionally, the base model is obtained by the following operations: A first pre-trained language model is obtained by training on a standard corpus based on a predetermined rule; Acquire entertainment data of a specific field or platform, wherein the entertainment data includes one or more of bullet comments, comments, and manuscript texts; Based on the entertainment data, training the first pre-trained model to obtain a second pre-trained model; The second pre-trained model is determined as the base model.

[0011] Another aspect of an embodiment of the present application provides a role-playing method, the method comprising: Determine target role playing and question text; The target role-playing role and the question text are input into a role-playing model trained by any one of the training methods described above, so as to obtain a reply text of the target role-playing role to the question text through the role-playing model.

[0012] Optionally, the role-playing method further comprises: Determine a dialogue data set and a role setting set of a custom role, wherein the dialogue data set includes one or more dialogue rounds associated with the custom role; A predetermined question text, a dialogue data set of the custom role, and a role setting set are input into the role-playing model, so as to obtain a reply text of the custom role to the question text through the role-playing model.

[0013] Another aspect of an embodiment of the present application provides a training device for a role-playing model, the device comprising: A construction module, used to obtain dialogue data sets and role setting sets of multiple characters, and construct multiple sets of training data based on the dialogue data sets and role setting sets of the multiple characters; A supervised fine-tuning module, configured to perform supervised fine-tuning on the pre-trained base model based on the plurality of sets of training data to obtain an initial role-playing model; A preference optimization module is used to obtain a preference data set, and based on the preference data set, perform preference optimization on the initial role-playing model to obtain a target role-playing model; wherein the preference data set is used to define preferred dialogues and non-preferred dialogues; and the target role-playing model is used to play the multiple roles or new roles for dialogue.

[0014] Another aspect of an embodiment of the present application provides a role-playing device, the device comprising: A determination module is used to determine the target role and question text; The acquisition module is used to input the target role-playing role and the question text into the role-playing model obtained by the training device as described above, so as to obtain the response text of the target role-playing role to the question text through the role-playing model.

[0015] Another aspect of an embodiment of the present application provides a computer device, including: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein: the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0016] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0017] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the method described above when executed by a processor.

[0018] The above technical solution adopted in the embodiment of the present application may have the following advantages: Based on the dialogue data sets and role setting sets of multiple roles, multiple sets of training data are constructed to supervise and fine-tune the pre-trained base model, so that the model can learn and simulate the setting characteristics and speaking styles of different roles, achieve alignment for role-playing tasks, and obtain an initial role-playing model that performs better in the role-playing field, meeting the application requirements in most cases. Further, a preference data set is obtained and the initial role-playing model is directly optimized based on the preference data set, so that the model can align human preferences, values, etc., thereby obtaining a target role-playing model with stronger controllability and higher output quality. It can be seen that the training method provided in the embodiment of the present application is carried out in stages, and different training samples are constructed in each stage to optimize the base model, so that it can align role-playing tasks, human preferences and values, and finally obtain a role-playing model that performs well in terms of dialogue ability, role consistency and role-playing attractiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0020] Figure 1 A flowchart of a role-playing model training method according to Embodiment 1 of the present application is schematically shown; Figure 2 A flowchart of a training method for a base model according to Embodiment 1 of the present application is schematically shown; Figure 3 Schematically shows a role setting set and a dialogue data set according to the first embodiment of the present application; Figure 4 Schematically shows Figure 1Flow chart of sub-steps of step S100; Figure 5 The following schematically shows a flowchart of constructing a role knowledge base according to the first embodiment of the present application; Figure 6 The PPO and DPO according to the first embodiment of the present application are schematically shown; Figure 7 Schematically shows Figure 1 Flow chart of sub-steps of step S104; Figure 8 A framework diagram of a training method for a base model according to Embodiment 1 of the present application is schematically shown; Fig. 9 Schematically shows a role-playing dialogue according to the second embodiment of the present application; Fig.10 A flowchart of a role-playing method according to Embodiment 2 of the present application is schematically shown; Fig.11 A newly added flow chart of the role-playing method according to the second embodiment of the present application is schematically shown; Fig.12 A block diagram of a role-playing model training device according to Embodiment 3 of the present application is schematically shown; Fig.13 A block diagram schematically shows a role-playing device according to a fourth embodiment of the present application; and Fig.14 The hardware architecture diagram of the computer device according to the fifth embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0022] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0023] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0024] First, the following terms are explained: Benchmark: Benchmark is a standardized test used to evaluate model performance. It compares the performance of different models using fixed data sets and tasks.

[0025] CharacterEval: A Chinese benchmark dataset (evaluation set) for evaluating role-playing models. The evaluation set includes 77 characters and 1,785 dialogues, where the characters are from Chinese novels, scripts, movies, etc. The evaluation set has thirteen indicators that can evaluate the model from multiple dimensions such as dialogue ability, character consistency, and role-playing attractiveness.

[0026] Pedestal model: A universal language model.

[0027] SFT (Supervised Fine-Tuning): Use labeled data to perform targeted fine-tuning on a pre-trained model to optimize the model's performance in a specific field or task.

[0028] PPO (Proximal Policy Optimization): A reinforcement learning algorithm.

[0029] DPO (Direct Policy Optimization): a preference learning algorithm.

[0030] RAG (Retrieval-Augmented Generation): combines the retrieval mechanism with the generative model and provides the retrieved relevant information as additional context to the model.

[0031] STEM (Science, Technology, Engineering, Mathematics): Science, Technology, Engineering, Mathematics.

[0032] Secondly, in order to facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the relevant technologies are described below: A language model is a machine learning model based on a corpus that can be used to understand and generate natural language text. A language model can handle a variety of natural language tasks, such as question-answering, dialogue, translation, sentiment analysis, etc.

[0033] However, the applicant understands that the current language model still has obvious deficiencies in the field of role-playing, such as inaccurate understanding of the role, difficulty in maintaining the role's speaking style, etc., resulting in poor performance of the language model in many aspects such as dialogue ability, role consistency and role-playing attractiveness.

[0034] To this end, an embodiment of the present application provides a training scheme for a role-playing model. In this training scheme: (1) The base model absorbs a large amount of entertainment data and has unique advantages in the field of entertainment; (2) Training data is constructed based on business needs to improve the quality of training data. Based on the training data, the general base model is supervised and fine-tuned to align the role-playing tasks, so that it performs better in the role-playing field and meets the application needs in most cases; (3) The model after supervised fine-tuning is optimized based on the preference data set to align human preferences and values, thereby further improving the controllability and output quality of the model; (4) The trained role-playing model performs significantly better than other models of the same level in multiple dimensions such as dialogue ability, role consistency, and role-playing attractiveness; (5) While ensuring data desensitization, business data can also be used to customize and optimize specific role effects. Business data that is helpful to the model can be used to adjust the role-playing model, further improve the ability of the role-playing model, and form a closed loop. See below for details.

[0035] The technical solutions of the present application are described below through multiple embodiments. It should be noted that these embodiments can be implemented in a variety of different forms and should not be construed as being limited to the embodiments described here.

[0036] Embodiment 1 Figure 1 The flowchart of the role-playing model training method according to the first embodiment of the present application is schematically shown.

[0037] like Figure 1 As shown, the training method of the role-playing model may include steps S100 to S104, wherein: Step S100, obtaining dialogue data sets and role setting sets of multiple characters, and constructing multiple sets of training data based on the dialogue data sets and role setting sets of the multiple characters.

[0038] Step S102: Based on the multiple sets of training data, supervised fine-tuning is performed on the pre-trained base model to obtain an initial role-playing model.

[0039] Step S104, obtaining a preference data set, and performing preference optimization on the initial role-playing model based on the preference data set to obtain a target role-playing model; wherein the preference data set is used to define preferred dialogues and non-preferred dialogues; and the target role-playing model is used to play the multiple roles or new roles for dialogue.

[0040] The training method of the role-playing model provided in this embodiment: construct multiple sets of training data based on the dialogue data sets and role setting sets of multiple roles to supervise and fine-tune the pre-trained base model, so that the model can learn and simulate the setting characteristics and speaking styles of different roles, achieve alignment for role-playing tasks, and obtain an initial role-playing model that performs better in the role-playing field, meeting the application requirements in most cases. Further, obtain a preference data set and perform direct preference optimization on the initial role-playing model based on the preference data set, so that the model can align human preferences, values, etc., thereby obtaining a target role-playing model with stronger controllability and higher output quality. It can be seen that the training method provided in the embodiment of the present application is carried out in stages, and different training samples are constructed in each stage to optimize the base model so that it can align role-playing tasks, human preferences and values, and finally obtain a role-playing model that performs well in terms of dialogue ability, role consistency and role-playing attractiveness.

[0041] The following combination Figure 1 , each step in steps S100~S104 and other optional steps are explained in detail.

[0042] The training method provided in the embodiment of the present application can be divided into three stages that are executed successively, namely, the pre-training stage (PT), the supervised fine-tuning stage (SFT), and the preference optimization stage (PO). The optimization objectives of the model are different at different stages, so different training samples need to be constructed for each stage for model training. The following will further illustrate the various stages of model training.

[0043] The goal of the pre-training phase is to allow the model to learn general language knowledge and expressions, providing a solid foundation for subsequent fine-tuning based on specific tasks and optimization based on human preferences. At this stage, the language model can be trained based on a very large-scale corpus to initially master basic grammatical rules, logical abilities, and common sense knowledge. The corpus can be text data from various fields to help the model learn grammar, vocabulary, and common sense in different fields, and obtain a general large language model (base model) with good generalization capabilities. In practical applications, the training samples and training process in the pre-training phase can be further optimized according to business needs to enhance the capabilities of the base model. An exemplary solution is provided below.

[0044] In an optional embodiment, if Figure 2 As shown, the base model can be obtained by the following operations: Step S200: obtaining a first pre-trained language model by training on a standard corpus based on predetermined rules.

[0045] Step S202, obtaining entertainment data of a specific field or platform, wherein the entertainment data includes one or more of barrage, comments, and manuscript text.

[0046] Step S204: training the first pre-trained model based on the entertainment data to obtain a second pre-trained model.

[0047] Step S206: determine the second pre-trained model as the base model.

[0048] The standard corpus may include a large amount of publicly available corpus, which may come from books, encyclopedias, papers, web pages, STEM, etc. Exemplarily, the standard corpus may be used directly, or the corpus may be selected in a targeted manner according to actual business needs (such as the Internet, medical care, finance, education, etc.), and the corpus of the standard corpus may be selectively obtained according to business needs (such as: adjusting the proportion of corpus in different fields), and the first pre-trained language model may be trained to initially master basic grammatical rules, logical abilities, and common sense knowledge, etc. On this basis, data from a specific field or platform may be further obtained (such as bullet screens, comments, manuscript texts, etc. of a certain video platform), and the first pre-trained language model may be fine-tuned to obtain a second pre-trained language model that can better match a specific field or platform, i.e., a base model. It should be noted that all data used for model training in the embodiments of the present application are legal and compliant, and all data have been strictly cleaned, screened, and desensitized, and invalid, erroneous, or duplicated data may be removed, model training efficiency may be improved, and privacy and data security may be ensured. Among them, the cleaning method may include heuristic rules and classifier filtering. Screening may be based on business rules and data quality. Desensitization can be achieved through masking, encryption, hashing, data perturbation, and deletion of sensitive information.

[0049] In this embodiment, the pre-training can be divided into two stages: in the first stage, a large amount of general data is used to help the model learn general language knowledge and expressions. In the second stage, entertainment data in a specific domain or platform is introduced to fine-tune the model, improving its performance in the specific domain or platform and endowing the model with unique advantages. This phased training can reduce the risk of overfitting and improve the training efficiency, ensuring that the model has good performance in a wide range of language tasks and specific application scenarios.

[0050] The above embodiment exemplarily introduces the training method of the base model. However, the trained base model lacks the learning and understanding of roles and cannot generate dialogue content that matches the roles, so it is not suitable for direct use in role-playing scenarios. To solve the above problems, alignment (i.e., supervised fine-tuning) for role-playing tasks can be performed on the general large language model (base model) so that the model can have good performance in the field of role-playing. The following will further exemplarily introduce the remaining stages of model training.

[0051] Step S100 , obtain the dialogue data sets and role setting sets of multiple roles, and construct multiple groups of training data based on the dialogue data sets and role setting sets of the multiple roles.

[0052] The roles can come from literary works, film and television works, video games, animations / comics, all walks of life, history and culture, etc. For example: Sun Wukong, Mario, doctor, teacher, etc. The roles can be shaped through role setting sets and dialogue data sets. The role setting set can include various role characteristics such as the gender, age, height, nickname, self-address, occupation, personality, personal profile, relationship with other characters, language style, catchphrase, learning experience, work experience, etc. of the role. The dialogue data set can include multiple rounds of conversations associated with the role, and each round of conversation can be composed of a question text and a reply text. The conversation can better express the role characteristics and fully reflect the character / values / background, emotions and preference tendencies, catchphrases, dialects, speaking styles, favorite words and sentences, etc. of the role. Figure 3 Exemplarily provide the dialogue data set (only one round of conversation is shown) and role setting set of the role "Mengmeng Jiang". Exemplarily, the dialogue data sets and role setting sets of multiple roles can be obtained, and multiple groups of training data can be constructed based on the dialogue data sets and role setting sets of the multiple roles. There are various methods for constructing training data. For example, the dialogue data set and role setting set of the same role are spliced to form training data. Of course, methods such as embedding, tagging, and data augmentation can also be used to construct training data to meet different business requirements. The following will provide an exemplary solution.

[0053] In an alternative embodiment, as Figure 4 shown, step S100 may include: Step S400, based on a preset role knowledge base, obtaining a dialogue data set and a role setting set of the multiple roles, wherein the dialogue data set includes multiple rounds of dialogues sorted by time.

[0054] Step S402, performing the following operations for each round of dialogue of each character to construct the multiple sets of training data: S4021: Based on the generation time and content of the current dialogue, a reference dialogue is obtained from the dialogue data set of the current character, where the generation time of the reference dialogue is earlier than the generation time of the current dialogue.

[0055] S4022: When the reference dialogue is obtained, the current dialogue, the reference dialogue and the role setting set of the current character are concatenated to obtain training data.

[0056] S4023: In the case where the reference dialogue is not obtained from the dialogue data set of the current character: based on a preset reference dialogue set, the reference dialogue is obtained; the current dialogue, the reference dialogue and the role setting set of the current character are spliced ​​to obtain training data.

[0057] Exemplarily, the dialogue data sets and role setting sets of multiple roles can be obtained from the preset role knowledge base. The dialogue data set of each role includes multiple rounds of dialogues related to the role, wherein the generation time of each round of dialogue is different. For each round of dialogue of each role, the following operations can be performed to construct training data. Specifically: based on the generation time and dialogue content of the current (round) dialogue, RAG can be used to retrieve reference dialogues related to the current dialogue in the logarithmic data set of the current role, wherein the generation time of the reference dialogue is earlier than the current dialogue, and the content of the reference dialogue can be related to the content of the current dialogue. That is, the reference dialogue can be a past dialogue of the current role. For example, the dialogue data set of role A has five rounds of dialogue. If the current dialogue is the third round of dialogue, the reference dialogue may include the first round and / or the second round of dialogue. If the current dialogue is the first round of dialogue, there is no reference dialogue in the dialogue data set. In the case of obtaining the reference dialogue, the current dialogue, the reference dialogue and the role setting set can be spliced ​​to obtain the training data. In the case where the reference dialogue is not obtained from the dialogue data set of the current role, the reference dialogue can be obtained from the reference dialogue set prepared in advance. For example, if character A is a host, the reference dialogue set may include a plurality of classic quotes of the host. Similarly, the current dialogue, the reference dialogue, and the character setting set may be spliced ​​to obtain training data.

[0058] In this embodiment, the current dialogue, reference dialogue and role setting set are spliced ​​to form training data for training the model, which can improve the relevance and coherence of the generated dialogue, the accuracy of role playing, and the generalization and robustness of the model. In the absence of reference dialogue, splicing the current dialogue and role setting set can also ensure the quality of the generated dialogue and the consistency of the role.

[0059] The above embodiment exemplifies how to construct training data for the supervised fine-tuning phase based on the dialogue data sets and role setting sets of multiple roles in the preset role knowledge base. The following will exemplify how to pre-build a role knowledge base.

[0060] In an optional embodiment, if Figure 5 As shown, the role knowledge base can be obtained through the following operations: Step S500: Analyze the preset media content to determine multiple roles.

[0061] Step S502: extracting the initial dialogue data and role setting sets of the multiple roles from the media content.

[0062] Step S504: input the initial dialogue data of the multiple roles into a pre-trained dialogue screening model to obtain dialogue data sets of the multiple roles through the dialogue screening model.

[0063] Step S506: construct the role knowledge base based on the dialogue data sets and role setting sets of the multiple roles.

[0064] Media content can include a variety of open source data, such as movies, novels, scripts, ASR (automatic speech recognition), live broadcasts, etc. By analyzing the media content, multiple roles can be identified and the initial dialogue data and role setting sets of each role can be extracted. For example, a large number of literary and film and television works can be collected and processed, the lines and scripts and character setting data can be analyzed, and the role dialogues can be extracted from them to obtain the initial dialogue data and role setting sets of multiple roles. Among them, the initial dialogue data may include dialogues of poor quality, such as meaningless interjections, repeated dialogues, sensitive or inappropriate content, etc. The quality of training data is more important than the quantity. Therefore, the pre-trained dialogue screening model can be used to screen the initial dialogue data to obtain a high-quality dialogue data set. On this basis, a role knowledge base can be constructed based on the dialogue data sets and role setting sets of multiple roles.

[0065] In this embodiment, the initial dialogue data and role setting set of the character are obtained by analyzing media data from various sources, and the dialogue screening model is used to screen high-quality dialogue data to form a dialogue data set, so that a large-scale, high-quality role knowledge base can be constructed.

[0066] The above embodiments describe how to construct training data for the supervised fine-tuning stage. The supervised fine-tuning process will be described below.

[0067] Step S102 , based on the multiple sets of training data, supervised fine-tuning is performed on the pre-trained base model to obtain an initial role-playing model.

[0068] Based on the constructed multiple sets of training data, the base model is fine-tuned in a supervised manner to obtain the initial role-playing model. During the training process, the model not only learns how to generate contextual dialogues based on the dialogue dataset, but also adjusts the generated content based on the role setting set to ensure that the dialogue is consistent with the role setting. Based on this, the initial role-playing model has good performance in role-playing tasks and can meet the application requirements in most cases.

[0069] In this embodiment, by constructing the training data to supervise and fine-tune the base model, the model accuracy can be improved, making the model suitable for role-playing scenarios, better understanding and simulating the role, and outputting dialogue content that matches the role, with excellent performance in multiple dimensions such as dialogue ability, role consistency, and role-playing attractiveness.

[0070] In practical applications, in order to further improve the controllability and output quality of the model, the model can also be optimized for preferences, so that the model is aligned with human preferences or values ​​and outputs conversation content that better meets user or business needs. The following is a further illustrative introduction to the preference optimization stage.

[0071] Step S104 , obtaining a preference data set, and optimizing the initial role-playing model based on the preference data set to obtain a target role-playing model; wherein the preference data set is used to define preferred dialogues and non-preferred dialogues; and the target role-playing model is used to play the multiple roles or new roles for dialogue.

[0072] The preference data set may be used to define a preferred conversation and a non-preferred conversation. Specifically, the preference data set may include multiple preference data pairs. Figure 6As shown, each preference data pair may include an input instruction Prompt (also called question text, such as x: "write me a poem about the history of jazz") and two types of candidate results (such as: yw and yl), and each type of candidate result may include a reply text. Preference data pairs can be manually constructed and labeled, or labeled and generated through models. Preference optimization can use PPO or DPO. In order to save computing resources and improve training efficiency, this embodiment uses DPO (direct preference optimization) to perform preference optimization on the initial role-playing model. DPO can directly perform preference learning on the labeled preference data pairs, and there is no need to load 4 models like PPO (the language model to be trained, the critic model, the reward model, and the constraint model), so it can save a lot of resource consumption (such as Figure 6 As shown). Taking role-playing as an example, DPO does not directly use the initial role-playing model to generate dialogues, but instead provides the dialogues (preference data pairs) generated by other models to the initial role-playing model, and tells it which candidate results are good (yw, Chosen) and which candidate results are bad (yl, Reject), so that the initial role-playing model can learn and align human preferences and values ​​from them, and obtain a target role-playing model with higher output quality. It can be seen that DPO optimizes the initial role-playing model by comparing the dialogues of other models. If the level of other models is too different from the initial role-playing model, the comparison results may be inaccurate or unrepresentative, and cannot effectively guide model optimization, resulting in poor training results. Based on this, the method of obtaining the preference data set can be optimized. An exemplary solution is provided below.

[0073] In an optional embodiment, if Figure 7 As shown, step S104 may include: Step S700: matching a first reference model and a second reference model based on the performance of the initial role-playing model.

[0074] Step S702: input a preset question text into the first reference model to obtain a first reply text through the first reference model.

[0075] Step S704: input the question text into the second reference model to obtain a second reply text through the second reference model.

[0076] Step S706: Determine the preference value of the first reply text and the preference value of the second reply text based on the pre-trained quality inspection and screening model.

[0077] Step S708: construct the preference data set based on the question text, the preference value of the first reply text, and the preference value of the second reply text.

[0078] Exemplarily, the first reference model and the second reference model with a performance gap within an appropriate range can be matched according to the performance of the initial role-playing model. For example, five reference models are pre-trained (performance scores are distributed between 60 and 80), and the performance score of the initial role-playing model is 70, then the reference model with a performance score of 80 can be selected as the first reference model. Select a reference model with a performance score of 60 as the second reference model. Input the same question text into the first reference model and the second reference model respectively to obtain the first reply text and the second reply text. Use the pre-trained quality inspection screening model to score the first reply text and the second reply text, that is, mark yw and yl. Based on the preference value of the first reply text and the preference value of the second reply text, the question text and the corresponding first reply text and second reply text are combined to form a preference data pair. Multiple different question texts can be constructed, and the above process can be performed to obtain multiple preference data pairs, and finally form a preference data set.

[0079] In this embodiment, by matching reference models with similar performance, response texts with different preferences are generated and scored using a quality inspection screening model to construct preference data pairs, thereby forming a high-quality preference data set.

[0080] By using the preference data set constructed by the above embodiment to optimize the preferences of the initial role-playing model, the model can be taught which questions cannot be asked or answered, or which require positive guidance, and which outputs are in line with expectations (human preferences and values) and which are not. This can ultimately improve the accuracy and effectiveness of model optimization and obtain the target role-playing model.

[0081] In this embodiment, DPO is used to optimize the model's preferences, which has a fast training speed and consumes fewer resources. It can further improve the model's controllability and output quality (i.e., role-playing ability) and reduce the security risks of the conversation content.

[0082] In order to make this application easier to understand, the following Figure 8 and Table 1 provide an exemplary application.

[0083] S1: The first pre-trained language model is obtained based on the public dataset.

[0084] S2: Fine-tune the first pre-trained language model based on data from a specific field or platform to obtain the second pre-trained model, namely the base model (data-enhanced large model).

[0085] S3: Use data from multiple sources (such as open source dialogue data, film and television, novel dialogue data, ASR, live broadcast dialogue data, etc.) to build a character knowledge base. The character knowledge base can include character setting sets and dialogue data sets, and can also include language style references and other content.

[0086] S5: Use RAG to retrieve the role knowledge base to construct multiple sets of training data.

[0087] S6: Use multiple sets of training data to supervise fine-tune the base model and obtain the initial role-playing model (role-playing model-SFT).

[0088] S7: Use the quality inspection screening model to construct and annotate preference comparison data (preference dataset).

[0089] S8: Based on the preference data set, the initial role-playing model is optimized to obtain the target role-playing model (role-playing model-DPO).

[0090] The target role-playing model can be used to provide online services, that is, to play a designated role and communicate with users. If the business data generated by the online service is helpful to the model, the data can be added to the role knowledge base while ensuring data desensitization, and used to update the target role-playing model, further improving the role-playing model's dialogue ability and forming a closed loop.

[0091] In this exemplary application: (1) The base model absorbs a large amount of entertainment data and has unique advantages in the field of entertainment; (2) Training data is constructed based on business needs to improve the quality of training data. Based on the training data, the general base model is supervised and fine-tuned to align the role-playing tasks, so that it performs better in the role-playing field and meets the application needs in most cases; (3) The preference optimization of the supervised fine-tuned model is performed based on the preference data set to align human preferences and values, thereby further improving the controllability and output quality of the model; (4) The trained role-playing model performs well in multiple dimensions such as dialogue ability, role consistency, and role-playing attractiveness, as shown in Table 1; (5) While ensuring data desensitization, business data can also be used to customize and optimize specific role effects. Business data that is helpful to the model can be used to adjust the role-playing model, further improve the ability of the role-playing model, and form a closed loop.

[0092]

[0093] Table 1 Embodiment 2 The role-playing model used in this embodiment can be trained by any training method in the first embodiment.

[0094] Fig.10The flowchart of the role-playing method according to the first embodiment of the present application is schematically shown.

[0095] like Fig.10 As shown, the role-playing method may include steps S1000 to S1002, wherein: Step S1000, determining the target role and question text.

[0096] Step S1002: input the target role-playing character and the question text into a role-playing model, so as to obtain a reply text of the target role-playing character to the question text through the role-playing model.

[0097] The role-playing method provided in this embodiment can specify the role to be played as the target role and determine the question text. The target role and the question text are input into the role-playing model trained by any of the training methods provided in the above embodiments, and the response text of the target role to the question text can be obtained, such as Fig. 9 As shown. It can be seen that the embodiment of the present application can generate accurate responses that match the target role-playing role by inputting the target role-playing role and the question text into the trained role-playing model, thereby enhancing the realism of role-playing and user experience.

[0098] In some embodiments, a reference dialogue may be searched in a preset role database according to the target role and the question text. The reference dialogue may be a historical dialogue of the target role, and the historical dialogue may be related to the question text. The target role, the question text and the reference dialogue may be input into the role-playing model together to improve the output quality and dialogue ability of the model.

[0099] In an optional embodiment, if Fig.11 As shown, the role-playing method may also include: Step S1100, determining a dialogue data set and a role setting set of a custom role, wherein the dialogue data set includes one or more dialogue rounds associated with the custom role.

[0100] Step S1102, inputting a predetermined question text, a dialogue data set of the custom role, and a role setting set into the role-playing model, so as to obtain a reply text of the custom role to the question text through the role-playing model.

[0101] The role-playing model trained by the first embodiment can also provide a role customization function. Exemplarily, a role setting set and a dialogue data set of a custom role can be constructed, and the dialogue data set can include one or more rounds of dialogues associated with the custom role. The predetermined question text, the dialogue data set of the custom role, and the role setting set are input into the role-playing model, and the reply text of the custom role to the question text can be obtained.

[0102] In this embodiment, specific roles can also be customized and optimized to further enhance the model's conversational capabilities and user experience.

[0103] Embodiment 3 Fig.12 The block diagram of the role-playing model training device according to the third embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium, and are executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Fig.12 As shown, the apparatus 1200 may include: a construction module 1210, a supervision fine-tuning module 1220 and a preference optimization module 1230, wherein: A construction module 1210 is used to obtain dialogue data sets and role setting sets of multiple roles, and construct multiple sets of training data based on the dialogue data sets and role setting sets of the multiple roles; A supervised fine-tuning module 1220 is used to perform supervised fine-tuning on the pre-trained base model based on the multiple sets of training data to obtain an initial role-playing model; The preference optimization module 1230 is used to obtain a preference data set, and based on the preference data set, perform preference optimization on the initial role-playing model to obtain a target role-playing model; wherein the preference data set is used to define preferred dialogues and non-preferred dialogues; and the target role-playing model is used to play the multiple roles or new roles for dialogue.

[0104] As an optional embodiment, the step of obtaining dialogue data sets and role setting sets of multiple characters, and constructing multiple sets of training data based on the dialogue data sets and role setting sets of the multiple characters, includes: Based on a preset role knowledge base, obtaining a dialogue data set and a role setting set of the multiple roles, wherein the dialogue data set includes multiple rounds of dialogues sorted by time; The following operations are performed for each round of dialogue of each character to construct the multiple sets of training data: Based on the generation time and the content of the current dialogue, a reference dialogue is obtained from the dialogue data set of the current character, where the generation time of the reference dialogue is earlier than the generation time of the current dialogue; When the reference dialogue is obtained, the current dialogue, the reference dialogue and the character setting set of the current character are concatenated to obtain training data; In the case where the reference dialogue is not obtained from the dialogue data set of the current character: based on a preset reference dialogue set, the reference dialogue is obtained; the current dialogue, the reference dialogue and the role setting set of the current character are spliced ​​to obtain training data.

[0105] As an optional embodiment, the role knowledge base is obtained by the following operations: Analyze the preset media content and determine multiple roles; Extracting initial dialogue data and role setting sets of the plurality of roles from the media content; Inputting the initial dialogue data of the multiple characters into a pre-trained dialogue screening model to obtain dialogue data sets of the multiple characters through the dialogue screening model; The role knowledge base is constructed based on the dialogue data sets and role setting sets of the multiple roles.

[0106] As an optional embodiment, the obtaining the preference data set includes: matching a first reference model with a second reference model based on the performance of the initial role-playing model; Inputting a preset question text into the first reference model to obtain a first reply text through the first reference model; Inputting the question text into the second reference model to obtain a second reply text through the second reference model; Determining a preference value of the first reply text and a preference value of the second reply text based on a pre-trained quality inspection and screening model; The preference data set is constructed based on the question text, the preference value of the first reply text, and the preference value of the second reply text.

[0107] As an optional embodiment, the base model is obtained by the following operations: A first pre-trained language model is obtained by training on a standard corpus based on a predetermined rule; Acquire entertainment data of a specific field or platform, wherein the entertainment data includes one or more of bullet comments, comments, and manuscript texts; Based on the entertainment data, training the first pre-trained model to obtain a second pre-trained model; The second pre-trained model is determined as the base model.

[0108] Embodiment 4 Fig.13 The block diagram of the role-playing device according to the fourth embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium, and are executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Fig.13 As shown, the device 1300 may include: a determination module 1310 and an acquisition module 1320, wherein: Determination module 1310, for determining the target role and question text; The acquisition module 1320 is used to input the target role-playing role and the question text into the role-playing model trained by the training method provided in the above embodiment, so as to obtain the response text of the target role-playing role to the question text through the role-playing model.

[0109] As an optional embodiment, the device 1300 is further used for: Determine a dialogue data set and a role setting set of a custom role, wherein the dialogue data set includes one or more dialogue rounds associated with the custom role; A predetermined question text, a dialogue data set of the custom role, and a role setting set are input into the role-playing model, so as to obtain a reply text of the custom role to the question text through the role-playing model.

[0110] Embodiment 5 Fig.14 The hardware architecture diagram of the computer device 10000 suitable for implementing the training method of the role-playing model according to the third embodiment of the present application is schematically shown. In some embodiments, the computer device 10000 can be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 can be a rack server, a blade server, a tower server or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Fig.14 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them: The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 can be an internal storage module of the computer device 10000, such as a hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 can also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10010 can also include both the internal storage module of the computer device 10000 and its external storage device. In this embodiment, the memory 10010 is generally used to store the operating system and various application software installed in the computer device 10000, such as the program code of the training method of the role-playing model, etc. In addition, the memory 10010 can also be used to temporarily store various data that have been output or will be output.

[0111] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0112] The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0113] It should be pointed out that Fig.14 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementation of all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0114] In this embodiment, the training method of the role-playing model stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.

[0115] Embodiment 6 An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the role-playing model training method in the embodiment are implemented.

[0116] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium may also be an external storage device of a computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Of course, the computer-readable storage medium may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the training method of the role-playing model in the embodiment. In addition, the computer-readable storage medium may also be used to temporarily store various types of data that have been output or are to be output.

[0117] Embodiment 7 An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0118] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0119] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for training a role-playing model, characterized in that: The method comprises: Acquire dialogue data sets and role setting sets of multiple characters, and construct multiple sets of training data based on the dialogue data sets and role setting sets of the multiple characters; Based on the multiple sets of training data, supervised fine-tuning is performed on the pre-trained base model to obtain an initial role-playing model; Obtain a preference data set, and perform preference optimization on the initial role-playing model based on the preference data set to obtain a target role-playing model; wherein the preference data set is used to define preferred dialogues and non-preferred dialogues; and the target role-playing model is used to play the multiple roles or new roles for dialogue.

2. The method according to claim 1, characterized in that The step of obtaining dialogue data sets and role setting sets of multiple characters, and constructing multiple sets of training data based on the dialogue data sets and role setting sets of the multiple characters, includes: Based on a preset role knowledge base, obtaining a dialogue data set and a role setting set of the multiple roles, wherein the dialogue data set includes multiple rounds of dialogues sorted by time; The following operations are performed for each round of dialogue of each character to construct the multiple sets of training data: Based on the generation time and the content of the current dialogue, a reference dialogue is obtained from the dialogue data set of the current character, where the generation time of the reference dialogue is earlier than the generation time of the current dialogue; When the reference dialogue is obtained, the current dialogue, the reference dialogue and the character setting set of the current character are concatenated to obtain training data; In the case where the reference dialogue is not obtained from the dialogue data set of the current character: based on a preset reference dialogue set, the reference dialogue is obtained; the current dialogue, the reference dialogue and the role setting set of the current character are spliced ​​to obtain training data.

3. The method according to claim 1, characterized in that The role knowledge base is obtained by the following operations: Analyze the preset media content and determine multiple roles; Extracting initial dialogue data and role setting sets of the plurality of roles from the media content; Inputting the initial dialogue data of the multiple characters into a pre-trained dialogue screening model to obtain dialogue data sets of the multiple characters through the dialogue screening model; The role knowledge base is constructed based on the dialogue data sets and role setting sets of the multiple roles.

4. The method according to claim 1, characterized in that: The step of obtaining the preference data set includes: matching a first reference model with a second reference model based on the performance of the initial role-playing model; Inputting a preset question text into the first reference model to obtain a first reply text through the first reference model; Inputting the question text into the second reference model to obtain a second reply text through the second reference model; Determining a preference value of the first reply text and a preference value of the second reply text based on a pre-trained quality inspection and screening model; The preference data set is constructed based on the question text, the preference value of the first reply text, and the preference value of the second reply text.

5. The method according to any one of claims 1 to 4, characterized in that: The base model is obtained by the following operations: A first pre-trained language model is obtained by training on a standard corpus based on a predetermined rule; Acquire entertainment data of a specific field or platform, wherein the entertainment data includes one or more of bullet comments, comments, and manuscript texts; Based on the entertainment data, training the first pre-trained model to obtain a second pre-trained model; The second pre-trained model is determined as the base model.

6. A role-playing method, characterized in that: The method comprises: Determine target role playing and question text; The target role-playing character and the question text are input into a role-playing model trained by the training method according to any one of claims 1 to 5, so as to obtain a reply text of the target role-playing character to the question text through the role-playing model.

7. The role-playing method according to claim 6, characterized in that: Also includes: Determine a dialogue data set and a role setting set of a custom role, wherein the dialogue data set includes one or more dialogue rounds associated with the custom role; A predetermined question text, a dialogue data set of the custom role, and a role setting set are input into the role-playing model, so as to obtain a reply text of the custom role to the question text through the role-playing model.

8. A training device for a role-playing model, characterized in that: The device comprises: A construction module, used to obtain dialogue data sets and role setting sets of multiple characters, and construct multiple sets of training data based on the dialogue data sets and role setting sets of the multiple characters; A supervised fine-tuning module, configured to perform supervised fine-tuning on the pre-trained base model based on the plurality of sets of training data to obtain an initial role-playing model; A preference optimization module is used to obtain a preference data set, and based on the preference data set, perform preference optimization on the initial role-playing model to obtain a target role-playing model; wherein the preference data set is used to define preferred dialogues and non-preferred dialogues; and the target role-playing model is used to play the multiple roles or new roles for dialogue.

9. A role-playing device, characterized in that: The device comprises: A determination module is used to determine the target role and question text; An acquisition module is used to input the target role-playing role and the question text into a role-playing model obtained by the training method of any one of claims 1 to 5, so as to obtain the response text of the target role-playing role to the question text through the role-playing model.

10. A computer device, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claims 1 to 7 are implemented.

Images