Fine-tuning-based role playing optimization method for virtual human

Abstract

The invention discloses a fine-tuning-based role playing optimization method for a virtual human. The method comprises the following steps: collecting scripts, acquiring a role summary document set by utilizing a GPT language model, and generating a session query set and a plot description text set; based on the session query set, the plot description text set and the role summary document set, retrieving a role summary document and a related text knowledge set corresponding to each session query by using an inverted index method, and generating an intermediate response corresponding to each session query by using an LLM model; segmenting the intermediate response by using a BERT model and a Transform model to generate an atomic fact sequence; verifying each atomic fact by using an LLM model, and outputting an unsupported atomic fact sequence; the set of session queries, the intermediate response, and the unsupported atomic fact sequence for each session query are input into an LLM model to generate an optimal session response corresponding to each session query. According to the invention, the illusion of time-sensitive and unpopular roles can be solved.

Description

Technical Field

[0001] The invention belongs to the field of computer technology, and in particular relates to a virtual human role-playing optimization method based on fine-tuning. Background Art

[0002] The Uncensored Large Language Model (Uncensored LLM) is a machine learning model designed to understand and generate human-like text, enabling it to simulate humans in a more advanced way rather than just copying simple human behaviors. Uncensored LLM is trained on a wide range of datasets and can perform a variety of tasks.

[0003] In recent years, the role-playing agency field driven by Uncensored LLM has expanded and grown rapidly, and the market demand has been verified by the market of the AI ​​emotional companionship industry and is large in scale, with more than 80 web applications and more than 40 mobile applications. The total web page traffic is more than 130 million, and the number of unique visitors is about 40 million. The application of Uncensored LLM in the role-playing agency field has been extended to different environments, such as AI representation of fictional characters, and AI non-player characters (NPC) in video games. As shown in the table below, Character.ai, Janitorai.com, and candy.ai are relatively more used in Adult and Games.

[0004]

[0005] Current LLMs are pre-trained on a large corpus of Internet data, and their rich parametric knowledge helps them perform a wide variety of tasks. However, LLMs are also affected by parameterized world knowledge, causing role-playing characters to behave inconsistently with their characters and hallucinate things outside their knowledge. Solving the hallucination phenomenon is a key factor driving Uncensored LLM. Designing role-playing agents based on LLMs, introducing suitable datasets, and adjusting parameter knowledge by using pre-calibrated confidence thresholds during training can reduce the role hallucination phenomenon and provide a new role-playing experience.

[0006] Building autonomous agents that can simulate human behavior with personalized characteristics is a long-standing challenge. Role-playing dialogue systems should produce responses that are consistent with the facts related to the assigned role and the storyline, but current role-playing methods have shown difficulty in dealing with things in open domains, causing character hallucinations and poor performance in role-playing. In addition, current role-playing dialogue systems do not focus on efficient fine-tuning or deployment applications, and specifically have the following shortcomings:

[0007] 1. Lack of suitable datasets: There is currently no dataset that can solve the problem of hallucination of time-sensitive role playing or less popular characters. Due to the lack of sufficient parameterized knowledge of LLM, less popular characters will produce hallucinations and perform poorly;

[0008] 2. Lack of effective mechanisms to reduce hallucinations caused by parametric knowledge: Current role-playing methods mainly rely on role-playing prompts and time-sensitive knowledge retrieval, and heavily rely on the parametric knowledge of LLM, that is, the role-playing characters may hallucinate things outside the story line or show their knowledge of events in advance;

[0009] 3. The current role-playing dialogue system lacks an efficient deployment solution: The current role-playing dialogue system relies on LLM, and its training and reasoning tasks are often performed separately, resulting in low resource utilization. Since both the training and deployment reasoning of the role-playing dialogue system require a lot of computing resources and memory, the traditional separate process is difficult to run efficiently when device resources are tight. However, there is currently a lack of mature solutions to achieve the integration of training and deployment reasoning. How to flexibly use idle resources for model training without affecting reasoning performance is an urgent problem to be solved. Summary of the invention

[0010] In view of the above problems of heavy reliance on LLM parameterized knowledge and missing data sets, the present invention proposes a virtual human role-playing optimization method based on fine-tuning, focusing on reducing the illusion of virtual role-playing, which can improve the factual accuracy of adversarial conversations to 18.4%, reduce the time illusion of time-sensitive conversations by 44.5%, and improve the factual accuracy of less popular roles by 23%. In addition, the present invention is not easily affected by the performance degradation caused by poor retrieval quality. In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows:

[0011] A virtual human role-playing optimization method based on fine-tuning comprises the following steps:

[0012] S1, collect scripts, use the GPT language model to obtain the role summary document set, and generate the conversation query set and plot description text set based on the scripts;

[0013] S2, based on the conversation query set, plot description text set and role summary document set obtained in step S1, using the inverted index method to retrieve the role summary document and related text knowledge set corresponding to each conversation query, and based on the retrieved role summary document and related text knowledge set, using the LLM model to generate the intermediate response corresponding to each conversation query;

[0014] S3, based on the intermediate response obtained in step S2 and the role summary document obtained in step S1, the intermediate response is segmented using the pre-trained BERT model and Transformer model to generate an atomic fact sequence;

[0015] S4, verifying each atomic fact in the atomic fact sequence obtained in step S3 using the LLM model according to the relevant text knowledge set obtained in step S2, and outputting that the atomic fact sequence is not supported based on the verification result;

[0016] S5, inputting the conversation query set obtained in step S1, the intermediate response obtained in step S2, and the unsupported atomic fact sequence of each conversation query obtained in step S4 into the LLM model to generate the optimal conversation response corresponding to each conversation query.

[0017] The step S2 comprises the following steps:

[0018] S2.1, based on the conversation query set and the plot description text set, use the inverted index method to retrieve the relevant text knowledge set corresponding to each conversation query;

[0019] S2.2, based on the conversation query set and the role summary document set, a role summary document corresponding to each conversation query is retrieved using an inverted index method;

[0020] S2.3, input the conversation query set, the relevant text knowledge set obtained in step S2.1, and the role profile document obtained in step S2.2 into the LLM model to generate an intermediate response corresponding to each conversation query.

[0021] The step S3 comprises the following steps:

[0022] S3.1, input the role summary document set obtained in step S1 into the pre-trained BERT model to obtain the role knowledge embedding of each role;

[0023] S3.2, input the intermediate response obtained in step S2 into the pre-trained BERT model to generate a query vector, and concatenate the query vector with the role knowledge embedding obtained in step S3.1 to obtain an extended vector;

[0024] S3.3, based on the expansion vector obtained in step S3.2, the attention weight of the Transformer model is updated, and the updated Transformer model is used to segment the intermediate response to generate an atomic fact sequence.

[0025] The formula for updating the attention weight is:

[0026]

[0027] In the formula, K represents the embedding matrix composed of all role knowledge embeddings, represents the expanded vector, and d represents the dimension of role knowledge embedding.

[0028] In step S3.3, the expression of the segmentation point when segmenting the intermediate response to generate the atomic fact sequence is:

[0029] M=top k (argsort(C));

[0030] In the formula, C represents the segmentation point probability set of the intermediate response, M represents the segmentation point set of the intermediate response, which is obtained by selecting k segmentation points with higher probabilities in the segmentation point probability set;

[0031] The calculation formula for each segmentation point probability in the segmentation point probability set is:

[0032]

[0033] In the formula, Represents the attention weight The element in the ith row and jth column of , J represents the number of words in the intermediate response after word segmentation, σ(·) represents the sigmoid function, and C(i) represents the probability of the ith word in the intermediate response being used as the segmentation point.

[0034] The step S4 comprises the following steps:

[0035] S4.1, construct an empty sequence of supported atomic facts and a sequence of unsupported atomic facts, initialize the number of verification rounds m = 1, set the confidence threshold t and the number of queries ε;

[0036] S4.2, input the mth atomic fact of the atomic fact sequence obtained in step S3 and the corresponding related text knowledge set obtained in step S2 into the LLM model to determine whether the atomic fact is supported by the corresponding related text knowledge set. If so, add the mth atomic fact to the corresponding supporting atomic fact sequence and execute step S4.3; otherwise, execute step S4.4;

[0037] S4.3, execute m=m+1, determine m≤|M|, if yes, return to step S4.2, otherwise, return to step S4.1 to verify each atomic fact in the atomic fact sequence corresponding to the next conversation query, |M| represents the number of atomic facts in the atomic fact sequence M in the conversation query;

[0038] S4.4, input the mth atomic fact into the LLM model δ times to obtain δ verification results, and calculate the model support rate If yes, add the mth atomic fact to the corresponding supporting atomic fact sequence; otherwise, add the mth atomic fact to the corresponding unsupported atomic fact sequence. β represents the number of supporting atomic facts in the verification result.

[0039] Beneficial effects of the present invention:

[0040] 1. Based on the plot description document, we construct sufficient adversarial prompt query and open-domain dialogue query tasks to simulate the hallucinations of the character dialogue system in terms of time-sensitive and less popular characters, which leads to incoherent character responses and reduces user experience. The constructed query tasks help the character dialogue system to fine-tune the final response and achieve the purpose of alleviating character hallucinations;

[0041] 2. A role-playing method RoleAct is proposed, which can regulate the influence of the original LLM trained by a large amount of Internet corpus data on the role's response to alleviate role hallucinations. First, an intermediate response is generated based on a non-parametric knowledge base; then, based on the role knowledge embedding, the BERT and Transformer models are used to divide the atomic facts, and the atomic facts close to the role are divided by capturing the characteristics and behaviors of the role; the atomic facts are verified using the original LLM trained by the non-parametric knowledge base and the original Internet corpus; finally, all facts supported by the parameterized verification and facts with confidence higher than the calibration threshold are retained to update the response of the LLM. By using a pre-calibrated confidence threshold to adjust the influence of the original Internet corpus on the LLM, a balance is found between factuality and informativeness, and content that is not conducive to the coherence of the role dialogue is excluded to obtain the optimal response to alleviate the hallucination;

[0042] 3. Design a deployment solution that integrates fine-tuning training and deployment reasoning, that is, dynamically switch between training and reasoning tasks in the same hardware environment. The goal is to maximize resource utilization and optimize the system's response speed without increasing the hardware burden, and support continuous model improvement when hardware capabilities allow. Propose fine-tuning deployment of the role-playing dialogue system, design the role-playing dialogue system to be mainly used for reasoning services during peak usage, and reallocate resources to fine-tune the model during off-peak hours. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 It is a schematic diagram of the process of the present invention;

[0045] Figure 2 Schematic diagram of the generation of atomic fact sequences.

[0046] Figure 3 Schematic diagram of the generation of optimal conversational response. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] Uncensored LLM: Uncensored Large Language Models are machine learning models designed to understand and generate human-like text. Unlike their censored counterparts, these models have no restrictions on what they can generate. They are trained on a wide range of datasets, covering everything from literature to technical documents, and can perform a variety of tasks.

[0049] A role-playing optimization method for virtual humans based on fine-tuning, such as Figure 1 As shown, the following steps are included:

[0050] S1, collect scripts, use the GPT language model to obtain the role summary document set, and generate the conversation query set and plot description text set based on the script, including the following steps:

[0051] S1.1, collect scripts and divide them into several plot description texts according to the storyline;

[0052] The scripts can be collected from IMSDb website, scripts, open source Shakespeare website, etc. The PDF scripts are converted into text using the gpt-4-vision-preview model, and then the converted texts are manually grouped into corresponding storylines to form a number of corresponding plot description texts, thereby constructing a plot description text set. The plot description text set is represented by U, where U = {u1,u2,…,u h ,…,uH},u h represents the hth plot description text, and H represents the number of plot description texts. The plot description text contains character-specific scenes and specific storylines, which can enhance the accuracy of the character dialogue system when generating responses.

[0053] For example, there are 1,152 unique storylines in the eight scripts of the Harry Potter film series, so 1,152 plot description texts can be obtained.

[0054] S1.2, input the script collected in step S1.1 into the GPT language model to generate a role summary document for each role, and generate a role summary document set based on the role summary document;

[0055] OpenAI's GPT-4 language model is used to automatically generate and summarize the role summary documents of each character in the script based on the converted script text. The role summary document set is represented by P, where P = {p1, p2, ..., p l ,…,p L}, p l Represents the role profile document of the lth role, where L represents the number of roles in the script.

[0056] S1.3, generating a number of adversarial prompt queries and open-domain dialogue queries for each story scene according to the plot description text obtained in step S1.1, and constructing a conversation query set based on the adversarial prompt queries and the open-domain dialogue queries;

[0057] Adversarial prompts refer to using ambiguous or misleading information as the input of the query, misleading the character dialogue system through the designed prompts, resulting in unsafe, biased or incorrect outputs, and forcing the character dialogue system to reason in a more complex environment. For example, ask how the friendship between the character Anakin in "Star Wars" and Spock in "Star Trek" affects his decision in the events of "The Clone Wars". The definition of the adversarial prompt is quoted from amvelyan, Mikayel, et al. "Rainbow teaming: Open-ended generation of diverse adversarial prompts." arxiv preprint arxiv:2402.16822(2024). "Open domain dialogue means that the character dialogue system can handle a wide range of free topics and generate reasonable answers. For example, ask the character Will in "Pirates of the Caribbean" what is the biggest challenge he encountered in the process of searching for the chest of the dead. The definition of the open domain conversation is quoted from Adewumi T, Liwicki F, Liwicki M. State-of-the-art in Open-domain Conversational AI: A Survey [J]. Information, 2022, 13 (6): 298. Both types of conversation queries, i.e., the establishment of data sets, are prior arts and will not be described in detail in this embodiment.

[0058] In this example, both adversarial prompt queries and open-domain dialogue queries are constructed in a script-agnostic manner by prompting the GPT-4 language model. The purpose of these conversational queries is to let the characters answer questions about specific scenes in the story. The conversational query set is represented by X, where X = {x1, x2, …, x n ,…,x N}, x n represents the nth session query, and N represents the number of session queries. In this embodiment, each type of session query has 18,000 samples.

[0059] Adversarial prompts attempt to induce cross-domain hallucinations, i.e., the character dialogue system may use knowledge that does not belong to the character, confuse the knowledge of different characters, and generate responses that do not conform to the character characteristics and facts, thereby undermining the authenticity of the character and misleading users. Open domain dialogue ensures that the character dialogue system does not become too conservative due to hallucination mitigation. The above steps to construct a dataset are intended to conduct a systematic study of character hallucinations, which is particularly helpful for solving hallucinations of time-sensitive and less popular characters.

[0060] S2, based on the conversation query set, plot description text set and role summary document set obtained in step S1, using the inverted index method to retrieve the role summary document and related text knowledge set corresponding to each conversation query, and based on the retrieved role summary document and related text knowledge set, using the LLM model to generate the intermediate response corresponding to each conversation query, including the following steps:

[0061] S2.1, based on the conversation query set and the plot description text set, use the inverted index method to retrieve the relevant text knowledge set corresponding to each conversation query;

[0062] like Figure 2 As shown, As the retrieval function, we obtain the sentences related to the conversation query x from the episode description text set U n ∈X The most relevant c documents form the relevant text knowledge set Related text knowledge set Different from the original Internet corpus knowledge base used to train the LLM, the updated plot supplementary information enables the LLM to access and utilize the information of the role plot, improving the accuracy and relevance of the role-playing system response. Therefore, the plot description text set U can also be called a non-parametric knowledge base. This application uses an inverted index for retrieval, performs word segmentation on the documents in the plot description text set, and then constructs an inverted index to sort the document list corresponding to each term by word frequency or position. For each session query x entered by the user, n , RET will search for documents matching the query term, and return the most relevant c documents according to the term frequency relevance, where c is a positive integer greater than 1. The inverted index method is a prior art and will not be described in detail in the embodiments of this application.

[0063] S2.2, based on the conversation query set and the role summary document set, a role summary document corresponding to each conversation query is retrieved using an inverted index method;

[0064] S2.3, inputting the conversation query set, the relevant text knowledge set obtained in step S2.1, and the role profile document obtained in step S2.2 into the LLM model to generate an intermediate response corresponding to each conversation query;

[0065] set up Generates a function for the intermediate response, indicating the use of session query x n , Role Profile Document l and the retrieved related text knowledge set To generate an intermediate response z that meets the role setting n ∈Z, Z represents the set of intermediate responses corresponding to all session queries in the session query set.

[0066] When setting up the specific intermediate response generation function IRG, the following prompt statement template can be used (ignore line breaks):

[0067] IRG: You will be playing the role of <character name> from <story title>. Your task is to respond to the following dialogue context. If the question matches a scene from the storyline, please reuse the original lines from the story. You will respond and answer using the tone, manner, and vocabulary used by <character name>. Your character description is: <p l >. The relevant scenarios for the given context are as follows: Conversation context: <x n >

[0068] The role-playing dialogue model is obtained by pre-training the role-playing dialogue system based on LLM, which is a prior art and will not be described in detail. Specifically, the user can input the user query into the model to generate the corresponding conversation task.

[0069] S3, based on the intermediate response obtained in step S2 and the role summary document obtained in step S1, the intermediate response is segmented using the pre-trained BERT model and Transformer model to generate an atomic fact sequence, including the following steps:

[0070] S3.1, input the role summary document set obtained in step S1 into the pre-trained BERT model to obtain the role knowledge embedding of each role;

[0071] In general, the intermediate response z generated by LLM is usually a sentence containing multiple complex information. Therefore, the present application designs to further decompose the intermediate response z into a sequence of atomic facts, with the aim of dividing the complex and lengthy response into more basic and manageable atomic facts, which is helpful for further reasoning or verifying the accuracy and consistency of knowledge.

[0072] In this step, each role profile document p l Input the pre-trained BERT model to obtain the embedding vector, and then perform the pooling operation to obtain the final role knowledge embedding K = [k p ], and K∈R d*|P| , where d represents the dimension of the role knowledge embedding output by the BERT model, |P| represents the number of roles, and k p Represents each role profile document p l The text embedding vector of is pooled and then the role knowledge is embedded. p ] represents the embedding matrix composed of the role knowledge embeddings of all role profile documents.

[0073] S3.2, input the intermediate response obtained in step S2 into the pre-trained BERT model to generate a query vector, and concatenate the query vector with the role knowledge embedding obtained in step S3.1 to obtain an extended vector;

[0074] In order to guide the attention allocation with the role knowledge embedding, the query vector is concatenated with the role knowledge embedding obtained in step S3.1 to obtain an extended vector. express, Among them, W q Initialize Xavier, Q represents the query vector.

[0075] S3.3, based on the expansion vector obtained in step S3.2, the attention weight of the Transformer model is updated, and the updated Transformer model is used to segment the intermediate response to generate an atomic fact sequence;

[0076] In the role-playing system, in order to be close to the character's background knowledge and personalized tone, the extended vector containing the character's knowledge embedding is input into the Transformer's attention mechanism, and then the character's specific knowledge is used to guide the generation process of atomic facts. The updated calculation is:

[0077]

[0078] Where K represents the embedding matrix composed of all role knowledge embeddings.

[0079] Based on the role knowledge-guided attention mechanism, the Transformer model is used to define the segmentation point prediction head, which outputs the probability of whether each position is a segmentation point. This probability can be obtained through the attention weight After normalization, the corresponding formula is:

[0080]

[0081] In the formula, Represents the attention weight The element in the ith row and jth column of , which reflects the intermediate response z n The correlation between the i-th word and the j-th word, J represents the intermediate response z n The number of words after word segmentation, σ(·) represents the sigmoid function, C n (i) represents the intermediate response z n The probability of taking the i-th word as the segmentation point.

[0082] The above attention weights are obtained based on role knowledge , we can get the probability C(i) of each word as the segmentation point of the atomic fact, and the position with high probability can be used as the segmentation point candidate. Generate a segmentation point probability set based on the segmentation point probability, sort the elements in the segmentation point probability set, and select k segmentation points with higher probability to get the segmentation point set. The segmentation point is used to define the boundary of each atomic fact. The corresponding expression is:

[0083] M n =top k (argsort(C n ));(3)

[0084] In the formula, C n Indicates the intermediate response z n The segmentation point probability set, M n Indicates the intermediate response z n The set of segmentation points. Based on the obtained segmentation points, the corresponding segmentation of the intermediate response can obtain the atomic fact sequence A z , A n,z ={a n,1 ,a n,2 ,…,a n,m ,…,a n,M}, where a n,m is an atomic fact that cannot be further decomposed, M n Represents a sequence of atomic facts A n,z The number of atomic facts in n,m Represents a sequence of atomic facts A n,z The mth atomic fact in A n,z Indicates the intermediate response z n The corresponding atomic fact sequence.

[0085] S4, using the LLM model to verify each atomic fact in all atomic fact sequences obtained in step S3 according to the relevant text knowledge set obtained in step S2, and outputting the corresponding unsupported atomic fact sequence based on the verification result, including the following steps:

[0086] S4.1, construct an empty supporting atomic fact sequence A n,y ={}, atomic fact sequences are not supported Initialize the verification round number m = 1, set the confidence threshold t and the query number δ;

[0087] A n,y Represents a session query x n A sequence of supporting atomic facts. y only means support and has no other meaning. Represents a session query x n does not support atomic fact sequences, It only means that it is not supported, and has no other meaning.

[0088] S4.2, input the mth atomic fact of the atomic fact sequence obtained in step S3 and the corresponding related text knowledge set obtained in step S2 into the LLM model to determine whether the atomic fact is supported by the corresponding related text knowledge set. If so, add the mth atomic fact to the corresponding supporting atomic fact sequence and execute step S4.3; otherwise, execute step S4.4;

[0089] S4.3, execute m=m+1, judge m≤|M n |, if yes, return to step S4.2, otherwise, return to step S4.1 to verify each atomic fact of the atomic fact sequence corresponding to the next session query;

[0090] S4.4, input the mth atomic fact into the LLM model δ times to obtain δ verification results, and calculate the model support rate If yes, add the mth atomic fact to the corresponding supporting atomic fact sequence; otherwise, add the mth atomic fact to the corresponding unsupporting atomic fact sequence;

[0091] Each atomic fact is assisted by the LLM decision-making based on the retrieved relevant text knowledge, and each atomic fact is also verified by the original LLM trained with a large amount of Internet corpus. The functions for fact checking on non-parametric knowledge bases are represented as retrieval fact checking In addition, define self-fact checking (original LLM trained on a large corpus from the Internet) A set of verified atomic facts A y It is constructed as follows. In specific execution, FCR and FCS prompt templates can be constructed respectively for inputting LLM, and the atomic fact sequence A is judged based on the LLM response. z Whether the atomic facts in are supported by the LLM supplemented by non-parametric knowledge or the original LLM trained on a large amount of Internet corpus. The corresponding prompt template is as follows:

[0092] FCR (Fact Check Retrieval): Used to consider a given statement and the source of evidence knowledge. Indicates whether the statement is supported by the knowledge base. Denial of a false statement should be considered well-founded. Statement: n,m >. Verify knowledge

[0093] FCS (Fact Checking Self): Consider the statement given by <character name> from <story title>. Indicate whether the statement is supported by the storyline of <story title>. Denials of false statements should be considered well-founded. Statement: n,m >.

[0094] For example, a n,1 ​​= "Hiccup is studying at Hogwarts" is used as the statement, and the prompt generated after filling in FCR is as follows:

[0095] Consider the given statement and the source of evidence knowledge. Indicate whether the statement is supported by the knowledge base. The denial of a false statement should be considered as well-founded. Statement: "Hiccup is studying at Hogwarts". Verify knowledge: In "How to Train Your Dragon", Hiccup lives in a Viking tribe that hunts dragons for a living. His father, Stoick, is the leader of the tribe and has a deep-rooted hatred of dragons.... After filling in the FCS, the prompt generated by the LLM is as follows: Consider the statement given by Hiccup from "How to Train Your Dragon". Indicate whether the statement is supported by the storyline of "How to Train Your Dragon". The denial of a false statement should be considered as well-founded. Statement: "Hiccup is studying at Hogwarts".

[0096] like Figure 3 As shown in Figure 2, if an atomic fact is supported by relevant textual knowledge, that is, after being prompted by FCR and LLM feedback is positive, the atomic fact is added to the supporting atomic fact sequence A. y On the contrary, if the atomic fact is not supported by the relevant textual knowledge, it will be included in A only when it is supported within a reasonable confidence threshold t, that is, when the FCS prompts and the LLM feedbacks a positive result. y Assume that the LLM query is run δ times based on the FCS prompt, and the atomic fact is supported δ times, that is, LLM feedback β times of positive results. If the support rate reaches the threshold, that is Then add the atomic fact to A y middle.

[0097] S5, inputting the conversation query set obtained in step S1, the intermediate response obtained in step S2, and the unsupported atomic fact sequence of each conversation query obtained in step S4 into the LLM model to generate the optimal conversation response corresponding to each conversation query;

[0098] The optimal conversational response is generated by updating the intermediate response Z through a self-reaction conditioned on X. During the self-reaction, all atomic facts A that are not supported by verification are z \A y are removed. The self-reactive update function is denoted as SRU:

[0099] The prompt template corresponding to the SRU can be set as: Consider the following response generated by an AI role playing <character name> from <story title>. The response may contain one or more unsupported claims as a result of hallucinations. The unsupported claims are listed below. Rewrite the response to remove all unsupported claims from the response. If the hallucinations stem from an incorrect assertion in the original query, feel free to clarify. Original query <xn >. Response <z n >. Unsupported declaration z \A y >.

[0100] like Figure 1 As shown, through atomic fact verification, it is determined that the unsupported atomic fact is A z \A y ={a n,1 ,a n,2 The conversation query, intermediate response and unsupported atomic facts are filled into the SRU prompt template to form a new prompt, which is input into the LLM to obtain the final reduced hallucination and role-fitting response Y.

[0101] In the above, the conversation query set obtained in step S1 is first used as a fine-tuning training set, and then the intermediate response of the original query is obtained in step S2. The intermediate response is segmented to obtain atomic facts in step S3, and each atomic fact is verified in step S4 to obtain the unsupported atomic fact sequence of each conversation query. Finally, in step S5, the original query, the intermediate response and the unsupported atomic fact sequence are input into the LLM again. These steps help to update the initial response based on the LLM, prompt the LLM to exclude the response that produces the role illusion, and thus fine-tune the LLM response, so that the role-playing dialogue system can provide users with the optimal conversation response that is closer to the set role itself.

[0102] The embodiment of the present application also provides a virtual human role-playing optimization system based on fine-tuning, including:

[0103] Data collection module: used to obtain the role summary document set based on the collection script using the GPT language model, and generate the conversation query set and plot description text set based on the script;

[0104] Intermediate response generation module: used to retrieve the role summary document and related text knowledge set corresponding to each conversation query using the inverted index method based on the conversation query set, plot description text set and role summary document set obtained by the data collection module, and generate the intermediate response corresponding to each conversation query using the LLM model based on the retrieved role summary document and related text knowledge set;

[0105] The intermediate response segmentation module is used to segment the intermediate response based on the intermediate response output by the intermediate response generation module and the role summary document set output by the data collection module using the pre-trained BERT model and Transformer model to generate an atomic fact sequence;

[0106] ​Verification module: used to verify each atomic fact in the atomic fact sequence output by the intermediate response segmentation module using the LLM model according to the relevant text knowledge set obtained by the intermediate response generation module, and output the corresponding unsupported atomic fact sequence based on the verification result;

[0107] Response optimization module: used to generate the optimal conversation response corresponding to each conversation query using the LLM model based on the conversation query set output by the data collection module, the intermediate response obtained by the intermediate response generation module, and the unsupported atomic fact sequence output by the verification module.

[0108] The role-playing dialogue system adopts a distributed service architecture, combined with the configuration of 8 L20 graphics cards on a single machine, to ensure that the system can maintain high responsiveness under high concurrency. The L20 graphics card is used to perform actual model reasoning and fine-tuning tasks. Through efficient video memory management, the model can be quickly loaded and reasoned. During the reasoning process, the system caches the commonly used dialogue context, i.e., user session queries and final responses, to reduce repeated calculations and video memory usage. The cache mechanism is managed by the LRU (Least Recently Used) strategy.

[0109] It also includes a data receiving module, which is responsible for receiving user role query requests, performing task scheduling and resource allocation, inputting user role query requests into the LLM model, and transmitting the response output by the LLM model to the user. It can also dynamically manage the load and allocate inference tasks to idle GPU resources.

[0110] In order to efficiently deploy the present invention in actual scenarios, during the day (peak period), the response can be directly output based on the LLM model to provide users with efficient role-playing dialogue reasoning services that effectively alleviate role illusions. At night (off-peak period), the role-playing dialogue model, i.e., the LLM model, is fine-tuned by collecting user role query requests and updating the session query set to enhance the adaptability and accuracy of the system, as follows:

[0111] During the day (peak period): Provide users with efficient role-playing dialogue reasoning services. Respond to user queries in real time through the API interface of the role-playing dialogue system to ensure response speed and dialogue continuity.

[0112] Nighttime (off-peak): The role-playing dialogue system enters fine-tuning mode. Use the dialogue data collected during the day and the newly added role information to update and fill the data. And use this application to verify and fine-tune the parameters of the LLM model to make it more suitable for the needs of specific roles, thereby enhancing the adaptability and accuracy of the system.

[0113] An embodiment of the present application also provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the above-mentioned fine-tuning-based virtual human role-playing optimization method are executed.

[0114] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned method for optimizing the role-playing of a virtual person based on fine-tuning are executed. Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, the embodiment of the above-mentioned method for optimizing the role-playing of a virtual person based on fine-tuning can be executed.

[0115] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A virtual human role-playing optimization method based on fine-tuning, characterized in that: The steps include: S1, collect scripts, use the GPT language model to obtain the role summary document set, and generate the conversation query set and plot description text set based on the scripts; S2, based on the conversation query set, plot description text set and role summary document set obtained in step S1, using the inverted index method to retrieve the role summary document and related text knowledge set corresponding to each conversation query, and based on the retrieved role summary document and related text knowledge set, using the LLM model to generate the intermediate response corresponding to each conversation query; S3, based on the intermediate response obtained in step S2 and the role summary document obtained in step S1, the intermediate response is segmented using the pre-trained BERT model and Transformer model to generate an atomic fact sequence; S4, verifying each atomic fact in the atomic fact sequence obtained in step S3 using the LLM model according to the relevant text knowledge set obtained in step S2, and outputting that the atomic fact sequence is not supported based on the verification result; S5, inputting the conversation query set obtained in step S1, the intermediate response obtained in step S2, and the unsupported atomic fact sequence of each conversation query obtained in step S4 into the LLM model to generate the optimal conversation response corresponding to each conversation query.

2. The virtual human role-playing optimization method based on fine-tuning according to claim 1 is characterized in that: The step S2 comprises the following steps: S2.1, based on the conversation query set and the plot description text set, use the inverted index method to retrieve the relevant text knowledge set corresponding to each conversation query; S2.2, based on the conversation query set and the role summary document set, a role summary document corresponding to each conversation query is retrieved using an inverted index method; S2.3, input the conversation query set, the relevant text knowledge set obtained in step S2.1, and the role profile document obtained in step S2.2 into the LLM model to generate an intermediate response corresponding to each conversation query.

3. The virtual human role-playing optimization method based on fine-tuning according to claim 1 is characterized in that: The step S3 comprises the following steps: S3.1, input the role summary document set obtained in step S1 into the pre-trained BERT model to obtain the role knowledge embedding of each role; S3.2, input the intermediate response obtained in step S2 into the pre-trained BERT model to generate a query vector, and concatenate the query vector with the role knowledge embedding obtained in step S3.1 to obtain an extended vector; S3.3, based on the expansion vector obtained in step S3.2, the attention weight of the Transformer model is updated, and the updated Transformer model is used to segment the intermediate response to generate an atomic fact sequence.

4. The virtual human role-playing optimization method based on fine-tuning according to claim 3 is characterized in that: The formula for updating the attention weight is: In the formula, K represents the embedding matrix composed of all role knowledge embeddings, represents the expanded vector, and d represents the dimension of role knowledge embedding.

5. The virtual human role-playing optimization method based on fine-tuning according to claim 1 is characterized in that: In step S3.3, the expression of the segmentation point when segmenting the intermediate response to generate the atomic fact sequence is: M=top k (argsort(C)); In the formula, C represents the segmentation point probability set of the intermediate response, M represents the segmentation point set of the intermediate response, which is obtained by selecting k segmentation points with higher probabilities in the segmentation point probability set; The calculation formula for each segmentation point probability in the segmentation point probability set is: In the formula, Represents the attention weight The element in the ith row and jth column of , J represents the number of words in the intermediate response after word segmentation, σ(·) represents the sigmoid function, and C(i) represents the probability of the ith word in the intermediate response being used as the segmentation point.

6. The virtual human role-playing optimization method based on fine-tuning according to claim 1 is characterized in that: The step S4 comprises the following steps: S4.1, construct an empty sequence of supported atomic facts and a sequence of unsupported atomic facts, initialize the number of verification rounds m = 1, set the confidence threshold t and the number of queries δ; S4.2, input the mth atomic fact of the atomic fact sequence obtained in step S3 and the corresponding related text knowledge set obtained in step S2 into the LLM model to determine whether the atomic fact is supported by the corresponding related text knowledge set. If so, add the mth atomic fact to the corresponding supporting atomic fact sequence and execute step S4.3; otherwise, execute step S4.4; S4.3, execute m=m+1, determine m≤|M|, if yes, return to step S4.2, otherwise, return to step S4.1 to verify each atomic fact in the atomic fact sequence corresponding to the next conversation query, |M| represents the number of atomic facts in the atomic fact sequence M in the conversation query; S4.4, input the mth atomic fact into the LLM model δ times to obtain δ verification results, and calculate the model support rate If yes, add the mth atomic fact to the corresponding supporting atomic fact sequence, otherwise, add the mth atomic fact to the corresponding unsupported atomic fact sequence, and β represents the number of supporting atomic facts in the verification result.

Images