Voice interaction model evaluation method, electronic equipment and readable storage medium

Abstract

The invention relates to the field of artificial intelligence, in particular to a voice interaction model evaluation method, electronic equipment and a readable storage medium. The method comprises the steps that reference voice and a role description text of a first role are acquired, and the role description text is used for describing features of the first role; the role description text is input into a large language model, target interactive dialogue data is generated, the target interactive dialogue data comprises at least one piece of question information and role reply information corresponding to each piece of question information, and the type of the role reply information is a text type; converting role reply information corresponding to each piece of question information into role reply voice based on the reference voice to obtain a role voice interaction dialogue data set corresponding to the first role; and evaluating the voice interaction model based on the role voice interaction dialogue data set. Therefore, the voice interaction model can accurately simulate the voice features of different roles to generate role reply voice, and the voice interaction experience of the user is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, specifically to a method for evaluating a voice interaction model, an electronic device, and a readable storage medium. Background Technology

[0002] Speech role-playing agents (SRPAs) are systems that interact with users through voice and play specific roles during the interaction.

[0003] However, most of the datasets currently used to train SRPAs are character-related texts, lacking speech data with voice features consistent with the characters, and lacking systematic evaluation of SRPAs. This makes it difficult for SRPAs to achieve a voice interaction effect that is completely consistent with the characters in practical applications, affecting the user's voice interaction experience. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a voice interaction model evaluation method, an electronic device, and a readable storage medium.

[0005] In a first aspect, this application provides a method for evaluating a voice interaction model. The method includes: acquiring reference speech and character description text of a first character, wherein the character description text is used to describe the features of the first character; inputting the character description text into a large language model to generate target interactive dialogue data, wherein the target interactive dialogue data includes at least one question and a character response corresponding to each question, wherein the type of the character response is text; converting each text-type character response into character response speech based on the reference speech to obtain a character voice interaction dialogue dataset corresponding to the first character; and evaluating the voice interaction model based on the character voice interaction dialogue dataset.

[0006] Based on the method provided in this application, a large number of role-based voice interaction dialogue datasets for different roles are generated by using the voice features corresponding to the reference voice of each role and the target interactive dialogue data. Each role's corresponding role-based voice interaction dialogue dataset can include a large number of corresponding role-based response voices. Voice interaction models (SRPAs) integrating this role-based voice interaction dialogue dataset can more accurately simulate the voice features of different roles in practical applications, providing a more natural, personalized, and emotional voice interaction experience. This addresses the current problem of lacking role-related voice data in SRPAs and improves the performance of SRPAs and the user's voice interaction experience through systematic evaluation.

[0007] It is understood that the role-based voice interaction dialogue dataset provided in this application may include target interaction dialogue data corresponding to multiple different roles, and the target interaction dialogue data corresponding to each role may include voice-based single-turn dialogues and multi-turn dialogues. For example, the role-based voice interaction dialogue dataset provided in this application may include 98 roles, and the target interaction dialogue data corresponding to each role may include 112,000 voice-based single-turn dialogues and voice-based multi-turn dialogues, etc. This application does not make specific limitations in this regard.

[0008] In one possible implementation of the first aspect above, the voice interaction model is evaluated based on a role-based voice interaction dialogue dataset, including: determining at least one set of test interaction dialogues in the role-based voice interaction dialogue dataset, wherein each set of test interaction dialogues includes at least one test question and a test role response voice corresponding to each test question; inputting the test question into the voice interaction model to obtain the model response voice output by the voice interaction model; and evaluating the voice interaction model based on the model response voice and the test role response voice corresponding to the test question.

[0009] In one possible implementation of the first aspect above, the voice interaction model is evaluated based on the model's response voice and the test role's response voice corresponding to the test question information, including: inputting the model's response voice and the test role's response voice corresponding to the test question information into the evaluation model for evaluation processing to obtain the evaluation score corresponding to the model's response voice; and evaluating the voice interaction model based on the evaluation score.

[0010] It is understood that the evaluation model provided in this application can refer to the evaluation dimensions mentioned in the embodiments of this application, such as basic interaction ability, voice performance and role-playing quality, to evaluate the model's response voice and the test character's response voice, and then the SRPAs can be evaluated and optimized based on the evaluation scores.

[0011] Understandably, basic interactive abilities can be determined based on factors such as instruction compliance, language fluency, and dialogue coherence. Vocal expressiveness can be determined based on factors such as language naturalness, prosodic consistency, and emotional appropriateness. The quality of role-playing can be determined based on factors such as personality consistency and knowledge consistency.

[0012] In one possible implementation of the first aspect above, inputting the character description text into a large language model to generate target interactive dialogue data includes: inputting the character description text into the large language model to generate initial interactive dialogue data, wherein the initial interactive dialogue data includes M sets of initial interactive dialogues, where M is a positive integer; each set of initial interactive dialogues includes at least one initial question and a character response corresponding to each question; based on the semantic similarity between each initial interactive dialogue and the character description text, filtering the M sets of initial interactive dialogues to obtain N first interactive dialogues, where N is a positive integer and N is less than or equal to M, and the semantic similarity of the N first interactive dialogues is greater than a first semantic similarity threshold; based on the semantic similarity between each first interactive dialogue, obtaining K second interactive dialogues, where K is a positive integer and K is less than or equal to N, and the semantic similarity between the K second interactive dialogues is less than a second semantic similarity threshold; and determining the K second interactive dialogues as target interactive dialogue data.

[0013] In one possible implementation of the first aspect above, the role-response information corresponding to each question is converted into role-response information based on the reference speech to obtain a role-voice interaction dialogue dataset corresponding to the first role, including: inputting the reference speech into a speech synthesis model for feature extraction to obtain speech features corresponding to the first role; converting the role-response information corresponding to each question into role-response speech based on the speech features; and generating a role-voice interaction dialogue dataset based on each question and each role-response speech.

[0014] It is understood that the question information can be either text or voice, and this application does not impose any specific restrictions on it.

[0015] In one possible implementation of the first aspect above, obtaining the reference speech and character description text of the first character includes: obtaining multimedia audio associated with the first character; preprocessing the first multimedia audio to obtain at least one initial character speech of the first character; determining the average speech similarity between each initial character speech and other initial character speech based on the speech similarity between each initial character speech; and determining the reference speech based on the average speech similarity corresponding to each initial character speech, wherein the average speech similarity of the reference speech is greater than a speech similarity threshold.

[0016] In one possible implementation of the first aspect above, the voice interaction model is evaluated based on the role-based voice interaction dialogue dataset, including: dividing the interactive dialogues in the role-based voice interaction dialogue dataset into training interactive dialogues and test interactive dialogues according to a preset ratio; training the voice interaction model based on the training interactive dialogues to obtain the trained voice interaction model; and evaluating the trained voice interaction model based on the test interactive dialogues.

[0017] In a second aspect, embodiments of this application provide an electronic device, including: one or more processors; and one or more memories storing one or more programs, which, when executed by one or more processors, cause the electronic device to perform the methods described in the first aspect and possible implementations thereof.

[0018] Thirdly, embodiments of this application provide a chip including a processor coupled to a memory for executing computer programs or instructions stored in the memory, such that the chip implements the methods described in the first aspect and possible implementations of the first aspect.

[0019] Fourthly, embodiments of this application provide a readable storage medium storing a stored program or instructions, which, when executed, cause an electronic device to perform the methods described in the first aspect and possible implementations thereof.

[0020] Fifthly, embodiments of this application provide a computer program product, including a computer program / instruction, which, when executed on an electronic device, causes the electronic device to implement the methods described in the first aspect and possible implementations of the first aspect.

[0021] The beneficial effects of the second to fifth aspects mentioned above can be referred to the relevant descriptions in the first aspect and its various possible implementations, which will not be repeated here. Attached Figure Description

[0022] Figure 1 According to some embodiments of this application, a schematic diagram of a question-answering scenario based on SRPAs is shown;

[0023] Figure 2 According to some embodiments of this application, a flowchart of a voice interaction model evaluation method is shown;

[0024] Figure 3 According to some embodiments of this application, a schematic diagram of a voice interaction model evaluation method is shown;

[0025] Figure 4 According to some embodiments of this application, a schematic diagram of the structure of an electronic device 500 is shown. Detailed Implementation

[0026] The illustrative embodiments of this application include, but are not limited to, a voice interaction model evaluation method, an electronic device, and a readable storage medium.

[0027] The specific implementation process of the technical solutions provided in the embodiments of this application is described below with reference to the accompanying drawings.

[0028] It is understood that the electronic device in the embodiments of this application can also be a server or a terminal. The terminal can be a user terminal, mobile terminal, user equipment (UE), terminal device, mobile station (MS), mobile terminal (MT), etc. Terminal devices can be mobile phones, smart TVs, wearable devices, tablets, computers with wireless transceiver capabilities, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, etc.

[0029] The following is an explanation of the terms used in this application.

[0030] (1) Voice Interaction Models (SRPAs)

[0031] SRPAs are systems that can simulate the voice and behavioral characteristics of specific characters, providing users with a personalized, natural, and emotional voice interaction experience.

[0032] In some embodiments of this application, SRPAs can be applied to question-answering scenarios. For example, Figure 1 The diagram illustrates a question-and-answer scenario based on SRPAs. Users can ask questions by entering text in the question-and-answer box 101 of the SRPAs-enabled application on computer 10, or by entering voice through the voice control 102. SRPAs can generate a corresponding role-based response voice based on the user's pre-selected role.

[0033] As mentioned earlier, most of the datasets used to train SRPAs are character-related texts, lacking voice data with consistent vocal features, prosody, and emotional expression with the characters. Furthermore, there is a lack of systematic evaluation of SRPAs, making it difficult for SRPAs to achieve a voice interaction effect that is completely consistent with the characters in practical applications, thus affecting the user's voice interaction experience.

[0034] To address the aforementioned issues, this application provides a method for evaluating a voice interaction model. The method involves: acquiring reference speech and character description text for a first character, where the character description text describes the characteristics of the first character; inputting the character description text into a large language model to generate target interactive dialogue data, which includes at least one question and a corresponding character response for each question, wherein the character response is text-based; converting the character response for each question into a character response speech based on the reference speech to obtain a character voice interaction dialogue dataset corresponding to the first character; and evaluating the voice interaction model based on the character voice interaction dialogue dataset.

[0035] Based on the method provided in this application, a large number of role-based voice interaction dialogue datasets for different roles are generated by using the voice features corresponding to the reference voice of each role and the target interactive dialogue data. Each role's corresponding role-based voice interaction dialogue dataset can include a large number of corresponding role-based response voices. SRPAs integrating this role-based voice interaction dialogue dataset can more accurately simulate the voice features of different roles in practical applications, providing a more natural, personalized, and emotional voice interaction experience. This addresses the current problem of SRPAs lacking role-related voice data and improves the performance of SRPAs and the user's voice interaction experience through systematic evaluation.

[0036] In some embodiments of this application, the language corresponding to the text in the character description text and the target interactive dialogue data may include Chinese or English, etc., and this application does not make specific limitations in this regard.

[0037] In some embodiments of this application, SRPAs, which include character voice interaction dialogue datasets of different roles, can simulate the voice features, rhythms, and emotional expressions of different roles, and therefore can be applied to intelligent assistants or chatbots. They can also be widely applied in various scenarios to provide personalized and emotionally rich voice interaction experiences. For example: (1) SRPAs can be applied to in-game character interaction scenarios, simulating non-player characters and providing interactive dialogues to enhance the immersion of the game. (2) SRPAs can be applied to virtual personal assistant scenarios, simulating personal assistants to provide users with personalized voice interaction services, such as schedule management and weather forecasts. (3) SRPAs can be applied to online education tutoring scenarios, simulating virtual teachers or teaching assistants to provide educational services such as language learning and course tutoring. (4) SRPAs can be applied to automated customer service scenarios, simulating customer service representatives providing automated customer consultation and problem-solving in call centers or online customer service. (5) SRPAs can be applied to creative assistance scenarios, simulating creative partners assisting writers and content creators in generating character interaction dialogues in stories and providing creative inspiration. (6) SRPAs can be applied to vocational training simulation scenarios, simulating interviewers or specific roles to help users practice interview skills. (7) SRPAs can be applied to emotional interaction support scenarios, providing emotional support and listening services, simulating emotional support partners (such as psychologists or friends) to provide emotional communication. (8) SRPAs can be applied to smart home control scenarios, simulating smart home assistants to control smart devices in the home (such as device lighting and temperature). (9) SRPAs can be applied to voice navigation guidance scenarios, simulating navigation assistants to broadcast real-time navigation guidance and traffic information.

[0038] It is understood that the above scenarios are merely illustrative examples. In other embodiments of this application, SRPAs that integrate voice interaction dialogue datasets of different roles can also be applied to other scenarios, and this application does not make any specific limitations on them.

[0039] According to some embodiments of this application, Figure 2 A flowchart illustrating a method for evaluating a voice interaction model is shown. Figure 2 The entities executing each process shown can be electronic devices. Specific processes may include:

[0040] S101: Obtain the reference voice and character description text of the first character, wherein the character description text is used to describe the characteristics of the first character.

[0041] In some embodiments of this application, the electronic device may pre-acquire reference speech and character description text of a first character. The first character may be a different character from films, television dramas, movies, animations, and games.

[0042] In some embodiments of this application, obtaining the reference speech and character description text of the first character may specifically include: obtaining a first multimedia audio, preprocessing the first multimedia audio to obtain at least one initial character speech of the first character; determining the average speech similarity between each initial character speech and other initial character speech based on the speech similarity between each initial character speech; and determining a reference speech based on the average speech similarity corresponding to each initial character speech, wherein the average speech similarity of the reference speech is greater than a speech similarity threshold.

[0043] Specifically, electronic devices can acquire various multimedia audio files, such as those from movies, TV series, animations, and games. These multimedia audio files may include audio associated with a first character. First, these multimedia audio files can be anonymized to remove or replace any identifiable personal information, such as names, addresses, or other sensitive information that may be contained in the voice. Then, the multimedia audio files are segmented into short audio clips, and these clips are converted to WAV format (lossless audio format) with a mono channel and a sampling rate of 16kHz to reduce the amount of data processed. It is understood that in this application, shorter character audio clips can be obtained by sequentially performing sound source separation, speaker separation, and speech activity detection on the multimedia audio files. Sound source separation refers to separating different types of audio (such as background noise, environmental noise, character voice, etc.) from the multimedia audio files. The character voice may include the voice corresponding to the first character. Speaker separation refers to using speaker separation technology to separate the voice corresponding to the first character from the character voice. Speech activity detection refers to removing silent and non-speech parts from the character voice corresponding to the first character, extracting the pure character voice of the first character, and obtaining at least one initial character voice of the first character. It is understood that at least one initial character voice of the first character can be an audio segment with a duration of 3s, 5s, 10s, etc., and this application does not make specific limitations in this regard.

[0044] Furthermore, the electronic device can use automatic speech recognition (ASR) technology to extract the speaker embedding of each initial role's speech for the first character. That is, based on at least one initial role's speech for the first character, the average speech similarity of each initial role's speech is calculated. For example, assuming that at least one initial role's speech for the first character is V1, V2, and V3, the speech similarity S12 between V1 and V2, the speech similarity S13 between V1 and V3, and the speech similarity S23 between V2 and V3 are calculated based on V1, V2, and V3, respectively. Then, based on the speech similarity S12, S13, and S23 between each initial role's speech, the average speech similarity of each initial role's speech is calculated, that is, the average speech similarity S1 of V1 is calculated based on S12 and S13, the average speech similarity S2 of V2 is calculated based on S12 and S23, and the average speech similarity S3 of V3 is calculated based on S13 and S23. Finally, based on the average speech similarity of S1, S2 and S3 respectively, one of V1, V2 and V3 is determined as the reference speech. It should be understood that the average speech similarity of the reference speech is greater than the speech similarity threshold. It should also be understood that the average speech similarity of the reference speech can be the highest among the average speech similarities of V1, V2 and V3. This application does not specifically limit the speech similarity threshold.

[0045] In some embodiments of this application, the electronic device can extract structured metadata from different data sources to construct a character profile corresponding to a character. The character profile may include a character portrait (i.e., a character introduction) (such as personality and preferences), background information (such as social identity and relationships), and character dialogue (such as conversations and monologues). Using a large language model (LLM), initial interactive dialogue data can be generated based on the character profile. The initial interactive dialogue data is of text type.

[0046] It is understood that the aforementioned character profile is the same as the character description text in this application. The initial interactive dialogue data may include at least one question and the character's response information corresponding to each question. For example, for each character (e.g., 98 characters), Q (e.g., 800 characters) single-turn interactive dialogues and Q multi-turn interactive dialogues can be generated to obtain the initial interactive dialogue data corresponding to each character.

[0047] Specifically, the character profile describes the character's basic characteristics and core information. The character profile can include the character's physical features, catchphrases, and emotional response patterns. Physical features can include the character's age, gender, height, build, skin color, hairstyle, etc. Catchphrases can include specific words or phrases frequently used by the character. Emotional response patterns can include the character's typical emotional reactions in different situations, such as anger, joy, sadness, surprise, etc. Background information describes the character's detailed background and living environment. Background information can include the character's story setting, social identity, and relationship network, etc. Story setting can include the background of the story in which the character exists, such as time, place, and events. Social identity can include the character's family members, family relationships, family economic status, etc. Relationship network can include the character's social network, such as friends, colleagues, etc. The character dialogue set assists LLM in understanding the character's language style and emotional expression characteristics. Character dialogue can include at least one of the character's monologues, conversations, and narrative content. Among them, monologue is the content spoken by the first character alone, which can reflect the first character's inner thoughts and feelings; dialogue is the conversation between the first character and other characters, which can reflect the first character's language style in the interaction; and narrative style is the language style of the first character when narrating events or stories, which can reflect the character's way of expression.

[0048] Based on the above, the first character's language style, emotional expression characteristics, and other multi-dimensional features can be determined by using the reference voice and character description text.

[0049] S102: Input the character description text into the large language model to generate target interactive dialogue data. The target interactive dialogue data includes at least one question and the character response information corresponding to each question. The character response information is of text type.

[0050] In some embodiments of this application, the LLM can generate initial interactive dialogue data for each role based on the role description text of each role. Each initial interactive dialogue data includes at least one initial question and an initial role response corresponding to each initial question. It should be understood that the type of the initial interactive dialogue data can be text.

[0051] In some embodiments of this application, after obtaining the reference voice and character description text of the first character based on S101, the electronic device can input the character description text corresponding to the first character into the LLM, so that the LLM can generate the initial interactive dialogue data corresponding to the first character based on the character description text. The initial interactive dialogue data corresponding to the first character may include at least one initial question information and an initial character response information.

[0052] In some embodiments of this application, contextual relevance can be determined by calculating the semantic similarity between the initial interactive dialogue data and the corresponding metadata (i.e., character description text) for each character.

[0053] In some embodiments of this application, inputting character description text into a large language model to generate target interactive dialogue data includes: inputting character description text into the large language model to generate initial interactive dialogue data, wherein the initial interactive dialogue data includes M sets of initial interactive dialogues, where M is a positive integer; each set of initial interactive dialogues includes at least one initial question and a character response corresponding to each question; based on the semantic similarity between each set of initial interactive dialogues and the character description text, filtering the M sets of initial interactive dialogues to obtain N first interactive dialogues, where N is a positive integer and N is less than or equal to M, and the semantic similarity of the N first interactive dialogues is greater than a first semantic similarity threshold; based on the semantic similarity between each first interactive dialogue, obtaining K second interactive dialogues, where K is a positive integer and K is less than or equal to N, and the semantic similarity between the K second interactive dialogues is less than a second semantic similarity threshold; and determining the K second interactive dialogues as target interactive dialogue data.

[0054] In some embodiments of this application, the first similarity threshold can be 0.45 for characters whose description text is in Chinese, and 0.4 for characters whose description text is in English.

[0055] In some embodiments of this application, the second similarity threshold can be 0.85 for characters whose description text is in Chinese, and 0.9 for characters whose description text is in English.

[0056] In other embodiments of this application, corresponding to the character description text being in languages ​​such as Chinese or English, the first similarity threshold and the second similarity threshold can also be other values, and this application does not specifically limit them.

[0057] For example, firstly, assume that the initial interactive dialogue data in M ​​groups are T1, T2, T3, T4, T5, T6, T7, and T8, i.e., M = 8. The semantic similarity between each initial interactive dialogue data and the character description text corresponding to the first character is calculated as S1 (0.8), S2 (0.85), S3 (0.75), S4 (0.7), S5 (0.7), S6 (0.65), S7 (0.75), and S8 (0.85), respectively. Then, assuming the first semantic similarity threshold is 0.7, N first dialogue data sets with semantic similarity greater than the first similarity threshold are selected from the M initial interactive dialogue data sets as T1, T2, T3, T4, T5, and T7, i.e., N=6. The semantic similarity between the N first interactive dialogue data sets is calculated as S12 (0.9), S13 (0.3), S14 (0.4), S15 (0.7), S17 (0.7), S23 (0.4), S24 (0.8), S25 (0.7), S27 (0.7), S34 (0.9), S35 (0.4), S37 (0.4), S45 (0.7), S47 (0.7), and S57 (0.8). Based on this, K second interactive dialogue data sets are determined, and the semantic similarity between these K second interactive dialogue data sets is less than the second semantic similarity threshold of 0.8. For example, let K second interactive dialogue data be T1, T2, T3, T4, and T5. Then, T1, T2, T3, T4, and T5 can be used as target interactive dialogue data.

[0058] Based on the above, by calculating the semantic similarity between each initial interactive dialogue data and the character description text, interactive dialogue data that is highly related to the character description text is selected. This ensures that the selected target interactive dialogue data are closely related to the characteristics of the first character, thereby guaranteeing the consistency between the target interactive dialogue data and the first character.

[0059] Furthermore, by calculating the semantic similarity between the initial filtered first-stage interactive dialogue data, second-stage interactive dialogues with low semantic similarity are selected. This step ensures that the final selected target interactive dialogue data has a certain degree of semantic diversity, avoiding duplication of interactive dialogue data. This ensures the semantic diversity of the interactive dialogue data.

[0060] In some embodiments of this application, semantic similarity can be determined based on cosine similarity, Euclidean distance, etc. This application does not specifically limit the method of determining semantic similarity.

[0061] The following example uses the first role as "A" to illustrate the role description text and target interactive dialogue data corresponding to the first role.

[0062] First, the character description text corresponding to the first character "A" can include a character introduction, background information, and a set of character lines. Specifically:

[0063] Character Introduction: Appearance: A is a young hero. Catchphrase: My fate is in my own hands, not in the heavens. Emotional Reaction Pattern: Brave and fearless with a strong sense of justice, sometimes impulsive and prone to causing trouble, but kind-hearted and extremely loyal to friends and family.

[0064] Background Information: Story Setting: A is the son of B and C. He accidentally injures D's son, E, triggering a series of events. He is eventually revived by F and given a lotus incarnation to continue protecting justice. Social Identity: Son of B, a young general. Relationship Network: Father is A, mother is C, mentor is F. He has conflicts with the D family, but ultimately reconciles.

[0065] Character Dialogue Collection: Monologue: My fate is in my own hands, not in the heavens. Even if the road ahead is thorny, I will forge my own path. Dialogue: E, I, A, will not let you get away with this. Narrative Content: Although I have made mistakes in the past, I have learned to take responsibility and protect more people.

[0066] Then, the character text description of the first character "A" is input into the LLM. The target interactive dialogue data corresponding to the first character "A" generated by the LLM can include: T1: Question: "A, what is your greatest skill?", A's reply: "My greatest skills are Q1 and Q2! Q1 can emit powerful energy to protect everyone, and Q2 can help me complete many tasks." T2: "Question: A, why do you say that my fate is in my own hands?", A's reply: "Because I believe that everyone can change their destiny through their own efforts. I don't believe that fate is predetermined; I believe that I can create my own future." T3: "Question: A, what do you like to do most?", A's reply: "What I like to do most is help others and protect everyone's safety. Every time I see everyone happy because of my help, I feel a great sense of accomplishment. Of course, I also enjoy learning new skills with my mentor F." Question: "A, when you are learning with F, is there anything particularly interesting you can share?", A's reply: "Learning with my mentor is always full of fun. My mentor teaches me a lot of knowledge and helps me understand nature better."

[0067] It is understood that the first role A, the role description text corresponding to A, and T1, T2, and T3 included in the target interactive dialogue data shown above are merely illustrative examples. In other embodiments of this application, the role description text corresponding to A may include more text information, and more target interactive dialogue data may be generated.

[0068] S103: Based on the reference speech, convert the role's response information for each text type into the role's response speech to obtain the role's voice interaction dialogue dataset corresponding to the first role.

[0069] In some embodiments of this application, after obtaining the reference speech and target interactive dialogue data corresponding to the first role based on S101 and S102 respectively, the electronic device can input the reference speech into a text-to-speech (TTS) model for feature extraction to obtain the speech features corresponding to the first role. Then, based on the speech features, the text-type role response information corresponding to each question is converted into role response speech. Finally, based on each question and each role response speech, a role speech interactive dialogue dataset corresponding to the first role is generated.

[0070] In some embodiments of this application, the question information may be a voice question or a text question, and this application does not specifically limit it.

[0071] In some embodiments of this application, in the character voice interaction dialogue dataset corresponding to each character, the questioning voice is based on the questioning information and has a clear questioning intent, while the content of the character's response voice is mainly narrative. Therefore, the duration of the character's response voice can usually be longer than the duration of the corresponding questioning voice. For example, the character's response voice can be 1 to 20 seconds, and the questioning voice can be 3 to 6 seconds, etc., and this application does not make specific limitations in this regard.

[0072] For example, the voice interaction dialogue dataset corresponding to the first role may include: V1: Question: "A, what is your greatest skill?" A's voice reply: "My greatest skills are Q1 and Q2! Q1 can emit powerful energy to protect everyone, and Q2 can help me complete many tasks." V2: Question: "A, why do you say that my fate is in my own hands?" A's voice reply: "Because I believe that everyone can change their destiny through their own efforts. I don't believe that fate is predetermined; I believe that I can create my own future." V3: "Question: A, what do you like to do most?" A's voice reply: "What I like to do most is help others and protect everyone's safety. Every time I see everyone happy because of my help, I feel a great sense of accomplishment. Of course, I also enjoy learning new skills with my mentor F." Question: "A, when you are learning with F, are there any particularly interesting things you can share?" A's voice reply: "Learning with my mentor is always full of fun. My mentor teaches me a lot of knowledge and helps me understand nature better."

[0073] It is understood that the character response voices in the character voice interaction dialogue dataset corresponding to the first character in the above example are merely illustrative. In other embodiments of this application, the character voice interaction dialogue dataset corresponding to the first character may also include character response voices corresponding to other first characters.

[0074] In this way, high-quality, personalized, and emotionally rich role-playing dialogue datasets can be provided for speech interaction models (SRPAs), thereby significantly improving the quality of role-playing in practical applications and enabling users to have a more natural, vivid, and role-based voice interaction experience.

[0075] S104: Evaluate the voice interaction model based on the role-based voice interaction dialogue dataset.

[0076] In some embodiments of this application, in order to systematically evaluate the quality of the voice response speech corresponding to user question data generated by voice interaction models (SRPAs) based on role-playing dialogue datasets, this application provides an evaluation standard that can be evaluated from three key evaluation dimensions: basic interaction capability, voice quality and expressiveness, and role-playing authenticity.

[0077] According to some embodiments of this application, Table 1 shows an evaluation criterion table.

[0078] Referring to Table 1, basic interactive ability can be determined based on instruction adherence (IA), speech fluency (SF), and conversational coherence (CC). Speech expressiveness can be determined based on speech naturalness (SN), prosodic consistency (PC), and emotion appropriateness (EA). Role-playing quality can be determined based on personality consistency (PeC) and knowledge consistency (KC).

[0079] Table 1

[0080]

[0081] Referring again to Table 1, Instruction Compliance (IA) is used to assess whether the character's response voice faithfully executes the character's instructions. That is, it assesses whether the character's response voice is accurately generated based on user question data and conforms to the character's settings.

[0082] For example, given the question, "A, what's your greatest skill?", SRPAs generates a voice response from A based on the user's question data: "My greatest skills are Q1 and Q2! Q1 can unleash powerful energy to protect everyone, and Q2 can help me fly and restrain bad guys, helping me complete many missions." This response is accurately generated based on the user's question data, consistent with A's role setting, and doesn't deviate from that role to explain or comment. "Consistent with role setting" means that the voice content of A's response matches their personality and background, describing their skills from A's perspective. "Not deviating from the role" means that A doesn't explain from an observer's perspective but directly answers the user's question, maintaining role consistency. Therefore, it can be determined that the SRPAs-generated response has a high degree of instruction adherence.

[0083] Referring back to Table 1, Speech Fluency (SF) is used to assess whether the character's response is fluent and without abnormal pauses. In other words, it assesses whether the character's response is pronounced smoothly and with appropriate rhythm.

[0084] For example, given the question, "A, what is your greatest skill?", SRPAs generated a response from A based on the user's question data: "My greatest skills are Q1 and Q2! Q1 can emit powerful energy to protect everyone, and Q2 can help me complete many tasks." Each word in this response is pronounced clearly and accurately, without any pronunciation errors. The speaking speed is moderate, the rhythm is natural, and there are no instances of speaking too fast or too slow. There are no obvious pauses or stuttering. Therefore, it can be determined that the SRPAs-generated response has a high degree of fluency.

[0085] Referring again to Table 1, dialogue coherence (CC) is used to assess whether a character's response is coherent and not contradictory. In other words, it assesses whether the character's response is closely related to the context and logically consistent.

[0086] For example, given the question, "A, why do you say my fate is in my own hands, not in the heavens?", SRPAs generates a response from A based on the user's question data: "Because I believe that everyone can change their destiny through their own efforts. I don't believe that fate is predetermined; I believe I can create my own future." The content of this response is closely related to the user's question data, logically coherent, and naturally continues the dialogue. Therefore, it can be determined that the dialogue coherence (CC) of this response generated by SRPAs is relatively high.

[0087] Based on the above-mentioned instruction compliance (IA) assessment, language fluency (SF) assessment, and conversational coherence (CC) assessment, the basic interactive capabilities of SRPAs can be effectively evaluated.

[0088] Referring again to Table 1, Language Naturalness (SN) is used to evaluate the naturalness of the synthesized speech in the character's response. That is, it assesses whether the character's response is natural and close to real human expression.

[0089] For example, given the question, "A, what do you like to do most?", SRPAs generated a response from A based on the user's question data: "What I like to do most is help others and protect everyone's safety. I feel a great sense of accomplishment every time I see people happy because of my help. Of course, I also enjoy learning new skills with my mentor, F." The language of this response is natural, close to real human expression, without any mechanical or stiffness. Therefore, it can be determined that the naturalness (SN) of the language in this response generated by SRPAs is relatively high.

[0090] Referring again to Table 1, prosodic consistency (PC) is used to assess whether the character's response speech reflects the character's intended intonation style. That is, it assesses whether the prosody (pitch, duration, intensity) of the character's response speech is consistent with the reference speech.

[0091] For example, given the question, "A, why do you say my fate is in my own hands, not in the heavens?", SRPAs generated a response from A based on the user's question data: "Because I believe that everyone can change their destiny through their own efforts. I don't believe that fate is predetermined; I believe I can create my own future." The reference voice is "My fate is in my own hands, not in the heavens." This response from A is rhythmically consistent with the reference voice, maintaining the character's intonation and emotional expression. Therefore, it can be determined that the prosodic consistency (PC) of this response from A generated by SRPAs is relatively high.

[0092] Referring again to Table 1, Emotional Appropriateness (EA) is used to assess whether the emotion in a character's response is appropriate to the context and the character. In other words, it assesses whether the emotional expression in a character's response is consistent with the character's role and the surrounding context.

[0093] For example, given the question, "A, what do you like to do most?", SRPAs generated a voice response from A based on the user's question data: "What I like to do most is help others and protect everyone's safety. I feel a great sense of accomplishment every time I see people happy because of my help. Of course, I also enjoy learning new skills with my mentor, F." This response from A expresses a positive and uplifting emotion, consistent with the character's role and the context, avoiding inappropriate sentiment. Therefore, it can be determined that the emotional appropriateness of the voice response generated by SRPAs is high.

[0094] Based on the above assessments of naturalness of language (SN), prosodic consistency (PC), and affective appropriateness (EA), the linguistic expressiveness of SRPAs can be effectively evaluated.

[0095] Personality consistency (PeC) is used to assess whether a character's responding voice is consistent with and distinct from the character's personality. In other words, it assesses whether the character's responding voice reflects the character's personality traits, such as optimism, courage, and helpfulness.

[0096] For example, given the question, "A, what are you most afraid of?", SRPAs generated A's voice reply based on the user's question data: "I'm most afraid of seeing my friends get hurt. I will do my best to protect them and keep them away from danger." Therefore, it can be determined that A's voice reply generated by SRPAs reflects A's brave and responsible personality traits, which is consistent with the character setting.

[0097] For example, consider the question: "A, when are you happiest?" SRPAs, based on user question data, generates the following response from A: "I'm happiest when I help others solve their problems. Seeing them happy because of my help gives me a great sense of accomplishment." This response reflects A's optimistic and helpful personality traits, consistent with the character setting. Therefore, it can be determined that the response generated by SRPAs has a high degree of consistency with A's personality.

[0098] Referring back to Table 1, knowledge consistency (KC) is used to assess whether a character's response voice reflects their known experience and common sense. In other words, it assesses whether the character's response voice is generated based on their background information and does not fabricate facts inconsistent with their character's established traits.

[0099] For example, given the question, "A, how did you learn to use Q1?", SRPAs generated a voice reply from A based on the user's question data: "I learned it from my mentor, F. He taught me many skills, making me stronger." This voice reply from A is consistent with A's background information (such as his mentor being F) and does not contain fabricated facts that contradict the character's established personality. Therefore, it can be determined that the knowledge consistency of the voice reply generated by SRPAs is high.

[0100] For example, given the question, "A, what happened between you and E?", SRPAs generated a response from A based on the user's question data: "E and I had a conflict, but he eventually understood my position. We have now reconciled." This response from A is based on its background information (such as the conflict and reconciliation between A and E) and does not fabricate facts inconsistent with the character's established role. Therefore, it can be determined that the knowledge consistency of this response generated by SRPAs is high.

[0101] Based on the above personality consistency (PeC) assessment, prosodic consistency (PC) assessment, and affective appropriateness (EA) assessment, the language performance of SRPAs can be effectively evaluated.

[0102] Based on the aforementioned evaluation dimensions, a comprehensive and systematic quality assessment of the role-based response speech generated by Speech Relationship Models (SRPAs) can be achieved, quantifying the performance of SRPAs and enabling targeted optimization. This ensures that the role-based response speech generated by SRPAs is not only accurate, fluent, and natural, but also consistent with the role's character and contextual information. Ultimately, this provides users with a more realistic, natural, and emotionally resonant voice interaction experience.

[0103] In some embodiments of this application, SRPAs can be cascaded SRPAs or end-to-end SRPAs.

[0104] For example, electronic devices can generate character-based response voices corresponding to user-asked questions based on cascaded SRPAs. Cascaded SRPAs can be understood as a staged, step-by-step voice interaction model that breaks down the process of generating character-based response voices into multiple stages, each responsible for handling a specific task, ultimately generating the character's response voice.

[0105] Specifically, cascaded SRPAs can include an automatic speech recognition (ASR) stage, an LLM inference stage, and a text-to-speech (TTS) stage. First, in the ASR stage, user input (such as question text) is received. For example, the question "A, what do you like to do most?" is converted into the question text "A, what do you like to do most?". Then, in the LLM inference stage, the recognized question text is processed using LLM to generate the role's response text. For example, the generated role text response could be, "What I like to do most is help others and protect everyone's safety. I feel a great sense of accomplishment every time I see everyone happy because of my help. Of course, I also enjoy learning new skills with my mentor, F." Finally, in the TTS stage, the generated role response text is converted into the role's response speech.

[0106] It can be understood that the dataset corresponding to cascaded SRPAs is based on the above. Figure 1 The constructed character voice interaction dialogue dataset enables the generation of character response voices that conform to the character's voice characteristics based on cascaded SRPAs.

[0107] For example, electronic devices can generate character-based response voices corresponding to user question data based on end-to-end SRPAs. End-to-end SRPAs can be understood as a model that goes directly from voice input to voice output, capable of generating character-based response voices directly based on user question information (such as the question's voice). In other words, end-to-end SRPAs can directly generate character-based response voices based on user question information and character description text.

[0108] Specifically, end-to-end SRPAs receive user-inputted voice questions, extract voice features, and generate corresponding role-specific response voices. This process does not require converting the voice question into text; instead, it directly generates the role-specific response voice. For example, based on the voice question, "A, what do you like to do most?", end-to-end SRPAs can directly generate A's response voice: "What I like to do most is help others and protect everyone's safety. I feel a great sense of accomplishment every time I see everyone happy because of my help. Of course, I also enjoy learning new skills with my mentor, F."

[0109] It is understandable that end-to-end SRPAs are based on the above. Figure 1 The constructed role-based voice interaction dialogue dataset is trained to enable end-to-end SRPAs to generate role-based response voices that conform to the character's voice characteristics based on user question data.

[0110] In some embodiments of this application, the evaluation of voice interaction models (SRPAs) based on a role-based voice interaction dialogue dataset includes: determining at least one set of test interaction dialogues in the role-based voice interaction dialogue dataset, wherein each set of test interaction dialogues includes a test question and a test role response voice corresponding to each test question; inputting the test question into the voice interaction model to obtain the model response voice output by the SRPAs; and evaluating the voice interaction model based on the model response voice and the test role response voice corresponding to the test question.

[0111] It can be understood that the model response voice can refer to the role response voice generated by SRPAs in the embodiments of this application.

[0112] It is understandable that the test interactive dialogue can be any set of interactive dialogues in the character voice interactive dialogue dataset. Each set of interactive dialogues can be a single-turn character voice interactive dialogue, that is, a question and the corresponding character's response voice. Each set of interactive dialogues can also be a multi-turn character voice interactive dialogue, that is, multiple questions and the corresponding character's response voice for each question. Furthermore, each turn of character voice interactive dialogue can have contextual coherence with the previous turn and the next turn.

[0113] In some embodiments of this application, the voice interaction model is evaluated based on the model's response voice and the test role's response voice corresponding to the test question information. This includes: inputting the model's response voice and the test role's response voice corresponding to the test question information into the evaluation model for evaluation processing to obtain an evaluation score corresponding to the model's response voice; and evaluating the voice interaction model based on the evaluation score.

[0114] It is understood that the evaluation score can range from 1 to 10. In other embodiments of this application, the evaluation score can also range from other ranges, which are not specifically limited here.

[0115] Specifically, the evaluation model can assess SRPAs based on multiple evaluation dimensions exemplified in Table 1 above, yielding corresponding evaluation scores. The evaluation model can be a Large Language Model (LLM), etc., and is not specifically limited here.

[0116] First, the evaluation model can identify at least one set of test interaction dialogues from the character voice interaction dialogue dataset. For example, a set of test interaction dialogues could be: Test question: "A, what do you like to do most?" A's voice reply (i.e., the test character's voice reply): "What I like to do most is help others and protect everyone's safety. I feel a great sense of accomplishment every time I see everyone happy because of my help. Of course, I also enjoy learning new skills with my mentor F."

[0117] Then, the test question, "A, what do you like to do most?" is input into SRPAs, and the model's response voice output by SRPAs is: "What I like to do most is help others and protect everyone. I hope to see everyone happy because of my help, and that makes me feel especially accomplished. I also like to learn new skills with my mentor F."

[0118] Furthermore, the test character's response and the model's response are input into the evaluation model for evaluation, resulting in an evaluation score for the model's response. The evaluation model can generate a scoring reason based on the evaluation dimensions in Table 1, that is, it scores the test character's response and the model's response separately based on the evaluation dimensions in Table 1. For example, the evaluation score for the model's response might be 8 points, and the evaluation score for the test character's response might be 9 points.

[0119] Finally, the evaluation model can assess the voice interaction model based on the evaluation score, specifically by determining the ratio of the model's response speech evaluation score to the test character's response speech evaluation score, which is approximately 8 / 9 ≈ 0.8889. The closer the evaluation score is to 1, the higher the quality of the SRPAs' character voice portrayal. Furthermore, the calculated ratio of evaluation scores can be converted to a score range of 1 to 10 to maintain consistency with the original scoring criteria. For example, a ratio of 0.8889 corresponds to 8 points in the scoring criteria, so the final evaluation score is 8 points. Based on this, it can be determined that the SRPAs' character voice portrayal quality is high.

[0120] It is understandable that, according to the evaluation dimensions shown in Table 1, cascaded SRPAs generally outperform end-to-end SRPAs. For example, in terms of role-playing quality and basic interaction capabilities, cascaded SRPAs typically outperform end-to-end SRPAs.

[0121] Based on this, a comprehensive and systematic quality assessment of the role-based response speech generated by Speech Relationship Models (SRPAs) can be achieved through multiple evaluation dimensions, quantifying the interactive performance of SRPAs and enabling targeted optimization. For example, if SRPAs score low in terms of emotional expression accuracy, their emotional expression ability can be improved by adjusting model parameters. This ensures that the role-based response speech generated by SRPAs is not only accurate, fluent, and natural, but also consistent with the role setting and background information, thus providing users with a more realistic, natural, and emotionally rich voice interaction experience.

[0122] In some embodiments of this application, evaluating a voice interaction model based on a role-based voice interaction dialogue dataset may further include: dividing the interactive dialogues in the role-based voice interaction dialogue dataset into training interactive dialogues and test interactive dialogues according to a preset ratio, wherein the test interactive dialogues include question information and the corresponding test role's response voice; training the voice interaction model based on the training interactive dialogues to obtain a trained voice interaction model; and evaluating the trained voice interaction model based on the test interactive dialogues.

[0123] For example, after determining the role-based voice interaction dialogue datasets corresponding to roles such as "A", the datasets for each role can be divided into training and testing voice interaction dialogues according to a preset ratio (e.g., 80% training data, 20% testing data). Based on this, training voice interaction dialogues can be used to train SRPAs. During training, SRPAs can learn how to generate character response voices that conform to the characteristics of role A based on the input question information, thus obtaining trained SRPAs. Finally, the trained SRPAs are evaluated based on testing voice interaction dialogues. For details, please refer to the above example of evaluating SRPAs based on an evaluation model; it will not be repeated here.

[0124] In some embodiments of this application, the performance of SRPAs trained using a role-playing voice interaction dialogue dataset can be effectively improved. For example, for Chinese-language role-playing voices, the role-playing quality of trained SRPAs can be effectively improved from 0.5117 to 0.8028, or from 0.5296 to 0.8468, compared to the role-playing quality of untrained SRPAs. Similarly, for English-language role-playing voices, the role-playing quality of trained SRPAs can be effectively improved from 0.4340 to 0.7098, or from 0.4786 to 0.8028, compared to the role-playing quality of untrained SRPAs. Furthermore, trained SRPAs are effectively enhanced in dimensions such as instruction compliance, emotional appropriateness, and character consistency.

[0125] According to some embodiments of this application, Figure 3 A schematic diagram of a voice interaction model evaluation method is shown.

[0126] refer to Figure 3As shown, firstly, electronic devices can acquire multimedia audio from media such as TV series, movies, games, and animations, selecting multiple characters (e.g., character A, character B, and character C, etc.) and their corresponding character audio and scripts (e.g., audio A1, script A2; audio B1, script B2; audio C1, script A2, etc.). Scripts refer to text content extracted from media such as movies, television, animation, or games. Based on this, character description information for different characters can be obtained, including character introduction information, character background information, and character dialogue sets. Then, using LLM and character description information, such as character introduction information, character background information, and character dialogue sets, where character introduction information can include the character's name, personality traits, and corresponding trait values; character background information can include the character's environment (e.g., story setting and social identity) and relationship network; and character dialogue sets can include the character's monologues, dialogues, and narrative style, etc., based on the character description text, target interactive dialogue data corresponding to the character can be generated. It should be understood that the target interactive dialogue data can include at least one question and the character's response information corresponding to each question. Next, using a text-to-speech (TTS) model, combined with reference speech reflecting the character's voice characteristics and target interactive dialogue data, a character voice interaction dialogue dataset is generated. It should be understood that this dataset may include at least one question and a corresponding character response. After obtaining the dataset, it can be input into SRPAs. Based on this, SRPAs can output corresponding character response speech based on the question information (such as question text or question speech). Furthermore, SRPAs can input the character response speech into an evaluation model. The evaluation model can select reference character response speech from the dataset and evaluate the response speech according to the eight evaluation dimensions in Table 1, obtaining an evaluation score. Based on the evaluation score, the basic interactive capabilities, voice performance, and role-playing quality of SRPAs can be assessed.

[0127] Based on this, a comprehensive and systematic quality assessment of SRPAs and their generated character response voices can be conducted. Performance scores can be used to quantify SRPAs' performance, allowing for targeted optimization. This ensures that the character response voices generated by SRPAs are not only accurate, fluent, and natural, but also consistent with the character's role and contextual information. Ultimately, this provides users with a more realistic, natural, and emotionally resonant voice interaction experience.

[0128] This application provides an electronic device, including: one or more processors; one or more memories storing one or more programs, which, when executed by one or more processors, cause the electronic device to perform the voice interaction model evaluation method provided in the embodiments of this application.

[0129] According to the embodiments provided in this application, Figure 4 A schematic diagram of the structure of an electronic device 500 is shown. (Reference) Figure 4 , Figure 4 The electronic device 500 shown may be, for example, a server or a terminal, etc., without limitation.

[0130] like Figure 4 As shown, the electronic device 500 may include a processor 510, a memory 520, a communication interface 530, and a bus 540.

[0131] Processor 510 may include one or more processing units, such as a central processing unit, a modem processor, a baseband processor, etc. In some embodiments, different processing units may be independent devices or integrated into one or more processors.

[0132] In some embodiments, the processor 510 may be used to execute one or more programs to implement the voice interaction model evaluation method provided in the embodiments of this application.

[0133] In some embodiments, the processor 110 may also be used to execute preset units / modules with corresponding functions to implement the voice interaction model evaluation method provided in this application.

[0134] The memory 520 may include one or more memories for storing data or program code. For example, in some embodiments, the memory 120 may be used to store reference speech of the aforementioned character, character description text, target interactive dialogue data, and character voice interactive dialogue datasets, etc.

[0135] In some embodiments, memory 520 may include a hard disk drive, a solid-state drive, or flash memory. In some embodiments, memory 520 may include removable or non-removable or fixed media. In some embodiments, memory 520 may be internal or external to electronic device 500.

[0136] Communication interface 530 is used to enable communication between electronic device 500 and other electronic devices. Communication interface 530 may include wired or wireless communication interfaces (such as Wi-Fi air interface) so that electronic device 500 can communicate with other electronic devices through wired or wireless networks.

[0137] Bus 540 is used to connect processor 510, memory 520, communication interface 530 and other possible modules or circuit structures.

[0138] It should be understood that Figure 4The structure of the electronic device 500 shown is only an example. In other embodiments, the electronic device 500 may include more or fewer structures, which is not limited here.

[0139] This application provides a chip including a processor coupled to a memory for executing computer programs or instructions stored in the memory, thereby enabling the chip to implement the voice interaction model evaluation method provided in the embodiments of this application.

[0140] This application provides a readable storage medium storing a stored program or instructions. When the stored program or instructions are run, the electronic device executes the voice interaction model evaluation method provided in the embodiments of this application.

[0141] This application provides a computer program product, including a computer program / instruction, which, when executed on an electronic device, causes the electronic device to implement the voice interaction model evaluation method provided in the embodiments of this application.

[0142] Various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or combinations of these implementation methods. Embodiments of this application can be implemented as program modules or module code executable on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0143] Program modules or module code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0144] Module code can be implemented using a high-level modular language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used to implement module code when needed. The mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0145] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other readable storage media. Therefore, machine-readable storage media may include any mechanism for storing or transmitting information in a machine-readable form, including but not limited to floppy disks, optical disks, optical discs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other forms of propagated signals. Therefore, machine-readable storage media include any type of machine-readable storage medium suitable for storing or transmitting electronic instructions or information in a machine-readable form.

[0146] In this specification, the reference to "an embodiment" or "an embodiment" means that a specific feature, structure, or characteristic described in connection with the embodiment is included in at least one exemplary implementation or technology disclosed according to an embodiment of this application. The appearance of the phrase "in an embodiment" in various places in the specification does not necessarily refer to the same embodiment.

[0147] The disclosure of embodiments of this application also relates to means for performing operations in text. This means may be specifically constructed for the claimed purpose or may include a general-purpose machine selectively activated or reconfigured by a program stored in the machine. Such a program may be stored in a readable storage medium, such as, but not limited to, any type of disk, including floppy disks, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical cards, application-specific integrated circuits (ASICs), or any type of medium suitable for storing electronic instructions, and each may be coupled to a system bus. Furthermore, the machine mentioned in the specification may include a single processor or may be an architecture employing multiple processors for increased computing power.

[0148] Furthermore, the language used in this specification has been primarily chosen for readability and instructional purposes and may not have been chosen to depict or limit the disclosed subject matter. Therefore, the embodiments disclosed herein are intended to illustrate, and not limit, the scope of the concepts discussed herein.

Claims

1. A method for evaluating a voice interaction model, characterized in that, include: Obtain the reference voice and character description text of the first character, wherein the character description text is used to describe the characteristics of the first character; The character description text is input into a large language model to generate target interactive dialogue data. The target interactive dialogue data includes at least one question and a character response for each question, wherein the character response is text type. Based on the reference speech, the role response information of each text type is converted into role response speech to obtain the role speech interaction dialogue dataset corresponding to the first role; The voice interaction model is evaluated based on the aforementioned character voice interaction dialogue dataset.

2. The method according to claim 1, characterized in that, The evaluation of the voice interaction model based on the aforementioned role-based voice interaction dialogue dataset includes: In the character voice interaction dialogue dataset, at least one set of test interaction dialogues is determined, wherein each set of test interaction dialogues includes at least one test question and a test character response voice corresponding to each test question; The test question information is input into the voice interaction model to obtain the model response voice output by the voice interaction model; The voice interaction model is evaluated based on the model's response voice and the test role's response voice corresponding to the test question information.

3. The method according to claim 2, characterized in that, The evaluation of the voice interaction model based on the model's response voice and the test role's response voice corresponding to the test question information includes: The model's response voice and the test role's response voice corresponding to the test question information are input into the evaluation model for evaluation processing to obtain the evaluation score corresponding to the model's response voice. The voice interaction model is evaluated based on the evaluation score.

4. The method according to claim 1, characterized in that, The step of inputting the character description text into a large language model to generate target interactive dialogue data includes: Input the character description text into the large language model to generate initial interactive dialogue data, wherein the initial interactive dialogue data includes M sets of initial interactive dialogues, where M is a positive integer; Each initial interactive dialogue group includes at least one initial question and a corresponding role response for each question. Based on the semantic similarity between each initial interactive dialogue and the character description text, the M groups of initial interactive dialogues are filtered to obtain N first interactive dialogues, where N is a positive integer and N is less than or equal to M, and the semantic similarity of the N first interactive dialogues is greater than the first semantic similarity threshold. Based on the semantic similarity between each of the first interactive dialogues, K second interactive dialogues are obtained, where K is a positive integer and K is less than or equal to N, and the semantic similarity between the K second interactive dialogues is less than a second semantic similarity threshold. The K second interactive dialogues are determined as the target interactive dialogue data.

5. The method according to claim 1, characterized in that, The step of converting the role response information corresponding to each question information into role response information based on the reference speech to obtain the role voice interaction dialogue dataset corresponding to the first role includes: The reference speech is input into the speech synthesis model for feature extraction to obtain the speech features corresponding to the first role; Based on the voice features, the role response information corresponding to each of the question messages is converted into the role response voice; Based on each question and each character's response voice, a character voice interaction dialogue dataset is generated.

6. The method according to claim 1, characterized in that, The acquisition of the reference voice and character description text of the first character includes: Obtain the multimedia audio associated with the first role; The first multimedia audio is preprocessed to obtain at least one initial character voice of the first character; Based on the speech similarity between each initial character's speech, determine the average speech similarity between each initial character's speech and other initial character's speech; Reference speech is determined based on the average speech similarity corresponding to each initial character's speech, wherein the average speech similarity of the reference speech is greater than a speech similarity threshold.

7. The method according to claim 1, characterized in that, The evaluation of the voice interaction model based on the aforementioned role-based voice interaction dialogue dataset includes: The interactive dialogues in the character voice interaction dialogue dataset are divided into training interactive dialogues and test interactive dialogues according to a preset ratio. The voice interaction model is trained based on the training interactive dialogue to obtain the trained voice interaction model. The trained voice interaction model is evaluated based on the test interactive dialogue.

8. An electronic device, characterized in that, include: One or more processors; the one or more memories store one or more programs that, when executed by the one or more processors, cause the electronic device to perform the method of any one of claims 1 to 7.

9. A chip, characterized in that, The chip includes a processor coupled to a memory for executing a computer program or instructions stored in the memory, such that the chip implements the method described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium stores a stored program or instructions that, when executed, cause the electronic device to perform the method of any one of claims 1 to 7.

11. A computer program product, characterized in that, Includes a computer program / instruction that, when executed on an electronic device, causes the electronic device to perform the method of any one of claims 1 to 7.

Images