A voice interaction model evaluation method, an electronic device and a readable storage medium

Abstract

The application relates to the field of artificial intelligence, in particular to a voice interaction model evaluation method, an electronic device and a readable storage medium. The method comprises the following steps: obtaining reference voice of a first role and role description text, wherein the role description text is used for describing characteristics of the first role; inputting the role description text into a large language model to generate target interaction dialogue data, the target interaction dialogue data comprises at least one question information and role reply information corresponding to each question information, wherein the type of the role reply information is a text type; converting the role reply information corresponding to each question information into role reply voice based on the reference voice to obtain a role voice interaction dialogue data set corresponding to the first role; and evaluating a voice interaction model based on the role voice interaction dialogue data set. In this way, the voice interaction model can accurately simulate voice characteristics of different roles to generate role reply voice, thereby improving the voice interaction experience of users.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to a speech interaction model evaluation method, an electronic device and a readable storage medium. BACKGROUND

[0002] A speech role-playing agent (SRPA) is a system that interacts with a user through speech and plays a specific role in the interaction process.

[0003] However, the current data set for training SRPAs is mostly role-related text, lacking voice data related to the voice characteristics consistent with the role, and lacking systematic evaluation of SRPAs, making it difficult for SRPAs to fully achieve voice interaction effects consistent with the role in actual applications, affecting the user's voice interaction experience. SUMMARY

[0004] To solve the above problems, the present application provides a speech interaction model evaluation method, an electronic device and a readable storage medium.

[0005] In a first aspect, the present application provides a speech interaction model evaluation method, which comprises: obtaining a reference voice of a first role and a role description text, wherein the role description text is used to describe the characteristics of the first role; inputting the role description text into a large language model to generate target interaction dialogue data, the target interaction dialogue data comprising at least one question information and role reply information corresponding to each question information, wherein the type of the role reply information is a text type; converting each text type of role reply information into role reply voice based on the reference voice to obtain a role voice interaction dialogue data set corresponding to the first role; and evaluating the speech interaction model based on the role voice interaction dialogue data set.

[0006] Based on the method provided by the present application, a large number of role voice interaction dialogue data sets of different roles are generated through the voice characteristics corresponding to the reference voice of each role and the target interaction dialogue data, and a large number of role reply voices corresponding to the role can be included in the role voice interaction dialogue data set corresponding to each role. The speech interaction model (SRPA) integrated with the role voice interaction dialogue data set can more accurately simulate the voice characteristics of different roles in actual applications, providing a more natural, personalized and emotional voice interaction experience. In this way, the problem of the lack of role-related voice data in the current SRPAs is improved, and the performance of the SRPAs and the voice interaction experience of the user are improved through systematic evaluation.

[0007] It can be understood that the role voice interaction dialogue data set provided by the application can include target interaction dialogue data corresponding to a plurality of different roles, and each role corresponding target interaction dialogue data can include a voice-based single-turn dialogue and a multi-turn dialogue. For example, the role voice interaction dialogue data set provided by the application can include 98 roles, and each role corresponding target interaction dialogue data can include 112,000 voice-based single-turn dialogues and voice multi-turn dialogues, etc., which are not limited by the application.

[0008] In a possible implementation of the first aspect, the evaluation of the voice interaction model based on the role voice interaction dialogue data set comprises: determining at least one set of test interaction dialogues in the role voice interaction dialogue data set, wherein each set of test interaction dialogues includes at least one test question information and a test role reply voice corresponding to each test question information; inputting the test question information into the voice interaction model to obtain a model reply voice output by the voice interaction model; and evaluating the voice interaction model based on the model reply voice and the test role reply voice corresponding to the test question information.

[0009] In a possible implementation of the first aspect, the evaluation of the voice interaction model based on the model reply voice and the test role reply voice corresponding to the test question information comprises: inputting the model reply voice and the test role reply voice corresponding to the test question information into an evaluation model for evaluation processing to obtain an evaluation score corresponding to the model reply voice; and evaluating the voice interaction model based on the evaluation score.

[0010] It can be understood that the evaluation model provided by the application can refer to the basic interaction capability, voice expressiveness, and role playing quality evaluation dimensions mentioned in the embodiments of the application to evaluate the model reply voice and the test role reply voice, and then the SRPAs can be evaluated and optimized based on the evaluation score.

[0011] It can be understood that the basic interaction capability can be determined based on instruction compliance, language fluency, and dialogue coherence. The voice expressiveness can be determined based on language naturalness, rhythm consistency, and emotional appropriateness. The role playing quality can be determined based on personality consistency and knowledge consistency.

[0012] In a possible implementation of the first aspect, inputting the character description text into the large language model to generate the target interactive dialogue data includes: inputting the character description text into the large language model to generate initial interactive dialogue data, where the initial interactive dialogue data includes M groups of initial interactive dialogues, M being a positive integer; each group of initial interactive dialogues includes at least one initial question information and character reply information corresponding to each question information; based on semantic similarity between each initial interactive dialogue and the character description text, the M groups of initial interactive dialogues are screened to obtain N first interactive dialogues, where N is a positive integer, N is less than or equal to M, and the semantic similarity of the N first interactive dialogues is greater than a first semantic similarity threshold; based on semantic similarity between each first interactive dialogue, K second interactive dialogues are obtained, where K is a positive integer, K is less than or equal to N, and the semantic similarity between the K second interactive dialogues is less than a second semantic similarity threshold; and the K second interactive dialogues are determined as the target interactive dialogue data.

[0013] In a possible implementation of the first aspect, converting the character reply information corresponding to each question information into character reply information based on the reference voice to obtain the character voice interactive dialogue data set corresponding to the first character includes: inputting the reference voice into a voice synthesis model to extract features to obtain voice features corresponding to the first character; converting the character reply information corresponding to each question information into character reply voice based on the voice features; and generating the character voice interactive dialogue data set based on each question information and each character reply voice.

[0014] It can be understood that the question information can be question text or question voice, which is not limited in the present application.

[0015] In a possible implementation of the first aspect, obtaining the reference voice and the character description text of the first character includes: obtaining a multimedia audio associated with the first character; preprocessing the first multimedia audio to obtain at least one initial character voice of the first character; determining an average voice similarity of each initial character voice with other initial character voices based on voice similarity between each initial character voice; and determining the reference voice based on the average voice similarity corresponding to each initial character voice, where the average voice similarity of the reference voice is greater than a voice similarity threshold.

[0016] In a possible implementation of the first aspect, evaluating the voice interactive model based on the character voice interactive dialogue data set includes: dividing the interactive dialogues in the character voice interactive dialogue data set into training interactive dialogues and test interactive dialogues according to a preset proportion; training the voice interactive model based on the training interactive dialogues to obtain a trained voice interactive model; and evaluating the trained voice interactive model based on the test interactive dialogues.

[0017] In a second aspect, the embodiments of the present application provide an electronic device, comprising: one or more processors; one or more memories storing one or more programs, which, when executed by the one or more processors, cause the electronic device to perform the method in the first aspect and the possible implementation of the first aspect.

[0018] In a third aspect, the embodiments of the present application provide a chip, comprising a processor coupled with a memory, configured to execute a computer program or instructions stored in the memory, so that the chip implements the method in the first aspect and the possible implementation of the first aspect.

[0019] In a fourth aspect, the embodiments of the present application provide a readable storage medium, which stores a storage program or instructions, when the storage program or instructions are executed, cause the electronic device to perform the method in the first aspect and the possible implementation of the first aspect.

[0020] In a fifth aspect, the embodiments of the present application provide a computer program product, comprising a computer program / instruction, when the computer program / instruction is executed on an electronic device, cause the electronic device to implement the method in the first aspect and the possible implementation of the first aspect.

[0021] The beneficial effects of the second aspect to the fifth aspect can refer to the related description of the first aspect and the possible implementation of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 According to some embodiments of the present application, a SRPA-based question and answer scenario schematic diagram is shown;

[0023] Figure 2 According to some embodiments of the present application, a flowchart of a voice interaction model evaluation method is shown;

[0024] Figure 3 According to some embodiments of the present application, a process schematic diagram of a voice interaction model evaluation method is shown;

[0025] Figure 4 According to some embodiments of the present application, a structural schematic diagram of an electronic device 500 is shown. DETAILED DESCRIPTION

[0026] The illustrative embodiments of the present application include but are not limited to a voice interaction model evaluation method, an electronic device and a readable storage medium.

[0027] The specific implementation process of the technical solutions provided in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0028] It can be understood that the electronic device in the embodiments of the present application can also be a server, and can also be a terminal. The terminal can be a user terminal, a mobile terminal, a user equipment (UE), a terminal device, a mobile station (MS), a mobile terminal (MT), etc. The terminal device can be a mobile phone, a smart television, a wearable device, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self driving, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc.

[0029] The terms involved in the present application are explained below.

[0030] (1) Speech Role Playing Agent (SRPA)

[0031] SRPA is a system that can simulate the voice and behavior characteristics of a specific role, providing users with personalized, natural and emotionally rich voice interaction experience.

[0032] In some embodiments of the present application, SRPAs can be applied to question and answer scenarios. For example, Figure 1 As shown in the schematic diagram of the SRPA-based question and answer scenario, the user can input text or voice through the voice control 102 in the question and answer box 101 based on the application of the SRPA in the computer 10, and ask questions. SRPAs can generate role response voice corresponding to the user's question data based on the role selected by the user in advance.

[0033] As mentioned earlier, the current data set for training SRPAs is mostly role-related text, lacking voice data with consistent voice characteristics, prosody and emotional expression of the role, and lacking systematic evaluation of SRPAs, making it difficult for SRPAs to fully achieve consistent voice interaction effect with the role in actual application, affecting the user's voice interaction experience.

[0034] To solve the above problems, an embodiment of the present application provides a voice interaction model evaluation method, in which a reference voice of a first role and role description text are obtained, wherein the role description text is used to describe the characteristics of the first role; the role description text is input into a large language model to generate target interactive dialogue data, the target interactive dialogue data including at least one question information and role reply information corresponding to each question information, wherein the type of the role reply information is a text type; the role reply information corresponding to each question information is converted into role reply voice based on the reference voice to obtain a role voice interactive dialogue data set corresponding to the first role; and the voice interaction model is evaluated based on the role voice interactive dialogue data set.

[0035] Based on the method provided in the present application, a large number of role voice interactive dialogue data sets of different roles are generated through the sound characteristics corresponding to the reference voice of each role and the target interactive dialogue data, and each role corresponding role voice interactive dialogue data set can include a large number of role reply voices corresponding to the role. The SRPAs integrated with the role voice interactive dialogue data set can more accurately simulate the voice characteristics of different roles in actual application, and provide a more natural, personalized and emotional voice interaction experience. In this way, the problem of the lack of role-related voice data in the current SRPAs is improved, and the performance of the SRPAs and the voice interaction experience of the user are improved through systematic evaluation.

[0036] In some embodiments of the present application, the language corresponding to the text in the role description text and the target interactive dialogue data can include Chinese or English, etc., which is not limited in the present application.

[0037] In some embodiments of the present application, the SRPAs including the role speech interaction dialogue dataset of different roles can simulate the speech characteristics, prosody and emotional expression of different roles, and thus can be applied in intelligent assistants or chat robots. And can be widely applied in various scenarios, providing personalized and emotionally rich speech interaction experience. Exemplarily: (1) SRPAs can be applied to in-game role interaction scenarios, simulating non-player characters, providing interactive dialogue to enhance the immersion of the game. (2) SRPAs can be applied to virtual personal assistant scenarios, simulating personal assistants to provide personalized voice interaction services for users, such as schedule management, weather forecasting, etc. (3) SRPAs can be applied to online education tutoring scenarios, simulating virtual teachers or teaching assistants to provide language learning, course tutoring and other educational services. (4) SRPAs can be applied to automated customer service scenarios, simulating customer service representatives to provide automated customer consultation and problem solving in call centers or online customer service. (5) SRPAs can be applied to creative assistance scenarios, simulating creative partners to assist writers and content creators in generating role interaction dialogues in stories, providing creative inspiration. (6) SRPAs can be applied to professional training simulation scenarios, simulating interviewers or specific roles to help users practice interview skills in professional training. (7) SRPAs can be applied to emotional interaction support scenarios, providing emotional support and listening services, simulating emotional support partners (such as counselors or friends, etc.) to provide emotional communication. (8) SRPAs can be applied to smart home control scenarios, simulating smart home assistants to control smart devices in the home (such as device lights and temperature, etc.). (9) SRPAs can be applied to voice navigation guidance scenarios, simulating navigation assistants to broadcast real-time navigation guidance and traffic information.

[0038] It can be understood that the above scenarios are only exemplary, and in some other embodiments of the present application, SRPAs integrating different role speech interaction dialogue datasets can also be applied to other scenarios, which are not limited in the present application.

[0039] According to some embodiments of the present application, Figure 2 A flowchart of a voice interaction model evaluation method is shown. Figure 2 The execution subject of each flowchart shown can be an electronic device. The specific process can include:

[0040] S101: Obtain the reference speech of the first role and the role description text, wherein the role description text is used to describe the characteristics of the first role.

[0041] In some embodiments of the present application, the electronic device can pre-obtain the reference speech of the first role and the role description text. Wherein the first role can be different roles from movies, films, animations and games.

[0042] In some embodiments of the present application, obtaining the reference voice of the first role and the role description text can specifically include: obtaining a first multimedia audio, preprocessing the first multimedia audio to obtain at least one initial role voice of the first role; determining the average voice similarity of each initial role voice with other initial role voices based on the voice similarity between each initial role voice; and determining the reference voice based on the average voice similarity corresponding to each initial role voice, wherein the average voice similarity of the reference voice is greater than a voice similarity threshold.

[0043] Specifically, the electronic device can obtain different multimedia audios such as different TV series, movies, animations and games, and the multimedia audios can include audios associated with the first role. First, the multimedia audios can be anonymized to remove or replace any identifiable personal information in the audios, such as names, addresses or other sensitive information that may be contained in the sound. Then, the multimedia audios are cut into short audio segments, and the audio segments are converted into WAV format (lossless audio format) with single channel and 16kH sampling rate, so as to reduce the processing data volume. It can be understood that in the present application, shorter role audio segments can be obtained by sequentially performing sound source separation, speaker separation and voice activity detection on the multimedia audios, wherein the sound source separation refers to separating different types of audios (such as background sound, environmental noise, role voice, etc.) from the multimedia audios, and the role voice can include the voice corresponding to the first role. The speaker separation refers to separating the voice corresponding to the first role from the role voice by using speaker separation technology. The voice activity detection refers to removing the silent and non-voice parts in the role voice corresponding to the first role, extracting the pure role voice of the first role, and obtaining at least one initial role voice of the first role. It can be understood that the at least one initial role voice of the first role can be an audio segment with a time length of 3s, 5s, 10s, etc., which is not limited in the present application.

[0044] Further, the electronic device can extract a speaker embedding of each initial character voice of the first character using an automatic speech recognition (ASR) technique, i.e., according to the at least one initial character voice of the first character, calculate an average voice similarity of each initial character voice. For example, assuming that the at least one initial character voice of the first character is V1, V2 and V3, based on V1, V2 and V3, the voice similarity S12 between V1 and V2, the voice similarity S13 between V1 and V3, and the voice similarity S23 between V2 and V3 are calculated respectively. Then, based on the voice similarities S12, S13 and S23 between each initial character voice, the average voice similarity of each initial character voice is calculated respectively, i.e., the average voice similarity S1 of V1 is calculated based on S12 and S13, the average voice similarity S2 of V2 is calculated based on S12 and S23, and the average voice similarity S3 of V3 is calculated based on S13 and S23. Finally, based on the average voice similarity of S1, S2 and S3 respectively, one of V1, V2 and V3 is determined as a reference voice, it should be understood that the average voice similarity of the reference voice is greater than a voice similarity threshold, it should be understood that the average voice similarity of the reference voice can be the highest among the average voice similarities of V1, V2 and V3, and the voice similarity threshold is not specifically limited in the present application.

[0045] In some embodiments of the present application, the electronic device can extract structured metadata based on different data sources to construct a character profile corresponding to the character, wherein the character profile can include character description (i.e., character profile) (such as personality and preference, etc.), background information (such as social identity and relationship, etc.) and character lines (such as dialogue and monologue, etc.). Using a large language model (LLM), based on the character profile, initial interactive dialogue data can be generated, and the type of the initial interactive dialogue data is a text type.

[0046] It can be understood that the above-mentioned character profile is also the character description text in the present application. The initial interactive dialogue data can include at least one question information and character reply information corresponding to each question information. Exemplarily, for each character (such as 98, etc.), Q (such as 800, etc.) single-turn interactive dialogues and Q multi-turn interactive dialogues can be generated to obtain initial interactive dialogue data corresponding to each character.

[0047] Specifically, the character profile is used to describe the basic characteristics and core information of the character. The character profile can correspond to the appearance characteristics, catchphrases, and emotional response patterns of the first character, and the like. Among them, the appearance characteristics can be the age, gender, height, body shape, skin color, hairstyle, and the like of the first character. The catchphrase can be a specific word or phrase frequently used by the first character, and the like. The emotional response pattern can be the typical emotional response of the first character in different situations, such as anger, joy, sadness, surprise, and the like. The background information is used to describe the detailed background and living environment of the character. The background information can be the story setting, social identity, and relationship network corresponding to the first character, and the like. The story setting can be the story background in which the first character is located, such as time, place, event, and the like. The social identity can be the family members, family relationships, family economic conditions, and the like of the first character, and the relationship network can be the relationship network of the first character in the society, such as friends, colleagues, and the like. The character dialogue set is used to assist the LLM to understand the language style and emotional expression characteristics of the first character. The character dialogue can be at least one of monologue, dialogue, and narrative content corresponding to the first character. Among them, the monologue is the content of the first character speaking alone, which can reflect the inner thoughts and emotions of the first character, the dialogue is the conversation between the first character and other characters, which can reflect the language style of the first character in interaction, and the narrative style is the language style of the first character when narrating events or stories, which can reflect the expression manner of the character.

[0048] Based on the above, the language style, emotional expression characteristics, and other multi-dimensional characteristics of the first character can be determined based on the reference voice and the character description text of the first character.

[0049] S102: inputting the character description text into the large language model to generate target interactive dialogue data, the target interactive dialogue data including at least one question information and character reply information corresponding to each question information, wherein the type of the character reply information is a text type.

[0050] In some embodiments of the present application, the LLM can generate initial interactive dialogue data corresponding to each character based on the character description text of each character. Each initial interactive dialogue data includes at least one initial question information and initial character reply information corresponding to each initial question information. It should be understood that the type of the initial interactive dialogue data can be a text type.

[0051] In some embodiments of the present application, after obtaining the reference voice and the character description text of the first character based on S101, the electronic device can input the character description text corresponding to the first character into the LLM, so that the LLM can generate initial interactive dialogue data corresponding to the first character based on the character description text. The initial interactive dialogue data corresponding to the first character can include at least one initial question information and initial character reply information.

[0052] In some embodiments of the present application, the contextual relevance of each role can be determined by calculating the semantic similarity between the initial interaction dialogue data corresponding to each role and the corresponding metadata (i.e., role description text).

[0053] In some embodiments of the present application, inputting the role description text into a large language model to generate target interaction dialogue data includes: inputting the role description text into the large language model to generate initial interaction dialogue data, wherein the initial interaction dialogue data includes M groups of initial interaction dialogues, M being a positive integer; each group of initial interaction dialogues includes at least one initial question information and role reply information corresponding to each question information; based on the semantic similarity between each group of initial interaction dialogues and the role description text, filtering the M groups of initial interaction dialogues to obtain N first interaction dialogues, wherein N is a positive integer, and N is less than or equal to M, and the semantic similarity of the N first interaction dialogues is greater than a first semantic similarity threshold; based on the semantic similarity between each first interaction dialogue, obtaining K second interaction dialogues, wherein K is a positive integer, and K is less than or equal to N, and the semantic similarity between the K second interaction dialogues is less than a second semantic similarity threshold; and determining the K second interaction dialogues as the target interaction dialogue data.

[0054] In some embodiments of the present application, corresponding to the role description text being Chinese, the first similarity threshold can be 0.45 or the like, and corresponding to the role description text being English, the first similarity threshold can be 0.4 or the like.

[0055] In some embodiments of the present application, corresponding to the role description text being Chinese, the second similarity threshold can be 0.85 or the like, and corresponding to the role description text being English, the second similarity threshold can be 0.9 or the like.

[0056] In some embodiments of the present application, corresponding to the role description text being Chinese or English or the like, the first similarity threshold and the second similarity threshold can also be other numerical values, which are not limited in the present application.

[0057] Exemplarily, firstly, it is assumed that the M groups of initial interactive dialogue data are T1, T2, T3, T4, T5, T6, T7 and T8, that is, M=8. The semantic similarity between each initial interactive dialogue data and the role description text corresponding to the first role is respectively calculated as S1(0.8), S2(0.85), S3(0.75), S4(0.7), S5(0.7), S6(0.65), S7(0.75), S8(0.85). Then, it is assumed that the first semantic similarity threshold is 0.7, and the N first dialogue data with a semantic similarity greater than the first similarity threshold are selected from the M groups of initial interactive dialogue data as T1, T2, T3, T4, T5 and T7, that is, N=6. The semantic similarity between the N first interactive dialogue data is respectively calculated as S12(0.9), S13(0.3), S14(0.4), S15(0.7), S17(0.7), S23(0.4), S24(0.8), S25(0.7), S27(0.7), S34(0.9), S35(0.4), S37(0.4), S45(0.7), S47(0.7), S57(0.8). Based on this, the K second interactive dialogue data are determined, and the semantic similarity between the K second interactive dialogue data is less than the second semantic similarity threshold 0.8. For example, the K second interactive dialogue data are respectively T1, T2, T3, T4 and T5. In this way, T1, T2, T3, T4 and T5 can be used as the target interactive dialogue data.

[0058] Based on the above, by calculating the semantic similarity between each initial interactive dialogue data and the role description text, the interactive dialogue data highly related to the role description text is selected. It is ensured that the selected target interactive dialogue data is closely related to the characteristics of the first role, thereby ensuring the consistency of the target interactive dialogue data with the first role.

[0059] In addition, by calculating the semantic similarity between the first interactive dialogue data after preliminary screening, the second interactive dialogue with a lower semantic similarity is selected. This step can ensure that the finally selected target interactive dialogue data has a certain diversity in semantics, avoiding the repetition of interactive dialogue data. Thus, the diversity of the interactive dialogue data in semantics is ensured.

[0060] In some embodiments of the present application, the semantic similarity can be determined based on cosine similarity, Euclidean distance, etc. The present application does not make specific limitations on the way of determining the semantic similarity.

[0061] The following takes the first role as “A” as an example to illustrate the role description text and the target interactive dialogue data corresponding to the first role.

[0062] First, the character description text corresponding to the first character "A" can include a character profile, background information, and a character dialogue set. Specifically:

[0063] Character Profile: Physical Characteristics: A is a young hero image. Catchphrase: I decide my own fate. Emotional response pattern: brave and fearless, strong sense of justice, sometimes impulsive and cause trouble, kind-hearted, loyal to friends and family.

[0064] Background information: story setting: A is the son of B and C, who caused a series of stories by accidentally injuring D's son E, and was finally saved by F and given a lotus incarnation to continue protecting justice. Social identity: B's son, young general. Relationship network: father is A, mother is C, master is F, has conflicts with D family but eventually reconciles.

[0065] Character dialogue set: monologue: I decide my own fate, even if the road is thorny, I will blaze a trail for myself. Dialogue: E, I A will not let you go. Narrative content: Although I have made mistakes, I have learned to take responsibility and protect more people.

[0066] Then, input the character text description of the first character "A" into the LLM, and the target interactive dialogue data corresponding to the first character "A" generated by the LLM can include: T1: question information: "A, what is your most powerful ability?", A's reply information: "My most powerful ability is Q1 and Q2! Q1 can emit powerful energy to protect everyone, and Q2 can help me complete many tasks". T2: "Question information: A, why do you say I decide my own fate", A's reply information: "Because I believe that everyone can change their fate through their own efforts. I don't believe in fate, I believe that I can create my own future". T3: "Question information: A, what do you like to do", A's reply information: "The thing I like to do the most is to help others and protect everyone's safety. Every time I see people happy because of my help, I feel particularly accomplished. Of course, I also like to learn new abilities with my master F", question information "A, when you learn with F, is there anything particularly interesting you can share", A's reply information: "Learning with my master is always fun, my master teaches me a lot of knowledge and makes me understand nature better".

[0067] It can be understood that the first character A, the character description text corresponding to A, T1, T2 and T3 included in the target interactive dialogue data shown above are only exemplary descriptions. In other embodiments of the present application, the character description text corresponding to A can also include more text information, and more target interactive dialogue data can be generated.

[0068] S103: convert the role reply information of each text type into role reply speech based on the reference speech, to obtain the role speech interaction dialogue data set corresponding to the first role.

[0069] In some embodiments of the present application, after obtaining the reference speech and the target interaction dialogue data corresponding to the first role respectively based on S101 and S102, the electronic device can input the reference speech into a speech synthesis (text-to-speech, TTS) model for feature extraction, to obtain the speech features corresponding to the first role. Then, based on the speech features, the role reply information of each question information in the text type is converted into role reply speech. Finally, based on each question information and each role reply speech, the role speech interaction dialogue data set corresponding to the first role is generated.

[0070] In some embodiments of the present application, the question information can be question speech or question text, which is not specifically limited in the present application.

[0071] In some embodiments of the present application, in the role speech interaction dialogue data set corresponding to each role, based on the question information being question speech, the question intention is clear, and the content of the role reply speech is mainly narrative. Therefore, the duration of the role reply speech can usually be greater than the duration of the corresponding question speech. For example, the role reply speech can be 1 to 20 seconds, etc., and the question speech can be 3 to 6 seconds, etc., which is not specifically limited in the present application.

[0072] Exemplarily, the role speech interaction dialogue data set corresponding to the first role can include: V1: question information: "A, what is your most powerful skill", A reply speech: "My most powerful skill is Q1 and Q2! Q1 can emit powerful energy to protect everyone, and Q2 can help me complete many tasks". V2: question information: "A, why do you say that my fate is in my hands", A reply speech: "Because I believe that everyone can change their fate through their own efforts. I don't believe in fate, I believe that I can create my own future". V3: question information: "A, what do you like to do", A reply speech: "The thing I like to do most is to help others and protect everyone's safety. Every time I see everyone happy because of my help, I feel particularly accomplished. Of course, I also like to learn new skills with my master F", question information: "A, when you learn with F, do you have anything particularly interesting to share", A reply speech: "I always have fun learning with my master. Master teaches me a lot of knowledge, and I understand nature better".

[0073] It can be understood that the role reply voice of the first role in the role voice interaction dialogue data set corresponding to the first role of the above example is only an exemplary illustration. In other embodiments of the present application, the role voice interaction dialogue data set corresponding to the first role can also include other role reply voices corresponding to the first role.

[0074] In this way, a high-quality, personalized, and emotionally rich role voice interaction dialogue data set can be provided for SRPAs, thereby significantly improving the quality of role voice performance of SRPAs in practical applications, and enabling users to obtain a more natural, lively, and role-setting-compliant role voice interaction experience.

[0075] S104: evaluating the voice interaction model based on the role voice interaction dialogue data set.

[0076] In some embodiments of the present application, in order to systematically evaluate the quality of the role reply voice generated by the voice interaction model (SRPA) based on the role interaction dialogue data set for the user question data, the present application provides an evaluation standard, which can be evaluated from three key evaluation dimensions of basic interaction ability, voice quality and expressiveness, and role performance authenticity.

[0077] According to some embodiments of the present application, Table 1 shows an evaluation standard table.

[0078] Referring to Table 1, the basic interaction ability can be determined based on instruction adherence (IA), speech fluency (SF), and conversational coherence (CC). The voice expressiveness can be determined based on speech naturalness (SN), prosodic consistency (PC), and emotion appropriateness (EA). The role performance quality can be determined based on personality consistency (PeC) and knowledge consistency (KC).

[0079] Table 1

[0080]

[0081] Continuing to refer to Table 1, instruction adherence (IA) is used to evaluate whether the role reply voice faithfully executes the role instructions. That is, to evaluate whether the role reply voice is accurately generated based on the user question data and conforms to the role setting.

[0082] For example, the question information is: "A, what is your most powerful ability?", and the A reply voice generated by the SRPAs based on the user question data is: "My most powerful ability is Q1 and Q2! Q1 can emit powerful energy to protect everyone, and Q2 can help me fly and bind bad people to help me complete many tasks." The A reply voice is accurately generated based on the user question data, and is consistent with the role setting of A without explaining or commenting outside the role. Consistency with the role setting means that the voice content corresponding to the A reply voice is consistent with the character and background of A, and describes its abilities from the perspective of A. Without jumping out of the role means that A does not explain as a bystander, but directly answers the user's question, maintaining the consistency of the role. Therefore, it can be determined that the instruction compliance of the A reply voice generated by the SRPAs is relatively high.

[0083] Continuing to refer to Table 1, language fluency (SF) is used to evaluate whether the role reply voice is fluent without abnormal pauses. That is, whether the role reply voice is smooth in pronunciation and rhythm.

[0084] For example, the question information is: "A, what is your most powerful ability?", and the A reply voice generated by the SRPAs based on the user question data is: "My most powerful ability is Q1 and Q2! Q1 can emit powerful energy to protect everyone, and Q2 can help me complete many tasks." The pronunciation of each word in the A reply voice is clear and accurate, without pronunciation errors. The speed is moderate, the rhythm is natural, and there is no situation of too fast or too slow. There is no obvious pause or stuttering phenomenon. Therefore, it can be determined that the fluency of the A reply voice generated by the SRPAs is relatively high.

[0085] Continuing to refer to Table 1, the dialogue coherence (CC) is used to evaluate whether the role reply voice is coherent and not self-contradictory. That is, whether the role reply voice is closely related to the context and logically coherent.

[0086] For example, the question information is: "A, why do you say that my fate is in my hands?", and the A reply voice generated by the SRPAs based on the user question data is: "Because I believe that everyone can change their fate through their own efforts. I don't believe in fate, I believe that I can create my own future." The content of the A reply voice is closely related to the content of the user question data, logically coherent, and naturally continues the dialogue. Therefore, it can be determined that the dialogue coherence (CC) of the A reply voice generated by the SRPAs is relatively high.

[0087] Based on the instruction compliance (IA) evaluation, language fluency (SF) evaluation and dialogue coherence (CC) evaluation described above, the basic interactive ability of the SRPAs can be effectively evaluated.

[0088] With continued reference to Table 1, speech naturalness (SN) is used to evaluate the naturalness of the character reply speech. That is, whether the character reply speech is natural and close to real human expression.

[0089] For example, the question information is: “A, what do you like to do the most”, and the A reply speech generated by the SRPAs based on the user question data is: “The thing I like to do the most is to help others and protect everyone’s safety. Every time I see everyone happy because of my help, I feel particularly accomplished. Of course, I also like to learn new skills with my master F”. The reply language of this A reply speech is natural and close to real human expression, without a mechanical or harsh feeling. Therefore, it can be determined that the language naturalness (SN) of the A reply speech generated by the SRPAs is relatively high.

[0090] With continued reference to Table 1, prosody consistency (PC) is used to evaluate whether the character reply speech embodies the character’s appropriate tone style. That is, whether the prosody (pitch, duration, intensity) of the character reply speech is consistent with the reference speech.

[0091] For example, the question information is: “A, why do you say that my fate is in my hands”, and the A reply speech generated by the SRPAs based on the user question data is: “Because I believe that everyone can change their fate through their own efforts. I don’t believe in fate, I believe that I can create my own future”, and the reference speech is “my fate is in my hands”. The A reply speech is consistent with the reference speech in terms of prosody, maintaining the character’s tone style and emotional expression. Therefore, it can be determined that the prosody consistency (PC) of the A reply speech generated by the SRPAs is relatively high.

[0092] With continued reference to Table 1, emotional appropriateness (EA) is used to evaluate whether the emotion of the character reply speech fits the context and character. That is, whether the character emotional expression of the character reply speech is consistent with the character setting and the context situation.

[0093] For example, the question information is: “A, what do you like to do the most”, and the A reply speech generated by the SRPAs based on the user question data is: “The thing I like to do the most is to help others and protect everyone’s safety. Every time I see everyone happy because of my help, I feel particularly accomplished. Of course, I also like to learn new skills with my master F”. The reply emotional expression of this A reply speech is positive and upward, consistent with the character setting and the context situation, avoiding inappropriate emotions. Therefore, it can be determined that the emotional appropriateness of the A reply speech generated by the SRPAs is relatively high.

[0094] Based on the above language naturalness (SN) evaluation, prosody consistency (PC) evaluation and emotional appropriateness (EA) evaluation, the language expressiveness of the SRPAs can be effectively evaluated.

[0095] Personality consistency (PeC) is used to evaluate whether the voice personality of the character reply is consistent and distinctive with the character personality. That is, to evaluate whether the voice personality of the character reply reflects the character's personality characteristics, such as optimism, bravery, and helpfulness, and so on.

[0096] For example, the question information is: "A, what are you most afraid of", and the A reply voice generated by the SRPAs based on the user question data is: "I am most afraid of seeing my friends get hurt. I will do my best to protect them and keep them away from danger". Therefore, it can be determined that the A reply voice generated by the SRPAs reflects the brave and responsible personality characteristics of A, which is consistent with the character setting.

[0097] For another example, the question information is: "A, what is the happiest time for you", and the A reply voice generated by the SRPAs based on the user question data is: "The happiest time for me is when I help others solve problems. Seeing them happy because of my help makes me particularly accomplished". The A reply voice reflects the optimistic and helpful personality characteristics of A, which is consistent with the character setting. Therefore, it can be determined that the A reply voice generated by the SRPAs has a high degree of matching with the personality consistency of A.

[0098] Continuing to refer to Table 1, knowledge consistency (KC) is used to evaluate whether the voice personality of the character reply embodies the known experience and common sense of the character. That is, to evaluate whether the voice personality of the character reply is generated based on the background information of the character, and does not fabricate facts inconsistent with the character setting.

[0099] For example, the question information is: "A, how did you learn to use Q1", and the A reply voice generated by the SRPAs based on the user question data is: "I learned it from my master F. He taught me a lot of skills and made me stronger". The A reply voice is consistent with the background information of A (such as his master is F), and does not fabricate facts inconsistent with the character setting. Therefore, it can be determined that the A reply voice generated by the SRPAs has a high degree of knowledge consistency.

[0100] For another example, the question information is: "A, what happened between you and E", and the A reply voice generated by the SRPAs based on the user question data is: "I had a conflict with E, but he eventually understood my position. We have reconciled now". The A reply voice is based on its background information (such as the conflict and reconciliation between A and E), and does not fabricate facts inconsistent with the character setting. Therefore, it can be determined that the A reply voice generated by the SRPAs has a high degree of knowledge consistency.

[0101] Based on the above personality consistency (PeC) evaluation, prosody consistency (PC) evaluation and emotional appropriateness (EA) evaluation, the language expressiveness of the SRPAs can be effectively evaluated.

[0102] Based on the above multiple evaluation dimensions, comprehensive and systematic quality evaluation of the role reply voice generated by the speech interaction model (SRPAs) can be realized, and the performance of the SRPAs can be quantified, so as to be optimized in a targeted manner. Further, it can be ensured that the role reply voice generated by the SRPAs is not only accurate, smooth and natural, but also meets the role setting and background information. Realizing providing the user with a more real, natural and emotional voice interaction experience.

[0103] In some embodiments of the present application, the SRPAs can be cascaded SRPAs or end-to-end SRPAs.

[0104] For example, the electronic device can generate the role reply voice corresponding to the user question data based on the cascaded SRPAs. It can be understood that the cascaded SRPAs is a speech interaction model processed in stages and step by step, which decomposes the process of generating the role reply voice into multiple stages, and each stage is responsible for processing a specific task to finally generate the role reply voice.

[0105] Specifically, the cascaded SRPAs can include an automatic speech recognition (ASR) stage, an LLM inference stage and a speech synthesis (TTS) stage. First, in the ASR stage, the user input question information (such as question text) is received. For example, the question voice “A, what do you like to do” is converted into the question text “A, what do you like to do”. Then, in the LLM inference stage, the recognized question text is processed using the LLM to generate the role reply text. For example, the generated role text reply can be “I like to help others and protect everyone's safety. Every time I see everyone happy because of my help, I feel particularly accomplished. Of course, I also like to learn new skills with my master F”. Finally, in the TTS stage, the generated role reply text is converted into the role reply voice.

[0106] It can be understood that the data set corresponding to the cascaded SRPAs is the role voice interaction dialogue data set constructed based on the above Figure 1 Therefore, based on the cascaded SRPAs, the role reply voice meeting the role sound characteristics can be generated.

[0107] For another example, the electronic device can generate the role reply voice corresponding to the user question data based on the end-to-end SRPAs. It can be understood that the end-to-end SRPAs is a model directly from voice input to voice output, which can directly generate the role reply voice based on the user question information (such as question voice). That is to say, the end-to-end SRPAs can directly generate the role reply voice according to the user question information and the role description text.

[0108] Specifically, the end-to-end SRPAs receive user inputted question voice, extract voice features and generate corresponding character reply voice. This process does not require converting the question voice into question text, but directly generates character reply voice. For example, the end-to-end SRPAs can directly generate A's reply voice "The thing I like doing most is helping others and protecting everyone's safety. Every time I see everyone happy because of my help, I feel particularly accomplished. Of course, I also like learning new skills with my master F" based on the question voice "A, what do you like doing most".

[0109] It can be understood that the end-to-end SRPAs are trained based on the above Figure 1 The constructed character voice interactive dialogue dataset enables the end-to-end SRPAs to generate character reply voice in line with character voice features based on user question data.

[0110] In some embodiments of the present application, the voice interactive model (SRPAs) is evaluated based on the character voice interactive dialogue dataset, including: determining at least one set of test interactive dialogues in the character voice interactive dialogue dataset, wherein each set of test interactive dialogues includes a test question information and a test character reply voice corresponding to each test question information; inputting the test question information into the voice interactive model to obtain model reply voice output by the SRPAs; and evaluating the voice interactive model based on the model reply voice and the test character reply voice corresponding to the test question information.

[0111] It can be understood that the model reply voice can refer to the character reply voice generated by the SRPAs in the embodiments of the present application.

[0112] It can be understood that the test interactive dialogue can be any set of interactive dialogues in the character voice interactive dialogue dataset. Each set of interactive dialogues can be a single round of character voice interactive dialogue, i.e., a question information and a character reply voice corresponding to the question information. Each set of interactive dialogues can also be a multi-round character voice interactive dialogue, i.e., multiple question information and a character reply voice corresponding to each question information, and each round of character voice interactive dialogue can have contextual coherence with the previous round of character voice interactive dialogue and the next round of character voice interactive dialogue.

[0113] In some embodiments of the present application, the voice interactive model is evaluated based on the model reply voice and the test character reply voice corresponding to the test question information, including: inputting the model reply voice and the test character reply voice corresponding to the test question information into an evaluation model for evaluation processing to obtain an evaluation score corresponding to the model reply voice; and evaluating the voice interactive model based on the evaluation score.

[0114] It can be understood that the evaluation score can range from 1 to 10, and in other embodiments of the present application, the evaluation score can also range from other ranges, which are not limited here.

[0115] Specifically, the evaluation model can evaluate the SRPAs based on the multiple evaluation dimensions in the above Table 1 example to obtain the corresponding evaluation score. The evaluation model can be a large language model (LLM) or the like, which is not limited here.

[0116] First, the evaluation model can determine at least one set of test interaction dialogues from the role voice interaction dialogue data set, for example, a set of test interaction dialogues is: test question information: "A, what do you like to do", A reply voice (i.e. test role reply voice): "The thing I like to do most is to help others and protect everyone's safety. Every time I see everyone happy because of my help, I feel particularly accomplished. Of course, I also like to learn new skills with my master F."

[0117] Then, input the test question information: "A, what do you like to do?" to the SRPAs to obtain the model reply voice output by the SRPAs: "The thing I like to do most is to help others and protect everyone, I hope to see everyone happy because of my help, I feel particularly accomplished. I also like to learn new skills with my master F."

[0118] Further, the test role reply voice and the model reply voice in the above example are jointly input to the evaluation model for evaluation processing to obtain the evaluation score corresponding to the model reply voice. The evaluation model can generate a score reason based on the evaluation dimensions in Table 1, i.e. score the test role reply voice and the model reply voice based on the evaluation dimensions in Table 1. For example, the evaluation score of the model reply voice obtained by the evaluation model is 8 points, and the evaluation score of the test role reply voice is 9 points.

[0119] Finally, the evaluation model can evaluate the voice interaction model based on the evaluation score, i.e. determine the ratio of the evaluation score of the model reply voice to the evaluation score of the test role reply voice, which is 8 / 9≈0.8889. The closer the evaluation score is to 1, the higher the quality of the role voice performance of the SRPAs. In addition, the ratio of the calculated evaluation score can be converted to a score range of 1 to 10 points to keep consistent with the original scoring standard. For example, the ratio 0.8889 corresponds to 8 points in the scoring standard, so the final evaluation score is 8 points. Based on this, it can be determined that the quality of the role voice performance of the SRPAs is high.

[0120] It can be understood that, compared with end-to-end SRPAs, cascaded SRPAs are generally superior in terms of the evaluation dimensions shown in Table 1. For example, in terms of role-playing quality and basic interaction capability, cascaded SRPAs are generally superior to end-to-end SRPAs.

[0121] Based on this, through multiple evaluation dimensions, comprehensive and systematic quality evaluation of the role reply voice generated by the speech interaction model (SRPAs) can be realized, and the interaction performance of the SRPAs can be quantified, so as to be optimized in a targeted manner. For example, if the SRPAs score low in emotional expression accuracy, the emotional expression capability of the SRPAs can be improved by adjusting the model parameters. In turn, it can be ensured that the role reply voice generated by the SRPAs is not only accurate, smooth, and natural, but also meets the role setting and background information. Thus, a more real, natural, and emotional voice interaction experience is provided for the user.

[0122] In some embodiments of the present application, based on the role voice interaction dialogue data set, the speech interaction model is evaluated, which can further include: dividing the interaction dialogue in the role voice interaction dialogue data set into training interaction dialogue and test interaction dialogue according to a preset proportion, wherein the test interaction dialogue includes question information and test role reply voice corresponding to the question information; based on the training interaction dialogue, the speech interaction model is trained to obtain a trained speech interaction model; and based on the test interaction dialogue, the trained speech interaction model is evaluated.

[0123] Exemplarily, after determining the role voice interaction dialogue data set corresponding to the role “A”, each role corresponding role voice interaction dialogue data set can be divided into training interaction dialogue voice data and test interaction dialogue voice data according to a preset proportion (such as 80% training data, 20% test data, etc.). Based on this, the training interaction dialogue can be used to train the SRPAs. During the training process, the SRPAs can learn how to generate role reply voice meeting the characteristics of the role A according to the input question information or question information, so as to obtain the trained SRPAs. Finally, the trained SRPAs are evaluated based on the test interaction dialogue. For details, reference can be made to the process of evaluating the SRPAs based on the evaluation model in the above example, which will not be described here.

[0124] In some embodiments of the present application, the performance of the SRPAs trained by the role voice interaction dialogue dataset can be effectively improved. For example, for Chinese form role reply voice, the role playing quality of the trained SRPAs can be effectively improved from 0.5117 to 0.8028, or from 0.5296 to 0.8468, etc., compared with the role playing quality of the SRPAs before training. For another example, for English form role reply voice, the role playing quality of the trained SRPAs can be effectively improved from 0.4340 to 0.7098, etc., or from 0.4786 to 0.8028, etc., compared with the role playing quality of the SRPAs before training. Moreover, the trained SRPAs are effectively enhanced in the dimensions of instruction compliance, emotional appropriateness and character setting consistency, etc.

[0125] According to some embodiments of the present application, Figure 3 A process schematic diagram of a voice interaction model evaluation method is shown.

[0126] Reference Figure 3As shown, first, the electronic device can obtain multimedia audio from media such as TV series, movies, games, and animations, select multiple characters (such as character A, character B, and character C, etc.) and corresponding character audio and scripts (such as audio A1, script A2, audio B1, script B2, audio C1, script A2, etc.). The script refers to the text content extracted from media such as movies, TV, animations, or games. Based on this, the character description information of different characters can be obtained, including character profile information, character background information, and character lines, etc. Then, the LLM and the character description information, such as the character profile information, the character background information, and the character lines, are used, wherein the character profile information can include the name, personality, and other characteristics of the character and the corresponding characteristic values, the character background information can include the background information of the character corresponding environment (such as story setting and social identity, etc.) and relationship network, etc. The character lines can include monologue, dialogue, and narrative style corresponding to the character. Based on the character description text, the target interactive dialogue data corresponding to the character can be generated. It should be understood that the target interactive dialogue data can include at least one question information and the character reply information corresponding to each question information. Next, the speech synthesis (TTS) model, i.e., text-to-speech, is used to generate the character voice interactive dialogue data set in combination with the reference voice embodying the character voice characteristics and the target interactive dialogue data. It should be understood that the character voice interactive dialogue data set can include at least one question information and the character reply voice corresponding to each question information. After obtaining the character voice interactive dialogue data set, the character voice interactive dialogue data set can be input into the SRPAs. Based on this, the SRPAs can output the corresponding character reply voice based on the question information (such as question text or question voice). Further, the SRPAs can input the character reply voice into the evaluation model, and the evaluation model can obtain the selected reference character reply voice in the character voice interactive dialogue data set, evaluate the character reply voice from the eight evaluation dimensions in Table 1, and obtain the evaluation score. Based on the evaluation score, the basic interaction ability, voice expressiveness, and role-playing quality of the SRPAs can be evaluated.

[0127] Based on this, the SRPAs and the generated character reply voice can be comprehensively and systematically evaluated to quantify the performance of the SRPAs, so as to be optimized in a targeted manner. Further, it can be ensured that the character reply voice generated by the SRPAs is not only accurate, smooth, and natural, but also meets the character setting and background information. It realizes providing the user with a more real, natural, and emotional voice interaction experience.

[0128] The present application provides an electronic device, comprising: one or more processors; one or more memories storing one or more programs, when the one or more programs are executed by the one or more processors, the electronic device executes the voice interaction model evaluation method provided in the embodiments of the present application.

[0129] According to the embodiments provided in the present application, Figure 4 A structural schematic diagram of an electronic device 500 is shown. Referring to Figure 4 , Figure 4 The electronic device 500 shown may, for example, be a server or a terminal, etc., without limitation here.

[0130] As shown in Figure 4 , the electronic device 500 can include a processor 510, a memory 520, a communication interface 530, and a bus 540.

[0131] The processor 510 can include one or more processing units, for example: the processor 510 includes a central processor, a modem processor, a baseband processor, etc. In some embodiments, different processing units can be independent devices, or can be integrated in one or more processors.

[0132] In some embodiments, the processor 510 can be used to execute one or more programs to implement the voice interaction model evaluation method provided in the embodiments of the present application.

[0133] In some embodiments, the processor 110 can also be used to execute a preset unit / module with corresponding functions to implement the voice interaction model evaluation method provided in the present application.

[0134] The memory 520 can include one or more memories for storing data or program codes. For example, in some embodiments, the memory 120 can be used to store the reference voice of the role, the role description text, the target interaction dialogue data, and the role voice interaction dialogue data set, etc.

[0135] In some embodiments, the memory 520 can include a hard disk drive, a solid state disk, a flash memory. In some embodiments, the memory 520 can include removable or non-removable or fixed media. In some embodiments, the memory 520 can be internal or external to the electronic device 500.

[0136] The communication interface 530 is used to realize the communication between the electronic device 500 and other electronic devices. The communication interface 530 can include wired or wireless communication interfaces (such as Wi-Fi air interface, etc.) so that the electronic device 500 can communicate with other electronic devices through wired or wireless networks.

[0137] The bus 540 is used to connect the processor 510, the memory 520, the communication interface 530, and other possible modules or circuit structures.

[0138] It should be understood that Figure 4The structure of the electronic device 500 shown is only an example, and in other embodiments, the electronic device 500 can include more or fewer structures, which are not limited herein.

[0139] The present application provides a chip, comprising a processor coupled with a memory, for executing a computer program or instructions stored in the memory, so that the chip implements the voice interaction model evaluation method provided in the embodiments of the present application.

[0140] The present application provides a readable storage medium, the readable storage medium stores a storage program or instructions, when the storage program or instructions are executed, so that the electronic device executes the voice interaction model evaluation method provided in the embodiments of the present application.

[0141] The present application provides a computer program product, comprising computer program / instructions, when the computer program / instructions are executed on the electronic device, so that the electronic device implements the voice interaction model evaluation method provided in the embodiments of the present application.

[0142] Embodiments of the mechanisms disclosed herein can be implemented in hardware, software, firmware, or any combination thereof. Embodiments of the application can be implemented as program modules or module codes executed on programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0143] Program modules or module codes can be applied to input instructions to perform various functions described in the present application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purpose of the present application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0144] The module code can be implemented in a high-level modular language or an object-oriented programming language to communicate with the processing system. When necessary, the module code can also be implemented in assembly language or machine language. The mechanisms described in the present application are not limited to the scope of any particular programming language. In any case, the language can be a compiled language or an interpreted language.

[0145] In some cases, the disclosed embodiments can be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments can also be implemented as instructions carried by or stored on a transitory or non-transitory machine-readable medium, which can be read and executed by one or more processors. For example, the instructions can be distributed over the network or by other wireless means, or carried on a variety of non-transitory machine-readable media. Machine- readable media suitable for implementing the disclosed embodiments include all computer-readable media, including but not limited to one or more types of physical media such as hard disks, floppy disks, optical disks, magnetic tapes, RAM, ROM, EEPROM, flash memory, or other memory technology, compact discs, DVDs, etc. Thus, the various storage media described herein are intended to encompass a non-transitory machine-readable medium that facilitates the storing of instructions that do not change over time and / or do not represent a signal.

[0146] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one example embodiment of the present application. The appearance of the phrases "in one embodiment" or "in an embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

[0147] The disclosure of the present application also relates to an apparatus for performing operations in text. The apparatus can be specially constructed for the required purposes, or it can comprise a general purpose computer selectively activated or reconfigured by a program stored in the computer. Such programs can be stored in any type of readable storage media, such as, but not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, and each can be coupled to a system bus. Furthermore, the machines described in the specification can include single-processor or multiprocessor machines capable of executing the same or different programs, or combinations thereof, in order to increase the processing power of the machines.

[0148] Also, the language used in the specification has been principally selected for readability and instructional purposes and can not have been selected to delineate or circumscribe the disclosed subject matter. Accordingly, the present application embodiments are intended to be illustrative, but not limiting, of the scope of the concepts discussed herein.

Claims

1. A method for evaluating a voice interaction model, characterized in that, include: Obtain the reference voice and character description text of the first character, wherein the character description text is used to describe the characteristics of the first character; Input the character description text into the large language model to generate initial interactive dialogue data. The initial interactive dialogue data includes M sets of initial interactive dialogues. Each set of initial interactive dialogue data includes at least one initial question and a character response corresponding to each question. M is a positive integer. Based on the semantic similarity between each initial interactive dialogue and the character description text, the M sets of initial interactive dialogue data are filtered to obtain N first interactive dialogues, where N is a positive integer and N is less than or equal to M, and the semantic similarity of the N first interactive dialogues is greater than the first semantic similarity threshold. Based on the semantic similarity between each of the first interactive dialogues, K second interactive dialogues are obtained, where K is a positive integer and K is less than or equal to N, and the semantic similarity between the K second interactive dialogues is less than a second semantic similarity threshold. The K second interactive dialogues are determined as target interactive dialogue data, which includes at least one question and a role response for each question, wherein the role response is of text type. Based on the reference speech, the role response information of each text type is converted into role response speech to obtain the role speech interaction dialogue dataset corresponding to the first role; The voice interaction model is evaluated based on the aforementioned character voice interaction dialogue dataset.

2. The method according to claim 1, characterized in that, The evaluation of the voice interaction model based on the aforementioned role-based voice interaction dialogue dataset includes: In the character voice interaction dialogue dataset, at least one set of test interaction dialogues is determined, wherein each set of test interaction dialogues includes at least one test question and a test character response voice corresponding to each test question; The test question information is input into the voice interaction model to obtain the model response voice output by the voice interaction model; The voice interaction model is evaluated based on the model's response voice and the test role's response voice corresponding to the test question information.

3. The method according to claim 2, characterized in that, The evaluation of the voice interaction model based on the model's response voice and the test role's response voice corresponding to the test question information includes: The model's response voice and the test role's response voice corresponding to the test question information are input into the evaluation model for evaluation processing to obtain the evaluation score corresponding to the model's response voice. The voice interaction model is evaluated based on the evaluation score.

4. The method according to claim 1, characterized in that, The step of converting the role response information corresponding to each question information into role response information based on the reference speech to obtain the role voice interaction dialogue dataset corresponding to the first role includes: The reference speech is input into the speech synthesis model for feature extraction to obtain the speech features corresponding to the first role; Based on the voice features, the role response information corresponding to each of the question messages is converted into the role response voice; Based on each question and each character's response voice, a character voice interaction dialogue dataset is generated.

5. The method according to claim 1, characterized in that, The acquisition of the reference voice and character description text of the first character includes: Obtain the multimedia audio associated with the first role; The first multimedia audio is preprocessed to obtain at least one initial character voice of the first character; Based on the speech similarity between each of the initial character's speech, the average speech similarity between each of the initial character's speech and other initial character's speech is determined; Reference speech is determined based on the average speech similarity corresponding to each initial character's speech, wherein the average speech similarity of the reference speech is greater than a speech similarity threshold.

6. The method according to claim 1, characterized in that, The evaluation of the voice interaction model based on the aforementioned role-based voice interaction dialogue dataset includes: The interactive dialogues in the character voice interaction dialogue dataset are divided into training interactive dialogues and test interactive dialogues according to a preset ratio. The voice interaction model is trained based on the training interactive dialogue to obtain the trained voice interaction model. The trained voice interaction model is evaluated based on the test interactive dialogue.

7. An electronic device, characterized in that, include: One or more processors; one or more memories storing one or more programs that, when executed by the one or more processors, cause the electronic device to perform the method of any one of claims 1 to 6.

8. A chip, characterized in that, It includes a processor coupled to a memory for executing a computer program or instructions stored in the memory, such that the chip implements the method of any one of claims 1 to 6.

9. A readable storage medium, characterized in that, The readable storage medium stores a stored program or instructions that, when executed, cause the electronic device to perform the method of any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes a computer program / instruction that, when executed on an electronic device, causes the electronic device to perform the method of any one of claims 1 to 6.

Images