Speech synthesis model evaluation method and device and storage medium

Abstract

The invention provides a speech synthesis model evaluation method and device and a storage medium, relates to the technical field of computers, and can evaluate a novel speech synthesis model. The method comprises the steps of obtaining a plurality of style instructions, wherein the style instructions are used for indicating voice features of voice data needing to be output; determining voice data corresponding to each style instruction based on a to-be-tested voice synthesis model; and evaluating the to-be-tested speech synthesis model according to the matching degree between the plurality of style instructions and the respective corresponding speech data.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a speech synthesis model evaluation method, device and storage medium. Background Art

[0002] With the development of speech synthesis technology, especially based on large language models, new speech synthesis models with multiple speaker roles, multiple emotions, and multiple styles have been developed, such as ChatTTS, PromptTTS, and StyleTTS. In traditional speech synthesis models, the input is only content instructions, and the output synthesized speech only needs to meet the content instructions. In new speech synthesis models, the input includes not only content instructions but also style instructions. The output synthesized speech needs to meet not only the content instructions but also the style instructions.

[0003] At present, traditional speech synthesis model evaluation methods evaluate speech synthesis models through subjective mean opinion score (MOS), comparative mean opinion score (CMOS) or ABTest. However, the above methods can only evaluate the matching degree between the speech synthesized by the speech synthesis model and the content instruction, but cannot evaluate the matching degree between the speech synthesized by the speech synthesis model and the style instruction, that is, they cannot evaluate new speech synthesis models. Summary of the invention

[0004] The present application provides a speech synthesis model evaluation method, device and storage medium, which can evaluate a new speech synthesis model.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, the present application provides a method for evaluating a speech synthesis model, the method comprising: obtaining a plurality of style instructions, the style instructions being used to indicate speech features of speech data to be output; determining the speech data corresponding to each style instruction based on the speech synthesis model to be tested; and evaluating the speech synthesis model to be tested based on the degree of match between the plurality of style instructions and their respective corresponding speech data.

[0007] In a possible implementation, the speech feature includes at least one of the following: timbre feature; emotion feature; speech speed feature; and pitch feature.

[0008] In one possible implementation, based on the speech synthesis model to be tested, the speech data corresponding to each style instruction is determined, including: obtaining multiple content instructions, the content instructions are used to indicate the text content of the speech data to be output; for each style instruction, the style instruction and any content instruction among the multiple content instructions are input into the speech synthesis model to be tested, and the speech data corresponding to the style instruction is obtained, so as to obtain the speech data corresponding to each style instruction.

[0009] In a possible implementation, the method further includes: for each style instruction, acquiring a semantic feature of the style instruction and an audio feature of speech data corresponding to the style instruction; and determining a matching degree between the style instruction and the speech data based on the semantic feature of the style instruction and the audio feature of the speech data.

[0010] In a possible implementation, determining the degree of match between the style instruction and the speech data based on the semantic features of the style instruction and the audio features of the speech data includes: taking the similarity between the semantic features of the style instruction and the audio features of the speech data as the degree of match between the style instruction and the speech data.

[0011] In one possible implementation, the speech synthesis model to be tested is evaluated according to the degree of match between multiple style instructions and their corresponding speech data, including: obtaining evaluation parameters according to the degree of match between multiple style instructions and their corresponding speech data, the evaluation parameters being the average value and standard deviation of the match; evaluating the speech synthesis model to be tested according to the evaluation parameters and a preset equalization coefficient, the preset equalization coefficient being used to adjust the weight of the average value and standard deviation of the match.

[0012] In a second aspect, the present application provides a speech synthesis model evaluation device, which includes: a communication unit and a processing unit; the communication unit is used to obtain multiple style instructions, the style instructions are used to indicate the speech features of speech data that need to be output; the processing unit is used to determine the speech data corresponding to each style instruction based on the speech synthesis model to be tested; the processing unit is also used to evaluate the speech synthesis model to be tested based on the matching degree between the multiple style instructions and the corresponding speech data.

[0013] In a possible implementation, the speech feature includes at least one of the following: timbre feature; emotion feature; speech speed feature; and pitch feature.

[0014] In one possible implementation, the communication unit is further used to obtain multiple content instructions, where the content instructions are used to indicate the text content of the voice data that needs to be output; for each style instruction, the processing unit is further used to input the style instruction and any content instruction among the multiple content instructions into the speech synthesis model to be tested, and obtain the voice data corresponding to the style instruction, so as to obtain the voice data corresponding to each style instruction.

[0015] In a possible implementation, the method further includes: for each style instruction, the communication unit is further used to obtain the semantic features of the style instruction and the audio features of the voice data corresponding to the style instruction; the processing unit is further used to determine the matching degree between the style instruction and the voice data based on the semantic features of the style instruction and the audio features of the voice data.

[0016] In a possible implementation, the processing unit is further configured to use the similarity between the semantic features of the style instruction and the audio features of the speech data as the matching degree between the style instruction and the speech data.

[0017] In one possible implementation, the processing unit is further used to obtain evaluation parameters based on the matching degree between multiple style instructions and their corresponding speech data, and the evaluation parameters are the average value and standard deviation of the matching degree; the processing unit is also used to evaluate the speech synthesis model to be tested based on the evaluation parameters and a preset equalization coefficient, and the preset equalization coefficient is used to adjust the weight of the average value and standard deviation of the matching degree.

[0018] In a third aspect, the present application provides a speech synthesis model evaluation device, which includes: a processor and a communication interface; the communication interface and the processor are coupled, and the processor is used to run a computer program or instructions to implement the speech synthesis model evaluation method described in the first aspect and any possible implementation method of the first aspect.

[0019] In a fourth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the instructions are executed on a terminal, the terminal executes the speech synthesis model evaluation method described in the first aspect and any possible implementation method of the first aspect.

[0020] In a fifth aspect, the present application provides a computer program product comprising instructions. When the computer program product runs on a speech synthesis model evaluation device, the speech synthesis model evaluation device performs the speech synthesis model evaluation method described in the first aspect and any possible implementation method of the first aspect.

[0021] In a sixth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run a computer program or instructions to implement the speech synthesis model evaluation method as described in the first aspect and any possible implementation method of the first aspect.

[0022] Specifically, the chip provided in the present application also includes a memory for storing computer programs or instructions.

[0023] The above technical solution brings at least the following beneficial effects: obtaining multiple style instructions, determining the speech data corresponding to each style instruction based on the speech synthesis model to be tested, and evaluating the speech synthesis model to be tested according to the matching degree between the multiple style instructions and the corresponding speech data. In other words, the speech synthesis model evaluation method provided by the present application evaluates the speech synthesis model according to the matching degree between the style instruction input to the speech synthesis model and the speech data output by the speech synthesis model, thereby being able to evaluate the new speech synthesis model. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of the composition of a speech synthesis model evaluation device provided in an embodiment of the present application;

[0025] Figure 2 A flowchart of a speech synthesis model evaluation method provided in an embodiment of the present application;

[0026] Figure 3 A flowchart of another speech synthesis model evaluation method provided in an embodiment of the present application;

[0027] Figure 4 A flowchart for determining the degree of match between style instructions and voice data provided in an embodiment of the present application;

[0028] Figure 5 A flow chart for determining an evaluation model provided in an embodiment of the present application;

[0029] Figure 6 A flowchart for determining a target style text provided in an embodiment of the present application;

[0030] Figure 7 A schematic diagram of a model training provided in an embodiment of the present application;

[0031] Figure 8 A schematic diagram of an audio encoder provided in an embodiment of the present application;

[0032] Fig. 9 A flowchart of another speech synthesis model evaluation method provided in an embodiment of the present application;

[0033] Fig.10 A flowchart for determining system compliance provided in an embodiment of the present application;

[0034] Fig.11 A flowchart for evaluating a single piece of data on a PromptTTS model provided in an embodiment of the present application;

[0035] Fig.12 A schematic diagram of the structure of a speech synthesis model evaluation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The speech synthesis model evaluation method, device and storage medium provided in the embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0037] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0038] The terms "first" and "second" and the like in the specification and drawings of this application are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects.

[0039] In addition, the terms "including" and "having" and any variations thereof mentioned in the description of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0040] It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0041] In the description of the present application, unless otherwise specified, “plurality” means two or more.

[0042] As described in the background technology, traditional speech synthesis model evaluation methods are gradually unable to adapt to the performance of new speech synthesis models. For example, in traditional manual evaluation methods, subjective MOS can only evaluate the overall score of synthesized speech, CMOS can only evaluate the similarity of timbre, and ABTest can only compare two different speech synthesis models based on preference. These evaluation methods are biased towards the overall properties of the speech synthesis model. It is difficult to accurately evaluate the new features of the new speech synthesis model (i.e., whether the synthesized speech follows the input style instructions), and the labor and time costs are high.

[0043] Furthermore, in order to save the manpower cost of evaluation, a variety of model-assisted evaluation algorithms have been developed. For example, UTMOS is a representative model-assisted evaluation algorithm for subjective MOS. This type of method collects subjective MOS data from speech and manual evaluation, uses a deep learning model to perform fine-tuning based on pre-trained models such as wav2vec or wavLM, and then combines the results of multiple models with the model ensemble learning method to finally predict subjective MOS.

[0044] However, the above methods are unable to solve the evaluation problem of new speech synthesis models with multiple speaker roles, multiple emotions, and multiple styles. That is, the above methods can only evaluate the matching degree between the speech synthesized by the speech synthesis model and the content instructions, but cannot evaluate the matching degree between the speech synthesized by the speech synthesis model and the style instructions.

[0045] In view of this, the embodiment of the present application provides a speech synthesis model evaluation method that can adapt to the evaluation needs of the current status of speech synthesis technology development. The method includes: obtaining multiple style instructions, determining the speech data corresponding to each style instruction based on the speech synthesis model to be tested, and evaluating the speech synthesis model to be tested according to the matching degree between the multiple style instructions and the corresponding speech data. In other words, the speech synthesis model evaluation method provided by the present application evaluates the speech synthesis model according to the matching degree between the style instructions input into the speech synthesis model and the speech data output by the speech synthesis model, and thus can evaluate the new speech synthesis model.

[0046] For example, Figure 1 A schematic diagram of the composition of a speech synthesis model evaluation device 10 provided in an embodiment of the present application, wherein the speech synthesis model evaluation device 10 may include a processor 101 and a bus 102.

[0047] Furthermore, the speech synthesis model evaluation device 10 may also include a communication interface 103 and a memory 104. The processor 101, the memory 104 and the communication interface 103 may be connected via a bus 102.

[0048] The processor 101 is a central processing unit (CPU), a general processor, a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof. The processor 101 may also be other devices with processing functions, such as circuits, devices, or software modules, without limitation.

[0049] The bus 102 is used to transmit information between the components included in the speech synthesis model evaluation device 10.

[0050] The communication interface 103 is used to communicate with other devices or other communication networks. The other communication networks may be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The communication interface 103 may be a module, a circuit, a communication interface or any device capable of realizing communication.

[0051] The memory 104 is used to store instructions, where the instructions may be computer programs.

[0052] The memory 104 may be a read-only memory (ROM) or other types of static storage devices that can store static information and / or instructions, or a random access memory (RAM) or other types of dynamic storage devices that can store information and / or instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), magnetic disk storage media or other magnetic storage devices, etc., without limitation.

[0053] It should be noted that the memory 104 can exist independently of the processor 101, or can be integrated with the processor 101. The memory 104 can be used to store instructions or program codes or some data, etc. The memory 104 can be located in the speech synthesis model evaluation device 10, or can be located outside the speech synthesis model evaluation device 10, without limitation. The processor 101 is used to execute the instructions stored in the memory 104 to implement the speech synthesis model evaluation method provided in the following embodiments of the present application.

[0054] In an example, the processor 101 may include one or more CPUs, for example, CPU0 and CPU1 (not shown in the figure).

[0055] As an optional implementation, the speech synthesis model evaluation device 10 includes multiple processors.

[0056] As an optional implementation, the speech synthesis model evaluation device 10 further includes an output device and an input device. For example, the input device is a keyboard, a mouse, a microphone, a joystick, and the like, and the output device is a display screen, a speaker, and the like.

[0057] It should be noted that the speech synthesis model evaluation device 10 can be a desktop computer, a portable computer, a network server, a mobile phone, a tablet computer, a wireless terminal, an embedded device, a chip system or a computer with Figure 1 In addition, Figure 1 The composition shown in the Figure 1 The limitations of each device in Figure 1 In addition to the parts shown, Figure 1 The various devices in the drawings may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0058] In the embodiment of the present application, the chip system may be composed of a chip, or may include a chip and other discrete devices.

[0059] In addition, the actions, terms, etc. involved in the various embodiments of the present application can refer to each other without limitation. The message name or parameter name in the message exchanged between the various devices in the embodiments of the present application is only an example, and other names can also be used in the specific implementation without limitation.

[0060] The speech synthesis model evaluation method provided in the embodiment of the present application is described below in conjunction with the accompanying drawings. Among them, the actions, terms, etc. involved in the various embodiments of the present application can refer to each other without limitation. The message name or parameter name in the message exchanged between the various devices in the embodiment of the present application is only an example, and other names can also be used in the specific implementation without limitation. The actions involved in the various embodiments of the present application are only an example, and other names can also be used in the specific implementation, such as: "included in" in the embodiment of the present application can also be replaced by "carried on" or "carried in", etc.

[0061] It should be noted that the executing entity of the technical solution of this application is not limited.

[0062] like Figure 2 As shown, the embodiment of the present application proposes a speech synthesis model evaluation method, the method comprising:

[0063] S201. Obtain multiple style instructions.

[0064] The style instruction is used to indicate the voice features of the voice data to be output.

[0065] Exemplarily, the above-mentioned voice features may include at least one of the following: timbre features, emotion features, speech speed features or pitch features. Among them, the timbre features may include gender features and age features.

[0066] Optionally, the above-mentioned voice features may also include intonation style features or volume features.

[0067] For example, the gender feature is male or female. The age feature is infant, child, teenager, youth, middle-aged or elderly. The emotion feature is happiness, sadness, anger, fear or doubt. The speech speed feature is slow, medium or fast. The pitch feature is low, medium or high. The tone style feature is steady, high, deep or loud. The volume feature is low, medium or loud.

[0068] In one example, the style instruction may be: a young male voice is speaking in a happy and high-spirited mood, and the speech speed is also very fast. The voice features corresponding to the style instruction may include: male voice, young, happy, high-spirited, and fast-speech speed.

[0069] In another example, the style instruction may be: a middle-aged female voice slowly tells a story in a gentle and low tone. The voice features corresponding to the style instruction may include: female voice, middle-aged, gentle, slow speaking speed and low volume.

[0070] S202: Determine the speech data corresponding to each style instruction based on the speech synthesis model to be tested.

[0071] The speech synthesis model to be tested refers to a new speech synthesis model that needs to be evaluated. The model can output synthesized speech according to the input content instructions and style instructions. The content instruction is used to indicate the text content of the speech data to be output. The synthesized speech output by the speech synthesis model to be tested must not only conform to the text of the content instruction, but also meet the requirements of the style instruction, that is, express the speech according to the specified speech features.

[0072] In one possible implementation, multiple content instructions are obtained, and for each style instruction, the style instruction and any content instruction among the multiple content instructions are input into the speech synthesis model to be tested to obtain speech data corresponding to the style instruction and the selected content instruction, and then the speech data corresponding to each style instruction can be obtained.

[0073] For example, if the content instruction is "Hello everyone, welcome to the speech synthesis service", and the style instruction is "A young male voice is speaking in a happy and high-spirited mood, and the speaking speed is very fast", then the speech data output by the speech synthesis model to be tested is a segment of speech by a young male voice, and the content of the speech is "Hello everyone, welcome to the speech synthesis service", and the speech is presented in a happy and high-spirited mood, and the speaking speed is relatively fast.

[0074] S203: Evaluate the speech synthesis model to be tested according to the matching degree between the multiple style instructions and the corresponding speech data.

[0075] The matching degree between the style instruction and the corresponding voice data refers to the degree to which the voice features of the voice data match the voice features indicated by the style instruction.

[0076] In a possible implementation, for each style instruction, the semantic features of the style instruction and the audio features of the speech data corresponding to the style instruction are obtained, and the similarity between the semantic features of the style instruction and the audio features of the speech data is used as the matching degree between the style instruction and the speech data.

[0077] Furthermore, the degree of match between each style instruction and the corresponding voice data is taken as the single-unit compliance, which is an indicator for evaluating the speech synthesis model under test on a single piece of data (i.e., a pair of style instructions and voice data). The system compliance of the speech synthesis model under test is determined according to the average and standard deviation of all single-unit compliances, which is an indicator for evaluating the speech synthesis model under test on multiple pieces of data. The larger the system compliance, the better the performance of the speech synthesis model under test. For details, please refer to the following Fig. 9 The described embodiments are not described in detail here.

[0078] In the speech synthesis model evaluation method provided by the present application, multiple style instructions are obtained, and based on the speech synthesis model to be tested, the speech data corresponding to each style instruction is determined, and the speech synthesis model to be tested is evaluated according to the matching degree between the multiple style instructions and the corresponding speech data. In other words, the embodiment of the present application evaluates the speech synthesis model according to the matching degree between the style instruction input to the speech synthesis model and the speech data output by the speech synthesis model, and thus can evaluate the new speech synthesis model.

[0079] Furthermore, before S203, it is necessary to determine the matching degree between each style instruction and the corresponding speech data, so that the speech synthesis model to be tested can be evaluated according to the matching degree between the multiple style instructions and the corresponding speech data. Figure 3 As shown, the speech synthesis model evaluation method recorded in the embodiment of the present application may also include the following steps.

[0080] S301 . For each style instruction, obtain semantic features of the style instruction and audio features of speech data corresponding to the style instruction.

[0081] In one possible implementation, the style instruction and the speech data are input into the evaluation model to obtain the embedding vector E of the style instruction. t (i.e., the semantic features of the style instruction) and the embedding vector E of the speech data a (i.e., the audio features of the speech data).

[0082] It should be noted that the above evaluation model is obtained by training the model in a comparative learning mode based on the target speech dataset and the target style text dataset. Figure 5 The described embodiments are not described in detail here.

[0083] In another possible implementation, a text recognition tool is used to process the style instruction to obtain the semantic features of the style instruction, and an audio recognition tool is used to process the speech data to obtain the audio features of the speech data.

[0084] S302: Determine a matching degree between the style instruction and the voice data based on the semantic features of the style instruction and the audio features of the voice data.

[0085] In a possible implementation, combined with the above S301 E t and E a For example, E a and E t The similarity is taken as the matching degree between the style instruction and the speech data.

[0086] For example, E a and E t The cosine similarity of is used as the matching degree between the style instruction and the speech data, E a and E t The cosine similarity can be obtained by the following formula 1.

[0087]

[0088] Among them, cos(E a ,E t ) means E a and E t The cosine similarity of E a Represents the embedding vector of speech data, E t Embedding vector representing the style instruction.

[0089] Of course, you can also use E a and Et Other similarities may be used as the matching degree between the style instruction and the voice data, for example, dot product similarity, which is not limited in any way in this embodiment of the present application.

[0090] For example, Figure 4 A flow chart for determining the degree of match between style instructions and voice data provided in an embodiment of the present application. Figure 4 As shown, the speech data is input into the audio encoder and the multilayer perceptron (MLP) in sequence to obtain the embedding vector E of the speech data. a The style instruction is input into the text encoder and the multi-layer perceptron in turn to obtain the embedding vector E of the style instruction t Calculate E a and E t The cosine similarity of is used to obtain the matching degree between the style instruction and the speech data.

[0091] In one embodiment, Figure 5 As shown, the evaluation model in the above S301 can be determined through the following S501 to S503.

[0092] S501: Obtain a target speech data set.

[0093] In one possible implementation, a target speech dataset is obtained from an open source speech dataset, for example, a multi-modal multi-scene multi-label emotional dialogue database (M3ED).

[0094] In another possible implementation, a speech synthesis model is used to generate a large amount of synthesized speech as a target speech dataset.

[0095] In yet another possible implementation, a web crawler is used to obtain the target speech data set from the Internet.

[0096] It should be noted that in order to improve the accuracy of the evaluation speech synthesis model, the distribution of speech data in the target speech data set should be consistent with the actual usage scenario. Specifically, as many different speech data as possible should be obtained. For example, from the gender classification, male and female voices should be obtained. From the age classification, the voices of the elderly, middle-aged and young people, and children should be obtained. From the emotion classification, voices with various emotions such as happiness, surprise, sadness, disgust / disdain, anger / anger, fear / fear, neutrality, and doubt should be obtained. From the intonation classification, voices with various intonation styles such as smooth, ordinary, high-pitched, low, soothing, whispering, gentle, playful, serious, loud, cheerful, nervous, and confident should be collected. From the speed classification, voices with different speeds such as fast, medium, and slow should be collected. From the pitch classification, voices with different pitches such as low, medium, and high should be obtained.

[0097] It is understandable that obtaining as many different speech data as possible as the target speech data set makes the speech data in the target speech data set more abundant, comprehensive and representative, thereby improving the accuracy of the evaluation model subsequently trained using the target speech data set.

[0098] S502: Determine a target style text set corresponding to the target speech data set.

[0099] In one possible implementation, for each target speech data in the target speech dataset, multiple discrete labels of the target speech data are obtained according to multiple speech classification models, and a large language model (LLM) is used to convert the multiple discrete labels into target style text described in natural language, thereby obtaining a target style text set.

[0100] It should be noted that the above-mentioned multiple speech classification models include a gender classification model, an age classification model, an emotion classification model, an intonation classification model, a speech rate classification model and a pitch classification model, and the discrete labels corresponding to the target speech data are collected through different speech classification models. Specifically, the collection process of each discrete label is specifically described below.

[0101] 1-1. Gender label collection

[0102] Use a pre-trained speech model such as wav2vec2-large or wavLM as the base model, and fine-tune the base model using an open source speech dataset with gender labels (e.g., the emotional voices database (EmoV_DB)) to obtain a gender classification model. Input the target speech data into the gender classification model to obtain the corresponding gender label.

[0103] 1-2. Age label collection

[0104] Use manual annotation to obtain age labels for some speech data, use the speech data with age labels as training data, fine-tune models such as wav2vec2-large or wavLM, and obtain an age classification model. Input the above target speech data into the age classification model to obtain the corresponding age label.

[0105] 1-3. Emotion label collection

[0106] Input the above target speech data into an open source emotion classification model, such as the emotion2vec model, to obtain the corresponding emotion label.

[0107] 1-4. Intonation tag collection

[0108] Use manual annotation to obtain the intonation labels of some speech data, use the speech data with intonation labels as training data, fine-tune models such as wav2vec2-large or wavLM, and obtain an intonation classification model. Input the above target speech data into the intonation classification model to obtain the corresponding intonation labels.

[0109] 1-5. Speech rate label collection

[0110] Use an automatic speech recognition (ASR) model, such as the whisper model, to perform speech recognition and location on the target speech data, and then calculate the average number of words spoken per second to obtain the speaking rate. Count the speaking rate information of the entire target speech data set, and divide the speaking rate into 5 levels according to the distribution of the statistical results to obtain the corresponding speaking rate labels.

[0111] 1-6. Tone label collection

[0112] Use an audio feature extraction tool (e.g., WORLD tool or STRAIGHT tool) to extract the fundamental frequency (f0) feature of the target speech data. Count the f0 values ​​of the entire target speech data set, and divide the tones into multiple levels according to the distribution of the statistical results to obtain corresponding tone labels. For example, according to the statistical results, the tones are equally divided into three levels as three tone labels of "low tone", "medium tone" and "high tone".

[0113] It should be noted that the training data obtained by manual annotation are used in the above 1-2 and 1-4. When manually annotating speech data, the annotators need to be trained, and the sample speech and annotation points corresponding to various label categories need to be provided so that the annotators understand the annotation purpose and annotation standards. In addition, the annotated data should be delivered in stages for spot checks, so that the annotators can be retrained in time after the annotation problems are found, so as to improve the accuracy of manual annotation.

[0114] Optionally, manually annotated data sets are usually divided into training sets, test sets, and validation sets. The training set can be annotated by one annotator, and the test set and validation set can be jointly annotated by three annotators to further improve the accuracy of the annotation.

[0115] Optionally, according to the label collection methods of 1-1 to 1-6, other types of labels can also be collected, such as volume labels. Specifically, if there is an open source classification model with high accuracy, the model can be used directly for label collection. If there is no open source classification model with high accuracy, the training data can be obtained by using relevant open source speech data or manual annotation. According to the training data, models such as wav2vec2-large or wavLM are fine-tuned to obtain a speech classification model, and then the speech classification model can be used to collect labels corresponding to the target speech data.

[0116] Furthermore, after obtaining multiple discrete tags, since each tag is separate and not a complete natural language description, it is necessary to convert the discrete tags into a target style text described in natural language. Specifically, a large language model such as Tongyi Qianwen or Llama can be used to convert the discrete tags into a target style text described in natural language. The specific process includes the following steps 1 and 2.

[0117] Step 1: Convert multiple discrete tags into text described in natural language.

[0118] In a possible implementation, multiple discrete tags and a first prompt are input into a large language model to obtain a text described in a natural language.

[0119] In one example, the first prompt may be: Please convert the following words into a sentence. The multiple discrete tags may be: male voice, young, fast speaking, happy, and high-pitched. In combination with this example, the input of the large language model is: Please convert the following words into a sentence: [male voice, young, fast speaking, happy, high-pitched]. The output of the large language model may be: A young male voice is speaking with a happy and high-pitched emotion, and the speaking speed is also very fast.

[0120] In addition, in order to obtain diversified target style texts, discrete labels can also be adjusted. For example, an elderly male voice can be adjusted to a grandfather.

[0121] Step 2: Rewrite the text described in natural language to obtain the target style text.

[0122] In a possible implementation, the text described in natural language and the second prompt are input into a large language model to obtain a target style text.

[0123] In one example, the second prompt may be: Please rewrite the following sentence into a synonymous sentence. The text of the natural language description may be: A young man is speaking happily and excitedly, and his speech speed is very fast. In combination with this example, the input of the large language model is: Please rewrite the following sentence into a synonymous sentence: [A young man is speaking happily and excitedly, and his speech speed is very fast]. The output of the large language model may be: A young man is speaking happily and excitedly, and his speech speed is very fast.

[0124] It is understandable that using a large speech model to rewrite text described in natural language can produce more diverse target style texts.

[0125] For example, Figure 6 A flowchart for determining a target style text is provided in an embodiment of the present application. Figure 6 As shown, the target speech data is input into the gender classification model, the age classification model, the emotion classification model, the intonation classification model, the speech speed classification model and the pitch classification model to obtain a plurality of discrete labels. The plurality of discrete labels and the first prompt are input into the large language model to obtain a text described in natural language. The text described in natural language and the second prompt are input into the large language model to obtain a text of a target style.

[0126] S503: Determine an evaluation model based on the target speech data set and the target style text set.

[0127] In a possible implementation, the target speech data set and the target style text set obtained in S502 are used to perform model training in a contrastive learning paradigm to obtain an evaluation model.

[0128] Optionally, when using the above target speech dataset and target style text set for model training, the training data is usually divided into a training set, a test set, and a validation set. In order to improve the accuracy of the constructed evaluation model, the target style text in the test set and the validation set can be manually corrected, so as to accurately verify the effect of the evaluation model.

[0129] For example, Figure 7 A schematic diagram of a model training provided in an embodiment of the present application. Figure 7As shown, using a dual-tower structure model, the target speech data is input into the audio encoder and the multi-layer perceptron in turn to obtain the embedding vector of the target speech data. The target style text is input into the text encoder and the multi-layer perceptron in turn to obtain the embedding vector of the target style text. The model is optimized by comparing the learning loss function to obtain the final evaluation model. Among them, the role of the audio encoder is to extract hidden features from the target speech data. The role of the text encoder is to extract hidden features from the target style text. The role of the multi-layer perceptron is to normalize the extracted hidden features of the target speech data and the hidden features of the target style text to the same feature dimension.

[0130] Exemplarily, the contrastive learning loss function can refer to the contrastive language image pretraining (CLIP) model and is expressed as the following formula 2.

[0131]

[0132] Where L represents the contrastive learning loss function and N represents the total number of samples. Represents the embedding vector of the target speech data in the i-th pair of samples. Represents the embedding vector of the target speech data in the jth pair of samples. Represents the embedding vector of the target style text in the i-th pair of samples. represents the embedding vector of the target style text in the jth pair of samples. τ represents the trainable temperature parameter used to scale the loss.

[0133] It can be understood that by minimizing the contrastive learning loss function and adjusting the parameters of the audio encoder, text encoder and multi-layer perceptron, the evaluation model can learn how to map the speech data and style text into a common embedding space so that the matching speech data and style text have similar vector representations in this space.

[0134] For example, Figure 8 A schematic diagram of an audio encoder provided in an embodiment of the present application. The audio encoder is a hierarchical token-semantic audio transformer (HTS-AT) structure, which is a structure based on a transformer network structure and combined with a convolutional neural network (CNN). Figure 8As shown in the figure, the speech data is input into the CNN-based patch embedding, then processed by the swin transformers block N times, and finally input into the CNN-based token-semantic layer to obtain the hidden features of the speech data. The swin transformers block includes the swin transformer and the patch merging layer.

[0135] Exemplarily, the text encoder in the above evaluation model can be a RoBERTa model. The RoBERTa model is an improved BERT model, and its network structure is similar to BERT, but the training method has been optimized in many ways.

[0136] It should be noted that this application does not restrict the hyperparameters such as the feature dimension and number of layers of the transformer architecture used by the above-mentioned audio encoder and text encoder.

[0137] In one embodiment, Fig. 9 As shown, the above S203 can be specifically determined through the following S901 to S902.

[0138] S901: Obtain evaluation parameters according to the matching degrees between a plurality of style instructions and the corresponding voice data.

[0139] The evaluation parameters are the mean and standard deviation of the matching degree.

[0140] In a possible implementation, the matching degree between N style instructions and their corresponding voice data is taken as N single compliance degrees, denoted as S_single i , i is a number. The average value of the monomer compliance can be expressed by the following formula 3, and the standard deviation of the monomer compliance can be expressed by the following formula 4.

[0141]

[0142] Where μ represents the average value of monomer compliance. N represents the total number of monomer compliance. i Indicates the compliance degree of the monomer numbered i.

[0143]

[0144] Where σ represents the standard deviation of monomer compliance. N represents the total number of monomer compliance. i represents the compliance degree of the monomer numbered i. μ represents the average compliance degree of the monomers.

[0145] S902: Evaluate the speech synthesis model to be tested according to the evaluation parameters and the preset equalization coefficient.

[0146] The preset equalization coefficient is used to adjust the weight of the average value and standard deviation of the matching degree.

[0147] In a possible implementation, the system compliance of the speech synthesis model to be tested is calculated based on the evaluation parameters and the preset equalization coefficient, and the system compliance is used as the evaluation result of the speech synthesis model to be tested.

[0148] It is understandable that in order to compare and select different speech synthesis models, or to iterate the speech synthesis model under development, it is necessary not only to evaluate the individual compliance of a single voice data and style instruction, but also to evaluate the overall performance of the speech synthesis model, and then to use system compliance to perform a system-level evaluation of the speech synthesis model.

[0149] Exemplarily, the preset equalization coefficients include a and b, wherein a is used to adjust the weight of the average value of the matching degree, and b is used to adjust the weight of the standard deviation of the matching degree.

[0150] In combination with the above example, the system compliance of the speech synthesis model to be tested can be expressed by the following formula 5.

[0151]

[0152] Wherein, S_system represents the system compliance of the speech synthesis model to be tested. a represents the weight of the average value. μ represents the average value of the individual compliance. b represents the weight of the standard deviation. σ represents the standard deviation of the individual compliance.

[0153] It can be understood that since the monomer compliance is calculated based on the cosine similarity, and the value range of cosine similarity is [-1, 1], the value range of the average value μ of the monomer compliance is also [-1, 1]. The constant 1 is added to the denominator to prevent the denominator from being 0 in extreme cases.

[0154] It should be noted that by changing the values ​​of a and b, the weights of the mean and standard deviation can be adjusted, so that the calculated system compliance index can pay more attention to the average output effect of the speech synthesis model to be tested (i.e., the mean value), or pay more attention to the output stability of the speech synthesis model to be tested (i.e., the standard deviation). For example, when evaluating the speech synthesis model to be tested, if the application scenario focuses on the average output effect of the model, the a value can be increased and the b value can be reduced, so that the system compliance index is more inclined to the model with a larger average value μ. If the application scenario pays more attention to the output stability of the model, the b value can be increased and the a value can be reduced, so that the system compliance index is more inclined to the model with a smaller standard deviation σ.

[0155] For example, Fig.10A flow chart for determining system compliance provided in an embodiment of the present application. Fig.10 As shown, each pair of style instructions and corresponding voice data is input into the evaluation model to obtain multiple monomer compliances. The average and standard deviation of the monomer compliances are calculated, and the values ​​of a and b in the preset equalization coefficients are determined, and then the system compliance can be obtained according to the above formula 5.

[0156] The following takes the PromptTTS model as an example and uses the above speech synthesis model evaluation method to evaluate the PromptTTS model.

[0157] It should be noted that the PromptTTS model is a new multi-dimensional controllable speech synthesis model that can generate speech with specified gender, speaking speed, pitch, volume, and emotion. The input of the PromptTTS model is divided into Style Prompt and Content Prompt, and the output is a human voice audio. Among them, Style Prompt is the style instruction mentioned above, and Content Prompt refers to the content of the human voice, that is, the content instruction mentioned above.

[0158] For example, Fig.11 A flowchart of a single data evaluation of the PromptTTS model is provided in the embodiment of the present application. Fig.11 As shown, the content instruction and style instruction are input into the PromptTTS model to obtain speech data. The speech data and style instruction are input into the evaluation model to obtain the embedding vector E of the speech data. a and the embedding vector E of the style instruction t Calculate E a and E t The cosine similarity of is used to obtain the compliance index of the speech data and the style instruction, that is, the monomer compliance.

[0159] Optional, Fig.11 The evaluation model in can be the evaluation model constructed above, or it can be an evaluation model rebuilt for the PromptTTS model. Specifically, a large amount of speech data is synthesized through the PromptTTS model as the target speech data set. According to the gender classification model, speech rate classification model, pitch classification model, emotion classification model and volume classification model, multiple discrete labels of each target speech data are obtained. The Llama2 model and preset prompts are used to convert the multiple discrete labels of each target speech data into style text, and then the target style text set is obtained. Figure 7 The evaluation model is trained by contrastive learning.

[0160] Exemplarily, the preset prompt may be: Please convert the following words into a sentence: [a person, {gender label}, {speaking speed label}, {pitch label}, {emotion label}, {volume label}].

[0161] Furthermore, different content instructions and style instructions are combined and repeated. Fig.11 The process is repeated N times to obtain N monomer compliances. According to the descriptions of S901 and S902, the system compliance of the PromptTTS model can be obtained, and then the system compliance can be used as the evaluation result of the PromptTTS model.

[0162] It is understandable that the above-mentioned speech synthesis model evaluation method can be implemented by a speech synthesis model evaluation device. In order to realize the above-mentioned functions, the speech synthesis model evaluation device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the modules and algorithm steps of each example described in the embodiments disclosed herein, the embodiments disclosed in this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments disclosed in this application.

[0163] The embodiments disclosed in the present application can divide the functional modules of the speech synthesis model evaluation device generated by the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiments disclosed in the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0164] Fig.12 A schematic diagram of the structure of a speech synthesis model evaluation device provided in an embodiment of the present application. Fig.12 As shown, the speech synthesis model evaluation device 120 can be used to perform Figure 2 , Figure 3 as well as Fig. 9 The speech synthesis model evaluation method shown in the figure. The speech synthesis model evaluation device 120 includes: a communication unit 1201 and a processing unit 1202.

[0165] The communication unit 1201 is used to obtain multiple style instructions, where the style instructions are used to indicate the voice features of the voice data to be output; the processing unit 1202 is used to determine the voice data corresponding to each style instruction based on the voice synthesis model to be tested; the processing unit 1202 is also used to evaluate the voice synthesis model to be tested based on the matching degree between the multiple style instructions and the corresponding voice data.

[0166] In a possible implementation, the speech feature includes at least one of the following: timbre feature; emotion feature; speech speed feature; and pitch feature.

[0167] In one possible implementation, the communication unit 1201 is also used to obtain multiple content instructions, where the content instructions are used to indicate the text content of the voice data that needs to be output; for each style instruction, the processing unit 1202 is also used to input the style instruction and any content instruction among the multiple content instructions into the speech synthesis model to be tested, and obtain the voice data corresponding to the style instruction, so as to obtain the voice data corresponding to each style instruction.

[0168] In a possible implementation, the method further includes: for each style instruction, the communication unit 1201 is further used to obtain the semantic features of the style instruction and the audio features of the speech data corresponding to the style instruction; the processing unit 1202 is further used to determine the matching degree between the style instruction and the speech data based on the semantic features of the style instruction and the audio features of the speech data.

[0169] In a possible implementation, the processing unit 1202 is further configured to use the similarity between the semantic features of the style instruction and the audio features of the speech data as the matching degree between the style instruction and the speech data.

[0170] In one possible implementation, the processing unit 1202 is further used to obtain evaluation parameters based on the matching degree between multiple style instructions and their corresponding speech data, where the evaluation parameters are the average value and standard deviation of the matching degree; the processing unit 1202 is further used to evaluate the speech synthesis model to be tested based on the evaluation parameters and a preset equalization coefficient, where the preset equalization coefficient is used to adjust the weight of the average value and standard deviation of the matching degree.

[0171] Through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0172] The present disclosure also provides a computer-readable storage medium having instructions stored thereon. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the speech synthesis model evaluation method provided in the above-mentioned embodiment of the present disclosure.

[0173] The embodiments of the present disclosure also provide a computer program product including instructions, which, when executed on an electronic device, enables the electronic device to execute the speech synthesis model evaluation method provided by the embodiments of the present disclosure.

[0174] Among them, the computer readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a register, a hard disk, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above, or any other form of computer readable storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application-specific integrated circuit (ASIC). In the embodiments of the present application, a computer-readable storage medium may be any tangible medium that contains or stores a program, which may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0175] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A speech synthesis model evaluation method, characterized in that: The method comprises: Acquire a plurality of style instructions, where the style instructions are used to indicate voice features of voice data to be output; Determine the speech data corresponding to each style instruction based on the speech synthesis model to be tested; The speech synthesis model to be tested is evaluated according to the matching degree between the multiple style instructions and the corresponding speech data.

2. The method according to claim 1, characterized in that The voice feature includes at least one of the following: Timbre characteristics; Emotional characteristics; Speech speed characteristics; Tonal characteristics.

3. The method according to claim 1, characterized in that The step of determining the speech data corresponding to each style instruction based on the speech synthesis model to be tested includes: Acquire a plurality of content instructions, where the content instructions are used to indicate text content of the voice data to be output; For each style instruction, the style instruction and any one of the multiple content instructions are input into the speech synthesis model to be tested to obtain speech data corresponding to the style instruction, so as to obtain speech data corresponding to each style instruction.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: For each style instruction, obtaining a semantic feature of the style instruction and an audio feature of speech data corresponding to the style instruction; Based on the semantic features of the style instruction and the audio features of the voice data, a matching degree between the style instruction and the voice data is determined.

5. The method according to claim 4, characterized in that The determining the matching degree between the style instruction and the voice data based on the semantic feature of the style instruction and the audio feature of the voice data includes: The similarity between the semantic feature of the style instruction and the audio feature of the speech data is used as the matching degree between the style instruction and the speech data.

6. The method according to claim 1, characterized in that The step of evaluating the speech synthesis model to be tested according to the matching degree between the plurality of style instructions and the corresponding speech data comprises: Obtaining evaluation parameters according to the matching degrees between the plurality of style instructions and the voice data corresponding to each of them, wherein the evaluation parameters are the average value and standard deviation of the matching degrees; The speech synthesis model to be tested is evaluated according to the evaluation parameters and a preset equalization coefficient, and the preset equalization coefficient is used to adjust the weight of the average value and standard deviation of the matching degree.

7. A speech synthesis model evaluation device, characterized in that: The device comprises: a communication unit and a processing unit; The communication unit is used to obtain a plurality of style instructions, wherein the style instructions are used to indicate the voice features of the voice data to be output; The processing unit is used to determine the speech data corresponding to each style instruction based on the speech synthesis model to be tested; The processing unit is further used to evaluate the speech synthesis model to be tested according to the matching degree between the multiple style instructions and the corresponding speech data.

8. A speech synthesis model evaluation device, characterized in that: include: Memory and processor; The memory is coupled to the processor; The memory is used to store instructions executable by the processor; When the processor executes the instruction, it performs the speech synthesis model evaluation method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer executes the speech synthesis model evaluation method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product includes computer program instructions, and when the computer program instructions are executed by a processor, the speech synthesis model evaluation method as described in any one of claims 1 to 6 is implemented.

Images