Synthetic speech evaluation method, device, equipment and storage medium

Abstract

The present invention provides a method, apparatus, device, and storage medium for evaluating synthesized speech, involving artificial intelligence. The method comprises: inputting natural speech into a first speech neural network model, selecting at least one encoding layer, and obtaining encoding results extracted by the at least one encoding layer as a first encoding set; wherein the at least one encoding layer does not include the output layer of the first neural network model; inputting synthesized speech into the first speech neural network model, obtaining encoding results extracted by the at least one encoding layer as a second encoding set; and obtaining a speech perception distance as an evaluation result of the synthesized speech based on the first and second encoding sets. This method does not rely on human effort, thereby improving the efficiency of evaluating the naturalness of synthesized speech.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a synthetic speech evaluation method, device, equipment and storage medium. Background Art

[0002] With the development of deep learning, speech synthesis systems have achieved great success in the authenticity of synthesized speech. The synthesized speech generated by some models is almost indistinguishable from the real thing.

[0003] In the past, evaluating the naturalness of synthesized speech (e.g., whether the synthesized speech sounds like a real person, whether the speaker's tone is natural, etc.) often relied on subjective human evaluation, such as the traditional MOS score evaluation method. Because this reliance on human effort can result in subjective evaluation results, it can also be slow and inefficient when evaluating large amounts of synthesized speech. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, embodiments of the present invention provide a method, apparatus, device and storage medium for synthesized speech evaluation.

[0005] On the one hand, an embodiment of the present invention provides a method for evaluating synthetic speech, comprising: inputting natural speech into a first speech neural network model, selecting at least one coding layer, and obtaining the coding result extracted by the at least one coding layer as a first coding set; wherein the at least one coding layer does not include the output layer of the first neural network model; inputting synthetic speech into the first speech neural network model, and obtaining the coding result extracted by the at least one coding layer as a second coding set; and obtaining the speech perception distance as an evaluation result of the synthetic speech based on the first coding set and the second coding set.

[0006] The synthetic speech evaluation method according to the embodiment of the present invention has at least the following beneficial effects: by inputting natural speech and synthetic speech into the same speech neural network, obtaining the encoding results of at least one selected layer to obtain two encoding sets, and calculating the speech perception distance based on the results of the encoding sets to obtain the evaluation results of the synthetic speech; the entire process can be carried out automatically without relying on manpower, thereby improving the evaluation efficiency of the naturalness evaluation of the synthetic speech; in addition, the speech neural network used for evaluation can be arbitrary, and the content of the natural speech and the synthetic speech does not necessarily correspond or have a connection, thereby improving the scope of adaptability.

[0007] According to some embodiments of the present invention, obtaining the speech perception distance based on the first code set and the second code set includes: obtaining the mean and covariance matrix of the first code set; obtaining the mean and covariance matrix of the second code set; and deriving the speech perception distance based on the mean of the first code set, the covariance matrix of the first code set, the mean of the second code set, and the covariance matrix of the second code set.

[0008] According to some embodiments of the present invention, deriving the speech perception distance according to the mean of the first code set, the covariance matrix of the first code set, the mean of the second code set, and the covariance matrix of the second code set includes:

[0009]

[0010] Among them, SFID(S real ,S generated ) represents the speech perception distance, S real Represents the first code set, S generated represents the second code set, tr() represents the trace of the matrix, that is, the sum of the main diagonal elements of the matrix, represents the mean value of the encoding result of the i-th encoding layer in the first encoding set, represents the covariance matrix of the i-th coding layer in the first coding set, represents the mean value of the encoding result of the i-th encoding layer in the second encoding set, represents the covariance matrix of the i-th coding layer in the second coding set, i=1,2..,k, where k is the total number of selected coding layers.

[0011] According to some embodiments of the present invention, the at least one encoding layer includes at least: a first activation layer of the first speech neural network model from the input side to the output side.

[0012] According to some embodiments of the present invention, the at least one encoding layer is configured as: the first activation layer of the first speech neural network model from the input side to the output side, the first feature layer of the first speech neural network model from the output side to the input side, and the second feature layer of the first speech neural network model from the output side to the input side.

[0013] On the other hand, an embodiment of the present invention provides a synthetic speech evaluation device, comprising: a first module, used to input natural speech into a first speech neural network model, select at least one coding layer, and obtain the coding result extracted by the at least one coding layer as a first coding set; wherein the at least one coding layer does not include the output layer of the first neural network model; a second module, used to input synthetic speech into the first speech neural network model, and obtain the coding result extracted by the at least one coding layer as a second coding set; and a third module, used to obtain the speech perception distance as an evaluation result of the synthetic speech based on the first coding set and the second coding set.

[0014] According to the synthetic speech evaluation device of the embodiment of the present invention, there are at least the following beneficial effects: by inputting natural speech and synthetic speech into the same speech neural network, obtaining the encoding results of at least one selected layer to obtain two encoding sets, and calculating the speech perception distance based on the results of the encoding sets to obtain the evaluation results of the synthetic speech; the entire process can be carried out automatically without relying on manpower, thereby improving the evaluation efficiency of the naturalness evaluation of the synthetic speech; in addition, the speech neural network used for evaluation can be arbitrary, and the content of the natural speech and the synthetic speech does not necessarily correspond or have a connection, thereby improving the scope of adaptability.

[0015] On the other hand, an embodiment of the present invention provides a computer device, comprising: at least one processor; at least one memory for storing at least one program; when the at least one program is executed by the at least one processor, the at least one processor implements the synthetic speech evaluation method as described above.

[0016] On the other hand, the present invention provides a computer-readable storage medium having program instructions stored therein. When the program instructions are executed by a processor, the above-mentioned synthetic speech evaluation method is implemented.

[0017] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0019] Figure 1 The figure is a flowchart of the steps of the synthetic speech evaluation method provided by an embodiment of the present invention.

[0020] Figure 2 4 is a flow chart of the steps for calculating the speech perception distance in an embodiment of the present invention.

[0021] Figure 3The figure shows the experimental results of evaluating the synthesized speech obtained by different speech synthesis models according to the method of the embodiment of the present invention.

[0022] Figure 4 3 is a module diagram of a synthetic speech evaluation device provided by an embodiment of the present invention.

[0023] Figure 5 It is a schematic diagram of a module of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. In the subsequent description, suffixes such as "module," "component," or "unit" used to represent elements are used solely to facilitate the description of the present invention and have no specific meaning in themselves. Therefore, "module," "component," or "unit" may be used interchangeably. "First," "second," and the like are used solely to distinguish technical features and should not be construed as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features. In this subsequent description, the consecutive numbering of method steps is for ease of review and understanding. In conjunction with the overall technical solution of the present invention and the logical relationship between the various steps, adjusting the order of implementation of the steps does not affect the technical effects achieved by the technical solution of the present invention. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and should not be construed as limiting the present invention.

[0025] Reference Figure 1 , this embodiment discloses a synthetic speech evaluation method involving artificial intelligence, including but not limited to the following steps S100 to S300.

[0026] Step S100: input natural speech into a first speech neural network model, select at least one coding layer, and obtain the coding result extracted by the at least one coding layer as a first coding set; wherein at least one coding layer does not include the output layer of the first neural network model.

[0027] The first speech neural network model can be any multi-layer neural network. In this embodiment, there is no specific limitation on the first speech neural network model. As long as the speech neural network model has other layers in addition to the output layer, the corresponding encoding results can be extracted. The encoding results, or embeddings, refer to the embedded mapping formed by the natural original low-dimensional data stream in the high-dimensional space.

[0028] The at least one selected encoding layer can be any layer in the first speech neural network model except the output layer, for example, the first activation layer of the first neural network model from the input side to the output side, and the encoding layer of any layer after the first activation layer.

[0029] Experiments have shown that by using only the embeddings (encoding results) extracted from the first activation layer or any encoding layer after the first activation layer, a speech perception distance (SFID) evaluation result that is positively correlated with the MOS score can be obtained.

[0030] The at least one encoding layer is configured as: the first activation layer of the first speech neural network model from the input side to the output side, the first feature layer of the first speech neural network model from the output side to the input side (equivalent to the last feature layer from the input side to the output side), and the second feature layer of the first speech neural network model from the output side to the input side (equivalent to the second to last feature layer from the input side to the output side). This can make the speech perception distance evaluation result obtained closest to the evaluation result of human ear discrimination when obtaining less data.

[0031] In the embodiment of the present invention, the purpose of using multiple coding layers to extract embeddings (coding results) is to make the speech perception distance evaluation results show a stronger correlation with the traditional MOS score. Therefore, for different models, the selection of coding layers can be different. The total number of selected coding layers can be one or more, for example, k, where k is a positive integer. The first coding set obtained is also called the real data set S real .

[0032] Step S200: Input the synthesized speech into the first speech neural network model, and obtain the encoding result extracted by the at least one encoding layer as the second encoding set.

[0033] Specifically, the synthesized speech is input into the first speech neural network model, and the same coding layers as in the step of obtaining the first coding set are selected, and the coding results extracted by these coding layers are obtained as each coding set. The obtained second coding set is also called the synthesized data set S generated .

[0034] In the embodiments of the present invention, there is no limitation on the number of natural speech and synthesized speech. That is, the number of natural speech inputs to the first speech neural network model can be one or more, and the number of synthesized speech inputs to the first speech neural network model can be one or more. Furthermore, the speech content of the natural speech and synthesized speech inputs to the first speech neural network model does not necessarily need to correspond or have a certain correlation. For example, the text content does not need to be completely identical or partially identical.

[0035] Step S300 : Acquire a speech perception distance as an evaluation result of the synthesized speech according to the first code set and the second code set.

[0036] Specifically, refer to Figure 2 , including the following steps S320 to S330.

[0037] S310, obtaining a first coding set S according to the coding layer real The mean and the covariance matrix . Where i represents the th coding layer in the selected coding layer.

[0038] S320, obtaining a second coding set S according to the coding layer generated The mean encoding result of and the covariance matrix

[0039] S330, according to the mean of the first code set , the covariance matrix of the first coding set , the mean of the second coding set and the covariance matrix of the second coding set , and the speech perception distance is obtained.

[0040]

[0041] Among them, SFID(S real ,S generated ) represents the speech perception distance, tr() represents the trace of the matrix, that is, summing the main diagonal elements of the input matrix, S real Represents the first code set, S generated represents the second code set, represents the mean value of the encoding result of the i-th encoding layer in the first encoding set, represents the covariance matrix of the i-th coding layer in the first coding set, represents the mean value of the encoding result of the i-th encoding layer in the second encoding set, represents the covariance matrix of the i-th coding layer in the second coding set, i = 1, 2.., k, where k is the total number of selected coding layers.

[0042] Reference Figure 3, the synthetic speech generated by different speech synthesis models, and the speech perception distance results obtained by the above-mentioned synthetic speech evaluation method (i.e., SFID column). In the experiment, the four speech synthesis models trained on the LJSpeech dataset are: lj-taco2-hifigan, lj-cfs2-hifigan, lj-fs2-hifigan, and lj-taco2-griffin-lim. Several synthetic speech are randomly generated from the synthesis results of each module to calculate the speech perception distance. In addition, real speech is randomly selected to calculate the speech distance of the real speech, i.e. Figure 3 The rows in self-distance refer to the distance from the real speech to the real speech.

[0043] Before the experiment began, it was known that the three models lj-taco2-hifigan, lj-cfs2-hifigan, and lj-fs2-hifigan had similar synthesis quality (referring to the naturalness of the speech), and their corresponding MOS scores were close, while the naturalness of the speech synthesized by lj-taco2-griffin-lim was relatively poor. Figure 3 It can be seen that the speech synthesis evaluation method provided in this embodiment can be used to evaluate the differences in the naturalness of different synthesized speech, accurately reflect the quality of the naturalness of the synthesized speech, and show a strong correlation with the MOS score, which can quantify the naturalness of the speech.

[0044] The synthesized speech evaluation method provided in this embodiment inputs natural speech and synthesized speech into the same speech neural network, obtains the encoding results of at least one selected layer to obtain two encoding sets, and calculates the speech perception distance based on the encoding set results to obtain the synthesized speech evaluation result. The entire process can be performed automatically without human intervention and can be used to evaluate the naturalness of more difficult to define and abstract speech. The evaluation is more objective, thereby improving the efficiency of synthesized speech naturalness evaluation. In addition, the speech neural network used for evaluation can be arbitrary, and the content of natural speech and synthesized speech does not necessarily correspond or relate to each other, thereby increasing the scope of application.

[0045] Reference Figure 4 This embodiment discloses a synthetic speech evaluation device involving artificial intelligence, including but not limited to the following first module 410, second module 420 and third module 430.

[0046] The first module 420 is used to input natural speech into the first speech neural network model, select at least one coding layer, and obtain the coding result extracted by the at least one coding layer as the first coding set; wherein the at least one coding layer does not include the output layer of the first neural network model.

[0047] The second module 420 is used to input the synthesized speech into the first speech neural network model, and obtain the encoding result extracted by the at least one encoding layer as the second encoding set.

[0048] The third module 430 is configured to obtain a speech perception distance as an evaluation result of the synthesized speech according to the first code set and the second code set.

[0049] Figure 4 The synthetic speech evaluation device shown is Figure 1 The synthesized speech evaluation method shown is based on the same inventive concept. Therefore, the synthesized speech evaluation device, working in concert with the aforementioned modules, inputs natural speech and synthesized speech into the same speech neural network, obtains the encoding results of at least one selected layer, and generates two encoding sets. Based on the encoding set results, the speech perception distance is calculated to obtain the synthesized speech evaluation result. This entire process can be performed automatically without human intervention, making the evaluation more objective and improving the efficiency of synthesized speech naturalness assessment. Furthermore, the speech neural network used for evaluation can be arbitrary, and the content of the natural and synthesized speech does not necessarily correspond or relate to each other, thereby increasing the scope of application.

[0050] Reference Figure 5 This embodiment provides a computer device comprising: at least one processor 510; at least one memory 520 for storing at least one program; when the at least one program is executed by the at least one processor 510, the at least one processor 510 implements the above-mentioned synthetic speech evaluation method. The processor 510 may also be referred to as a CPU (Central Processing Unit). The processor 510 may be an integrated circuit chip with signal processing capabilities. The processor 510 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The general-purpose processor may be a microprocessor, or any conventional processor. The memory 520 may include various components (e.g., machine-readable media), including but not limited to random access memory components, read-only components, and any combination thereof. The memory 520 may also include instructions (e.g., software) (e.g., stored on one or more machine-readable media); the instructions implement the above-mentioned synthetic speech evaluation method of the present invention. It should be understood that, in order to avoid redundancy, the contents not involved in this embodiment may refer to the above-mentioned synthetic speech evaluation method.

[0051] This embodiment provides a computer-readable storage medium storing program instructions that, when executed by a processor, implement the aforementioned synthesized speech evaluation method. It should be understood that, to avoid redundant description, any content not covered in this embodiment may be referred to in conjunction with the aforementioned synthesized speech evaluation method.

[0052] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above and the functional modules / units in the devices may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0053] In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium (storage medium for short), which may include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those of ordinary skill in the art, the term computer-readable medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0054] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention shall fall within the scope of the present invention.

Claims

1. A synthetic speech evaluation method, characterized in that: include: Inputting natural speech into a first speech neural network model, selecting at least one encoding layer, and obtaining encoding results extracted by the at least one encoding layer as a first encoding set; wherein the at least one encoding layer does not include an output layer of the first neural network model; Inputting the synthesized speech into the first speech neural network model, and obtaining the encoding result extracted by the at least one encoding layer as a second encoding set; Obtaining a speech perception distance as an evaluation result of synthesized speech according to the first code set and the second code set; The acquiring the speech perception distance according to the first code set and the second code set includes: obtaining the speech perception distance according to the mean of the first code set, the covariance matrix of the first code set, the mean of the second code set, and the covariance matrix of the second code set; In which, the at least one encoding layer is configured as: the first activation layer of the first speech neural network model from the input side to the output side, the first feature layer of the first speech neural network model from the output side to the input side, and the second feature layer of the first speech neural network model from the output side to the input side.

2. The synthetic speech evaluation method according to claim 1, wherein: The acquiring the speech perception distance according to the first code set and the second code set further includes: Obtaining the mean and covariance matrix of the first coding set according to the coding layer; The mean and covariance matrix of the second coding set are obtained according to the coding layer.

3. The synthetic speech evaluation method according to claim 2, wherein: The obtaining the speech perception distance according to the mean of the first code set, the covariance matrix of the first code set, the mean of the second code set, and the covariance matrix of the second code set includes: Among them, SFID(S real ,S generated ) represents the speech perception distance, S real Represents the first code set, S generated represents the second code set, tr() represents the trace of the matrix, that is, the sum of the main diagonal elements of the matrix, represents the mean value of the encoding result of the i-th encoding layer in the first encoding set, represents the covariance matrix of the i-th coding layer in the first coding set, represents the mean value of the encoding result of the i-th encoding layer in the second encoding set, represents the covariance matrix of the i-th coding layer in the second coding set, i=1,2..,k, where k is the total number of selected coding layers.

4. A synthetic speech evaluation device, characterized in that: include: A first module is configured to input natural speech into a first speech neural network model, select at least one encoding layer, and obtain an encoding result extracted by the at least one encoding layer as a first encoding set; wherein the at least one encoding layer does not include an output layer of the first neural network model; A second module is configured to input the synthesized speech into the first speech neural network model and obtain the encoding result extracted by the at least one encoding layer as a second encoding set; A third module is configured to obtain a speech perception distance as an evaluation result of the synthesized speech based on the first code set and the second code set; The acquiring the speech perception distance according to the first code set and the second code set includes: obtaining the speech perception distance according to the mean of the first code set, the covariance matrix of the first code set, the mean of the second code set, and the covariance matrix of the second code set; In which, the at least one encoding layer is configured as: the first activation layer of the first speech neural network model from the input side to the output side, the first feature layer of the first speech neural network model from the output side to the input side, and the second feature layer of the first speech neural network model from the output side to the input side.

5. A computer device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the synthetic speech evaluation method according to any one of claims 1 to 3.

6. A computer-readable storage medium, wherein program instructions are stored in the storage medium, and when the program instructions are executed by a processor, the synthetic speech evaluation method according to any one of claims 1 to 3 is implemented.

Images