Speech synthesis method, related device, equipment and storage medium

Abstract

The invention discloses a speech synthesis method, a related device, equipment and a storage medium, and the method comprises the steps: obtaining a pronunciation feature sequence of a to-be-synthesized text; performing a first step number of reasoning decoding on the pronunciation feature sequence based on the teacher CFM model to obtain a first feature sequence, performing a second step number of reasoning decoding on the first feature sequence based on the student CFM model to obtain a second feature sequence, and performing a third step number of reasoning decoding on the second feature sequence based on the teacher CFM model to obtain a third feature sequence; obtaining a third feature sequence as an acoustic feature sequence; wherein the sum of the first step number, the second step number and the third step number is the target step number, and the parameter quantity of the student CFM model is smaller than that of the teacher CFM model; and reconstructing the acoustic feature sequence based on a vocoder to obtain a synthetic voice signal. According to the scheme, computing resources and time cost can be reduced as much as possible on the premise of ensuring the naturalness of the synthesized voice as much as possible.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech synthesis method and related devices, equipment and storage media. Background Art

[0002] In recent years, artificial intelligence models with extremely large parameter amounts, such as large language models, have been increasingly used in speech synthesis systems to generate speech signals with higher naturalness.

[0003] However, due to the large number of parameters and the need for multiple decoding steps, the computing resources and time cost increase dramatically, making it difficult to deploy in some scenarios. In view of this, how to reduce computing resources and time cost as much as possible while ensuring the naturalness of synthesized speech as much as possible has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a speech synthesis method and related devices, equipment and storage media, which can reduce computing resources and time costs as much as possible while ensuring the naturalness of the synthesized speech as much as possible.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a speech synthesis method, including: obtaining a pronunciation feature sequence of a text to be synthesized; wherein the pronunciation feature sequence contains the pronunciation feature information of each phoneme in the text to be synthesized; performing a first number of inference decoding on the pronunciation feature sequence based on a teacher CFM model to obtain a first feature sequence, and performing a second number of inference decoding on the first feature sequence based on a student CFM model to obtain a second feature sequence, and performing a third number of inference decoding on the second feature sequence based on the teacher CFM model to obtain a third feature sequence as an acoustic feature sequence; wherein the sum of the first number of steps, the second number of steps, and the third number of steps is the target number of steps, and the number of parameters of the student CFM model is less than that of the teacher CFM model; reconstructing the acoustic feature sequence based on a vocoder to obtain a synthesized speech signal.

[0006] In order to solve the above technical problems, the second aspect of the present application provides a speech synthesis device, including: a sequence acquisition module, an inference decoding module and a signal reconstruction module, the sequence acquisition module is used to obtain the pronunciation feature sequence of the text to be synthesized; wherein the pronunciation feature sequence contains the pronunciation feature information of each phoneme in the text to be synthesized; the inference decoding module is used to perform the first number of inference decoding on the pronunciation feature sequence based on the teacher CFM model to obtain the first feature sequence, and perform the second number of inference decoding on the first feature sequence based on the student CFM model to obtain the second feature sequence, and perform the third number of inference decoding on the second feature sequence based on the teacher CFM model to obtain the third feature sequence as the acoustic feature sequence; wherein the sum of the first number of steps, the second number of steps and the third number of steps is the target number of steps, and the number of parameters of the student CFM model is less than that of the teacher CFM model; the signal reconstruction module is used to reconstruct the acoustic feature sequence based on the vocoder to obtain a synthesized speech signal.

[0007] In order to solve the above technical problems, the third aspect of the present application provides an electronic device, which at least includes a memory and a processor coupled to each other, the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the speech synthesis method in the above first aspect.

[0008] In order to solve the above technical problem, the fourth aspect of the present application provides a computer-readable storage medium, which stores program instructions that can be executed by a processor, and the program instructions are used to implement the speech synthesis method of the first aspect.

[0009] The above scheme obtains the pronunciation feature sequence of the text to be synthesized, and the pronunciation feature sequence contains the pronunciation feature information of each phoneme in the text to be synthesized, and then performs the first step of inference decoding on the pronunciation feature sequence based on the teacher CFM model to obtain the first feature sequence, and performs the second step of inference decoding on the first feature sequence based on the student CFM model to obtain the second feature sequence, and performs the third step of inference decoding on the second feature sequence based on the teacher CFM model to obtain the third feature sequence as the acoustic feature sequence, and the sum of the first step, the second step, and the third step is the target step, and the number of parameters of the student CFM model is less than that of the teacher CFM model. M model, based on the vocoder, reconstructs the acoustic feature sequence to obtain the synthesized speech signal. Therefore, on the one hand, since the CFM model as a decoder has been proven to have excellent performance in converting the pronunciation feature sequence into the acoustic feature sequence, the use of the CFM model in speech synthesis can ensure the naturalness of the synthesized speech as much as possible. On the other hand, since the teacher CFM model and the student CFM model with fewer parameters are jointly inferred during the speech synthesis process, and the teacher CFM model infers the number of steps at the beginning and end, and the student CFM model infers the number of steps in the middle, it can greatly reduce the number of parameters and decoding time while ensuring the performance of the CFM model. Therefore, it is possible to reduce computing resources and time costs as much as possible while ensuring the naturalness of the synthesized speech as much as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flowchart of an embodiment of the speech synthesis method of the present application; Figure 2 It is a schematic diagram of the framework of an embodiment of a speech synthesis device of the present application; Figure 3 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application; Figure 4 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0011] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0012] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0013] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the fragment " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0014] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the speech synthesis method of the present application. Specifically, it may include the following steps: Step S11: Obtain the pronunciation feature sequence of the text to be synthesized.

[0015] In the disclosed embodiment, the pronunciation feature sequence includes the pronunciation feature information of each phoneme in the text to be synthesized. It should be noted that the text to be synthesized can be Chinese text, English text, etc., and the language involved in the text to be synthesized is not limited here. In addition, the text to be synthesized can be directly input by the user through voice, text and other input methods; or, the text to be synthesized can also be extracted from scripts, novels and other chapters. The above examples are only several possible ways to obtain the text to be synthesized, and the way to obtain the text to be synthesized is not limited here, and other possible ways to obtain the text to be synthesized are not given one by one. As a possible example, an encoder such as a network architecture including Transformer can be used to encode the phoneme sequence of the text to be synthesized to obtain the pronunciation feature sequence of the text to be synthesized. It should be noted that in addition to the Transformer network architecture, the encoder can also adopt other network architectures, and the network architecture of the encoder is not limited here. In addition, the above example is only a possible way to obtain the pronunciation feature sequence in the actual application process, and it does not lead to other possible ways of obtaining it, and other possible ways of obtaining it are not given one by one here.

[0016] Step S12: Based on the teacher CFM model, perform the first number of inference decoding on the pronunciation feature sequence to obtain a first feature sequence, and based on the student CFM model, perform the second number of inference decoding on the first feature sequence to obtain a second feature sequence, and based on the teacher CFM model, perform the third number of inference decoding on the second feature sequence to obtain the third feature sequence as an acoustic feature sequence.

[0017] It should be noted that CFM (Conditional Flow Matching) is a fast method for training continuous normalized flow models. It is used for simulation-free training objectives of continuous normalized processes, allowing conditional generative modeling and accelerating training and reasoning. Conditional flow matching converts the uncomputable marginal flow matching problem into a computable conditional problem by using conditional probability paths and conditional vector fields, simplifying the training process. For the implementation principle of the CFM model, please refer to the technical details of conditional flow matching, which will not be repeated here.

[0018] In the disclosed embodiment, the student CFM model has fewer parameters than the teacher CFM model. As a possible example, the student CFM can be obtained by trimming the teacher CFM model and performing distillation training with the teacher CFM model. In other words, the teacher CFM model can be trimmed first as the student CFM model, and then the student CFM model can be trained by knowledge distillation with the teacher CFM model.

[0019] In one implementation scenario, as a possible example, before the teacher CFM model is trimmed to obtain the student CFM model, the teacher CFM can be pre-trained. Specifically, the sample pronunciation feature sequence of the sample text, the sample acoustic feature sequence of the sample speech to which the sample text belongs, and the sample vector field of the sample acoustic feature sequence output based on the sample pronunciation feature sequence can be obtained. For ease of description, the sample pronunciation feature sequence can be recorded as , the sample acoustic feature sequence can be recorded as , the sample vector field can be written as In addition, the initial parameters of the teacher CFM model can be written as It should be noted that the sample acoustic feature sequence may include the acoustic features of each audio frame in the sample speech (such as MFCC, etc.). Based on this, the latent variable can be sampled and obtained. , time step and conditional data , and thus after being processed by the teacher CFM model, the output vector field of the teacher CFM model can be obtained , and then based on the difference between the target vector field and the output vector field, the pre-training loss of the teacher CFM model can be measured: …… (1) In the above formula (1), represents the L2 norm, which can be obtained by calculating the sum of the squares of the differences between the elements at the corresponding positions in the output vector field and the target vector field, and is not limited here. On this basis, the network parameters of the teacher CFM model can be adjusted based on the pre-training loss of the teacher CFM model.

[0020] In one implementation scenario, after obtaining the teacher CFM model, the target network layer in the teacher CFM model can be trimmed to obtain the student CFM model, and the target network layer at least includes the attention layer. It should be noted that in addition to the attention layer, the target network layer can also include other network layers. Specifically, multiple rounds of experiments can be conducted to confirm that the relatively minor network layer in the teacher CFM model can be used as the target network layer. For example, the importance of the network layer can be determined by measuring the output accuracy of the teacher CFM model after trimming. In the above manner, the target network layer in the teacher CFM model is trimmed to obtain the student CFM model, and the target network layer at least includes attention, which helps to minimize the number of parameters of the student CFM model.

[0021] In an implementation scenario, as described above, the CFM model outputs a vector field (indicating the direction and magnitude of change), and the feature sequence output when performing inference decoding can be obtained by superimposing the vector field on the feature sequence input when performing inference decoding. For example, a pronunciation feature sequence can be processed by the CFM model for one step to obtain a vector field, and the vector field can be superimposed on the pronunciation feature sequence to obtain a new pronunciation feature sequence, and the new pronunciation feature sequence can be processed by the CFM model for another step to obtain a new vector field, and the new vector field can be superimposed on the aforementioned new pronunciation feature sequence to obtain another new pronunciation feature sequence, and this cycle is repeated continuously, and finally when the target number of steps is passed, the latest pronunciation feature sequence can be used as the acoustic feature sequence. In this case, during the distillation training process, a sample pronunciation feature sequence of the sample text can be obtained, and a target vector field that outputs a target acoustic feature sequence under the condition of the sample pronunciation feature sequence can be obtained, and the target acoustic feature sequence is the sample acoustic feature sequence of the sample speech to which the sample text belongs, and then the first vector field output after the sample pronunciation feature sequence is input into the teacher CFM model for processing is obtained, and the second vector field output after the sample pronunciation feature sequence is input into the student CFM model is obtained, the target vector field is selected with a first probability and the first vector field is selected with a second probability, and a reference vector field is obtained, and the sum of the first probability and the second probability is 1, so as to adjust the network parameters of the student CFM model at least based on the difference between the second vector field and the reference vector field. Compared with directly using the first vector field output after the teacher CFM model is processed as the training target during the distillation training process, the above method can minimize the error accumulation between the teacher CFM model and the true target that may be introduced due to multiple distillations, and help improve the robustness of the student CFM model.

[0022] In a specific implementation scenario, for ease of description, the target vector field of the target acoustic feature sequence output under the condition of the sample pronunciation feature sequence can be recorded as (See the sample vector field above). The first vector field output by the teacher CFM model after processing can be recorded as ,in, represents the network parameters of the teacher CFM model. Similarly, the second vector field output by the student CFM model can be recorded as ,in, represents the network parameters of the student CFM model. In addition, the first probability can be recorded as p EC , the second probability can be recorded as 1-p EC , then the reference vector field It can be expressed as: …… (2) In a specific implementation scenario, before adjusting the network parameters of the student CFM model based at least on the difference between the second vector field and the reference vector field, the first probability distribution of the first vector field output by the teacher CFM model can be obtained, and the second probability distribution of the second vector field output by the student CFM model can be obtained. On this basis, the network parameters of the student CFM model can be adjusted based on the difference between the second vector field and the reference vector field, and the difference between the first probability distribution and the second probability distribution. For ease of description, the first probability distribution of the first vector field output by the teacher CFM model can be recorded as The second probability distribution of the second vector field output by the student CFM model can be recorded as q student , then the training loss of the student CFM model is It can be expressed as: …… (3) In the above formula (3), represents the L2 norm, Represents KL divergence. On this basis, the training loss of the student CFM model can be , adjust the network parameters of the student CFM model. In the above method, in the process of adjusting the network parameters of the student model, further referring to the difference between the probability distribution of the output vector field of the student CFM model and the probability distribution of the output vector field of the teacher CFM model is helpful to constrain the student CFM model to learn the teacher CFM model at the probability distribution level.

[0023] In a specific implementation scenario, similar to the pre-training process of the teacher CFM mentioned above, the relevant data can be sampled before formal training. For details, please refer to the pre-training process of the teacher CFM model mentioned above, which will not be repeated here. In addition, in each step of the student CFM model inference decoding, the target drop rate can be used. Abandon the output feature sequence of the previous step as the input feature sequence of the current step, and abandon the output feature sequence of the previous step ( ) as the input feature sequence of the current step, unconditional sampling is performed based on the target acoustic feature sequence , as the input feature sequence of the current step number, and the target drop rate is negatively correlated with the cumulative number of training rounds up to the current training round. Exemplarily, the target drop rate can be expressed as: …… (4) In the above formula (4), represents the target dropout rate for the kth training round, represents the target dropout rate for the k+1th training round, The attenuation coefficient representing the target dropout rate can be set to 0.9, 0.8, etc., and the specific value of the attenuation coefficient is not limited here. In the above method, during the inference decoding process of the training phase, the output feature sequence of the previous step is abandoned at the target dropout rate as the input feature sequence of the current step, and when the output feature sequence of the previous step is abandoned as the input feature sequence of the current step, unconditional sampling is performed based on the target acoustic feature sequence as the input feature sequence of the current step, and the target dropout rate is negatively correlated with the cumulative number of training rounds up to the current training round, which can reduce the target dropout rate with continuous training, and help improve the robustness of the student CFM model.

[0024] In the disclosed embodiment, the sum of the first step number, the second step number, and the third step number is the target step number. As a possible example, the target step number and the preset ratio can be preset, and the product of the preset ratio and the target step number can be obtained as the first step number and the third step number, and the difference between the target step number and the first step number and the third step number can be obtained as the second step number. For ease of description, the target step number can be recorded as N, and the first step number and the third step number can be expressed as: …… (5) …… (6) In the above formulas (5) and (6), Indicates the preset ratio, represents the first step number, Indicates the third step number, Indicates rounding down. On this basis, the second step number can be expressed as: …… (7) In the above formula (7), represents the first step number, Indicates the third step number, Represents the second step number. That is to say, in this example, the first step number and the third step number can be the same, so that the setting method of the first step number, the second step number and the third step number can be simplified as much as possible. It should be noted that the specific numerical value of the preset ratio can be set according to the actual application needs, and the specific numerical value of the preset ratio is not limited here. Of course, the above example is only a possible setting method of the first step number, the second step number and the third step number, and the setting method of the first step number, the second step number and the third step number is not limited here. For example, the first step number and the third step number may also be different, such as the first step number may be greater than the third step number, or the first step number may be less than the third step number, and examples are not given one by one here. On this basis, for each step number ,if , then at these steps the teacher CFM model can perform inference decoding, which can be expressed as: …… (8) In the above formula (8), represents the number of steps t, represents the network parameters of the teacher CFM model, represents the output result of the teacher CFM model at step t-1, It indicates the output result of the teacher CFM model at step t, that is, the output result of the previous step is used as the model input of the next step. It should be noted that the output result of any step inference decoding can be obtained by superimposing the vector field of the corresponding step CFM model on the input feature sequence. For details, please refer to the above related description, which will not be repeated here. As the number of steps increases, if , then these steps can be performed by the student CFM model to perform inference decoding, which can be expressed as: …… (9) In the above formula (9), represents the network parameters of the student CFM model. The specific meanings of the other parameters can be found in the relevant explanation of formula (8) and will not be repeated here. , then these steps can be performed by the teacher CFM model to perform inference decoding, which can be expressed as: …… (10) In the above formula (10), the specific meaning of each parameter can be found in the relevant explanation of formula (8), which will not be repeated here. It should be noted that when the number of steps t is equal to the target number of steps N, the output result of the inference decoding can be used as an acoustic feature sequence, which can be expressed as: …… (11) From the above process steps, it can be seen that the initial step defines the basis of the output. Errors in the starting area will propagate and amplify in subsequent steps. The higher capacity of the teacher CFM model ensures accurate initialization. The steps in the middle area refine the output based on the infrastructure. Errors here are unlikely to propagate significantly. The optimized student CFM model is sufficient to handle this part efficiently. The final step refines the quality of the output, which must meet high standards to ensure quality. Errors in this area directly affect the final output. The teacher CFM model is used for its accuracy and high capacity to reduce errors in the terminal area.

[0025] Step S13: reconstructing the acoustic feature sequence based on the vocoder to obtain a synthesized speech signal.

[0026] Specifically, after obtaining the acoustic feature sequence, the acoustic feature sequence can be directly reconstructed using a vocoder to obtain a synthesized speech signal. It should be noted that the specific process of reconstructing the acoustic feature sequence can refer to the technical details of the vocoder, which will not be repeated here. As a possible example, the system framework combining the teacher CFM model and the student CFM model in the disclosed embodiment can be integrated into a speech synthesis system, or it can also be integrated into a large language model (that is, the system framework combining the teacher CFM model and the student CFM model in the disclosed embodiment can be used as a part of the large language model). The specific integration method of the system framework combining the teacher CFM model and the student CFM model in the disclosed embodiment is not limited here.

[0027] The above scheme obtains the pronunciation feature sequence of the text to be synthesized, and the pronunciation feature sequence contains the pronunciation feature information of each phoneme in the text to be synthesized, and then performs the first step of inference decoding on the pronunciation feature sequence based on the teacher CFM model to obtain the first feature sequence, and performs the second step of inference decoding on the first feature sequence based on the student CFM model to obtain the second feature sequence, and performs the third step of inference decoding on the second feature sequence based on the teacher CFM model to obtain the third feature sequence as the acoustic feature sequence, and the sum of the first step, the second step, and the third step is the target step, and the number of parameters of the student CFM model is less than that of the teacher CFM model. M model, based on the vocoder, reconstructs the acoustic feature sequence to obtain the synthesized speech signal. Therefore, on the one hand, since the CFM model as a decoder has been proven to have excellent performance in converting the pronunciation feature sequence into the acoustic feature sequence, the use of the CFM model in speech synthesis can ensure the naturalness of the synthesized speech as much as possible. On the other hand, since the teacher CFM model and the student CFM model with fewer parameters are jointly inferred during the speech synthesis process, and the teacher CFM model infers the number of steps at the beginning and end, and the student CFM model infers the number of steps in the middle, it can greatly reduce the number of parameters and decoding time while ensuring the performance of the CFM model. Therefore, it is possible to reduce computing resources and time costs as much as possible while ensuring the naturalness of the synthesized speech as much as possible.

[0028] See also Figure 2 , Figure 2 It is a schematic diagram of a framework of an embodiment of a speech synthesis device of the present application. The speech synthesis device 20 includes: a sequence acquisition module 21, an inference decoding module 22 and a signal reconstruction module 23, the sequence acquisition module 21 is used to obtain the pronunciation feature sequence of the text to be synthesized; wherein the pronunciation feature sequence contains the pronunciation feature information of each phoneme in the text to be synthesized; the inference decoding module 22 is used to perform the first step of inference decoding on the pronunciation feature sequence based on the teacher CFM model to obtain the first feature sequence, and perform the second step of inference decoding on the first feature sequence based on the student CFM model to obtain the second feature sequence, and perform the third step of inference decoding on the second feature sequence based on the teacher CFM model to obtain the third feature sequence as the acoustic feature sequence; wherein the sum of the first step, the second step and the third step is the target step, and the number of parameters of the student CFM model is less than that of the teacher CFM model; the signal reconstruction module 23 is used to reconstruct the acoustic feature sequence based on the vocoder to obtain a synthesized speech signal.

[0029] In the above scheme, the speech synthesis device 20 obtains the pronunciation feature sequence of the text to be synthesized, and the pronunciation feature sequence includes the pronunciation feature information of each phoneme in the text to be synthesized, and then performs the first step of inference decoding on the pronunciation feature sequence based on the teacher CFM model to obtain the first feature sequence, and performs the second step of inference decoding on the first feature sequence based on the student CFM model to obtain the second feature sequence, and performs the third step of inference decoding on the second feature sequence based on the teacher CFM model to obtain the third feature sequence as the acoustic feature sequence, and the sum of the first step, the second step, and the third step is the target step, and the parameter amount of the student CFM model is less than The teacher CFM model reconstructs the acoustic feature sequence based on the vocoder to obtain the synthesized speech signal. Therefore, on the one hand, since the CFM model as a decoder has been proven to have excellent performance in converting the pronunciation feature sequence into the acoustic feature sequence, the use of the CFM model in speech synthesis can ensure the naturalness of the synthesized speech as much as possible. On the other hand, since the teacher CFM model and the student CFM model with fewer parameters are jointly inferred during the speech synthesis process, and the teacher CFM model infers the number of steps at the beginning and end, and the student CFM model infers the number of steps in the middle, it can greatly reduce the number of parameters and decoding time while ensuring the performance of the CFM model. Therefore, it is possible to reduce computing resources and time costs as much as possible while ensuring the naturalness of the synthesized speech as much as possible.

[0030] In some disclosed embodiments, the first step number is the same as the third step number, and the speech synthesis device 20 also includes a step partitioning module for obtaining the product of a preset ratio and a target step number as the first step number and the third step number, and obtaining the difference between the target step number and the first step number and the third step number as the second step number.

[0031] In some disclosed embodiments, the student CFM is obtained by pruning the teacher CFM model and performing distillation training with the teacher CFM model.

[0032] In some disclosed embodiments, the speech synthesis device 20 also includes a model pruning module for pruning the target network layer in the teacher CFM model to obtain a student CFM model; wherein the target network layer includes at least an attention layer.

[0033] In some disclosed embodiments, the CFM model outputs a vector field, and the feature sequence output when performing inference decoding is obtained by superimposing the vector field on the feature sequence input when performing inference decoding. The speech synthesis device 20 also includes a sequence acquisition module for acquiring a sample pronunciation feature sequence of the sample text, and acquiring a target vector field that outputs a target acoustic feature sequence under the condition of the sample pronunciation feature sequence; wherein the target acoustic feature sequence is a sample acoustic feature sequence of the sample speech to which the sample text belongs; the speech synthesis device 20 also includes a vector field acquisition module for acquiring a first vector field output after the sample pronunciation feature sequence is input into the teacher CFM model for processing, and acquiring a second vector field output after the sample pronunciation feature sequence is input into the student CFM model; the speech synthesis device 20 also includes a vector field selection module for selecting the target vector field with a first probability and selecting the first vector field with a second probability to obtain a reference vector field; wherein the sum of the first probability and the second probability is 1; the speech synthesis device 20 also includes a parameter adjustment module for adjusting the network parameters of the student CFM model based at least on the difference between the second vector field and the reference vector field.

[0034] In some disclosed embodiments, the speech synthesis device 20 also includes a probability distribution acquisition module for acquiring a first probability distribution of a first vector field output by the teacher CFM model, and acquiring a second probability distribution of a second vector field output by the student CFM model; the parameter adjustment module is specifically used to adjust the network parameters of the student CFM model based on the difference between the second vector field and the reference vector field, and the difference between the first probability distribution and the second probability distribution.

[0035] In some disclosed embodiments, the speech synthesis device 20 also includes a sequence discarding module, which is used to discard the output feature sequence of the previous step as the input feature sequence of the current step at a target discarding rate in each step of the student CFM model inference decoding during the process of the student CFM model processing the sample pronunciation feature sequence in the current training round, and to perform unconditional sampling based on the target acoustic feature sequence as the input feature sequence of the current step when the output feature sequence of the previous step is discarded as the input feature sequence of the current step; wherein the target discarding rate is negatively correlated with the cumulative number of training rounds up to the current training round.

[0036] See also Figure 3 , Figure 3: is a schematic diagram of the framework of an embodiment of an electronic device of the present application. The electronic device 30 includes at least a memory 31 and a processor 32 coupled to each other, the memory 31 stores at least program instructions, and the processor 32 is used to execute the program instructions to implement the steps in any of the above-mentioned speech synthesis method embodiments. For details, please refer to the aforementioned disclosed embodiments, which will not be repeated here. As a possible example, the electronic device 30 may include but is not limited to an office notebook, an e-book reader, a smart phone, a tablet computer, etc., and the specific type of the electronic device 30 is not limited here.

[0037] Specifically, the processor 32 is used to control itself and the memory 31 to implement the steps in any of the above-mentioned speech synthesis method embodiments. The processor 32 can also be called a CPU (Central Processing Unit). The processor 32 may be an integrated circuit chip with signal processing capabilities. The processor 32 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 32 can be implemented by an integrated circuit chip.

[0038] In the above scheme, the electronic device 30 obtains the pronunciation feature sequence of the text to be synthesized, and the pronunciation feature sequence includes the pronunciation feature information of each phoneme in the text to be synthesized, and then performs the first step of inference decoding on the pronunciation feature sequence based on the teacher CFM model to obtain the first feature sequence, and performs the second step of inference decoding on the first feature sequence based on the student CFM model to obtain the second feature sequence, and performs the third step of inference decoding on the second feature sequence based on the teacher CFM model to obtain the third feature sequence as the acoustic feature sequence, and the sum of the first step, the second step, and the third step is the target step, and the number of parameters of the student CFM model is less than that of the teacher. The teacher CFM model reconstructs the acoustic feature sequence based on the vocoder to obtain the synthesized speech signal. Therefore, on the one hand, since the CFM model as a decoder has been proven to have excellent performance in converting the pronunciation feature sequence into the acoustic feature sequence, the use of the CFM model in speech synthesis can ensure the naturalness of the synthesized speech as much as possible. On the other hand, since the teacher CFM model and the student CFM model with fewer parameters are jointly inferred during the speech synthesis process, and the teacher CFM model infers the number of steps at the beginning and end, and the student CFM model infers the number of steps in the middle, it can greatly reduce the number of parameters and decoding time while ensuring the performance of the CFM model. Therefore, it is possible to reduce computing resources and reduce time costs as much as possible while ensuring the naturalness of the synthesized speech as much as possible.

[0039] See also Figure 4 , Figure 4 4 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 40 of the present application. The computer-readable storage medium 40 stores program instructions 41 that can be executed by a processor, and the program instructions 41 are used to implement the steps in any of the above-mentioned speech synthesis method embodiments.

[0040] In the above scheme, the computer-readable storage medium 40 obtains the pronunciation feature sequence of the text to be synthesized, and the pronunciation feature sequence contains the pronunciation feature information of each phoneme in the text to be synthesized, and then performs the first step of inference decoding on the pronunciation feature sequence based on the teacher CFM model to obtain the first feature sequence, and performs the second step of inference decoding on the first feature sequence based on the student CFM model to obtain the second feature sequence, and performs the third step of inference decoding on the second feature sequence based on the teacher CFM model to obtain the third feature sequence as the acoustic feature sequence, and the sum of the first step, the second step, and the third step is the target step, and the number of parameters of the student CFM model is small. The teacher CFM model reconstructs the acoustic feature sequence based on the vocoder to obtain the synthesized speech signal. Therefore, on the one hand, since the CFM model as a decoder has been proven to have excellent performance in converting the pronunciation feature sequence into the acoustic feature sequence, the use of the CFM model in speech synthesis can ensure the naturalness of the synthesized speech as much as possible. On the other hand, since the teacher CFM model and the student CFM model with fewer parameters are jointly inferred during the speech synthesis process, and the teacher CFM model infers the number of steps at the beginning and end, and the student CFM model infers the number of steps in the middle, the number of parameters and decoding time can be greatly reduced while ensuring the performance of the CFM model. Therefore, it is possible to reduce computing resources and reduce time costs as much as possible while ensuring the naturalness of the synthesized speech as much as possible.

[0041] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0042] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0043] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0044] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0045] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0046] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0047] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A speech synthesis method, characterized in that: include: Acquire a pronunciation feature sequence of the text to be synthesized; wherein the pronunciation feature sequence includes pronunciation feature information of each phoneme in the text to be synthesized; Based on the teacher CFM model, the pronunciation feature sequence is subjected to the first number of inference decoding to obtain a first feature sequence, and based on the student CFM model, the first feature sequence is subjected to the second number of inference decoding to obtain a second feature sequence, and based on the teacher CFM model, the second feature sequence is subjected to the third number of inference decoding to obtain a third feature sequence as an acoustic feature sequence; wherein the sum of the first number of steps, the second number of steps, and the third number of steps is the target number of steps, and the number of parameters of the student CFM model is less than that of the teacher CFM model; The acoustic feature sequence is reconstructed based on a vocoder to obtain a synthetic speech signal.

2. The method according to claim 1, characterized in that The first step is the same as the third step. Before performing the first step of inference decoding on the pronunciation feature sequence based on the teacher CFM model to obtain the first feature sequence, the method further includes: The product of the preset ratio and the target number of steps is obtained as the first number of steps and the third number of steps, and the difference between the target number of steps and the first number of steps and the third number of steps is obtained as the second number of steps.

3. The method according to claim 1, characterized in that The student CFM is obtained by pruning the teacher CFM model and performing distillation training with the teacher CFM model.

4. The method according to claim 3, characterized in that The clipping operation includes: The target network layer in the teacher CFM model is trimmed to obtain the student CFM model; wherein the target network layer includes at least an attention layer.

5. The method according to claim 3, characterized in that: The CFM model outputs a vector field, and the feature sequence output when performing the inference decoding is obtained by superimposing the vector field on the feature sequence input when performing the inference decoding. The distillation training includes: Acquire a sample pronunciation feature sequence of a sample text, and acquire a target vector field that outputs a target acoustic feature sequence under the condition of the sample pronunciation feature sequence; wherein the target acoustic feature sequence is a sample acoustic feature sequence of a sample speech to which the sample text belongs; Obtaining a first vector field output after the sample pronunciation feature sequence is input into the teacher CFM model for processing, and obtaining a second vector field output after the sample pronunciation feature sequence is input into the student CFM model; Selecting the target vector field with a first probability and selecting the first vector field with a second probability to obtain a reference vector field; wherein the sum of the first probability and the second probability is 1; A network parameter of the student CFM model is adjusted based on at least a difference between the second vector field and the reference vector field.

6. The method according to claim 5, characterized in that Before adjusting the network parameters of the student CFM model based at least on the difference between the second vector field and the reference vector field, the method further includes: Obtain a first probability distribution of the first vector field output by the teacher CFM model, and obtain a second probability distribution of the second vector field output by the student CFM model; The adjusting the network parameters of the student CFM model based at least on the difference between the second vector field and the reference vector field comprises: Based on the difference between the second vector field and the reference vector field, and the difference between the first probability distribution and the second probability distribution, the network parameters of the student CFM model are adjusted.

7. The method according to claim 5, characterized in that In the process of processing the sample pronunciation feature sequence by the student CFM model in the current training round, the method further includes: In each step of the student CFM model inference decoding, the output feature sequence of the previous step is abandoned at a target discard rate as the input feature sequence of the current step, and in the case of abandoning the output feature sequence of the previous step as the input feature sequence of the current step, unconditional sampling is performed based on the target acoustic feature sequence as the input feature sequence of the current step; The target discard rate is negatively correlated with the cumulative number of training rounds up to the current training round.

8. A speech synthesis device, characterized in that: include: A sequence acquisition module, used to acquire a pronunciation feature sequence of the text to be synthesized; wherein the pronunciation feature sequence includes pronunciation feature information of each phoneme in the text to be synthesized; An inference decoding module, configured to perform a first number of inference decoding on the pronunciation feature sequence based on the teacher CFM model to obtain a first feature sequence, perform a second number of inference decoding on the first feature sequence based on the student CFM model to obtain a second feature sequence, and perform a third number of inference decoding on the second feature sequence based on the teacher CFM model to obtain a third feature sequence as an acoustic feature sequence; wherein the sum of the first number of steps, the second number of steps, and the third number of steps is a target number of steps, and the number of parameters of the student CFM model is less than that of the teacher CFM model; The signal reconstruction module is used to reconstruct the acoustic feature sequence based on a vocoder to obtain a synthetic speech signal.

9. An electronic device, characterized in that: The invention at least comprises a memory and a processor coupled to each other, wherein the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the speech synthesis method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the speech synthesis method according to any one of claims 1 to 7.

Images