A method, device, equipment and storage medium for generating audio feature information

Abstract

The present invention discloses a method, device, equipment and storage medium for generating audio feature information. The method includes receiving text information to be speech synthesized and a selected sound type; predicting acoustic feature information based on the text information; obtaining standardized parameters corresponding to the sound type; performing restoration processing on the acoustic feature information using the standardized parameters; generating audio feature information corresponding to the sound type based on the restored acoustic feature information; the audio feature information is used to generate a speech signal corresponding to the sound type. In the present invention, multiple sounds share the same set of predictors to predict acoustic feature information, and the predictor outputs regularized acoustic features. Through the standardized parameters corresponding to the selected sound, the acoustic features can be de-regularized to the original space, so that the decoder generates audio feature information corresponding to the selected sound type based on the de-regularized acoustic features, thereby enabling the speech synthesis model to have the function of predicting multiple sounds.

Description

Technical Field

[0001] The present invention relates to the field of audio processing technology, and in particular to a method, device, equipment and storage medium for generating audio feature information. Background Art

[0002] In some application scenarios, text information needs to be converted into voice signals so that users can hear the content of the text information without viewing the text information. For example, a voice assistant application has the function of converting text into voice.

[0003] In order to solve the problem of how to convert text information into speech signals, the TTS (Text To Speech) model came into being. After training, the TTS model can convert text information into speech signals. Furthermore, the TTS model of the Fastspeech class includes an acoustic model, which includes a feature adder. The predictor in the feature adder is used to predict acoustic features such as duration, pitch, and energy of the speech to improve the quality of speech synthesis. However, the current predictor can only predict one voice (one speaker) and cannot predict multiple voices, which results in the speech synthesis model being unable to synthesize multiple voices. This is because when training the model, the acoustic features of one voice are used as samples to output the acoustic features corresponding to the voice. In other words, the predictor directly maps the text information to the actual data of the acoustic features of the voice, and the actual data can only represent one voice. If you want to train multiple voices, because the duration, pitch, and energy of different voices are different, the feature value range of the acoustic features is also different, resulting in multiple voices being unable to share the predictor. Summary of the invention

[0004] The main purpose of the present invention is to propose an audio feature information generation method, device, equipment and storage medium, aiming to solve the problem that the predictor can only predict one sound but cannot predict multiple sounds, resulting in the inability of the speech synthesis model to synthesize multiple sounds.

[0005] To achieve the above technical problems, the present invention is implemented through the following technical solutions:

[0006] An embodiment of the present invention provides a method for generating audio feature information, and the steps performed in a speech synthesis model include: receiving text information to be speech synthesized and a selected sound type; predicting acoustic feature information based on the text information; obtaining standardized parameters corresponding to the sound type; using the standardized parameters, performing restoration processing on the acoustic feature information; generating audio feature information corresponding to the sound type based on the acoustic feature information after the restoration processing; wherein the audio feature information is used to generate a speech signal corresponding to the sound type.

[0007] There are multiple acoustic feature information; there are multiple standardized parameters corresponding to the sound types; each standardized parameter corresponds to an acoustic feature information; and the use of the standardized parameters to perform restoration processing on the acoustic feature information includes: for each acoustic feature information, using the standardized parameters corresponding to the acoustic feature information to perform restoration processing on the acoustic feature information.

[0008] The standardized parameters include mean and standard deviation.

[0009] Before receiving the text information to be speech synthesized and the selected sound type, the method also includes: obtaining a sample set for training the speech synthesis model; wherein the sample set includes a plurality of sample speech signals corresponding to each sound type; for each sound type, determining and recording the mean and standard deviation corresponding to each acoustic feature based on the plurality of sample speech signals corresponding to the sound type.

[0010] Among them, predicting the acoustic feature information based on the text information includes: performing a preprocessing operation on the text information; performing an encoding operation on the preprocessed text information to obtain a text latent vector corresponding to the text information; and predicting the acoustic feature information based on the text latent vector corresponding to the text information.

[0011] Among them, the generating of audio feature information according to the acoustic feature information after restoration includes: using a preset mapping method to map the acoustic feature information after restoration to a space with the same dimension as the text latent vector to obtain an acoustic latent vector; superimposing the acoustic latent vector and the text latent vector to obtain a superimposed vector; performing a decoding operation on the superimposed vector to generate audio feature information corresponding to the sound type.

[0012] The types of mapping methods include: linear mapping method and convolution mapping method.

[0013] An embodiment of the present invention also provides an audio feature information generating device, comprising: a receiving module, used to receive text information to be speech synthesized and a selected sound type; a prediction module, used to predict acoustic feature information based on the text information; an acquisition module, used to acquire standardized parameters corresponding to the sound type; a restoration module, used to perform restoration processing on the acoustic feature information using the standardized parameters; a generation module, used to generate audio feature information corresponding to the sound type based on the acoustic feature information after the restoration processing; wherein the audio feature information is used to generate a speech signal corresponding to the sound type.

[0014] An embodiment of the present invention further provides an audio feature information generating device, the audio feature information generating device comprising a processor and a memory; the processor is used to execute an audio feature information generating program stored in the memory to implement any of the audio feature information generating methods described above.

[0015] An embodiment of the present invention further provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement any of the above-mentioned methods for generating audio feature information.

[0016] The beneficial effects of the embodiments of the present invention are as follows:

[0017] In an embodiment of the present invention, a speech synthesis model receives text information to be speech synthesized and a selected sound type; predicts acoustic feature information based on the text information; obtains standardized parameters corresponding to the sound type; uses the standardized parameters to perform restoration processing on the acoustic feature information; generates audio feature information corresponding to the sound type based on the acoustic feature information after restoration processing; wherein the audio feature information is used to generate a speech signal corresponding to the sound type. In an embodiment of the present invention, multiple sounds can share the same set of predictors to predict acoustic feature information, and the acoustic feature information output by the predictor is a regularized acoustic feature. Through the standardized parameters corresponding to the selected sound, the acoustic feature can be de-regularized to the original space, so that the decoder generates audio feature information corresponding to the selected sound type based on the de-regularized acoustic features, thereby enabling the speech synthesis model to predict multiple sounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0019] Figure 1 is a flowchart of a method for generating audio feature information according to an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of the structure of a speech synthesis model according to an embodiment of the present invention;

[0021] Figure 3 is a structural diagram of an audio feature information generating device according to an embodiment of the present invention;

[0022] Figure 4 FIG. 4 is a structural diagram of an audio feature information generating device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0024] According to an embodiment of the present invention, a method for generating audio feature information is provided. The execution subject of this method embodiment is a speech synthesis model. Figure 1 FIG. 1 is a flow chart of a method for generating audio feature information according to an embodiment of the present invention. Figure 2 Schematic diagram of the structure of a speech synthesis model according to an embodiment of the present invention (only some modules in the speech synthesis model are schematically drawn).

[0025] Step S110, receiving text information to be speech synthesized and a selected sound type.

[0026] The content of the text information is the content that needs to be converted into a voice signal.

[0027] The voice type is the type of the speaker. There are many types of voice types, for example: the voice of boy A, the voice of boy B.

[0028] Step S120: predicting acoustic feature information based on the text information.

[0029] There are multiple acoustic feature information. The types of acoustic feature information include but are not limited to: duration, pitch and energy of speech.

[0030] Specifically, a preprocessing operation is performed on the text information; an encoding operation is performed on the preprocessed text information to obtain a text latent vector corresponding to the text information; and acoustic feature information is predicted based on the text latent vector corresponding to the text information.

[0031] Furthermore, after receiving the text information, the speech synthesis model can obtain the pinyin information corresponding to the text information by performing a preprocessing operation on the text information; the pinyin information is converted into a word vector and then input into an encoder, which performs an encoding operation on the word vector to obtain a text latent vector corresponding to the text information, and inputs the text latent vector into a feature adapter; the feature adapter includes multiple predictors, each predictor corresponds to an acoustic feature, for example: predictor 1 corresponds to duration, predictor 2 corresponds to pitch, and predictor 3 corresponds to energy; multiple predictors receive the text latent vectors respectively, and predict the acoustic feature information of their corresponding acoustic features based on the received text latent vectors and output them.

[0032] Step S130, obtaining standardized parameters corresponding to the sound type.

[0033] The standardized parameters include mean and standard deviation.

[0034] There are multiple standardized parameters corresponding to the sound type, wherein each standardized parameter corresponds to an acoustic feature information.

[0035] The normalization parameters are values ​​determined according to sample speech signals corresponding to multiple sound types during the speech synthesis model training phase or before the trained speech synthesis model is used.

[0036] Specifically, before receiving the text information to be speech synthesized and the selected sound type, a sample set for training the speech synthesis model is obtained; wherein the sample set includes a plurality of sample speech signals corresponding to each sound type; for each sound type, based on the plurality of sample speech signals corresponding to the sound type, the mean and standard deviation corresponding to each acoustic feature are determined and recorded.

[0037] For example: voice type 1 (i.e. speaker 1) corresponds to 10,000 sample voice signals, and the acoustic feature information of each of the 10,000 sample voice signals is determined, such as duration, pitch, and energy; the mean and standard deviation of duration are determined as the duration standardization parameter corresponding to voice type 1; the mean and standard deviation of pitch are determined as the pitch standardization parameter corresponding to voice type 1; the mean and standard deviation of energy are determined as the energy standardization parameter corresponding to voice type 1. Voice type 2 and voice type 3 can also obtain their own standardization parameters by referring to this example.

[0038] Step S140: using the standardized parameters, perform restoration processing on the acoustic feature information.

[0039] Restoration processing means that the feature adder uses pre-recorded standardized parameters to restore the acoustic feature information to the original space.

[0040] After obtaining a plurality of standardized parameters corresponding to the sound type, for each piece of acoustic feature information, restoration processing is performed on the acoustic feature information using the standardized parameters corresponding to the acoustic feature information.

[0041] Specifically, in the stage of training the speech synthesis model, the training predictor outputs standardized acoustic feature information, so that the acoustic features of multiple sounds can be normalized into the same space. The feature controls of each sound have a mean of 0 and a standard deviation of 1, so that multiple sounds can share the predictor. In order to make the sound heard by the user meet the expected needs, after the predictor predicts the acoustic feature information, it is necessary to use the standardized parameters corresponding to the sound type expected by the user to map the acoustic feature information to the original space, that is, to denormalize it to the original space, so that the final synthesized speech signal is the sound expected by the user.

[0042] Further, restoration processing is performed, including: multiplying the acoustic feature information by the corresponding standard deviation, and adding the corresponding mean value to obtain the acoustic feature information after restoration processing.

[0043] Step S150, generating audio feature information corresponding to the sound type according to the acoustic feature information after restoration processing; wherein the audio feature information is used to generate a speech signal corresponding to the sound type.

[0044] The audio feature information includes: Mel spectrogram.

[0045] The predictor can use a preset mapping method to map the restored acoustic feature information to a space with the same dimension as the text latent vector to obtain an acoustic latent vector; superimpose the acoustic latent vector and the text latent vector to obtain a superimposed vector; and the decoder can perform a decoding operation on the superimposed vector to generate audio feature information corresponding to the sound type. The types of the mapping method include but are not limited to: a linear mapping method and a convolutional mapping method.

[0046] The linear mapping method is, for example: the predictor converts the restored acoustic feature information into a vector, multiplies the vector by a preset linear mapping vector, and then realizes the linear mapping of the acoustic feature information to obtain the acoustic latent vector.

[0047] The convolution mapping method is, for example: the predictor converts the restored acoustic feature information into a vector, inputs the vector into a preset one or more layers of a neural network, and then implements the convolution mapping of the acoustic feature information to obtain an acoustic latent vector.

[0048] To ensure that the acoustic latent vector and the textual latent vector can be superimposed, the dimensions of the acoustic latent vector and the textual latent vector need to be the same during mapping.

[0049] Specifically, the result output by the existing predictor does not use the direct mapping method of this embodiment, but divides the possible range of the output result into multiple intervals (bins) in advance, and sets a corresponding value for each interval (such as onehot encoding). After the predictor outputs the result, the interval to which the output result belongs is quantized, and then the value corresponding to the output result is determined, and the value is input into the decoder so that the decoder generates audio feature information. However, the accuracy of this method is poor. For example, if the predictor outputs 11 and 19 respectively, if 10 is used as a bin, after the result is quantized, the results of the two values ​​converted to one hot encoding are both 2, and the actual difference between 11 and 19 is large. If the scale of the bin is reduced, the prediction accuracy can be increased, but it is more difficult to train. However, in this embodiment, the output result of the predictor is not quantized, but a direct mapping method is used to eliminate the error caused by quantization. For example, in the case of linear mapping, the mapping value basis of 11 and 19 will remain different, thereby increasing the prediction accuracy.

[0050] In this embodiment, the speech synthesis model receives text information to be speech synthesized and a selected sound type; predicts acoustic feature information based on the text information; obtains standardized parameters corresponding to the sound type; uses the standardized parameters to perform restoration processing on the acoustic feature information; generates audio feature information corresponding to the sound type based on the restored acoustic feature information; wherein the audio feature information is used to generate a speech signal corresponding to the sound type. In this embodiment, multiple sounds can share the same set of predictors to predict acoustic feature information, and the acoustic feature information output by the predictor is a regularized acoustic feature. Through the standardized parameters corresponding to the selected sound, the acoustic feature can be de-regularized to the original space, so that the decoder generates audio feature information corresponding to the selected sound type based on the de-regularized acoustic features, thereby enabling the speech synthesis model to predict multiple sounds.

[0051] The embodiment of the present invention also provides a device for generating audio feature information. Figure 3 FIG. 4 is a structural diagram of an audio feature information generating device according to an embodiment of the present invention.

[0052] The receiving module 310 is used to receive text information to be speech synthesized and a selected sound type.

[0053] The prediction module 320 is used to predict acoustic feature information based on the text information.

[0054] The acquisition module 330 is used to acquire the standardized parameters corresponding to the sound type.

[0055] The restoration module 340 is used to perform restoration processing on the acoustic feature information using the standardized parameters.

[0056] The generating module 350 is used to generate audio feature information corresponding to the sound type according to the acoustic feature information after restoration processing; wherein the audio feature information is used to generate a speech signal corresponding to the sound type.

[0057] Furthermore, the number of the acoustic feature information is multiple; the number of standardized parameters corresponding to the sound type is multiple; wherein each standardized parameter corresponds to an acoustic feature information; the restoration module 340 is used to perform restoration processing on the acoustic feature information for each acoustic feature information using the standardized parameters corresponding to the acoustic feature information.

[0058] Furthermore, the standardized parameters include: mean and standard deviation.

[0059] Furthermore, the device also includes a recording module (not shown in the figure). The recording module is used to obtain a sample set for training the speech synthesis model before receiving the text information to be speech synthesized and the selected sound type; wherein the sample set includes a plurality of sample speech signals corresponding to each sound type; for each sound type, according to the plurality of sample speech signals corresponding to the sound type, determine and record the mean and standard deviation corresponding to each acoustic feature.

[0060] Furthermore, the prediction module 320 is used to perform a preprocessing operation on the text information; perform an encoding operation on the preprocessed text information to obtain a text latent vector corresponding to the text information; and predict acoustic feature information based on the text latent vector corresponding to the text information.

[0061] Furthermore, the generation module 350 is used to use a preset mapping method to map the restored acoustic feature information to a space with the same dimension as the text latent vector to obtain an acoustic latent vector; superimpose the acoustic latent vector and the text latent vector to obtain a superimposed vector; and perform a decoding operation on the superimposed vector to generate audio feature information corresponding to the sound type.

[0062] Furthermore, the types of mapping methods include: linear mapping method and convolution mapping method.

[0063] The functions of the device described in the embodiment of the present invention have been described in the above method embodiment. Therefore, for details not provided in the description of this embodiment, please refer to the relevant description in the above embodiment and will not be repeated here.

[0064] This embodiment provides a device for generating audio feature information. Figure 4 FIG. 4 is a structural diagram of an audio feature information generating device according to an embodiment of the present invention.

[0065] In this embodiment, the audio feature information generating device includes but is not limited to: a processor 410 and a memory 420 .

[0066] The processor 410 is used to execute the audio feature information generation program stored in the memory 420 to implement the above-mentioned audio feature information generation method.

[0067] Specifically, the processor 410 is used to execute the audio feature information generation program stored in the memory 420 to implement the following steps: receiving text information to be speech synthesized and a selected sound type; predicting acoustic feature information based on the text information; obtaining standardized parameters corresponding to the sound type; using the standardized parameters, performing restoration processing on the acoustic feature information; generating audio feature information corresponding to the sound type based on the acoustic feature information after the restoration processing; wherein the audio feature information is used to generate a speech signal corresponding to the sound type.

[0068] There are multiple acoustic feature information; there are multiple standardized parameters corresponding to the sound types; each standardized parameter corresponds to an acoustic feature information; and the use of the standardized parameters to perform restoration processing on the acoustic feature information includes: for each acoustic feature information, using the standardized parameters corresponding to the acoustic feature information to perform restoration processing on the acoustic feature information.

[0069] The standardized parameters include mean and standard deviation.

[0070] Before receiving the text information to be speech synthesized and the selected sound type, the method also includes: obtaining a sample set for training the speech synthesis model; wherein the sample set includes a plurality of sample speech signals corresponding to each sound type; for each sound type, determining and recording the mean and standard deviation corresponding to each acoustic feature based on the plurality of sample speech signals corresponding to the sound type.

[0071] Among them, predicting the acoustic feature information based on the text information includes: performing a preprocessing operation on the text information; performing an encoding operation on the preprocessed text information to obtain a text latent vector corresponding to the text information; and predicting the acoustic feature information based on the text latent vector corresponding to the text information.

[0072] Among them, the generating of audio feature information according to the acoustic feature information after restoration includes: using a preset mapping method to map the acoustic feature information after restoration to a space with the same dimension as the text latent vector to obtain an acoustic latent vector; superimposing the acoustic latent vector and the text latent vector to obtain a superimposed vector; performing a decoding operation on the superimposed vector to generate audio feature information corresponding to the sound type.

[0073] The types of mapping methods include: linear mapping method and convolution mapping method.

[0074] The embodiment of the present invention also provides a computer-readable storage medium. The computer-readable storage medium here stores one or more programs. The computer-readable storage medium may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid-state drive; the memory may also include a combination of the above-mentioned types of memory.

[0075] When one or more programs in the computer-readable storage medium can be executed by one or more processors, the above-mentioned audio feature information generating method is implemented.

[0076] Specifically, the processor is used to execute an audio feature information generation program stored in a memory to implement the following steps: receiving text information to be speech synthesized and a selected sound type; predicting acoustic feature information based on the text information; obtaining standardized parameters corresponding to the sound type; using the standardized parameters, performing restoration processing on the acoustic feature information; generating audio feature information corresponding to the sound type based on the acoustic feature information after restoration processing; wherein the audio feature information is used to generate a speech signal corresponding to the sound type.

[0077] There are multiple acoustic feature information; there are multiple standardized parameters corresponding to the sound types; each standardized parameter corresponds to an acoustic feature information; and the use of the standardized parameters to perform restoration processing on the acoustic feature information includes: for each acoustic feature information, using the standardized parameters corresponding to the acoustic feature information to perform restoration processing on the acoustic feature information.

[0078] The standardized parameters include mean and standard deviation.

[0079] Before receiving the text information to be speech synthesized and the selected sound type, the method also includes: obtaining a sample set for training the speech synthesis model; wherein the sample set includes a plurality of sample speech signals corresponding to each sound type; for each sound type, determining and recording the mean and standard deviation corresponding to each acoustic feature based on the plurality of sample speech signals corresponding to the sound type.

[0080] Among them, predicting the acoustic feature information based on the text information includes: performing a preprocessing operation on the text information; performing an encoding operation on the preprocessed text information to obtain a text latent vector corresponding to the text information; and predicting the acoustic feature information based on the text latent vector corresponding to the text information.

[0081] Among them, the generating of audio feature information according to the acoustic feature information after restoration includes: using a preset mapping method to map the acoustic feature information after restoration to a space with the same dimension as the text latent vector to obtain an acoustic latent vector; superimposing the acoustic latent vector and the text latent vector to obtain a superimposed vector; performing a decoding operation on the superimposed vector to generate audio feature information corresponding to the sound type.

[0082] The types of mapping methods include: linear mapping method and convolution mapping method.

[0083] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of the claims of the present invention.

Claims

1. A method for generating audio feature information, characterized in that: The steps performed in the speech synthesis model include: obtaining a sample set for training the speech synthesis model; wherein the sample set includes a plurality of sample speech signals corresponding to each sound type; For each sound type, according to a plurality of sample speech signals corresponding to the sound type, determine and record a mean value and a standard deviation corresponding to each acoustic feature; Receiving text information to be speech synthesized and a selected voice type; Performing a preprocessing operation on the text information; Performing an encoding operation on the preprocessed text information to obtain a text latent vector corresponding to the text information; Predicting acoustic feature information according to a text latent vector corresponding to the text information; Obtain standardized parameters corresponding to the sound type; The number of standardized parameters corresponding to the sound type is multiple; wherein each standardized parameter corresponds to an acoustic feature information; The standardized parameters include: mean and standard deviation; The acoustic feature information is multiplied by the corresponding standard deviation, and then added to the corresponding mean value to obtain the acoustic feature information after restoration processing; the acoustic feature information after restoration processing is mapped to a space having the same dimension as the text latent vector using a preset mapping method to obtain an acoustic latent vector; Superimposing the acoustic latent vector and the text latent vector to obtain a superimposed vector; A decoding operation is performed on the superimposed vector to generate audio feature information corresponding to the sound type, wherein the audio feature information is used to generate a speech signal corresponding to the sound type.

2. The method according to claim 1, characterized in that The types of mapping methods include: linear mapping method and convolution mapping method.

3. An audio feature information generating device, characterized in that: include: A receiving module, used for receiving text information to be speech synthesized and a selected sound type; A prediction module, performing a preprocessing operation on the text information; Performing an encoding operation on the preprocessed text information to obtain a text latent vector corresponding to the text information; Predicting acoustic feature information according to a text latent vector corresponding to the text information; An acquisition module is used to acquire a sample set for training the speech synthesis model; wherein the sample set includes a plurality of sample speech signals corresponding to each sound type; For each sound type, according to a plurality of sample speech signals corresponding to the sound type, determine and record a mean value and a standard deviation corresponding to each acoustic feature; Acquire a standardized parameter corresponding to the sound type; the number of standardized parameters corresponding to the sound type is multiple; wherein each standardized parameter corresponds to an acoustic feature information; The standardized parameters include: mean and standard deviation; For each sound type, according to a plurality of sample speech signals corresponding to the sound type, determine and record a mean value and a standard deviation corresponding to each acoustic feature; The restoration module multiplies the acoustic feature information by the corresponding standard deviation and adds the corresponding mean to obtain the acoustic feature information after restoration. The generation module uses a preset mapping method to map the acoustic feature information after restoration to a space with the same dimension as the text latent vector to obtain the acoustic latent vector. Superimposing the acoustic latent vector and the text latent vector to obtain a superimposed vector; A decoding operation is performed on the superimposed vector to generate audio feature information corresponding to the sound type; wherein the audio feature information is used to generate a speech signal corresponding to the sound type.

4. An audio feature information generating device, characterized in that: The audio feature information generating device comprises a processor and a memory; the processor is used to execute the audio feature information generating program stored in the memory to implement the audio feature information generating method according to any one of claims 1 to 2.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the audio feature information generating method according to any one of claims 1 to 2.

Images