Speech synthesis method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of speech synthesis, and discloses a speech synthesis method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining reference speech data and a to-be-synthesized text; encoding the reference voice data through a preset rhythm encoder to obtain a rhythm embedding vector; performing feature extraction processing on the reference voice data through a preset voice recognition model to obtain a compensation feature vector; encoding the compensation feature vector through a preset rhythm compensation encoder to obtain a rhythm compensation vector; fusing the rhythm embedding vector and the rhythm compensation vector to obtain a rhythm feature vector; and synthesizing the to-be-synthesized text and the rhythm feature vector through a preset speech synthesis model to generate a target synthesized speech. According to the method, the voice synthesis is assisted through rhythm embedding and rhythm compensation, so that the loss of rhythm information is reduced, the voice synthesis effect is improved, intelligent and humanized services are embodied in a financial service scene, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of speech synthesis technology, and in particular to a speech synthesis method, device, computer equipment and storage medium. Background Art

[0002] Text To Speech (TTS) is a technology that converts text into speech. It can convert text, numbers and other non-speech information generated by computers or input by humans in real time into natural speech output. Speech synthesis gives machines the ability to speak freely like humans. It is an important basis for human-computer voice interaction and is widely used in various scenarios. For example: intelligent voice assistants (such as Siri, iFlytek Voice Point), telephone customer service information query, in-car navigation, audio e-book reading, station real-time information broadcast assistant, and communication assistant for people with visual or speech impairments.

[0003] As human-computer voice interaction scenarios in work and life become more and more abundant, users not only require that the audio of speech synthesis be understandable, fluent and natural, but also put forward requirements for rhythm / emotion. In different financial service business scenarios, user needs and business backgrounds are also different. For example, the business backgrounds of sales and claims business in insurance services are different, and the emotional requirements are also different. In the prior art, emotional speech synthesis methods with rhythm include emotional speech synthesis based on waveform splicing, statistical parameter emotional speech synthesis and emotional speech synthesis based on neural networks. Among them, deep neural networks have excellent modeling capabilities, and speech synthesis methods based on deep learning have been widely used. However, in the process of speech synthesis through neural networks, when the speaker information and prosody information are separated, the prosody information is often lost. When there is a lack of sufficient prosody information, voice customer service shows mechanical and single emotions in different business scenarios, the effect of speech synthesis is poor, and it cannot reflect intelligent and humanized services, which affects user experience. Summary of the invention

[0004] Based on this, it is necessary to provide a speech synthesis method, apparatus, computer equipment and storage medium to address the above technical issues, so as to solve the problem of serious prosodic information loss and poor speech synthesis effect during speech synthesis.

[0005] A speech synthesis method, comprising: Obtain reference speech data and text to be synthesized; Encoding the reference speech data by using a preset prosody encoder to obtain a prosody embedding vector; Performing feature extraction processing on the reference speech data through a preset speech recognition model to obtain a compensation feature vector; The compensation feature vector is encoded by a preset prosody compensation encoder to obtain a prosody compensation vector; fusing the prosody embedding vector and the prosody compensation vector to obtain a prosody feature vector; The text to be synthesized and the prosodic feature vector are synthesized by using a preset speech synthesis model to generate a target synthesized speech.

[0006] A speech synthesis device, comprising: A data acquisition module, used to acquire reference speech data and text to be synthesized; A prosody embedding vector encoding module, used for encoding the reference speech data through a preset prosody encoder to obtain a prosody embedding vector; A compensation feature vector extraction module, used to perform feature extraction processing on the reference speech data through a preset speech recognition model to obtain a compensation feature vector; A rhythm compensation vector encoding module, used for encoding the compensation feature vector through a preset rhythm compensation encoder to obtain a rhythm compensation vector; A vector fusion module, used for fusing the rhythm embedding vector and the rhythm compensation vector to obtain a rhythm feature vector; The speech synthesis module is used to synthesize the text to be synthesized and the prosodic feature vector through a preset speech synthesis model to generate a target synthesized speech.

[0007] A computer device comprises a memory, a processor and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the above-mentioned speech synthesis method when executing the computer-readable instructions.

[0008] A computer-readable storage medium stores computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the speech synthesis method as described above.

[0009] The above-mentioned speech synthesis method, device, computer equipment and storage medium, the method is as follows: obtaining reference speech data and text to be synthesized; encoding the reference speech data through a preset prosody encoder to obtain a prosody embedding vector; performing feature extraction processing on the reference speech data through a preset speech recognition model to obtain a compensation feature vector; encoding the compensation feature vector through a preset prosody compensation encoder to obtain a prosody compensation vector; fusing the prosody embedding vector and the prosody compensation vector to obtain a prosody feature vector; synthesizing the text to be synthesized and the prosody feature vector through a preset speech synthesis model to generate a target synthesized speech. While decoupling the prosody features of the reference speech from the speaker, the present invention assists speech synthesis through prosody embedding and prosody compensation, thereby reducing the loss of prosody information, obtaining synthesized speech with rhythm, and improving the speech synthesis effect, reflecting intelligent and humanized services in financial service business scenarios, and improving user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0011] Figure 1 is a schematic diagram of an application environment of a speech synthesis method according to an embodiment of the present invention; Figure 2 is a flow chart of a speech synthesis method according to an embodiment of the present invention; Figure 3 is another schematic flow chart of a speech synthesis method according to an embodiment of the present invention; Figure 4 is a structural schematic diagram of a speech synthesis device in one embodiment of the present invention; Figure 5 is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0013] The embodiments of the present application can acquire and process voice data and text data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Artificial intelligence technology mainly includes computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology and machine learning / deep learning. This embodiment mainly involves speech synthesis technology, speech recognition technology and deep learning technology.

[0014] The speech synthesis method provided in the embodiment of the present application can be applied in Figure 1 In an application environment, a client communicates with a server. The client includes but is not limited to various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers.

[0015] The speech synthesis method provided in the embodiments of the present application can be applied to service business scenarios in the financial field, especially intelligent voice customer service scenarios in the financial services field, such as intelligent sales services, intelligent consulting services, intelligent follow-up services, intelligent collection services, and intelligent after-sales services in the banking and insurance industries.

[0016] In one embodiment, if Figure 2 As shown, a speech synthesis method is provided, comprising the following steps S10-S60. S10: Obtain reference speech data and text to be synthesized.

[0017] Understandably, speech synthesis is a text-to-speech conversion process involving the speaker's voice and computer text, so that the synthesized artificial speech can achieve the effect of expressing the content of the computer text with the speaker's voice. Reference speech data refers to the speaker's real speech data, which serves as a reference source for sound feature information and prosodic feature information in the speech synthesis process. The reference speech data can be speech data collected in real time by the client through recording software or applet, or speech data in a database pre-saved by the client, or speech data uploaded directly to the client locally. The text to be synthesized refers to the content information used for synthesized speech, which can be text and characters generated by the computer itself, or text information input from the outside. In this embodiment, the client sends the reference speech data and the text to be synthesized to the server, and the server performs corresponding processing after obtaining the reference speech data and the text to be synthesized.

[0018] S20. Encode the reference speech data using a preset prosody encoder to obtain a prosody embedding vector.

[0019] Understandably, the reference speech data contains sound feature information and rhythm feature information. Speech is a special kind of sound, and the acoustic features of speech include four elements: timbre, pitch, intensity, and length. Timbre refers to the distinctive characteristics of different sounds in terms of waveform, also known as sound feature information. Each person has a different timbre due to differences in gender, age, larynx, and vocal tract structure. The ups and downs of speech in pitch, intensity, and length are also known as rhythm feature information. Figure 3 As shown, after the server obtains the reference speech data, in order to decouple the speaker's voice feature information and prosody feature information, the reference speech data is input into the preset speaker encoder for encoding processing, and the voice feature information is extracted as the speaker's voice vector; the reference speech data is input into the preset prosody encoder for encoding processing, and the prosody feature information is extracted as the prosody embedding vector. The encoder is used to transform an input sequence of variable length into a background variable of fixed length, and encode the information of the input sequence in the background variable. Encoding processing is the process of reconstructing the input sequence into an intermediate state, and the structure of the encoder can adopt a recurrent neural network. The preset speaker encoder is a pre-trained encoder for extracting voice feature information from the reference speech data. The preset prosody encoder is a pre-trained encoder for extracting prosody feature information from the reference speech data. The extracted vector expressing the prosody feature information is called a prosody embedding vector.

[0020] S30, performing feature extraction processing on the reference speech data through a preset speech recognition model to obtain a compensation feature vector.

[0021] Understandably, in the speech synthesis process, after the speaker's voice feature information and prosodic feature information are decoupled, prosodic embedding with sufficient prosodic feature information often leads to the speaker's timbre leakage, that is, the mutual influence between the speaker's voice feature information and prosodic feature information cannot be eliminated. In order to further eliminate the influence of prosodic embedding on the speaker's timbre leakage, it is necessary to remove some prosodic embedding vectors that have an influence between the voice feature information and the prosodic feature information, which may weaken the expressiveness of the prosodic feature. This will result in partial loss of prosodic feature information, for example, when the prosodic feature information in the reference speech data is 100%, the prosodic feature information in the prosodic embedding vector is 90%.

[0022] Speech recognition is a technology that converts speech into text. It is widely used, such as Siri in Apple phones, speech-to-text functions in WeChat chats, and voice control functions in smart speakers. The preset speech recognition model is a pre-trained automatic speech recognition model (Automatic Speech Recognition, ASR). The input of the ASR model is speech data, and the output is text information corresponding to the speech data. The ASR model filters out the speaker's voice feature information during the forward propagation process, while retaining some feature information related to rhythm. Since these retained rhythm feature information will not be affected by the speaker's voice feature information, they complement the rhythm feature information in the rhythm embedding vector and can be used as rhythm compensation feature information.

[0023] In this embodiment, the preset speech recognition model is a pre-trained neural network model for identifying text information in reference speech data. For example, the preset speech recognition model can use a speech static decoding network. There are several intermediate layers between the input layer and the output layer of the preset speech recognition model. In order to solve the problem of how to reduce the loss of rhythmic feature information, the preset speech recognition model is used to perform speech recognition on the reference speech data, but it is not necessary to obtain the final text information, but to extract the feature vector output by the intermediate layer as the compensation feature vector. Specifically, in the intelligent sales promotion service scenario of the bank, feature extraction processing is performed on the reference speech data expressing enthusiastic emotions to obtain the compensation feature vector corresponding to the enthusiastic emotions; in the intelligent return visit service scenario, feature extraction processing is performed on the reference speech data expressing gentle emotions to obtain the compensation feature vector corresponding to the gentle emotions; in the intelligent collection service scenario, feature extraction processing is performed on the reference speech data expressing severe emotions to obtain the compensation feature vector corresponding to the severe emotions.

[0024] S40 , encoding the compensation feature vector by using a preset prosody compensation encoder to obtain a prosody compensation vector.

[0025] Understandably, the prosody compensation vector refers to the vector output by the preset prosody compensation encoder, which is used to compensate for the lost part of the prosody feature information of the speaker. The prosody embedding vector is the output vector of the preset prosody encoder, and the compensation feature vector is the intermediate layer output vector of the preset speech recognition model. Due to the different neural network structures of the preset prosody encoder and the preset speech recognition model, the compensation feature vector and the prosody embedding vector may not be directly matched, resulting in errors in the prosody feature information. The preset prosody compensation encoder is a pre-trained encoder used to re-encode the compensation feature vector, so that the re-encoded prosody compensation vector can match the prosody embedding vector.

[0026] S50: Fusing the prosody embedding vector and the prosody compensation vector to obtain a prosody feature vector.

[0027] Understandably, the prosody embedding vector and the prosody compensation vector each contain two parts of the prosody feature information of the same speaker. Since these two parts of the prosody feature information are mutually complementary, the prosody embedding vector and the prosody compensation vector are fused to obtain the prosody feature vector. The prosody feature vector is a vector representing the prosody features of the speaker. For example, when the prosody feature information in the prosody embedding vector is 90% and the prosody feature information that complements each other in the prosody compensation vector is 8%, the prosody feature information in the prosody feature vector is 98%. In deep learning, vector fusion methods include addition and concatenation. Addition is the operation of performing point-by-point addition of the corresponding elements in the two vectors, and concatenation is the operation of performing linear transformation on the two vectors. In the fusion process, when the dimensions of the prosody embedding vector and the prosody compensation vector are the same, the addition method can be directly adopted; when the dimensions of the prosody embedding vector and the prosody compensation vector are different, the concatenation method needs to be adopted. In different service scenarios, this embodiment fuses the prosody embedding vector and the prosody compensation vector of different emotions in the synthesized speech according to different user needs. Specifically, in the bank's intelligent sales promotion service scenario, the rhythmic embedding vector and the rhythmic compensation vector corresponding to enthusiastic emotions are fused; in the intelligent follow-up service scenario, the rhythmic embedding vector and the rhythmic compensation vector corresponding to gentle emotions are fused; in the intelligent collection service scenario, the rhythmic embedding vector and the rhythmic compensation vector corresponding to harsh emotions are fused.

[0028] S60: synthesizing the text to be synthesized and the prosodic feature vector using a preset speech synthesis model to generate a target synthesized speech.

[0029] Understandably, speech synthesis needs to go through three parts: text processing, acoustic model and vocoder. The preset speech synthesis model is a pre-trained neural network model for converting the text to be synthesized into speech, that is, the acoustic model part. The input of the preset speech synthesis model includes two parts, in which the text to be synthesized is used as the basic input part, and the prosodic feature vector is used as the auxiliary input part. The output of the preset speech synthesis model is the speech mel spectrum feature. In this embodiment, Figure 3As shown in the figure, the preset speech recognition model uses the end-to-end speech synthesis model Tactotron, which is an autoregressive structure based on the text encoder (Encoder)-attention layer network (Attention)-text decoder (Decoder). The text to be synthesized enters the acoustic model part after the text processing part. In addition, the input of the acoustic model part also includes the rhythm feature vector. The output of the acoustic model part is the speech Mel spectrum. The speech Mel spectrum is passed through the vocoder to obtain the waveform of the synthesized speech, and then the target synthesized speech is generated through spectrum conversion. Specifically, in the intelligent sales service scenario of the bank, the target synthesized speech shows enthusiastic emotions; in the intelligent return service scenario, the target synthesized speech shows gentle emotions; in the intelligent collection service scenario, the target synthesized speech shows severe emotions.

[0030] This embodiment obtains reference speech data and text to be synthesized; encodes the reference speech data through a preset prosody encoder to obtain a prosody embedding vector; extracts features from the reference speech data through a preset speech recognition model to obtain a compensation feature vector; encodes the compensation feature vector through a preset prosody compensation encoder to obtain a prosody compensation vector; fuses the prosody embedding vector and the prosody compensation vector to obtain a prosody feature vector; synthesizes the text to be synthesized and the prosody feature vector through a preset speech synthesis model to generate a target synthesized speech. This embodiment decouples the prosody features of the reference speech data from the speaker through a prosody encoder to obtain a prosody embedding part, extracts the intermediate layer output with prosody features in the speech recognition model as a prosody compensation part, and assists speech synthesis based on the prosody embedding part and the prosody compensation part as prosody features, thereby reducing the loss of prosody information, obtaining synthesized speech with rhythm, and improving the speech synthesis effect. The speech synthesis method of this embodiment embodies the intelligence, humanity and diversification of financial services, improves the quality and efficiency of services, and enhances user experience.

[0031] Optionally, in step S20, encoding the reference speech data by a preset prosody encoder to obtain a prosody embedding vector includes: S201, obtaining Mel-spectrogram features of the reference speech data; S202: Encode the Mel frequency spectrum feature by using a preset prosody encoder to obtain a prosody embedding vector.

[0032] Understandably, the Mel spectrum feature is a representation of the sound feature, and converting the reference speech data into the Mel spectrum feature is the process of characterizing the reference speech data. The reference speech data is a speech signal, and the speech signal is a waveform signal sampled over time. The essence of the Fourier transform is the mapping and conversion of frequency domain functions and time domain functions. To obtain the Mel spectrum feature of the reference speech data, the reference speech data must first be Fourier transformed to obtain the frequency feature of the reference speech data. The frequency feature is then converted to a logarithmic scale to form a spectrogram, and finally the frequency feature is mapped to the Mel scale to form a Mel spectrum feature.

[0033] In deep learning, the encoder and decoder are the superposition of two neural networks. For a single-layer recurrent neural network, the input and output correspond one-to-one, but the representation ability of a single-layer recurrent neural network is limited. In order to improve the representation ability of the neural network, the length of the input and output can be changed at will, so the encoder-decoder architecture appears. In the encoder-decoder architecture, a hidden layer is set, and the length of the hidden layer is set to a fixed value, while the length of the encoder input and the decoder output can be changed. The decoder decodes the hidden layer to reconstruct the encoder input.

[0034] In one embodiment, the Mel-spectrogram features of the reference speech data are obtained, and the Mel-spectrogram features are input into a preset prosody encoder for encoding processing, and the preset prosody encoder outputs an intermediate hidden layer vector of a fixed length as a prosody embedding vector.

[0035] This embodiment encodes the Mel-spectrogram features through a preset prosody encoder, and converts the Mel-spectrogram features into a prosody embedding vector of a fixed length, which is conducive to embedding the prosody feature information into the text to be synthesized, ensuring that it has stronger representation ability after decoding by the speech synthesis model, thereby improving the effect of speech synthesis.

[0036] Optionally, in step S30, that is, performing feature extraction processing on the reference speech data by using a preset speech recognition model to obtain a compensation feature vector, includes: S301, obtaining Mel frequency spectrum features of the reference speech data; S302: Input the Mel frequency spectrum feature into the preset speech recognition model, extract the intermediate layer output vector of the preset speech recognition model, and record the intermediate layer output vector as a compensation feature vector.

[0037] Understandably, the neural network of the preset speech recognition model includes several layers of neurons, and the output of the neurons in the previous layer is the input of the neurons in the next layer. In deep learning, the intermediate layer output vector refers to the intermediate layer feature (Artificial Intelligence Feature, AIF), which is a set of high-level, abstract feature representations obtained by the model after a certain number of layer propagation. The input layer of the preset speech recognition model is speech data, and the output layer is text information corresponding to the speech data. There are several intermediate layers of neurons between the input layer and the output layer of the preset speech recognition model. The preset speech recognition model filters out the speaker's voice feature information during the forward propagation process, while retaining some feature information related to rhythm. With the increase of the propagation level, the closer the intermediate layer features are to the output layer, the more thorough the filtering of the sound feature information is, and the less likely the retained rhythm features are to be affected by the sound feature information, that is, the better the compensation effect on the rhythm features is. In one embodiment, the mel-spectrogram features of the reference speech data are obtained, the mel-spectrogram features are input into a preset speech recognition model, the output vector of the penultimate layer of the preset speech recognition model, i.e., the output vector of the middle layer closest to the output layer, is extracted, and the output vector of the penultimate layer is recorded as a compensation feature vector.

[0038] This embodiment is based on the filtering effect of the preset speech recognition model on the speaker's voice feature information, inputs the Mel spectrum feature into the preset speech recognition model, and extracts the intermediate layer output vector of the preset speech recognition model as the compensation feature vector, which can reduce the loss of prosodic information and improve the effect of speech synthesis.

[0039] Optionally, in step S50, that is, fusing the prosody embedding vector and the prosody compensation vector to obtain a prosody feature vector, includes: S501: Add the prosody embedding vector and the prosody compensation vector to obtain the prosody feature vector.

[0040] It is understandable that the structures of the preset prosody compensation encoder and the preset prosody encoder may be the same or different. When the structures of the preset prosody compensation encoder and the preset prosody encoder are the same, the dimensions of the prosody embedding vector and the prosody compensation vector are the same, and the prosody embedding vector and the prosody compensation vector are directly added to achieve fusion to obtain a prosody feature vector. When the structures of the preset prosody compensation encoder and the preset prosody encoder are different, the dimensions of the prosody embedding vector and the prosody compensation vector are different, and it is necessary to splice the prosody embedding vector and the prosody compensation vector to achieve fusion to obtain a prosody feature vector. In one embodiment, in order to simplify the process, the preset prosody compensation encoder and the preset prosody encoder have the same neuron structure, and the prosody embedding vector and the prosody compensation vector are added to obtain a prosody feature vector.

[0041] This embodiment adds the prosody embedding vector and the prosody compensation vector as the prosody feature vector, and while decoupling the prosody feature from the speaker's voice feature, compensates for the lost prosody feature, thereby achieving better decoupling and preservation of prosody information.

[0042] Optionally, the preset speech synthesis model includes a text encoder, an attention layer and a text decoder; in step S60, the to-be-synthesized text and the prosodic feature vector are synthesized by the preset speech synthesis model to generate a target synthesized speech, including: S601, inputting the text to be synthesized into the text encoder for encoding to obtain a text encoding vector; S602, fusing the prosodic feature vector and the text encoding vector to obtain a target encoding vector; S603, inputting the target encoding vector into the attention layer to obtain an attention vector output by the attention layer; S604, inputting the attention vector into the text decoder for decoding to obtain a text decoding vector; S605: Generate a speech mel-spectrogram according to the text decoding vector, perform spectral conversion on the speech mel-spectrogram, and obtain a target synthesized speech corresponding to the text to be synthesized.

[0043] Understandably, in one embodiment, the preset speech recognition model adopts the end-to-end speech synthesis model Tactotron, which is a sequence-to-sequence (Sequence-To-Sequence, Seq2Seq) generation model with an attention mechanism (Attention), including a text encoder module and a text decoder module with an attention layer. The text encoder is responsible for mapping each character of the input text sequence to a discrete One-Hot encoding vector, and then encoding it to a continuous embedded form (Embedding) hidden layer state. The text decoder is responsible for decoding the target encoding vector after attention allocation, and uses the Mel scale spectrogram as the prediction output in Tacotron, predicting the output Mel spectrogram one frame at a time, that is, the speech Mel spectrum. Tacotron also includes a post-processing network, which is used to convert the speech Mel spectrum output by the text decoder into a waveform signal of synthesized speech audio. After generating the speech Mel spectrum corresponding to the text to be synthesized, the speech Mel spectrum is spectrally converted, and the speech Mel spectrum feature expression is inversely transformed into a time domain waveform sample, thereby obtaining a synthesized speech. In the specific implementation process, WaveNet can be used to implement spectral conversion to generate a synthesized speech corresponding to the text to be synthesized.

[0044] In this embodiment, unlike ordinary Tacotron, the input of the attention layer includes not only a text encoding vector but also a prosodic feature vector. After the text to be synthesized is input into the text encoder for encoding to obtain a text encoding vector and before input into the attention layer, the prosodic feature vector and the text encoding vector are fused to obtain a target encoding vector.

[0045] In the speech synthesis process, this embodiment uses the prosody embedding part output by the prosody encoder and the prosody compensation part output by the prosody compensation encoder as prosody feature vectors, and decodes the text encoding vector output by the auxiliary text encoder to generate synthesized speech, thereby reducing the loss of prosody information, obtaining synthesized speech with richer prosody representation, and improving the speech synthesis effect.

[0046] Optionally, in step S601, that is, inputting the text to be synthesized into the text encoder for encoding to obtain a text encoding vector, includes: S6011, performing phoneme embedding on the text to be synthesized to obtain a text phoneme sequence; S6012. Input the text phoneme sequence into the text encoder for encoding to obtain the text encoding vector.

[0047] Understandably, the modal differences between speech and text are huge: speech signals have more and richer information than text information, such as rhythm, timbre, emotion, etc. In addition, speech is a continuous signal composed of a series of phonemes, while text is discrete data composed of words, numbers and characters. Therefore, before the text to be synthesized is input into the text encoder of the acoustic model part, it is also necessary to perform text processing of phoneme embedding on the text to be synthesized. Phoneme is the smallest speech unit divided according to the natural properties of speech, and is divided according to the pronunciation action in the syllable. One pronunciation action constitutes a phoneme. For example, the three characters "Mandarin" are composed of 3 syllables and 9 phonemes. The text to be synthesized can be a text text or a phonetic text. When the text to be synthesized is a text text, it is necessary to first convert the text text into the corresponding phonetic text, and then embed the phonetic text into a phoneme to obtain a text phoneme sequence. After the text phoneme sequence is input into the text encoder, the text encoder converts the text phoneme sequences of different lengths into a fixed-length hidden layer state feature representation to obtain a text encoding vector.

[0048] This embodiment eliminates the modal difference between speech and text through phoneme-embedded text processing, and implements hidden layer state feature mapping of text phoneme sequences through a text encoder, which is conducive to the fusion of text encoding vectors and prosodic feature vectors.

[0049] Optionally, in step S60, that is, before synthesizing the text to be synthesized and the prosodic feature vector by using a preset speech synthesis model, the following steps are included: S606, obtaining sample data to be trained, wherein the sample data to be trained includes a text to be trained and an audio to be trained corresponding to the text to be trained; S607, constructing a speech synthesis model to be trained, and training the speech synthesis model to be trained based on the text to be trained to obtain a synthesized mel spectrum; S608, obtaining the real Mel spectrum of the audio to be trained; S609, calculating a reconstruction loss based on the synthesized mel spectrum and the real mel spectrum to obtain a training loss value; S610. Update the model parameters of the speech synthesis model to be trained according to the training loss value to obtain the preset speech synthesis model.

[0050] Understandably, the sample data to be trained is a pair of sample data collected in advance, including a one-to-one corresponding text to be trained and audio to be trained. The audio to be trained is used as reference speech data to extract rhythmic feature information on the one hand, and as real speech data of the text to be trained on the other hand, to verify and converge the speech synthesis model to be trained. The speech synthesis model to be trained is a constructed initial speech synthesis model with several initial model parameters. After updating and adjusting the initial model parameters of the speech synthesis model to be trained, the preset speech synthesis model can be obtained. The loss function is used as a learning criterion in deep learning to evaluate the quality of the model. By minimizing the loss function to solve and adjust the model parameters, the purpose of optimizing the model can be achieved. The reconstruction loss is the difference between the reconstructed output and the input, that is, the difference between the synthesized Mel spectrum and the real Mel spectrum of the audio to be trained. The mean square error (MSE) can be used as the reconstruction loss value.

[0051] In one embodiment, a set of corresponding texts to be trained and audios to be trained are obtained, the texts to be trained are input into the speech synthesis model to be trained, and the rhythmic features extracted based on the audio to be trained are used as auxiliary inputs to train the speech synthesis model to obtain a synthesized Mel spectrum. The real Mel spectrum of the audio to be trained is obtained, and the current synthesized Mel spectrum is subtracted from the real Mel spectrum and the squares are summed to obtain a training loss value. The model parameters of the speech synthesis model to be trained are updated according to the training loss value, and then the model parameters of the updated speech synthesis model to be trained are iterated multiple times based on the same steps. When the reconstruction loss is close to zero or reaches a minimum, a trained preset speech synthesis model is obtained.

[0052] In this embodiment, the speech synthesis model to be trained is trained by using the sample data to be trained, and the model parameters of the speech synthesis model to be trained are adjusted based on the reconstruction loss, so that the synthesized Mel spectrum is closer to the real Mel spectrum, thereby ensuring the accuracy of the trained preset speech synthesis model and making the synthesized speech have better authenticity and naturalness.

[0053] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0054] In one embodiment, a speech synthesis device is provided, which corresponds one-to-one to the speech synthesis method in the above embodiment. Figure 4 As shown, the speech synthesis device includes a data acquisition module 10, a prosody embedding vector encoding module 20, a compensation feature vector extraction module 30, a prosody compensation vector encoding module 40, a vector fusion module 50 and a speech synthesis module 60. The functional modules are described in detail as follows: A data acquisition module 10 is used to acquire reference speech data and text to be synthesized; A prosody embedding vector encoding module 20, configured to encode the reference speech data through a preset prosody encoder to obtain a prosody embedding vector; A compensation feature vector extraction module 30 is used to perform feature extraction processing on the reference speech data through a preset speech recognition model to obtain a compensation feature vector; A rhythm compensation vector encoding module 40, configured to encode the compensation feature vector through a preset rhythm compensation encoder to obtain a rhythm compensation vector; A vector fusion module 50, configured to fuse the prosody embedding vector and the prosody compensation vector to obtain a prosody feature vector; The speech synthesis module 60 is used to synthesize the text to be synthesized and the prosodic feature vector by using a preset speech synthesis model to generate a target synthesized speech.

[0055] Optionally, the prosody embedding vector encoding module 20 includes: A first Mel frequency spectrum feature acquisition unit, used to acquire the Mel frequency spectrum feature of the reference speech data; The prosody embedding vector encoding unit is used to encode the Mel frequency spectrum feature through a preset prosody encoder to obtain a prosody embedding vector.

[0056] Optionally, the compensation feature vector extraction module 30 includes: A second Mel frequency spectrum feature acquisition unit, used to acquire the Mel frequency spectrum features of the reference speech data; The intermediate layer output vector extraction unit is used to input the Mel spectrum feature into the preset speech recognition model, extract the intermediate layer output vector of the preset speech recognition model, and record the intermediate layer output vector as a compensation feature vector.

[0057] Optionally, the vector fusion module 50 includes: A vector adding unit is used to add the prosody embedding vector and the prosody compensation vector to obtain the prosody feature vector.

[0058] Optionally, the preset speech synthesis model includes a text encoder, an attention layer and a text decoder; the speech synthesis module 60 includes: A text encoding unit, used for inputting the text to be synthesized into the text encoder for encoding to obtain a text encoding vector; A target coding vector fusion unit, used for fusing the prosodic feature vector and the text coding vector to obtain a target coding vector; An attention vector generating unit, configured to input the target encoding vector into the attention layer to obtain an attention vector output by the attention layer; A text decoding unit, used for inputting the attention vector into the text decoder for decoding to obtain a text decoding vector; The speech synthesis unit is used to generate a speech mel spectrum according to the text decoding vector, perform spectral conversion on the speech mel spectrum, and obtain a target synthesized speech corresponding to the text to be synthesized.

[0059] Optionally, the speech synthesis module 60 further includes: A phoneme embedding unit, used for performing phoneme embedding on the text to be synthesized to obtain a text phoneme sequence; The text encoding vector generating unit is used to input the text phoneme sequence into the text encoder for encoding to obtain the text encoding vector.

[0060] Optionally, the speech synthesis module 60 further includes: A to-be-trained sample data acquisition unit, used to acquire to-be-trained sample data, wherein the to-be-trained sample data includes a to-be-trained text and an to-be-trained audio corresponding to the to-be-trained text; A synthesized mel-spectrogram acquisition unit is used to construct a speech synthesis model to be trained, train the speech synthesis model to be trained based on the text to be trained, and obtain a synthesized mel-spectrogram; A real mel-spectrogram acquisition unit, used to acquire the real mel-spectrogram of the audio to be trained; A reconstruction loss calculation unit, used to calculate the reconstruction loss according to the synthesized Mel spectrum and the real Mel spectrum to obtain a training loss value; A model parameter updating unit is used to update the model parameters of the speech synthesis model to be trained according to the training loss value to obtain the preset speech synthesis model.

[0061] For the specific definition of the speech synthesis device, please refer to the definition of the speech synthesis method above, which will not be repeated here. Each module in the above-mentioned speech synthesis device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0062] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a readable storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer-readable instructions. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer-readable instructions are executed by the processor, a speech synthesis method is implemented. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0063] In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer-readable instructions: Obtain reference speech data and text to be synthesized; Encoding the reference speech data by using a preset prosody encoder to obtain a prosody embedding vector; Performing feature extraction processing on the reference speech data through a preset speech recognition model to obtain a compensation feature vector; The compensation feature vector is encoded by a preset prosody compensation encoder to obtain a prosody compensation vector; fusing the prosody embedding vector and the prosody compensation vector to obtain a prosody feature vector; The text to be synthesized and the prosodic feature vector are synthesized by using a preset speech synthesis model to generate a target synthesized speech.

[0064] In one embodiment, one or more computer-readable storage media storing computer-readable instructions are provided. The readable storage media provided in this embodiment include non-volatile readable storage media and volatile readable storage media. The readable storage media store computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the following steps are implemented: Obtain reference speech data and text to be synthesized; Encoding the reference speech data by using a preset prosody encoder to obtain a prosody embedding vector; Performing feature extraction processing on the reference speech data through a preset speech recognition model to obtain a compensation feature vector; The compensation feature vector is encoded by a preset prosody compensation encoder to obtain a prosody compensation vector; fusing the prosody embedding vector and the prosody compensation vector to obtain a prosody feature vector; The text to be synthesized and the prosodic feature vector are synthesized by using a preset speech synthesis model to generate a target synthesized speech.

[0065] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through computer-readable instructions, and the computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When the computer-readable instructions are executed, they may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0066] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0067] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A speech synthesis method, characterized in that: include: Obtain reference speech data and text to be synthesized; Encoding the reference speech data by using a preset prosody encoder to obtain a prosody embedding vector; Performing feature extraction processing on the reference speech data through a preset speech recognition model to obtain a compensation feature vector; The compensation feature vector is encoded by a preset prosody compensation encoder to obtain a prosody compensation vector; fusing the prosody embedding vector and the prosody compensation vector to obtain a prosody feature vector; The text to be synthesized and the prosodic feature vector are synthesized by using a preset speech synthesis model to generate a target synthesized speech.

2. The speech synthesis method according to claim 1, characterized in that: The encoding process of the reference speech data by a preset prosody encoder to obtain a prosody embedding vector includes: Acquire the Mel frequency spectrum features of the reference speech data; The mel spectrum feature is encoded by a preset prosody encoder to obtain a prosody embedding vector.

3. The speech synthesis method according to claim 1, characterized in that: The step of performing feature extraction processing on the reference speech data by using a preset speech recognition model to obtain a compensation feature vector includes: Acquire the Mel frequency spectrum features of the reference speech data; The Mel frequency spectrum feature is input into the preset speech recognition model, the intermediate layer output vector of the preset speech recognition model is extracted, and the intermediate layer output vector is recorded as a compensation feature vector.

4. The speech synthesis method according to claim 1, wherein: The step of fusing the prosody embedding vector and the prosody compensation vector to obtain a prosody feature vector includes: The prosody embedding vector and the prosody compensation vector are added to obtain the prosody feature vector.

5. The speech synthesis method according to claim 1, wherein: The preset speech synthesis model includes a text encoder, an attention layer and a text decoder; The method of synthesizing the text to be synthesized and the prosodic feature vector by using a preset speech synthesis model to generate a target synthesized speech includes: Inputting the text to be synthesized into the text encoder for encoding to obtain a text encoding vector; Merging the prosodic feature vector and the text encoding vector to obtain a target encoding vector; Inputting the target encoding vector into the attention layer to obtain an attention vector output by the attention layer; Input the attention vector into the text decoder for decoding to obtain a text decoding vector; A speech mel spectrum is generated according to the text decoding vector, and the speech mel spectrum is spectrally converted to obtain a target synthesized speech corresponding to the text to be synthesized.

6. The speech synthesis method according to claim 5, characterized in that: The step of inputting the text to be synthesized into the text encoder for encoding to obtain a text encoding vector comprises: Performing phoneme embedding on the text to be synthesized to obtain a text phoneme sequence; The text phoneme sequence is input into the text encoder for encoding to obtain the text encoding vector.

7. The speech synthesis method according to claim 1, wherein: Before synthesizing the text to be synthesized and the prosodic feature vector by using a preset speech synthesis model, the method includes: Acquire sample data to be trained, wherein the sample data to be trained includes text to be trained and audio to be trained corresponding to the text to be trained; Constructing a speech synthesis model to be trained, and training the speech synthesis model to be trained based on the text to be trained to obtain a synthesized mel spectrum; Obtaining a true Mel-spectrogram of the audio to be trained; Calculate the reconstruction loss based on the synthesized Mel spectrum and the real Mel spectrum to obtain a training loss value; The model parameters of the speech synthesis model to be trained are updated according to the training loss value to obtain the preset speech synthesis model.

8. A speech synthesis device, characterized in that: include: A data acquisition module, used to acquire reference speech data and text to be synthesized; A prosody embedding vector encoding module, used for encoding the reference speech data through a preset prosody encoder to obtain a prosody embedding vector; A compensation feature vector extraction module, used to perform feature extraction processing on the reference speech data through a preset speech recognition model to obtain a compensation feature vector; A rhythm compensation vector encoding module, used for encoding the compensation feature vector through a preset rhythm compensation encoder to obtain a rhythm compensation vector; A vector fusion module, used for fusing the rhythm embedding vector and the rhythm compensation vector to obtain a rhythm feature vector; The speech synthesis module is used to synthesize the text to be synthesized and the prosodic feature vector through a preset speech synthesis model to generate a target synthesized speech.

9. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that: When the processor executes the computer-readable instructions, the speech synthesis method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing computer-readable instructions, characterized in that: When the computer-readable instructions are executed by one or more processors, the one or more processors perform the speech synthesis method according to any one of claims 1 to 7.

Images