Emotional speech synthesis method, device, equipment and storage medium

Abstract

The present invention relates to the field of artificial intelligence technology, and discloses a synthesis method, device, equipment and storage medium for emotional speech, which are used to solve the problem that the synthesized speech is bland and lacks emotion, and increase the diversity of the synthesized speech. The synthesis method of emotional speech includes: obtaining speech data to be recognized and corresponding text data; inputting the speech data to be recognized into a pre-trained emotion recognition network to generate Mel spectrum features and position codes, and combining the Mel spectrum features and the position codes for processing in the emotion recognition network to generate emotional embedding features; inputting the emotional embedding features and the text data into a pre-trained speech synthesis network to generate target Mel spectrum data; using a neural vocoder to perform speech conversion on the target Mel spectrum data to generate target emotional speech. In addition, the present invention also relates to blockchain technology, and the speech data to be recognized and the text data can be stored in the blockchain.

Description

Technical Field

[0001] The present invention relates to the technical field of speech synthesis, and in particular to a synthesis method, device, equipment and storage medium for emotional speech. Background Art

[0002] With the development of science and technology, artificial intelligence services such as smart customer service centers, chatbots, and smart speakers have entered our daily lives and are playing an increasingly important role. This type of artificial intelligence server usually involves speech synthesis technology, so speech synthesis technology has also been more widely used.

[0003] In the prior art, speech synthesis methods are mainly based on hidden Markov speech synthesis or neural network-based speech synthesis. Although these two speech synthesis methods can obtain good synthesized speech, the generated synthesized speech is bland and lacks emotion, and thus speech full of emotion cannot be obtained. Summary of the invention

[0004] The present invention provides a method, device, equipment and storage medium for synthesizing emotional speech, which are used to solve the problem that the synthesized speech is bland and lacks emotion and to increase the diversity of the synthesized speech.

[0005] The first aspect of the present invention provides a method for synthesizing emotional speech, comprising: obtaining speech data to be recognized and corresponding text data; inputting the speech data to be recognized into a pre-trained emotion recognition network to generate mel-spectrogram features and position codes, and processing the mel-spectrogram features and the position codes in the emotion recognition network to generate emotion embedding features; inputting the emotion embedding features and the text data into a pre-trained speech synthesis network to generate target mel-spectrogram data; and performing speech conversion on the target mel-spectrogram data using a neural vocoder to generate target emotional speech.

[0006] Optionally, in a first implementation method of the first aspect of the present invention, the speech data to be recognized is input into a pre-trained emotion recognition network to generate mel-spectrogram features and position codes, and the mel-spectrogram features and the position codes are combined for processing in the emotion recognition network to generate emotion embedding features, which includes: inputting the speech data to be recognized into a pre-trained emotion recognition network to generate mel-spectrogram features; generating position codes based on the mel-spectrogram features and a preset position conversion formula; inputting the mel-spectrogram features and the position codes into an encoder of the emotion recognition network for encoding to generate emotion embedding features.

[0007] Optionally, in a second implementation method of the first aspect of the present invention, the inputting the speech data to be recognized into a pre-trained emotion recognition network to generate Mel-spectrogram features includes: performing windowing processing on the speech data to be recognized to generate windowed speech data; performing short-time Fourier transform on the windowed speech data to generate Fourier transformed speech data; and using a Mel filter group to process the Fourier transformed speech data to generate Mel-spectrogram features.

[0008] Optionally, in a third implementation manner of the first aspect of the present invention, generating a position code according to the mel spectrum feature and a preset position conversion formula includes: reading the length of the mel spectrum feature and reading the position of the mel spectrum feature; generating a position input value based on the length of the mel spectrum feature and the position of the mel spectrum feature; and inputting the position input vector into a preset position conversion formula to generate a position code.

[0009] Optionally, in a fourth implementation method of the first aspect of the present invention, the step of inputting the mel spectrum feature and the position code into the encoder of the emotion recognition network for encoding to generate emotion embedding features includes: inputting the mel spectrum feature and the position code into the multi-head self-attention layer of the emotion recognition network, combining with residual connection, to generate an initial emotion feature vector; inputting the initial emotion feature vector into the forward propagation layer of the emotion recognition network for convolution to generate emotion embedding features.

[0010] Optionally, in a fifth implementation manner of the first aspect of the present invention, the inputting the emotion embedding features and the text data into a pre-trained speech synthesis network to generate target mel-spectrogram data includes: converting the text data into text embedding features in the pre-trained speech synthesis network; and splicing the text embedding features and the emotion embedding features in time sequence to generate target mel-spectrogram data.

[0011] Optionally, in a sixth implementation manner of the first aspect of the present invention, before obtaining the speech data to be recognized and the corresponding text data, the method for synthesizing emotional speech includes: obtaining emotional speech training data, emotional label data and text training data; using the emotional speech training data and the emotional label data, combined with a layer regularization mechanism, to perform model training to generate a pre-trained emotion recognition network, and using the emotional speech training data and the text training data to perform model training to generate a pre-trained speech synthesis network.

[0012] The second aspect of the present invention provides an emotional speech synthesis device, comprising: an acquisition module, used to acquire speech data to be recognized and corresponding text data; an embedded feature generation module, used to input the speech data to be recognized into a pre-trained emotion recognition network, generate Mel-spectrogram features and position codes, and process the Mel-spectrogram features and the position codes in the emotion recognition network to generate emotional embedded features; a Mel-spectrogram data generation module, used to input the emotional embedded features and the text data into a pre-trained speech synthesis network to generate target Mel-spectrogram data; a speech conversion module, used to perform speech conversion on the target Mel-spectrogram data using a neural vocoder to generate target emotional speech.

[0013] Optionally, in a first implementation method of the second aspect of the present invention, the embedded feature generation module includes: a mel-spectrogram feature generation unit, used to input the speech data to be recognized into a pre-trained emotion recognition network to generate a mel-spectrogram feature; a position code generation unit, used to generate a position code according to the mel-spectrogram feature and a preset position conversion formula; an encoding unit, used to input the mel-spectrogram feature and the position code into an encoder of the emotion recognition network for encoding to generate an emotion embedded feature.

[0014] Optionally, in a second implementation manner of the second aspect of the present invention, the Mel spectrum feature generation unit can also be specifically used to: perform windowing processing on the speech data to be recognized to generate windowed speech data; perform short-time Fourier transform on the windowed speech data to generate Fourier transformed speech data; and use a Mel filter group to process the Fourier transformed speech data to generate Mel spectrum features.

[0015] Optionally, in a third implementation manner of the second aspect of the present invention, the position code generation unit can also be specifically used to: read the length of the mel spectrum feature, and read the position of the mel spectrum feature; generate a position input value based on the length of the mel spectrum feature and the position of the mel spectrum feature; input the position input vector into a preset position conversion formula to generate a position code.

[0016] Optionally, in a fourth implementation method of the second aspect of the present invention, the encoding unit can also be specifically used to: input the Mel spectrum features and the position code into the multi-head self-attention layer of the emotion recognition network, combined with residual connection, to generate an initial emotion feature vector; input the initial emotion feature vector into the forward propagation layer of the emotion recognition network for convolution to generate emotion embedding features.

[0017] Optionally, in a fifth implementation of the second aspect of the present invention, the mel-spectrogram data generation module can also be specifically used to: convert the text data into text embedding features in a pre-trained speech synthesis network; and concatenate the text embedding features and the emotion embedding features in time sequence to generate target mel-spectrogram data.

[0018] Optionally, in a sixth implementation of the second aspect of the present invention, the emotional speech synthesis device also includes: a training data acquisition module, used to acquire emotional speech training data, emotional label data and text training data; a training module, used to use the emotional speech training data and the emotional label data, combined with a layer regularization mechanism to perform model training to generate a pre-trained emotion recognition network, and use the emotional speech training data and the text training data to perform model training to generate a pre-trained speech synthesis network.

[0019] The third aspect of the present invention provides an emotional speech synthesis device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the emotional speech synthesis device executes the above-mentioned emotional speech synthesis method.

[0020] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned emotional speech synthesis method.

[0021] In the technical solution provided by the present invention, the speech data to be recognized and the corresponding text data are obtained; the speech data to be recognized is input into a pre-trained emotion recognition network to generate mel spectrum features and position codes, and the mel spectrum features and the position codes are combined for processing in the emotion recognition network to generate emotion embedding features; the emotion embedding features and the text data are input into a pre-trained speech synthesis network to generate target mel spectrum data; a neural vocoder is used to perform speech conversion on the target mel spectrum data to generate target emotional speech. In the embodiment of the present invention, the emotion embedding features are generated by combining the mel spectrum features and the position codes with the pre-trained emotion recognition network, and then the emotion embedding features and the text data are spliced ​​to generate the target emotional speech, which solves the problem of blandness and lack of emotion in the synthesized speech and increases the diversity of the synthesized speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of an embodiment of a method for synthesizing emotional speech in an embodiment of the present invention;

[0023] Figure 2 Schematic diagram of another embodiment of the method for synthesizing emotional speech in an embodiment of the present invention;

[0024] Figure 3 Schematic diagram of an embodiment of a device for synthesizing emotional speech in an embodiment of the present invention;

[0025] Figure 4 It is a schematic diagram of another embodiment of the device for synthesizing emotional speech in an embodiment of the present invention;

[0026] Figure 5 The figure is a schematic diagram of an embodiment of a device for synthesizing emotional speech in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The embodiments of the present invention provide a method, apparatus, device and storage medium for synthesizing emotional speech, which are used to solve the problem that the synthesized speech is bland and lacks emotion, and to increase the diversity of the synthesized speech.

[0028] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of the method for synthesizing emotional speech in the embodiment of the present invention includes:

[0030] 101. Acquire speech data to be recognized and corresponding text data;

[0031] The server obtains the voice data to be recognized and the text data corresponding to the text data to be recognized. It should be emphasized that in order to further ensure the privacy and security of the voice data to be recognized and the text data, the voice data to be recognized and the text data can also be stored in a node of a blockchain.

[0032] The speech data to be recognized is speech data to be recognized with emotion, and may be speech data to be recognized with a happy emotion, speech data to be recognized with a surprised emotion, and / or speech data to be recognized with an angry emotion. When the server obtains the speech data to be recognized with emotion, it also obtains the corresponding text data. For example, if the speech data to be recognized with emotion is “Really? Congratulations!”, when the server obtains the speech data to be recognized of “Really? Congratulations!”, it also obtains the text data of “Really? Congratulations!”.

[0033] It is understandable that the execution subject of the present invention may be a synthesis device for emotional speech, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0034] 102. Input the speech data to be recognized into a pre-trained emotion recognition network to generate mel spectrum features and position codes, and process the mel spectrum features and position codes in the emotion recognition network to generate emotion embedding features;

[0035] The server inputs the speech data to be recognized into the pre-trained emotion recognition network, first generates mel spectrum features and position codes, and then processes the mel spectrum features and the position codes in the emotion recognition network to generate emotion embedding features.

[0036] The server inputs the speech data to be recognized "Really? Congratulations!" into the pre-trained emotion recognition network for calculation, and generates Mel-spectrogram features [B1, T1, D1] and position code P, where the position code P is generated based on the Mel-spectrogram features. The position code P is actually a hidden layer output. The server then combines the Mel-spectrogram features [B1, T1, D1] and the position code P for calculation to generate the emotion embedding features [B2, T2, D2].

[0037] 103. Input the emotion embedding features and text data into the pre-trained speech synthesis network to generate target mel spectrum data;

[0038] The server inputs the emotion embedding features and text data into the pre-trained speech synthesis network for calculation to generate the target mel-spectrogram data.

[0039] The server inputs the emotion embedding features of [B2, T2, D2] and the text data of "Really? Congratulations!" into a pre-trained speech synthesis network for calculation. In this embodiment, the speech synthesis network includes an encoder, in which the text data of "Really? Congratulations!" is feature-weighted to generate an extraction result, and the extraction result is concatenated with the emotion embedding features [B2, T2, D2] to generate target mel-spectrogram data [B2, T2, D2+D].

[0040] 104. Use a neural vocoder to perform speech conversion on the target mel spectrum data to generate the target emotional speech.

[0041] The server uses a neural vocoder to convert the target mel-spectrogram data into target emotional speech.

[0042] It should be noted that, in this embodiment, the neural vocoder is Wave Glow, and the target Mel-spectrogram data is the input of the neural vocoder. The frame length of the input is 1024, and the frame shift is 256. The target Mel-spectrogram data is first input into the affine coupling layer of the neural vocoder for scaling and conversion to generate emotional speech features, and then the emotional speech features are reversibly convolved to generate the target emotional speech "Really? (Surprised emotion) Congratulations! (Happy emotion)".

[0043] In an embodiment of the present invention, a pre-trained emotion recognition network is used to generate emotion embedding features in combination with mel-spectrogram features and position coding, and then the emotion embedding features and text data are concatenated to generate target emotional speech, thereby solving the problem of blandness and lack of emotion in synthesized speech and increasing the diversity of synthesized speech.

[0044] See also Figure 2 Another embodiment of the method for synthesizing emotional speech in the embodiment of the present invention includes:

[0045] 201. Acquire emotional speech training data, emotional label data, and text training data;

[0046] The server obtains emotional speech training data, emotional label data and text training data from a big data platform or database.

[0047] It should be noted that the emotional speech training data can be divided into emotional speech training data including noise and emotional speech training data not including noise.

[0048] The emotional speech training data can be speech training data with emotions such as "That's too much", "Really?" or "That's great", and emotional label data and text training data are obtained, among which the emotional speech training data of "That's too much" corresponds to the emotional label data of "anger", and the text training data of "That's too much"; the emotional speech training data of "Really?" corresponds to the emotional label data of "surprise" and the text training data of "Really?"; the emotional speech training data of "That's great" corresponds to the emotional label data of "happy" and the text training data of "That's great".

[0049] 202. Using the emotional speech training data and the emotional label data, combined with the layer regularization mechanism, to perform model training, to generate a pre-trained emotion recognition network, and using the emotional speech training data and the text training data to perform model training, to generate a pre-trained speech synthesis network;

[0050] The server performs training based on the emotional speech training data and the emotional label data in combination with the regularization mechanism to generate a pre-trained emotion recognition network, and then performs model training based on the emotional speech training data and the text training data to generate a pre-trained speech synthesis network.

[0051] It should be noted that the pre-trained emotion recognition network is used to extract emotion features, so the emotion recognition network is trained using emotion speech training data and emotion label data; the pre-trained speech synthesis network is used to synthesize emotion speech, so the emotion speech training data and text training data are used to train the speech synthesis network. Although the above training processes all include emotion speech training data, when training the emotion recognition network, the emotion speech training data can be either emotion speech training data including noise or emotion speech training data excluding noise, but when training the speech synthesis network, high-quality training data, i.e. emotion speech training data excluding noise, needs to be called for training.

[0052] In the process of training the emotion recognition network, the server combines the layer regularization mechanism to train the emotion recognition network. The main thing is to add a layer regularization mechanism after each sublayer. The layer regularization mechanism calculates the mean and variance of the output of layer i in the channel dimension, and then subtracts the mean from the output of layer i and divides it by the variance, so that the mean of the output of layer i is 0 and the variance is 1. The layer regularization mechanism can make the distribution of training data consistent and make the training process stable.

[0053] 203. Acquire the voice data to be recognized and the corresponding text data;

[0054] The server obtains the voice data to be recognized and the text data corresponding to the text data to be recognized. It should be emphasized that in order to further ensure the privacy and security of the voice data to be recognized and the text data, the voice data to be recognized and the text data can also be stored in a node of a blockchain.

[0055] The speech data to be recognized is speech data to be recognized with emotion, and may be speech data to be recognized with a happy emotion, speech data to be recognized with a surprised emotion, and / or speech data to be recognized with an angry emotion. When the server obtains the speech data to be recognized with emotion, it also obtains the corresponding text data. For example, if the speech data to be recognized with emotion is “Really? Congratulations!”, when the server obtains the speech data to be recognized of “Really? Congratulations!”, it also obtains the text data of “Really? Congratulations!”.

[0056] 204. Input the speech data to be recognized into a pre-trained emotion recognition network to generate mel spectrum features and position codes, and process the mel spectrum features and position codes in the emotion recognition network to generate emotion embedding features;

[0057] The server inputs the speech data to be recognized into the pre-trained emotion recognition network, first generates mel spectrum features and position codes, and then processes the mel spectrum features and the position codes in the emotion recognition network to generate emotion embedding features.

[0058] The server inputs the speech data to be recognized "Really? Congratulations!" into the pre-trained emotion recognition network for calculation, and generates Mel-spectrogram features [B1, T1, D1] and position code P, where the position code P is generated based on the Mel-spectrogram features. The position code P is actually a hidden layer output. The server then combines the Mel-spectrogram features [B1, T1, D1] and the position code P for calculation to generate the emotion embedding features [B2, T2, D2].

[0059] Specifically, the server inputs the speech data to be recognized into a pre-trained emotion recognition network to generate mel-spectrogram features; the server generates position codes based on the mel-spectrogram features and a preset position conversion formula; the server inputs the mel-spectrogram features and the position codes into the encoder of the emotion recognition network for encoding to generate emotion embedding features.

[0060] The server inputs the speech data to be recognized of "Really? Congratulations!" into the pre-trained emotion recognition network. First, "Really? Congratulations!" is input into the trained emotion recognition network for feature extraction to generate the Mel spectrum feature [B1, T1, D1]; then the server calculates the position encoding of the Mel spectrum feature of [B1, T1, D1] according to the preset position conversion formula to generate the position encoding P; finally, the server inputs the Mel spectrum feature [B1, T1, D1] and the position encoding P into the encoder of the emotion recognition network for encoding to generate the emotion embedding feature, wherein the encoder of the emotion recognition network includes five identical modules and a long short-term memory artificial neural network layer, wherein each module includes two sub-layers, namely a multi-head self-attention layer and a forward propagation layer, and [B1, T1, D1] and P are encoded in the encoder to generate the emotion embedding feature [B2, T2, D2].

[0061] The server inputs the speech data to be recognized into the pre-trained emotion recognition network to generate mel-spectrogram features including:

[0062] First, the server performs windowing processing on the speech data to be recognized to generate windowed speech data; then, the server performs short-time Fourier transform on the windowed speech data to generate Fourier transformed speech data; finally, the server uses a Mel filter bank to process the Fourier transformed speech data to generate Mel spectrum features.

[0063] The server uses a window function to perform windowing processing on the speech data to be recognized "Really? Congratulations!" to generate windowed speech data; then the server performs Fourier transform on the windowed speech data to determine the frequency and phase of the windowed speech data, thereby generating Fourier transformed speech data; finally, the server uses a Mel filter group to process the Fourier transformed speech data into Mel spectrum features.

[0064] The server generates position codes based on the Mel spectrum features and the preset position conversion formula, including:

[0065] The server reads the length of the Mel-spectrogram feature and the position of the Mel-spectrogram feature; the service generates a position input value based on the length of the Mel-spectrogram feature and the position of the Mel-spectrogram feature; the server inputs the position input vector into a preset position conversion formula to generate a position code.

[0066] In this embodiment, the preset position conversion formula is:

[0067]

[0068]

[0069] Among them, pos is the position of the Mel spectrum feature, 2i represents the even dimension, 2i+1 represents the odd dimension, and d mode Indicates the preset dimension vector corresponding to the position of the mel spectrum feature, for example, 256.

[0070] For example, the server reads the length of the Mel spectrum feature as 5, and the position of the Mel spectrum feature as 0. Then, based on the length of the Mel spectrum feature of "5" and the position of the Mel spectrum feature of "0", the server determines the position input value as [0,1,2,3,4], and then calculates the position input value [0,1,2,3,4] based on the above formula to generate the position code P. It should be noted that in this embodiment, P is only a reference term and is not specific position code data.

[0071] The server inputs the mel spectrum features and position encoding into the encoder of the emotion recognition network for encoding, and generates the emotion embedding features including:

[0072] The server inputs the position encoding of the Mel-spectrogram feature and the position encoding into the multi-head self-attention layer of the emotion recognition network, and combines the residual connection to generate the initial emotion feature vector; the server inputs the initial emotion feature vector into the forward propagation layer of the emotion recognition network for convolution to generate the emotion embedding feature.

[0073] The server first inputs the Mel spectrum features [B1, T1, D1] into the multi-head self-attention layer for calculation, and combines the residual connection to generate the initial emotion feature vector. The formula for the multi-head self-attention layer design is as follows:

[0074]

[0075] head i =Attention(QW i Q ,KW i K ,KW i V )

[0076] c t =MultiHead(Q,K,V)=Concat(head1,…,head h )W O ;

[0077] Among them, Q, K, V are input, namely Mel spectrum features, d k is a preset dimension vector, for example, 256, head i is the i-th head. Each calculation in the multi-head self-attention layer is one head. If 4 Attention(QW i Q ,KW i K ,KV V ) calculation, then 4 heads are generated, W i Q ,KW i K ,KW i V is the weight, which is generated during the training process. Concat refers to concatenating multiple heads along the last dimension. For example, the dimension vectors of these four heads are [B t ,T t ,D t ], the server concatenates them together to generate the initial sentiment feature vector [B t ,T t ,4D t ],W OTo learn the parameters in advance. After generating the initial emotion feature vector, the initial emotion feature vector and the Mel spectrum feature are input into the corresponding forward propagation layer for convolution to generate the first module emotion feature vector. Since the encoder includes five identical modules, five calculations are performed according to the above calculation method. The output result of the last module is input into a long short-term memory artificial neural network layer to generate the emotion embedding feature [B2, T2, D2].

[0078] It should be noted that the residual connection adds the input of each multi-head self-attention layer to the output as the input of the next forward propagation layer. In the first multi-head self-attention layer, the Mel spectrum feature is added to the initial emotion feature vector to generate the input of the forward propagation layer. This improves the relevance of the generated emotion embedding features.

[0079] 205. Input the emotion embedding features and text data into a pre-trained speech synthesis network to generate target mel spectrum data;

[0080] The server inputs the emotion embedding features and text data into the pre-trained speech synthesis network for calculation to generate the target mel-spectrogram data.

[0081] The server inputs the emotion embedding features of [B2, T2, D3] and the text data of "Really? Congratulations!" into a pre-trained speech synthesis network for calculation. In this embodiment, the speech synthesis network includes an encoder, in which the text data of "Really? Congratulations!" is feature-weighted to generate an extraction result, and the extraction result is concatenated with the emotion embedding features [B2, T2, D2] to generate target mel-spectrogram data [B2, T2, D2+D].

[0082] Specifically, the server converts text data into text embedding features in a pre-trained speech synthesis network; the server concatenates the text embedding features and the emotion embedding features in time sequence to generate target mel-spectrogram data.

[0083] In the pre-trained speech synthesis network, the server first converts the text data into text embedding features in the same form as the emotion embedding features in time order, and then the server splices the text embedding features and the emotion embedding features in time order to generate the target mel spectrum data. In this embodiment, for example, the emotion embedding feature is [B2, T2, D2] and the text embedding feature is [B2, T2, D3]. The server splices [B2, T2, D2] with [B2, T2, D] based on the time order to generate the target mel spectrum data as [B2, T2, D2+D]. In other embodiments, if the emotion embedding feature dimension is [B2, D2], the server expands the emotion embedding feature to [B2, 1, D2], and then splices the emotion embedding feature with the cost embedding feature.

[0084] 206. Use a neural vocoder to perform speech conversion on the target mel spectrum data to generate the target emotional speech.

[0085] The server uses a neural vocoder to convert the target mel-spectrogram data into target emotional speech.

[0086] It should be noted that, in this embodiment, the neural vocoder is Wave Glow, and the target Mel-spectrogram data is the input of the neural vocoder. The frame length of the input is 1024, and the frame shift is 256. The target Mel-spectrogram data is first input into the affine coupling layer of the neural vocoder for scaling and conversion to generate emotional speech features, and then the emotional speech features are reversibly convolved to generate the target emotional speech "Really? (Surprised emotion) Congratulations! (Happy emotion)".

[0087] In an embodiment of the present invention, a pre-trained emotion recognition network is used to generate emotion embedding features in combination with mel-spectrogram features and position coding, and then the emotion embedding features and text data are concatenated to generate target emotional speech, thereby solving the problem of blandness and lack of emotion in synthesized speech and increasing the diversity of synthesized speech.

[0088] The above describes the method for synthesizing emotional speech in the embodiment of the present invention. The following describes the device for synthesizing emotional speech in the embodiment of the present invention. Figure 3 , an embodiment of the synthesis device of emotional speech in the embodiment of the present invention includes:

[0089] The to-be-recognized data acquisition module 301 is used to acquire the to-be-recognized speech data and the corresponding text data;

[0090] The embedded feature generation module 302 is used to input the speech data to be recognized into a pre-trained emotion recognition network to generate Mel spectrum features and position codes, and process the Mel spectrum features and the position codes in the emotion recognition network to generate emotion embedded features;

[0091] A mel spectrum data generation module 303 is used to input the emotion embedding feature and the text data into a pre-trained speech synthesis network to generate target mel spectrum data;

[0092] The speech conversion module 304 is used to perform speech conversion on the target mel spectrum data using a neural vocoder to generate a target emotional speech.

[0093] In an embodiment of the present invention, a pre-trained emotion recognition network is used to generate emotion embedding features in combination with mel-spectrogram features and position coding, and then the emotion embedding features and text data are concatenated to generate target emotional speech, thereby solving the problem of blandness and lack of emotion in synthesized speech and increasing the diversity of synthesized speech.

[0094] See also Figure 4 Another embodiment of the emotional speech synthesis device in the embodiment of the present invention includes:

[0095] The to-be-recognized data acquisition module 301 is used to acquire the to-be-recognized speech data and the corresponding text data;

[0096] The embedded feature generation module 302 is used to input the speech data to be recognized into a pre-trained emotion recognition network to generate Mel spectrum features and position codes, and process the Mel spectrum features and the position codes in the emotion recognition network to generate emotion embedded features;

[0097] A mel spectrum data generation module 303 is used to input the emotion embedding feature and the text data into a pre-trained speech synthesis network to generate target mel spectrum data;

[0098] The speech conversion module 304 is used to perform speech conversion on the target mel spectrum data using a neural vocoder to generate a target emotional speech.

[0099] Optionally, the embedding feature generation module 302 includes:

[0100] A mel spectrum feature generating unit 3021 is used to input the speech data to be recognized into a pre-trained emotion recognition network to generate mel spectrum features;

[0101] A position code generating unit 3022, configured to generate a position code according to the mel spectrum feature and a preset position conversion formula;

[0102] The encoding unit 3023 is used to input the mel spectrum feature and the position code into the encoder of the emotion recognition network for encoding to generate an emotion embedding feature.

[0103] Optionally, the mel spectrum feature generating unit 3021 may also be specifically used for:

[0104] Performing windowing processing on the speech data to be recognized to generate windowed speech data;

[0105] Performing short-time Fourier transform on the windowed speech data to generate Fourier transformed speech data;

[0106] The speech data after the Fourier transformation is processed by using a Mel filter bank to generate Mel spectrum features.

[0107] Optionally, the position code generating unit 3022 may also be specifically used for:

[0108] Read the length of the Mel spectrum feature and read the position of the Mel spectrum feature;

[0109] Generate a position input value based on the length of the mel spectrum feature and the position of the mel spectrum feature;

[0110] The position input vector is input into a preset position conversion formula to generate a position code.

[0111] Optionally, the encoding unit 3023 may also be specifically used for:

[0112] Input the Mel spectrum features and the position encoding into the multi-head self-attention layer of the emotion recognition network, and combine with residual connection to generate an initial emotion feature vector;

[0113] The initial emotion feature vector is input into the forward propagation layer of the emotion recognition network for convolution to generate emotion embedding features.

[0114] Optionally, the mel spectrum data generating module 303 may also be specifically used for:

[0115] In a pre-trained speech synthesis network, converting the text data into text embedding features;

[0116] The text embedding features and the emotion embedding features are concatenated in time sequence to generate target mel spectrum data.

[0117] Optionally, the emotional speech synthesis device also includes:

[0118] The training data acquisition module 305 is used to acquire emotional speech training data, emotional label data and text training data;

[0119] The training module 306 is used to use the emotional speech training data and the emotional label data, combined with the layer regularization mechanism to perform model training to generate a pre-trained emotion recognition network, and use the emotional speech training data and the text training data to perform model training to generate a pre-trained speech synthesis network.

[0120] In an embodiment of the present invention, a pre-trained emotion recognition network is used to generate emotion embedding features in combination with mel-spectrogram features and position coding, and then the emotion embedding features and text data are concatenated to generate target emotional speech, thereby solving the problem of blandness and lack of emotion in synthesized speech and increasing the diversity of synthesized speech.

[0121] above Figure 3 and Figure 4 The emotional speech synthesis device in the embodiment of the present invention is described in detail from the perspective of modular functional entities, and the emotional speech synthesis device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0122] Figure 5 1 is a schematic diagram of the structure of an emotional speech synthesis device provided by an embodiment of the present invention. The emotional speech synthesis device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 533 or data 532. Among them, the memory 520 and the storage medium 530 can be temporary storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the emotional speech synthesis device 500. Furthermore, the processor 510 can be configured to communicate with the storage medium 530 to execute a series of instruction operations in the storage medium 530 on the emotional speech synthesis device 500.

[0123] The emotional speech synthesis device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 5 The structure of the emotional speech synthesis device shown does not constitute a limitation on the emotional speech synthesis device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0124] The present invention also provides a device for synthesizing emotional speech, wherein the computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the method for synthesizing emotional speech in the above-mentioned embodiments.

[0125] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the emotional speech synthesis method.

[0126] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0127] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0128] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0129] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for synthesizing emotional speech, characterized in that: The method for synthesizing the emotional speech comprises: Obtaining speech data to be recognized with emotion and corresponding text data; Inputting the speech data to be recognized into a pre-trained emotion recognition network to generate mel spectrum features and position codes, and processing the mel spectrum features and the position codes in the emotion recognition network to generate emotion embedding features, wherein the emotion recognition network is trained by emotion speech training data including noise and emotion speech training data not including noise; The emotion embedding feature and the text data are input into a pre-trained speech synthesis network to generate target mel spectrum data: the emotion recognition network is input into a pre-trained emotion recognition network for calculation to generate mel spectrum features [B1, T1, D1] and position code P, wherein the position code P is generated based on the mel spectrum features, and the position code P is a hidden layer output, and then the mel spectrum features [B1, T1, D1] and the position code P are combined for calculation to generate the emotion embedding feature representation: ; Perform feature weighting on text data, generate extraction results, and embed the extraction results with sentiment features Splicing is performed to generate target Mel spectrum data, wherein the result of extracting the text data is represented as: D, and the target Mel spectrum data is represented as: ; A neural vocoder is used to perform speech conversion on the target mel-spectrogram data to generate target emotional speech.

2. The method for synthesizing emotional speech according to claim 1, characterized in that: The step of inputting the speech data to be recognized into a pre-trained emotion recognition network to generate mel spectrum features and position codes, and combining the mel spectrum features and the position codes for processing in the emotion recognition network to generate emotion embedding features includes: Inputting the speech data to be recognized into a pre-trained emotion recognition network to generate mel-spectrogram features; Generate a position code according to the Mel spectrum feature and a preset position conversion formula; The mel spectrum feature and the position code are input into the encoder of the emotion recognition network for encoding to generate an emotion embedding feature.

3. The method for synthesizing emotional speech according to claim 2, characterized in that: The step of inputting the speech data to be recognized into a pre-trained emotion recognition network to generate mel-spectrogram features comprises: Performing windowing processing on the speech data to be recognized to generate windowed speech data; Performing short-time Fourier transform on the windowed speech data to generate Fourier transformed speech data; The speech data after the Fourier transformation is processed by using a Mel filter bank to generate Mel spectrum features.

4. The method for synthesizing emotional speech according to claim 2, characterized in that: The generating of the position code according to the Mel spectrum feature and the preset position conversion formula includes: Read the length of the Mel spectrum feature and read the position of the Mel spectrum feature; Generate a position input vector based on the length of the mel spectrum feature and the position of the mel spectrum feature; The position input vector is input into a preset position conversion formula to generate a position code.

5. The method for synthesizing emotional speech according to claim 2, characterized in that: The step of inputting the mel spectrum feature and the position code into the encoder of the emotion recognition network for encoding to generate the emotion embedding feature comprises: Input the Mel spectrum features and the position encoding into the multi-head self-attention layer of the emotion recognition network, and combine with residual connection to generate an initial emotion feature vector; The initial emotion feature vector is input into the forward propagation layer of the emotion recognition network for convolution to generate emotion embedding features.

6. The method for synthesizing emotional speech according to claim 1, characterized in that: The step of inputting the emotion embedding feature and the text data into a pre-trained speech synthesis network to generate target mel-spectrogram data comprises: In a pre-trained speech synthesis network, converting the text data into text embedding features; The text embedding features and the emotion embedding features are concatenated in time sequence to generate target mel spectrum data.

7. The method for synthesizing emotional speech according to any one of claims 1 to 6, characterized in that: Before acquiring the speech data to be recognized and the corresponding text data, the emotional speech synthesis method includes: Obtain emotional speech training data, emotional label data, and text training data; The emotional speech training data and the emotional label data are used in combination with a layer regularization mechanism to perform model training to generate a pre-trained emotion recognition network, and the emotional speech training data and the text training data are used to perform model training to generate a pre-trained speech synthesis network.

8. An emotional speech synthesis device, characterized in that: The emotional speech synthesis device comprises: An acquisition module is used to acquire speech data with emotions to be recognized and corresponding text data; An embedding feature generation module is used to input the speech data to be recognized into a pre-trained emotion recognition network to generate a mel spectrum feature and a position code, and process the mel spectrum feature and the position code in the emotion recognition network to generate an emotion embedding feature, wherein the emotion recognition network is trained by emotion speech training data including noise and emotion speech training data not including noise; The Mel spectrum data generation module is used to input the emotion embedding feature and the text data into a pre-trained speech synthesis network to generate target Mel spectrum data: input into a pre-trained emotion recognition network for calculation to generate Mel spectrum features [B1, T1, D1] and position code P, wherein the position code P is generated based on the Mel spectrum feature, and the position code P is a hidden layer output, and then combined with the Mel spectrum feature [B1, T1, D1] and the position code P for calculation to generate the emotion embedding feature representation: ; Perform feature weighting on text data, generate extraction results, and embed the extraction results with sentiment features Splicing is performed to generate target Mel spectrum data, wherein the result of extracting the text data is represented as: D, and the target Mel spectrum data is represented as: ; The speech conversion module is used to perform speech conversion on the target Mel spectrum data using a neural vocoder to generate target emotional speech.

9. An emotional speech synthesis device, characterized in that: The emotional speech synthesis device comprises: a memory and at least one processor, wherein the memory stores instructions; The at least one processor calls the instructions in the memory to enable the emotional speech synthesis device to execute the emotional speech synthesis method as described in any one of claims 1-7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the method for synthesizing emotional speech as described in any one of claims 1-7 is implemented.

Images