Speech synthesis method and device based on cross-domain information adaptive fusion, equipment and medium

Abstract

The application provides a speech synthesis method and device based on cross-domain information adaptive fusion, a database of sample texts and corresponding sample audios is established by using a phoneme sequence as text domain information and a log mel spectrum as frequency domain information, a phoneme sequence is obtained from the sample texts and a log mel spectrum is extracted from the sample audios; a speech synthesis model is constructed according to a phoneme feature extraction module, a spectrum feature compression module, a cross-domain feature alignment module and a spectrum detail filling module, the phoneme sequence and the log mel spectrum are input into the speech synthesis model for training, and a trained speech synthesis model is obtained; text data to be synthesized is obtained, a phoneme sequence is obtained from the text data to be synthesized, and a constant matrix is initialized; the phoneme sequence and the constant matrix are input into the trained speech synthesis model, and a synthesized log mel spectrum corresponding to the text data to be synthesized is obtained. In addition, the existing autoregressive speech synthesis model is limited in parallel ability and increases the training time cost due to the long short-term memory network in the training.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and speech synthesis, and particularly relates to a speech synthesis method and device for cross-domain information adaptive fusion, equipment and medium. BACKGROUND

[0002] Speech synthesis technology (Text-to-Speech, TTS) is a very important task in user interaction, aiming to synthesize clear and natural audio that is difficult to distinguish from human recordings. Recently, with the rapid development of deep learning, autoregressive speech synthesis models based on neural networks, such as Tacotron2 (J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan, R. A. Saurous, Y. Agiomyrgiannakis, and Y. Wu. Natural TTS Synthesis by Conditioning Waveneton MEL Spectrogram Predictions, Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Calgary, AB, Canada, 2018, pp. 4779-4783), use a single neural network to convert text to mel spectrum, and then use a vocoder such as WaveNet (A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. W. Senior, and K. Kavukcuoglu. WaveNet: A generative model for raw audio, CoRR, vol. abs / 1609.03499, 2016) to synthesize audio. This greatly improves the quality of synthesized audio, and even on some datasets it can be comparable to human recordings.

[0003] While current autoregressive models improve the quality of synthesized audio, they significantly increase training time. Specifically, the Tacotron2 model consists of an encoder and a decoder. Given an input sequence (words or phonemes), the encoder attempts to map them into a semantic space and generates a series of encoder hidden states. The decoder uses these hidden states as contextual information, constructing decoder hidden states through an attention mechanism, and then outputs Mel-spectrum frames. However, because both the encoder and decoder use Long Short-Term Memory (LSTM) networks, they can only process inputs and generate outputs sequentially, and constructing the current hidden state requires both the previous hidden state and the current input. This sequential processing characteristic limits the parallel capabilities of the training process, greatly increasing time costs.

[0004] In view of the above, this application is hereby submitted. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a speech synthesis method, apparatus, device, and medium for adaptive fusion of cross-domain information, involving text domain and frequency domain information. It uses phoneme sequences as text domain information and log-Mel spectrum as frequency domain information. It utilizes the temporal characteristics and parallel capabilities of temporal convolutional networks and the ability of cross-attention to focus on the relationships between cross-domain information. It can effectively solve the problem of increased training time cost caused by the limitation of parallel capabilities in the training process due to long short-term memory networks, while ensuring the quality of synthesized speech.

[0006] This invention discloses a speech synthesis method based on cross-domain information adaptive fusion, comprising:

[0007] Obtain the text data to be synthesized, and extract the phoneme sequence from the text to be synthesized. At the same time, initialize a constant matrix.

[0008] The pre-trained speech synthesis model is invoked to preprocess the phoneme sequence of the text to be synthesized and the constant matrix to generate a synthesized log-Mel spectrum corresponding to the text to be synthesized. The speech synthesis model consists of a phoneme feature extraction module, a spectral feature compression module, a cross-domain feature alignment module, and a spectral detail filling module.

[0009] Preferably, before calling the trained speech synthesis model to preprocess the phoneme sequence of the text to be synthesized and the constant matrix, the method further includes:

[0010] Obtain sample text and corresponding sample audio; establish a sample database based on the sample text and sample audio; and extract phoneme sequences and log-Mel spectra from the sample text and sample audio, respectively.

[0011] The preset phoneme feature extraction module, spectral feature compression module, cross-domain feature alignment module, and spectral detail filling module are processed to construct and generate an initial speech synthesis model;

[0012] The phoneme sequence of the sample text and the log-Mel spectrum of the sample audio are used as input data and fed into the initial speech synthesis model for training to obtain the trained speech synthesis model. The phoneme feature extraction module is configured to obtain the phoneme features of the phoneme sequence, the spectral feature compression module is configured to obtain the spectral features of the log-Mel spectrum, the cross-domain feature alignment module is configured to obtain the cross-domain fusion features corresponding to the phonemes using the phoneme features and spectral features, and the spectral detail filling module is configured to perform detail enrichment processing on the predicted log-Mel spectrum.

[0013] Preferably, the phoneme feature extraction module consists of a phoneme embedding layer, a convolutional block, and a temporal residual convolutional block. The execution steps of the phoneme feature extraction module are as follows:

[0014] The phoneme sequence of the sample text As input to the phoneme feature extraction module, the phoneme embedding layer of the phoneme feature extraction module is invoked. The phoneme sequence of the sample text Processing is performed to generate word embeddings of phoneme sequences. The process is formalized as follows: ;

[0015] Using the convolution block Word embedding of phoneme sequences Processing is performed to generate local phoneme features of the phoneme sequence. The process is formalized as follows: ,in, This is the set of network parameters for the convolutional block.

[0016] To achieve both fast processing speed and complete long-term features, a temporal residual convolution block with causal convolution as its core is utilized. Shallow features of phoneme sequences Perform context analysis to generate phoneme features of the phoneme sequence. The process is formalized as follows: ,in, This refers to the set of network parameters for the temporal residual convolutional block.

[0017] Preferably, the spectral feature compression module is composed of The spectral feature compression module consists of several fully connected modules, and its execution steps are as follows:

[0018] The log-Mel spectrum of the audio in the sample text data. As the spectral feature compression module The input is used to call the fully connected module of the spectral feature compression module to perform log-Mel spectrum analysis on the audio in the sample text data. The process is performed to generate the spectral characteristics of the log-Mel spectrum. The process is formalized as follows: ,in, This refers to the set of network parameters for the spectrum feature compression module.

[0019] Preferably, the cross-domain feature alignment module consists of a multi-head alignment attention mechanism and a multi-head self-attention mechanism. The execution steps of the cross-domain feature alignment module are as follows:

[0020] The phoneme features of the phoneme sequence and the spectral characteristics of the log-Mel spectrum As input to the cross-domain feature alignment module, the multi-head alignment attention mechanism is invoked. Phonetic features of the phoneme sequence and the spectral characteristics of the log-Mel spectrum Perform feature alignment processing to generate phoneme spectrum alignment features. The process is formalized as follows: ,in, This refers to the set of network parameters for the multi-head alignment attention mechanism.

[0021] Utilizing the aforementioned multi-head self-attention mechanism The phoneme spectrum alignment features Perform feature fusion processing to generate cross-domain fused features. The process is formalized as follows: ,in, This refers to the set of network parameters for the multi-head self-attention mechanism.

[0022] Preferably, the spectral detail filling module is composed of A fully connected layer and The module consists of several convolutional modules, and the execution steps of the spectral detail filling module are as follows:

[0023] The cross-domain fusion feature As the spectral detail filling module The input is used to call the spectral detail filling module to perform spectral fusion on the phoneme features. The composite logarithmic Mel spectrum was obtained after processing. The process is formalized as follows: ,in, The set of network parameters for the spectrum detail filling module.

[0024] The present invention also discloses a speech synthesis device for cross-domain information adaptive fusion, comprising:

[0025] The data acquisition unit is used to acquire the text data to be synthesized, extract the phoneme sequence from the text to be synthesized, and initialize a constant matrix.

[0026] The speech synthesis model unit is used to call the trained speech synthesis model to preprocess the phoneme sequence of the text to be synthesized and the constant matrix to generate a synthesized log-Mel spectrum corresponding to the text to be synthesized. The speech synthesis model consists of a phoneme feature extraction module, a spectral feature compression module, a cross-domain feature alignment module, and a spectral detail filling module.

[0027] The present invention also discloses a speech synthesis device with cross-domain information adaptive fusion, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the speech synthesis method with cross-domain information adaptive fusion as described above.

[0028] The present invention also discloses a readable storage medium storing a computer program that can be executed by a processor of the device in which the storage medium is located, to realize the speech synthesis method of cross-domain information adaptive fusion as described above.

[0029] In summary, the cross-domain adaptive fusion speech synthesis method, apparatus, device, and medium provided in this embodiment use phoneme sequences as text domain information and log-Mel spectra as frequency domain information. The method includes: establishing a database of sample texts and their corresponding sample audios, obtaining phoneme sequences from the sample texts, and extracting log-Mel spectra from the sample audios; constructing a speech synthesis model based on a phoneme encoder, a spectrum encoder, a fusion encoder, a spectrum mapping layer, and a post-processing network, and inputting the phoneme sequences of the sample texts and the log-Mel spectra of the sample audios into the speech synthesis model for training to obtain a trained speech synthesis model; obtaining text data to be synthesized, obtaining phoneme sequences from the text to be synthesized, and initializing a constant matrix; inputting the phoneme sequences of the text to be synthesized and the constant matrix into the trained speech synthesis model to obtain the synthesized log-Mel spectra corresponding to the text to be synthesized. This aims to solve the problem of increased training time costs caused by the limitation of long short-term memory networks in the parallel processing of autoregressive speech synthesis models during training, while ensuring the quality of the synthesized audio. Attached Figure Description

[0030] Figure 1This is a flowchart illustrating the speech synthesis method for cross-domain information adaptive fusion provided in an embodiment of the present invention.

[0031] Figure 2 This is a schematic diagram of the speech synthesis model provided in an embodiment of the present invention.

[0032] Figure 3 This is a schematic diagram of the phoneme preprocessing layer provided in an embodiment of the present invention.

[0033] Figure 4 This is a schematic diagram of a temporal convolutional layer provided in an embodiment of the present invention.

[0034] Figure 5 This is a schematic diagram of the spectrum feature compression module provided in an embodiment of the present invention.

[0035] Figure 6 This is a schematic diagram of the multi-head alignment attention mechanism provided in an embodiment of the present invention.

[0036] Figure 7 This is a schematic diagram of the multi-head self-attention mechanism provided in an embodiment of the present invention.

[0037] Figure 8 This is a schematic diagram of the spectrum detail filling module provided in an embodiment of the present invention.

[0038] Figure 9 This is a schematic diagram illustrating the evaluation results of the Biaobei Chinese Standard Female Voice Database dataset provided in this embodiment of the invention.

[0039] Figure 10 This is a schematic diagram of the mean score provided in an embodiment of the present invention.

[0040] Figure 11 This is a schematic diagram of a speech synthesis device for cross-domain information adaptive fusion provided in an embodiment of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0043] Please see Figures 1 to 2 The first embodiment of the present invention provides a speech synthesis method for cross-domain information adaptive fusion, which can be executed by a speech synthesis device (hereinafter referred to as the synthesis device), and in particular by one or more processors within the synthesis device, to achieve the following steps:

[0044] In this embodiment, the synthesis device may be a user terminal device (such as a smartphone, smart computer, or other smart device), which can establish a communication connection with a server in the cloud to achieve data interaction.

[0045] S101, Obtain the text data to be synthesized, and extract the text data to be synthesized to obtain the phoneme sequence from the text data to be synthesized. At the same time, initialize a constant matrix.

[0046] S102, the trained speech synthesis model is invoked to preprocess the phoneme sequence of the text to be synthesized and the constant matrix to generate a synthesized log-Mel spectrum corresponding to the text to be synthesized. The speech synthesis model consists of a phoneme feature extraction module, a spectral feature compression module, a cross-domain feature alignment module, and a spectral detail filling module.

[0047] In one possible embodiment of the present invention, before calling the trained speech synthesis model to preprocess the phoneme sequence of the text to be synthesized and the constant matrix, the method further includes:

[0048] Obtain sample text and corresponding sample audio; establish a sample database based on the sample text and sample audio; and extract phoneme sequences and log-Mel spectra from the sample text and sample audio, respectively.

[0049] The preset phoneme feature extraction module, spectral feature compression module, cross-domain feature alignment module, and spectral detail filling module are processed to construct and generate an initial speech synthesis model;

[0050] The phoneme sequence of the sample text and the log-Mel spectrum of the sample audio are used as input data and fed into the initial speech synthesis model for training to obtain the trained speech synthesis model. The phoneme feature extraction module is configured to obtain the phoneme features of the phoneme sequence, the spectral feature compression module is configured to obtain the spectral features of the log-Mel spectrum, the cross-domain feature alignment module is configured to obtain the cross-domain fusion features corresponding to the phonemes using the phoneme features and spectral features, and the spectral detail filling module is configured to perform detail enrichment processing on the predicted log-Mel spectrum.

[0051] Specifically, in this embodiment, a phoneme refers to the basic phonetic segment that constitutes words and languages. For example, "zhuan1" and "li4". The phoneme sequence of a text is obtained by arranging the phonemes corresponding to the characters in the text in the order of pronunciation. In specific implementation, the phoneme sequence of a text can be obtained by parsing the pronunciation of the characters in the text. For example, for the sample text "speech synthesis", by parsing the pinyin, tones, etc. of each character in the text, the phoneme sequence of the text can be obtained as "yu3 yin1 he2 cheng2", where the number for each character represents the tone, 1 for the first tone, 2 for the second tone, 3 for the third tone, 4 for the fourth tone, and 5 for the light tone. It should be noted that in other embodiments, other types of implementation methods for determining the phoneme sequence of a text can also be adopted, which are not specifically limited here, but these solutions are all within the protection scope of the present invention.

[0052] Specifically, in this embodiment, the log Mel spectrogram is a commonly used feature representation method in the fields of speech signal processing and audio processing. It is usually used in speech recognition, speech synthesis, speaker recognition, etc., and can provide reliable feature descriptions under different environments and pronunciation conditions. In specific implementation, the log Mel spectrogram of the sample audio is obtained through the Python toolkit (TorchAudio). It should be noted that in other embodiments, other types of implementation methods for extracting the log Mel spectrogram of audio can also be adopted, which are not specifically limited here, but these solutions are all within the protection scope of the present invention.

[0053] Specifically, in this embodiment, the phoneme feature extraction module is composed of a phoneme embedding layer, a convolutional block, and a temporal residual convolutional block. The execution steps of the phoneme feature extraction module are specifically as follows:

[0054] Take the phoneme sequence of the sample text as the input of the phoneme feature extraction module, and call the phoneme embedding layer of the phoneme feature extraction module to process the phoneme sequence of the sample text to generate the word embedding of the phoneme sequence , and this process is formalized as: ;

[0055] Use the convolutional block to process the word embedding of the phoneme sequence to generate the phoneme local feature of the phoneme sequence , and this process is formalized as: , where is the network parameter set of the convolutional block.

[0056] To achieve both fast processing speed and complete long-term features, a temporal residual convolution block with causal convolution as its core is utilized. Shallow features of phoneme sequences Perform context analysis to generate phoneme features of the phoneme sequence. The process is formalized as follows: ,in, This refers to the set of network parameters for the temporal residual convolutional block.

[0057] Please see Figure 3 In this embodiment, the number of convolutional blocks is set to 3. The embedding vector length of the embedding layer is 360, and the dimension of the embedding vector is 512. Each convolutional block consists of a one-dimensional convolution. Normalization Activation function (Tanh function), deactivation layer Composition, which can be represented as The input dimensions of the one-dimensional convolution are 512, 256, and 256, and the output dimensions are 256, 256, and 256, respectively. The kernel size is 5, the padding length is 2, and the deactivation probability of the deactivation layer is 0.2.

[0058] Please see Figure 4 In this embodiment, the number of temporal residual convolutional blocks is set to 2. Each temporal residual convolutional block... Composed of 2 convolutional modules and an activation function (ReLU function) is composed of, and can be represented as ,in, This represents the input features of the temporal residual block. (Convolutional module) By causal convolution Weight normalization Cutting layer Activation function (ReLU function) and deactivation layer Composition, which can be represented as In this case, the input and output dimensions of the causal convolution are both 256, the kernel size is 2, and the padding length is 1.

[0059] Specifically, in this embodiment, the spectral feature compression module is composed of... The spectral feature compression module consists of several fully connected modules, and its execution steps are as follows:

[0060] The log-Mel spectrum of the audio in the sample text data. As the spectral feature compression module The input is used to call the fully connected module of the spectral feature compression module to perform log-Mel spectrum analysis on the audio in the sample text data. The process is performed to generate the spectral characteristics of the log-Mel spectrum. The process is formalized as follows: ,in, This refers to the set of network parameters for the spectrum feature compression module.

[0061] Please see Figure 5 In this embodiment, the spectral feature compression module is composed of A fully connected module Composition, which can be represented as A fully connected module consists of a fully connected layer. Activation function (ReLU function) and deactivation layer Composition, which can be represented as The input dimensions of the connection layers are 80, 256, and 256, respectively, and the output dimensions are 256, 256, and 256. The deactivation probability of the deactivation layer is 0.2.

[0062] Specifically, in this embodiment, the cross-domain feature alignment module consists of a multi-head alignment attention mechanism and a multi-head self-attention mechanism. The execution steps of the cross-domain feature alignment module are as follows:

[0063] The phoneme features of the phoneme sequence and the spectral characteristics of the log-Mel spectrum As input to the cross-domain feature alignment module, the multi-head alignment attention mechanism is invoked. Phonetic features of the phoneme sequence and the spectral characteristics of the log-Mel spectrum Perform feature alignment processing to generate phoneme spectrum alignment features. The process is formalized as follows: ,in, This refers to the set of network parameters for the multi-head alignment attention mechanism.

[0064] Utilizing the aforementioned multi-head self-attention mechanism The phoneme spectrum alignment features Perform feature fusion processing to generate cross-domain fused features. The process is formalized as follows: ,in, This refers to the set of network parameters for the multi-head self-attention mechanism.

[0065] Please see Figure 6In this embodiment, considering the constraints between dataset size and network parameter size, the number of cross-domain feature alignment modules is set to three. The multi-head alignment attention mechanism... By fully connected layer Softmax function Feature splicing layer and layer normalization Composition, which can be represented as The specific workflow is as follows: The first step is to use fully connected layers with both input and output dimensions of 256 to process the phoneme features. and spectral characteristics The first step involves feature mapping; the second step is to calculate the dot product of the two mapping results from the first step and normalize it to obtain the attention weights; the third step is to process the phoneme features through a fully connected layer with both input and output dimensions of 256. The first step involves feature mapping; the second step uses the attention weights from the third step to perform a dot product operation on the mapping result, resulting in weighted features; the third step inputs the weighted features into a fully connected layer with both input and output dimensions of 256 for feature mapping; the fourth step combines the mapping result from the fifth step with the spectral features. The first step involves concatenating the features to obtain the concatenated features. The second step is to input the concatenated features into a fully connected layer with an input dimension of 512 and an output dimension of 256 for feature dimensionality reduction. The third step is to calculate the dimensionality reduction result and spectral features from the seventh step. The sum of the two is then subjected to layer normalization to obtain the phoneme spectral alignment features. .

[0066] Please see Figure 7 In this embodiment, the multi-head self-attention mechanism It consists of fully connected layers f FC Softmax function Layer normalization One-dimensional convolution and activation function The specific workflow of the ReLU function is as follows: The first step is to align the phoneme spectrum features through two fully connected layers with input and output dimensions of 256. The first step involves feature mapping; the second step is to calculate and normalize the dot product of the two mapping results from the first step to obtain the attention weights; the third step is to align the prime spectrum with the features through a fully connected layer with both input and output dimensions of 256. The feature mapping process is performed again; the fourth step is to use the attention weights from the second step to perform a dot product operation on the mapping result from the third step to obtain weighted features; the fifth step is to input the weighted features into a fully connected layer with both input and output dimensions of 256 for feature mapping; the sixth step is to calculate the dimensionality reduction result from the fifth step and the prime spectrum alignment features. The first step is to sum the results of the first two steps and then perform layer normalization on the sum. The seventh step is to perform a one-dimensional convolution on the normalized result of the sixth step, where the input dimension of the one-dimensional convolution is 256, the output dimension is 1024, the kernel size is 1, and the padding length is 0. The eighth step is to apply an activation function. The ReLU function activates the convolution result from step seven. Step nine involves convolving the activation result from step eight with a one-dimensional convolution, where the input dimension is 1024, the output dimension is 256, the kernel size is 1, and the padding length is 0. Step ten calculates the sum of the convolution result from step nine and the layer normalization result from step six, and then performs layer normalization on the sum to obtain the phoneme spectrum fusion feature. .

[0067] Specifically, in this embodiment, the spectral detail filling module is composed of... A fully connected layer and The module consists of several convolutional modules, and the execution steps of the spectral detail filling module are as follows:

[0068] The cross-domain fusion feature As the spectral detail filling module The input is used to call the spectral detail filling module to process the cross-domain fusion features. The composite logarithmic Mel spectrum was obtained after processing. The process is formalized as follows: ,in, The set of network parameters for the spectrum detail filling module.

[0069] In this embodiment, considering the constraints between the dataset size and the network parameter size, n MP =1, the input dimension of the fully connected layer is 256, the output dimension is 80, and the cross-domain fusion features are used as the fully connected layer. The input is used to obtain preliminary predicted features of the log-Mel spectrum. The process is formalized as follows: ,in, This refers to the set of network parameters for the fully connected layer.

[0070] Please see Figure 8 In this embodiment, =5, where the convolutional module It can be represented as Each convolutional module One-dimensional convolution Normalization Activation function (Tanh function), deactivation layer Composition, which can be represented as The input dimensions of the one-dimensional convolution are 80, 512, 512, 512, and 512, respectively, and the output dimensions are 512, 512, 512, 512, and 80, respectively. The kernel size is 5, the padding length is 2, and the deactivation probability of the deactivation layer is 0.1.

[0071] In this embodiment, preliminary prediction features of the log-Mel spectrum are calculated respectively. Logarithmic Mel spectrum of sample audio L1 loss Synthetic logarithmic Mel spectrum The log-Mel spectra of the samples and sample audio. L1 loss The calculation process can be formalized as follows:

[0072]

[0073] in, N The number of elements; L1 loss is combined using a weighted summation method. and L1 loss Thus, the global loss is obtained. , ,in, α, β ∈[0,1] is an empirical parameter, and Specifically, to simplify the model training process, empirical parameters are set. α = β =1 / 2; The global loss is gradually reduced through iterative training, and the model from the last training round is saved during this process. As a model for the synthesis stage.

[0074] Specifically, in this embodiment, it is assumed that a phoneme sequence is obtained from the text to be synthesized. Assume the constant matrix Among them, constant matrix The constant value is set to 0. The first step is to set the phoneme sequence of the text to be synthesized. and the constant matrix Input the trained speech synthesis model The first predicted logarithmic Mel spectrum mP1 and the first synthesized logarithmic Mel spectrum were obtained. The constant matrix Compared with the first predicted logarithm Mel spectrum The last frame is spliced ​​to obtain the first input logarithmic Mel spectrum. The second step is to synthesize the phoneme sequence of the text to be synthesized. And the first input logarithmic Mel spectrum Input the trained speech synthesis model fB to obtain the second predicted log-Mel spectrum. Second synthetic log-Mel spectrum The first input logarithmic Mel spectrum Compared with the second predicted logarithm Mel spectrum The last frame is spliced ​​to obtain the second input logarithmic Mel spectrum. Repeat step two Nfr times to obtain the predicted log-Mel spectrum. and the composite logarithmic Mel spectrum Among them, the synthetic logarithmic Mel spectrum The synthesized logarithmic Mel spectrum is the synthetic logarithmic spectrum corresponding to the text to be synthesized.

[0075] In this embodiment, considering the number of characters in the text to be synthesized, N fr =198. Please see below. Figure 9 The cross-domain information adaptive fusion speech synthesis method was evaluated using the Biaobei Chinese standard female voice database dataset, which was divided into two subsets: a training set and a test set. The training set contained 9000 sample text and audio pairs; the test set contained 1000 sample text and audio pairs. The applicant trained the model on the training set and saved the last model during training. The model's performance was evaluated on the test set and compared with natural speech and autoregressive acoustic model algorithms, such as Tacotron2. Specifically, the applicant selected 40 synthesized speech samples and 40 natural speech samples. Each speech sample was evaluated by at least 10 testers, and each tester evaluated at least 20 speech samples. To intuitively evaluate the model's performance, the applicant adopted the mainstream evaluation metric: Mean Opinion Score (MOS). The Mean Opinion Score is a subjective evaluation of the quality of synthesized speech, reflecting a subjective impression of the quality of sound. The evaluation criteria are as follows: Figure 10 As shown.

[0076] The cross-domain information adaptive fusion speech synthesis method consumes approximately 0.08 seconds in a single training step, which is 11 times faster than Tacotron2 (approximately 0.96 seconds) with the same batch size (32 samples per batch). For 200,000 training steps, Tacotron2 requires approximately 53 hours, while the cross-domain information adaptive fusion speech synthesis method only requires approximately 4.7 hours. The number of trainable parameters for the Tacotron2 model is 28.30M, while the number of trainable parameters for the cross-domain information adaptive fusion speech synthesis method model is 6.94M.

[0077] In summary, the proposed cross-domain information adaptive fusion speech synthesis method uses phoneme sequences as text domain information and log-Mel spectra as frequency domain information. It includes: establishing a database of sample texts and their corresponding sample audios, and extracting phoneme sequences from the sample texts and log-Mel spectra from the sample audios; constructing a speech synthesis model based on a phoneme feature extraction module, a spectral feature compression module, a cross-domain feature alignment module, and a spectral detail filling module, and inputting the phoneme sequences of the sample texts and the log-Mel spectra of the sample audios into the speech synthesis model for training, resulting in a trained speech synthesis model; acquiring the text data to be synthesized, extracting the phoneme sequences from the text to be synthesized, and initializing a constant matrix; inputting the phoneme sequences of the text to be synthesized and the constant matrix into the trained speech synthesis model to obtain the synthesized log-Mel spectra corresponding to the text to be synthesized. This method aims to address the problem of increased training time costs in autoregressive speech synthesis models due to the limitation of parallel training capabilities caused by long short-term memory networks, while ensuring the quality of the synthesized audio.

[0078] Please see Figure 11 A second embodiment of the present invention provides a speech synthesis apparatus for cross-domain information adaptive fusion, comprising:

[0079] The data acquisition unit 201 is used to acquire the text data to be synthesized, and to extract the phoneme sequence from the text to be synthesized. At the same time, it initializes a constant matrix.

[0080] The speech synthesis model unit 202 is used to call the trained speech synthesis model to preprocess the phoneme sequence of the text to be synthesized and the constant matrix to generate a synthesized log-Mel spectrum corresponding to the text to be synthesized. The speech synthesis model consists of a phoneme feature extraction module, a spectral feature compression module, a cross-domain feature alignment module, and a spectral detail filling module.

[0081] A third embodiment of the present invention provides a speech synthesis device with cross-domain information adaptive fusion, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the speech synthesis method with cross-domain information adaptive fusion as described above.

[0082] The fourth embodiment of the present invention provides a readable storage medium storing a computer program that can be executed by a processor of the device in which the storage medium is located, so as to realize the speech synthesis method of cross-domain information adaptive fusion as described above.

[0083] Exemplary examples show that the computer program described in the third and fourth embodiments of the present invention can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the cross-domain information adaptive fusion speech synthesis device. For example, the apparatus described in the second embodiment of the present invention.

[0084] The processor referred to can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. This processor is the control center of the cross-domain information adaptive fusion speech synthesis method, connecting various parts of the method via various interfaces and lines.

[0085] The memory can be used to store the computer programs and / or modules. The processor, by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory, realizes various functions of the cross-domain information adaptive fusion speech synthesis method. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, text conversion function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, text message data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0086] If the implemented module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0087] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0088] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions that fall within the scope of the present invention are within the scope of protection of the present invention.

Claims

1. A speech synthesis method based on cross-domain information adaptive fusion, characterized in that, include: Obtain the text data to be synthesized, and extract the phoneme sequence from the text to be synthesized. At the same time, initialize a constant matrix. The pre-trained speech synthesis model is invoked to preprocess the phoneme sequence of the text to be synthesized and the constant matrix to generate a synthesized log-Mel spectrum corresponding to the text to be synthesized. The speech synthesis model consists of a phoneme feature extraction module, a spectral feature compression module, a cross-domain feature alignment module, and a spectral detail filling module. Before calling the trained speech synthesis model to preprocess the phoneme sequence of the text to be synthesized and the constant matrix, the process also includes: Obtain sample text and corresponding sample audio; establish a sample database based on the sample text and sample audio; and extract phoneme sequences and log-Mel spectra from the sample text and sample audio, respectively. The preset phoneme feature extraction module, spectral feature compression module, cross-domain feature alignment module, and spectral detail filling module are processed to construct and generate an initial speech synthesis model; The phoneme sequence of the sample text and the log-Mel spectrum of the sample audio are used as input data and fed into the initial speech synthesis model for training to obtain the trained speech synthesis model. The phoneme feature extraction module is configured to obtain the phoneme features of the phoneme sequence, the spectral feature compression module is configured to obtain the spectral features of the log-Mel spectrum, the cross-domain feature alignment module is configured to obtain the cross-domain fusion features corresponding to the phonemes using the phoneme features and spectral features, and the spectral detail filling module is configured to perform detail enrichment processing on the log-Mel spectrum. The cross-domain feature alignment module consists of a multi-head alignment attention mechanism and a multi-head self-attention mechanism. The execution steps of the cross-domain feature alignment module are as follows: The phoneme features of the phoneme sequence and the spectral characteristics of the log-Mel spectrum As input to the cross-domain feature alignment module, the multi-head alignment attention mechanism is invoked. Phonetic features of the phoneme sequence and the spectral characteristics of the log-Mel spectrum Perform feature alignment processing to generate phoneme spectrum alignment features. The process is formalized as follows: ,in, This refers to the set of network parameters for the multi-head alignment attention mechanism. Utilizing the aforementioned multi-head self-attention mechanism The phoneme spectrum alignment features Perform feature fusion processing to generate cross-domain fused features. The process is formalized as follows: ,in, This refers to the set of network parameters for the multi-head self-attention mechanism.

2. The speech synthesis method with cross-domain information adaptive fusion according to claim 1, characterized in that, The phoneme feature extraction module consists of a phoneme embedding layer, a convolutional block, and a temporal residual convolutional block. The execution steps of the phoneme feature extraction module are as follows: The phoneme sequence of the sample text As input to the phoneme feature extraction module, the phoneme embedding layer of the phoneme feature extraction module is invoked. The phoneme sequence of the sample text Processing is performed to generate word embeddings of phoneme sequences. The process is formalized as follows: ; Using the convolution block Word embedding of phoneme sequences Processing is performed to generate local phoneme features of the phoneme sequence. The process is formalized as follows: ,in, The set of network parameters for the convolutional block; To achieve both fast processing speed and complete long-term features, temporal residual convolutional blocks with causal convolution as the core are utilized. Shallow features of phoneme sequences Perform context analysis to generate phoneme features of the phoneme sequence. The process is formalized as follows: ,in, This refers to the set of network parameters for the temporal residual convolutional block.

3. The speech synthesis method with cross-domain information adaptive fusion according to claim 1, characterized in that, The spectral feature compression module consists of The spectral feature compression module consists of several fully connected modules, and its execution steps are as follows: The log-Mel spectrum of the audio in the sample database As the spectral feature compression module The input calls the fully connected module of the spectral feature compression module to process the log-Mel spectrum of the audio in the sample database. The process is performed to generate the spectral characteristics of the log-Mel spectrum. The process is formalized as follows: ,in, This refers to the set of network parameters for the spectrum feature compression module.

4. The speech synthesis method with cross-domain information adaptive fusion according to claim 1, characterized in that, The spectrum detail filling module consists of A fully connected layer and The module consists of several convolutional modules, and the execution steps of the spectral detail filling module are as follows: The cross-domain fusion feature As the spectral detail filling module The input is used to call the spectral detail filling module to process the cross-domain fusion features. The composite log-Mel spectrum was obtained after processing. The process is formalized as follows: ,in, The set of network parameters for the spectrum detail filling module.

5. A speech synthesis device with cross-domain information adaptive fusion, characterized in that, include: The data acquisition unit is used to acquire the text data to be synthesized, extract the phoneme sequence from the text to be synthesized, and initialize a constant matrix. The speech synthesis model unit is used to call the trained speech synthesis model to preprocess the phoneme sequence of the text to be synthesized and the constant matrix to generate a synthesized log-Mel spectrum corresponding to the text to be synthesized. The speech synthesis model consists of a phoneme feature extraction module, a spectral feature compression module, a cross-domain feature alignment module, and a spectral detail filling module. Before calling the trained speech synthesis model to preprocess the phoneme sequence of the text to be synthesized and the constant matrix, the process also includes: Obtain sample text and corresponding sample audio; establish a sample database based on the sample text and sample audio; and extract phoneme sequences and log-Mel spectra from the sample text and sample audio, respectively. The preset phoneme feature extraction module, spectral feature compression module, cross-domain feature alignment module, and spectral detail filling module are processed to construct and generate an initial speech synthesis model; The phoneme sequence of the sample text and the log-Mel spectrum of the sample audio are used as input data and fed into the initial speech synthesis model for training to obtain the trained speech synthesis model. The phoneme feature extraction module is configured to obtain the phoneme features of the phoneme sequence, the spectral feature compression module is configured to obtain the spectral features of the log-Mel spectrum, the cross-domain feature alignment module is configured to obtain the cross-domain fusion features corresponding to the phonemes using the phoneme features and spectral features, and the spectral detail filling module is configured to perform detail enrichment processing on the log-Mel spectrum. The cross-domain feature alignment module consists of a multi-head alignment attention mechanism and a multi-head self-attention mechanism. The execution steps of the cross-domain feature alignment module are as follows: The phoneme features of the phoneme sequence and the spectral characteristics of the log-Mel spectrum As input to the cross-domain feature alignment module, the multi-head alignment attention mechanism is invoked. Phonetic features of the phoneme sequence and the spectral characteristics of the log-Mel spectrum Perform feature alignment processing to generate phoneme spectrum alignment features. The process is formalized as follows: ,in, This refers to the set of network parameters for the multi-head alignment attention mechanism. Utilizing the aforementioned multi-head self-attention mechanism The phoneme spectrum alignment features Perform feature fusion processing to generate cross-domain fused features. The process is formalized as follows: ,in, This refers to the set of network parameters for the multi-head self-attention mechanism.

6. A speech synthesis device with cross-domain information adaptive fusion, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the speech synthesis method of cross-domain information adaptive fusion as described in any one of claims 1 to 4.

7. A readable storage medium, characterized in that, The device contains a computer program that can be executed by a processor of the device where the storage medium is located, to implement the speech synthesis method of cross-domain information adaptive fusion as described in any one of claims 1 to 4.

Images