A speech synthesis method, structure, terminal and storage medium

Abstract

The present invention discloses a speech synthesis method, structure, terminal and storage medium. The method comprises: inputting a phoneme sequence of a text to be synthesized into a pre-trained speech synthesis model, and extracting local features of the phoneme sequence through the speech synthesis model; wherein the speech synthesis model is a speech synthesis model based on CNN and Transformer, using a multi-head attention mechanism to obtain global context information of the phoneme sequence, and using a deep separable convolution to obtain local information of the phoneme sequence, and obtaining local phoneme features based on the global context information and local information of the phoneme sequence; mapping the local phoneme features into a mel spectrum to obtain a speech synthesis result of the text to be synthesized. The present invention uses the multi-head attention mechanism of Transformer to model the relationship between different phonemes, obtain global context information, and uses deep separable convolution to refine the local features within the phonemes, thereby improving the granularity of local information and reducing the number of model parameters, the amount of calculation and the training time.

Description

Technical Field

[0001] The present invention relates to the field of speech synthesis technology, and in particular to a speech synthesis method, structure, terminal and storage medium. Background Art

[0002] Neural network-based end-to-end speech synthesis systems (TTS) have made tremendous progress in recent years. Existing speech synthesis technologies mainly include speech synthesis based on the Transformer structure and speech synthesis based on CNN (Convolutional Neural Networks). However, both Transformer and CNN models have certain limitations. The Transformer structure can capture long-term dependencies and has high training efficiency, but its local feature extraction is less granular. CNN gradually captures local contextual information through layer-by-layer local receptive fields, but CNN requires more layers or parameters to capture global contextual information. Summary of the Invention

[0003] The present invention provides a speech synthesis method, structure, terminal and storage medium, aiming to solve technical problems such as poor granularity of local feature extraction in existing Transformer-based speech synthesis and the need for more layers or parameters in CNN-based speech synthesis.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] A speech synthesis method, comprising:

[0006] Inputting a phoneme sequence of the text to be synthesized into a pre-trained speech synthesis model, and extracting local features of the phoneme sequence through the speech synthesis model; wherein the speech synthesis model is a speech synthesis model based on CNN and Transformer, using a multi-head attention mechanism to obtain global context information of the phoneme sequence, and using a depthwise separable convolution to obtain local information of the phoneme sequence, and obtaining local phoneme features based on the global context information and local information of the phoneme sequence;

[0007] The local features of the phonemes are mapped into mel spectrograms to obtain a speech synthesis result of the text to be synthesized.

[0008] The technical solution adopted in the embodiment of the present invention further includes: the speech synthesis model includes a first one-dimensional convolutional layer and a multi-head attention layer, and the use of the multi-head attention mechanism to obtain global context information of the phoneme sequence includes:

[0009] Utilizing the first one-dimensional convolutional layer to calculate local features between phonemes on the input phoneme sequence, to obtain a first local feature x1;

[0010] Based on the first local feature x1, the multi-head attention layer is used to calculate the global correlation feature x2 between phonemes.

[0011] The technical solution adopted by the embodiment of the present invention further includes: the speech synthesis model further includes a depthwise separable convolution layer, and the acquisition of local information of the phoneme sequence by using the depthwise separable convolution includes:

[0012] The depthwise separable convolutional layer is used to calculate the local features within the phoneme along the time domain on the global correlation feature x2 to obtain a second local feature x3.

[0013] The technical solution adopted by the embodiment of the present invention further includes: the speech synthesis model further includes a second one-dimensional convolutional layer, and the obtaining of the phoneme local features based on the global context information and local information of the phoneme sequence includes:

[0014] Based on the second local feature x3, the second one-dimensional convolutional layer is used to calculate the phoneme local feature of the phoneme sequence, and the final phoneme local feature is output.

[0015] The technical solution adopted in the embodiment of the present invention further includes: the calculation formulas of the first local feature x1 and the global correlation feature x2 are respectively:

[0016] x1=0.5*CNN1(x)+x

[0017] x2=MHSA(x1)+x1

[0018] Among them, CNN1 represents the first one-dimensional convolutional layer, MHSA represents the multi-head attention layer, and 0.5 is the attenuation coefficient.

[0019] The technical solution adopted in the embodiment of the present invention further includes: the calculation formula of the second local feature x3 is:

[0020] x3=DWC(x2)+x2

[0021] Among them, DWC represents the depth-wise separable convolutional layer.

[0022] The technical solution adopted in the embodiment of the present invention also includes: the final phoneme local feature calculation formula is:

[0023] y=0.5*CNN2(x3)+x3

[0024] Among them, CNN2 represents the second one-dimensional convolutional layer.

[0025] Another technical solution adopted by an embodiment of the present invention is: a speech synthesis system, comprising:

[0026] Feature extraction module: used to input the phoneme sequence of the text to be synthesized into a pre-trained speech synthesis model, and extract local features of the phoneme sequence through the speech synthesis model; wherein the speech synthesis model is a speech synthesis model based on CNN and Transformer, using a multi-head attention mechanism to obtain global context information of the phoneme sequence, and using a depthwise separable convolution to obtain local information of the phoneme sequence, and obtain local phoneme features based on the global context information and local information of the phoneme sequence;

[0027] Speech synthesis module: used to map the local features of the phonemes into mel spectrograms to obtain the speech synthesis result of the text to be synthesized.

[0028] Another technical solution adopted by an embodiment of the present invention is: a terminal, the terminal including a processor and a memory coupled to the processor, wherein:

[0029] The memory stores program instructions for implementing the above-mentioned speech synthesis method;

[0030] The processor is configured to execute the program instructions stored in the memory to perform the end-to-end speech synthesis operation.

[0031] Another technical solution adopted by an embodiment of the present invention is: a storage medium storing program instructions executable by a processor, wherein the program instructions are used to execute the above-mentioned speech synthesis method.

[0032] The speech synthesis method, structure, terminal, and storage medium of the embodiments of the present invention improve the existing speech synthesis model based on the Fastspeech network structure by combining CNN and Transformer, thereby obtaining an optimized speech synthesis model based on CNN and Transformer. The optimized speech synthesis model uses the multi-head attention mechanism in Transformer to model the relationship between different phonemes, obtains global context information, and uses deep separable convolution to refine the local features within the phonemes, thereby improving the granularity of local information, improving the naturalness of synthesized speech, and reducing problems such as missing words or skipping in synthesis. At the same time, due to the combination of global and local information, there is no need to increase the number of network layers to expand the receptive field range, which can significantly reduce the number of network layers, the number of model parameters, the amount of calculation, and the training time. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flowchart of a speech synthesis method according to a first embodiment of the present invention;

[0034] Figure 2is a flowchart of a speech synthesis method according to a second embodiment of the present invention;

[0035] Figure 3 This is a schematic diagram of the existing speech synthesis model structure based on the Fastspeech network structure;

[0036] Figure 4 Schematic diagram of the existing Transformer network structure;

[0037] Figure 5 This is a schematic diagram of the network structure of a speech synthesis model based on CNN and Transformer according to an embodiment of the present invention;

[0038] Figure 6 is a flowchart of a speech synthesis method according to a third embodiment of the present invention;

[0039] Figure 7 is a schematic structural diagram of a speech synthesis system according to an embodiment of the present invention;

[0040] Figure 8 is a schematic diagram of the terminal structure of an embodiment of the present invention;

[0041] Figure 9 It is a schematic diagram of the storage medium structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0043] The terms "first", "second" and "third" in the present invention are used only for descriptive purposes and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, a feature defined as "first", "second" and "third" may explicitly or implicitly include at least one of such features. In the description of the present invention, "multiple" means at least two, for example, two, three, etc., unless otherwise clearly and specifically defined. All directional indications in the embodiments of the present invention (such as up, down, left, right, front, back...) are only used to explain the relative positional relationship, movement, etc. between the components under a specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0044] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0045] See also Figure 1 , is a flow chart of a speech synthesis method according to a first embodiment of the present invention. The speech synthesis method according to the first embodiment of the present invention comprises the following steps:

[0046] S100: Inputting the phoneme sequence of the text to be synthesized into a pre-trained speech synthesis model, and extracting local features of the phoneme sequence through the speech synthesis model; wherein the speech synthesis model is a speech synthesis model based on CNN and Transformer, using a multi-head attention mechanism to obtain global context information of the phoneme sequence, and using a depthwise separable convolution to obtain local information of the phoneme sequence, and obtaining local phoneme features based on the global context information and local information of the phoneme sequence;

[0047] In this step, the speech synthesis model is based on the CNN and Transformer network structure, which includes a first one-dimensional convolutional layer CNN1, a multi-head attention layer MHSA, a depth-wise separable convolutional layer DWC, and a second one-dimensional convolutional layer CNN2. The feature extraction process of the speech synthesis model based on CNN and Transformer in the embodiment of the present invention is as follows: first, the first one-dimensional convolutional layer CNN1 is used to calculate the local features between phonemes of the input phoneme sequence x to obtain the first local feature x1; the calculation method of the first local feature x1 is as follows:

[0048] x1=0.5*CNN1(x)+x (1)

[0049] In order to avoid gradient attenuation during back propagation, the attenuation coefficient is multiplied by 0.5 and the jump connection is used.

[0050] Then, based on the first local feature x1, the multi-head attention layer MHSA is used to calculate the global correlation feature x2 between phonemes. The calculation formula of the global correlation feature x2 is:

[0051] x2=MHSA(x1)+x1 (2)

[0052] Then, the depthwise separable convolutional layer DWC is used to calculate the local features within the phoneme along the time domain of the global correlation feature x2 to obtain the second local feature x3. The calculation formula of the second local feature x3 is:

[0053] x3=DWC(x2)+x2 (3)

[0054] Finally, based on the second local feature x3, the second one-dimensional convolutional layer CNN2 is used to calculate the phoneme local feature of the input phoneme x and output the final phoneme local feature y:

[0055] y=0.5*CNN2(x3)+x3 (4)

[0056] Based on the above network structure, an embodiment of the present invention combines the multi-head attention mechanism and the depthwise separable convolution to improve the FFT Block network structure, uses the multi-head attention mechanism to model global context information, and uses the depthwise separable convolution to capture local information, thereby improving the refinement of the local features of the phonemes.

[0057] S101: Mapping the local features of the phonemes into mel spectrograms to obtain a speech synthesis result of the text to be synthesized.

[0058] See also Figure 2 , is a flow chart of a speech synthesis method according to a second embodiment of the present invention. The speech synthesis method according to the second embodiment of the present invention comprises the following steps:

[0059] S200: Obtaining a phoneme sequence of a text to be synthesized;

[0060] S201: Inputting the phoneme sequence of the text to be synthesized into a pre-trained speech synthesis model, the speech synthesis model uses one-dimensional convolution to calculate local features between phonemes on the input phoneme sequence to obtain a first local feature, and based on the first local feature, uses a multi-head attention mechanism to calculate global correlation features between phonemes, and uses depthwise separable convolution to calculate local features within phonemes along the time domain on the global correlation features to obtain a second local feature, and based on the second local feature, uses one-dimensional convolution to calculate phoneme local features on the input phonemes, and outputs the final phoneme local feature;

[0061] In this step, the embodiment of the present invention improves the existing speech synthesis model based on the Fastspeech network structure by combining CNN and Transformer to obtain an optimized speech synthesis model based on CNN and Transformer. Figure 3 The figure shows a schematic diagram of the existing speech synthesis model structure based on the Fastspeech network structure. The FastSpeech network structure is a feedforward structure based on self-attention and 1D convolution in the Transformer. The model takes a text (phoneme) sequence as input and generates a Mel spectrogram in a non-autoregressive manner. It uses a feedforward network based on self-attention in the Transformer and 1D convolution. Since the Mel spectrogram sequence is much longer than its corresponding phoneme sequence, in order to solve the problem of inconsistent matching lengths of the two sequences, FastSpeech uses a length regulator, which upsamples the phoneme sequence according to the duration of the phoneme (the number of Mel spectrograms corresponding to each phoneme) to match the length of the Mel spectrogram sequence. The regulator is based on a phoneme duration predictor and can predict the duration of each phoneme.

[0062] exist Figure 3 In the speech synthesis model shown in the figure, an N-layer FFT Block is first used to extract the hidden state representing the global context information of the text from the input phoneme, and then the hidden state is aligned with the audio Mel Spectrum through the Length Regulator module. Finally, an N-layer FFT (fast Fourier transform) Block is used to map the aligned hidden state to the Mel Spectrum. Among them, the FFT Block is a Transformer structure, and its structure is as follows: Figure 4 As shown in Figure 2, the FFT Block consists of a multi-head attention mechanism and two layers of one-dimensional convolution, and utilizes residual connections and layer normalization (LN).

[0063] like Figure 5 The figure shows a schematic diagram of the network structure of the speech synthesis model based on CNN and Transformer according to an embodiment of the present invention, which includes a first one-dimensional convolutional layer CNN1, a multi-head attention layer MHSA, a depth-wise separable convolutional layer DWC, and a second one-dimensional convolutional layer CNN2. The feature extraction process of the speech synthesis model based on CNN and Transformer according to an embodiment of the present invention is as follows: first, the first one-dimensional convolutional layer CNN1 is used to calculate the local features between phonemes of the input phoneme sequence x to obtain the first local feature x1; the calculation method of the first local feature x1 is as follows:

[0064] x1=0.5*CNN1(x)+x (1)

[0065] In order to avoid gradient attenuation during back propagation, the attenuation coefficient is multiplied by 0.5 and the jump connection is used.

[0066] Then, based on the first local feature x1, the multi-head attention layer MHSA is used to calculate the global correlation feature x2 between phonemes. The calculation formula of the global correlation feature x2 is:

[0067] x2=MHSA(x1)+x1 (2)

[0068] Then, the depthwise separable convolutional layer DWC is used to calculate the local features within the phoneme along the time domain of the global correlation feature x2 to obtain the second local feature x3. The calculation formula of the second local feature x3 is:

[0069] x3=DWC(x2)+x2 (3)

[0070] Finally, based on the second local feature x3, the second one-dimensional convolutional layer CNN2 is used to calculate the phoneme local feature of the input phoneme x and output the final phoneme local feature y:

[0071] y=0.5*CNN2(x3)+x3 (4)

[0072] Based on the above network structure, an embodiment of the present invention combines the multi-head attention mechanism and the depthwise separable convolution to improve the FFT Block network structure, uses the multi-head attention mechanism to model global context information, and uses the depthwise separable convolution to capture local information, thereby improving the refinement of the local features of the phonemes.

[0073] S202: Mapping the phoneme local feature y to a mel spectrum through a linear layer to obtain a speech synthesis result of the text to be synthesized.

[0074] Based on the above, the speech synthesis method of the second embodiment of the present invention combines CNN and Transformer to improve the existing speech synthesis model based on the Fastspeech network structure, obtaining an optimized speech synthesis model based on CNN and Transformer. The optimized speech synthesis model uses the multi-head attention mechanism in Transformer to model the relationship between different phonemes, obtain global context information, and uses deep separable convolution to refine the local features within the phonemes, thereby improving the granularity of local information, improving the naturalness of synthesized speech, and reducing problems such as missing words or skipping in synthesis. At the same time, due to the combination of global and local information, there is no need to increase the number of network layers to expand the receptive field range, which can significantly reduce the number of network layers, reducing the number of model parameters, the amount of calculation, and the training time.

[0075] See also Figure 6 , is a flow chart of a speech synthesis method according to a third embodiment of the present invention. The speech synthesis method according to the third embodiment of the present invention comprises the following steps:

[0076] S300: Obtaining a phoneme sequence of a text to be synthesized;

[0077] S301: Input the phoneme sequence of the text to be synthesized into a pre-trained speech synthesis model, and calculate the local features between phonemes on the input phoneme sequence through the first one-dimensional convolutional layer of the speech synthesis model to obtain the first local feature x1;

[0078] In this step, the embodiment of the present invention improves the existing speech synthesis model based on the Fastspeech network structure by combining CNN and Transformer to obtain an optimized speech synthesis model based on CNN and Transformer. Figure 3 The figure shows a schematic diagram of the existing speech synthesis model structure based on the Fastspeech network structure. The FastSpeech network structure is a feedforward structure based on self-attention and 1D convolution in the Transformer. The model takes a text (phoneme) sequence as input and generates a Mel spectrogram in a non-autoregressive manner. It uses a feedforward network based on self-attention in the Transformer and 1D convolution. Since the Mel spectrogram sequence is much longer than its corresponding phoneme sequence, in order to solve the problem of inconsistent matching lengths of the two sequences, FastSpeech uses a length regulator, which upsamples the phoneme sequence according to the duration of the phoneme (the number of Mel spectrograms corresponding to each phoneme) to match the length of the Mel spectrogram sequence. The regulator is based on a phoneme duration predictor and can predict the duration of each phoneme.

[0079] exist Figure 3In the speech synthesis model shown in the figure, an N-layer FFT Block is first used to extract the hidden state representing the global context information of the text from the input phoneme, and then the hidden state is aligned with the audio Mel Spectrum through the Length Regulator module. Finally, an N-layer FFT (fast Fourier transform) Block is used to map the aligned hidden state to the Mel Spectrum. Among them, the FFT Block is a Transformer structure, and its structure is as follows: Figure 4 As shown in Figure 2, the FFT Block consists of a multi-head attention mechanism and two layers of one-dimensional convolution, and utilizes residual connections and layer normalization (LN).

[0080] In the embodiment of the present application, the first local feature x1 is calculated as follows:

[0081] x1=0.5*CNN1(x)+x (1)

[0082] In order to avoid gradient attenuation during back propagation, the attenuation coefficient is multiplied by 0.5 and the jump connection is used.

[0083] S302: Based on the first local feature x1, the multi-head attention layer MHSA of the speech synthesis model is used to calculate the global correlation feature x2 between phonemes;

[0084] In this step, the calculation formula of the global correlation feature x2 is:

[0085] x2=MHSA(x1)+x1 (2)

[0086] S303: Using a depthwise separable convolutional layer DWC, the local feature calculation of the phoneme is performed on the global correlation feature x2 along the time domain to obtain a second local feature x3;

[0087] In this step, the calculation formula of the second local feature x3 is:

[0088] x3=DWC(x2)+x2 (3)

[0089] S304: Based on the second local feature x3, the second one-dimensional convolutional layer CNN2 is used to calculate the phoneme local feature of the input phoneme x, and output the final phoneme local feature y:

[0090] y=0.5*CNN2(x3)+x3 (4)

[0091] Based on the above network structure, an embodiment of the present invention combines the multi-head attention mechanism and the depthwise separable convolution to improve the FFT Block network structure, uses the multi-head attention mechanism to model global context information, and uses the depthwise separable convolution to capture local information, thereby improving the refinement of the local features of the phonemes.

[0092] S305: Map the phoneme local feature y to a mel spectrum through a linear layer to obtain a speech synthesis result of the text to be synthesized.

[0093] Based on the above, the speech synthesis method of the third embodiment of the present invention combines CNN and Transformer to improve the existing speech synthesis model based on the Fastspeech network structure, obtaining an optimized speech synthesis model based on CNN and Transformer. The optimized speech synthesis model uses the multi-head attention mechanism in Transformer to model the relationship between different phonemes, obtain global context information, and uses deep separable convolution to refine the local features within the phonemes, thereby improving the granularity of local information, improving the naturalness of synthesized speech, and reducing problems such as missing words or skipping in synthesis. At the same time, due to the combination of global and local information, there is no need to increase the number of network layers to expand the receptive field range, which can significantly reduce the number of network layers, reducing the number of model parameters, the amount of calculation, and the training time.

[0094] In an optional embodiment, it is also possible to upload the result of the speech synthesis method to the blockchain.

[0095] Specifically, corresponding summary information is obtained based on the results of the speech synthesis method. Specifically, the summary information is obtained by hashing the results of the speech synthesis method, such as using the sha256s algorithm. Uploading the summary information to the blockchain ensures its security and fairness and transparency to users. Users can download the summary information from the blockchain to verify whether the results of the speech synthesis method have been tampered with. The blockchain referred to in this example is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a string of data blocks generated using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity of the information (to prevent counterfeiting) and generate the next block. Blockchain can include the blockchain underlying platform, the platform product service layer, and the application service layer.

[0096] See also Figure 7 , is a schematic diagram of the structure of an end-to-end speech synthesis system according to an embodiment of the present invention. The end-to-end speech synthesis system 40 according to an embodiment of the present invention includes:

[0097] Data acquisition module 41: used to obtain the phoneme sequence of the text to be synthesized;

[0098] Feature extraction module 42: used to input the phoneme sequence of the text to be synthesized into a pre-trained speech synthesis model. The speech synthesis model uses a multi-head attention mechanism to obtain the global context information of the phoneme sequence, and uses a deep separable convolution to obtain the local information of the phoneme sequence. The local features of the phoneme are obtained based on the global context information and local information of the phoneme sequence. Among them, the embodiment of the present invention improves the existing speech synthesis model based on the Fastspeech network structure by combining CNN and Transformer, and obtains an optimized speech synthesis model based on CNN and Transformer. Specifically, Figure 3 The figure shows a schematic diagram of the existing speech synthesis model structure based on the Fastspeech network structure. The FastSpeech network structure is a feedforward structure based on self-attention and 1D convolution in the Transformer. The model takes a text (phoneme) sequence as input and generates a Mel spectrogram in a non-autoregressive manner. It uses a feedforward network based on self-attention in the Transformer and 1D convolution. Since the Mel spectrogram sequence is much longer than its corresponding phoneme sequence, in order to solve the problem of inconsistent matching lengths of the two sequences, FastSpeech uses a length regulator, which upsamples the phoneme sequence according to the duration of the phoneme (the number of Mel spectrograms corresponding to each phoneme) to match the length of the Mel spectrogram sequence. The regulator is based on a phoneme duration predictor and can predict the duration of each phoneme.

[0099] exist Figure 3 In the speech synthesis model shown in the figure, an N-layer FFT Block is first used to extract the hidden state representing the global context information of the text from the input phoneme, and then the hidden state is aligned with the audio Mel Spectrum through the Length Regulator module. Finally, an N-layer FFT (fast Fourier transform) Block is used to map the aligned hidden state to the Mel Spectrum. Among them, the FFT Block is a Transformer structure, and its structure is as follows: Figure 4 As shown in Figure 2, the FFT Block consists of a multi-head attention mechanism and two layers of one-dimensional convolution, and utilizes residual connections and layer normalization (LN).

[0100] like Figure 5The figure shows a schematic diagram of the network structure of the speech synthesis model based on CNN and Transformer according to an embodiment of the present invention, which includes a first one-dimensional convolutional layer CNN1, a multi-head attention layer MHSA, a depth-wise separable convolutional layer DWC, and a second one-dimensional convolutional layer CNN2. The feature extraction process of the speech synthesis model based on CNN and Transformer according to an embodiment of the present invention is as follows: first, the first one-dimensional convolutional layer CNN1 is used to calculate the local features between phonemes of the input phoneme sequence x to obtain the first local feature x1; the calculation method of the first local feature x1 is as follows:

[0101] x1=0.5*CNN1(x)+x (1)

[0102] In order to avoid gradient attenuation during back propagation, a coefficient of 0.5 and a skip connection are used.

[0103] Then, based on the first local feature x1, the multi-head attention layer MHSA is used to calculate the global correlation feature x2 between phonemes. The calculation formula of the global correlation feature x2 is:

[0104] x2=MHSA(x1)+x1 (2)

[0105] Then, the depthwise separable convolutional layer DWC is used to calculate the local features within the phoneme along the time domain of the global correlation feature x2 to obtain the second local feature x3. The calculation formula of the second local feature x3 is:

[0106] x3=DWC(x2)+x2 (3)

[0107] Finally, based on the second local feature x3, the second one-dimensional convolutional layer CNN2 is used to calculate the phoneme local feature of the input phoneme x and output the final phoneme local feature y:

[0108] y=0.5*CNN2(x3)+x3 (4)

[0109] Based on the above network structure, an embodiment of the present invention combines the multi-head attention mechanism and the depthwise separable convolution to improve the FFT Block network structure, uses the multi-head attention mechanism to model global context information, and uses the depthwise separable convolution to capture local information, thereby improving the refinement of the local features of the phonemes.

[0110] The speech synthesis module 43 is used to map the local phoneme feature y into a mel spectrum through a linear layer to obtain a speech synthesis result of the text to be synthesized.

[0111] Based on the above, the speech synthesis system of the embodiment of the present invention improves the existing speech synthesis model based on the Fastspeech network structure by combining CNN and Transformer, thereby obtaining an optimized speech synthesis model based on CNN and Transformer. The optimized speech synthesis model uses the multi-head attention mechanism in Transformer to model the relationship between different phonemes, obtains global context information, and uses deep separable convolution to refine the local features within the phonemes, thereby improving the granularity of local information, improving the naturalness of synthesized speech, and reducing problems such as missing words or skipping in synthesis. At the same time, due to the combination of global and local information, there is no need to increase the number of network layers to expand the receptive field range, which can significantly reduce the number of network layers, reducing the number of model parameters, the amount of calculation, and the training time.

[0112] See also Figure 8 , which is a schematic diagram of a terminal structure according to an embodiment of the present invention. The terminal 50 includes a processor 51 and a memory 52 coupled to the processor 51.

[0113] The memory 52 stores program instructions for implementing the above-mentioned speech synthesis method.

[0114] The processor 51 is configured to execute program instructions stored in the memory 52 to perform end-to-end speech synthesis operations.

[0115] The processor 51 may also be referred to as a CPU (Central Processing Unit). The processor 51 may be an integrated circuit chip having signal processing capabilities. The processor 51 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor or any conventional processor.

[0116] See also Figure 9 , Figure 9Schematic diagram of the structure of the storage medium of an embodiment of the present invention. The storage medium of the embodiment of the present invention stores a program file 61 that can implement all the above methods, wherein the program file 61 can be stored in the above storage medium in the form of a software product, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or terminal devices such as a computer, a server, a mobile phone, and a tablet.

[0117] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0118] In addition, the functional units in the various embodiments of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software functional units. The above is only an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A speech synthesis method, characterized in that: include: Inputting a phoneme sequence of the text to be synthesized into a pre-trained speech synthesis model, and extracting local features of the phoneme sequence through the speech synthesis model; wherein the speech synthesis model is a speech synthesis model based on CNN and Transformer, using a multi-head attention mechanism to obtain global context information of the phoneme sequence, and using a depthwise separable convolution to obtain local information of the phoneme sequence, and obtaining local phoneme features based on the global context information and local information of the phoneme sequence; Mapping the local phoneme features into mel-spectrograms to obtain a speech synthesis result of the text to be synthesized; The speech synthesis model includes a first one-dimensional convolutional layer and a multi-head attention layer, and the multi-head attention mechanism is used to obtain the global context information of the phoneme sequence, including: Utilizing the first one-dimensional convolutional layer to calculate local features between phonemes on the input phoneme sequence, to obtain a first local feature x1; Based on the first local feature x1, using the multi-head attention layer to calculate the global correlation feature x2 between phonemes; The speech synthesis model further includes a depthwise separable convolution layer, and the method of obtaining local information of a phoneme sequence by using the depthwise separable convolution layer includes: The depthwise separable convolutional layer is used to calculate the local features within the phoneme along the time domain on the global correlation feature x2 to obtain a second local feature x3.

2. The speech synthesis method according to claim 1, wherein: The speech synthesis model further includes a second one-dimensional convolutional layer, and obtaining the phoneme local features according to the global context information and local information of the phoneme sequence includes: Based on the second local feature x3, the second one-dimensional convolutional layer is used to calculate the phoneme local feature of the phoneme sequence, and the final phoneme local feature is output.

3. The speech synthesis method according to claim 1, wherein: The calculation formulas of the first local feature x1 and the global correlation feature x2 are respectively: x1=0.5*CNN1(x)+x x2=MHSA(x1)+x1 Here, x represents the phoneme sequence, CNN1 represents the first one-dimensional convolutional layer, MHSA represents the multi-head attention layer, and 0.5 is the attenuation coefficient.

4. The speech synthesis method according to claim 1, wherein: The calculation formula of the second local feature x3 is: x3=DWC(x2)+x2 Among them, DWC represents the depth-wise separable convolutional layer.

5. The speech synthesis method according to claim 2, wherein: The final phoneme local feature calculation formula is: y=0.5*CNN2(x3)+x3 Among them, CNN2 represents the second one-dimensional convolutional layer.

6. A speech synthesis system, the system being used to implement the steps of the speech synthesis method according to any one of claims 1 to 5, characterized in that: include: Feature extraction module: used to input the phoneme sequence of the text to be synthesized into a pre-trained speech synthesis model, and extract local features of the phoneme sequence through the speech synthesis model; wherein the speech synthesis model is a speech synthesis model based on CNN and Transformer, using a multi-head attention mechanism to obtain global context information of the phoneme sequence, and using a depthwise separable convolution to obtain local information of the phoneme sequence, and obtain local phoneme features based on the global context information and local information of the phoneme sequence; Speech synthesis module: used to map the local features of the phonemes into mel spectrograms to obtain the speech synthesis result of the text to be synthesized.

7. A terminal, characterized in that: The terminal includes a processor and a memory coupled to the processor, wherein: The memory stores program instructions for implementing the speech synthesis method according to any one of claims 1 to 5; The processor is configured to execute the program instructions stored in the memory to perform the speech synthesis method.

8. A storage medium, characterized in that: Program instructions executable by a processor are stored, and the program instructions are used to execute the speech synthesis method according to any one of claims 1 to 5.

Images