Speech synthesis method and apparatus

Abstract

The application provides a speech synthesis method and device, the method comprising: based on a current self-attention module in an encoder, extracting key information from previous text features output by a previous self-attention module to determine current text features; based on a current cross-attention module in a decoder corresponding to a level of the current self-attention module, aligning features of previous speech features output by a previous cross-attention module and the current text features to obtain current speech features; and based on speech features output by a last cross-attention module in the decoder, performing speech synthesis on to-be-synthesized text to obtain synthesized speech. The application reduces coupling of semantic information and acoustic information in speech synthesis, improves alignment stability of the semantic information and the acoustic information, reduces alignment instability caused by cross-attention, and thus improves speech synthesis effect.

Description

Technical Field

[0001] This invention relates to the field of speech synthesis technology, and in particular to a speech synthesis method and apparatus. Background Technology

[0002] Speech synthesis, also known as text-to-speech (TTS), aims to convert input text into fluent and natural output speech. It can be applied to assist visually impaired people in reading, voice navigation systems, automated customer service systems, audiobook production, and more.

[0003] Currently, speech synthesis is mostly based on speech synthesis models. During training, these models predict the first discrete unit (i.e., discrete token) of the speech signal based on the semantic information of the sample text, and convert the corresponding speech signal label into a second discrete unit. The parameters of the speech synthesis model are then updated based on the first and second discrete units to obtain the trained speech synthesis model. However, even after training, the speech synthesis model still suffers from stability issues, such as multiple readings, omissions, and misreadings. Summary of the Invention

[0004] This invention provides a speech synthesis method and apparatus to address the deficiencies in the prior art.

[0005] This invention provides a speech synthesis method, comprising the following steps:

[0006] Based on the current self-attention module in the encoder, key information is extracted from the previous text features output by the previous self-attention module to determine the current text features. The text features output by the first self-attention module are obtained by performing self-attention calculation on the text to be synthesized.

[0007] Based on the current cross-attention module in the decoder that corresponds to the current self-attention module level, the previous speech feature output by the previous cross-attention module is aligned with the current text feature to obtain the current speech feature. In the decoder, except for the level corresponding to the cross-attention module, the other levels are attention modules.

[0008] Based on the speech features output by the last cross-attention module in the decoder, the text to be synthesized is processed to obtain synthesized speech.

[0009] According to a speech synthesis method provided by the present invention, the alignment granularity of each cross-attention module in the decoder decreases as the self-attention module level in the encoder corresponding to the aligned text features increases.

[0010] According to a speech synthesis method provided by the present invention, the step of aligning the previous speech features output by the previous cross-attention module with the current text features based on the current cross-attention module corresponding to the current self-attention module level in the decoder to obtain the current speech features includes:

[0011] The number of blank frames is determined based on the alignment granularity of the current cross-attention module;

[0012] Add all blank frame features to the starting position of the previous speech feature to obtain the extended speech feature;

[0013] The extended speech features are aligned with the current text features to obtain the current speech features.

[0014] According to a speech synthesis method provided by the present invention, the step of aligning the extended speech features with the current text features to obtain the current speech features includes:

[0015] Based on the alignment granularity of the current cross-attention module, the frame features in the extended speech features are grouped.

[0016] The extended speech features after grouping are aligned with the current text features to obtain the current speech features.

[0017] According to a speech synthesis method provided by the present invention, the first cross-attention module input in the decoder is the target speech feature obtained by performing self-attention calculation on the speech data of the target speaker;

[0018] The position of each cross-attention module in the decoder is determined based on prior alignment information between the text to be synthesized and the speech data.

[0019] According to a speech synthesis method provided by the present invention, the encoder and the decoder share parameters.

[0020] The present invention also provides a speech synthesis device, comprising the following modules:

[0021] The extraction unit is used to extract key information from the previous text features output by the previous self-attention module based on the current self-attention module in the encoder, and determine the current text features. The text features output by the first self-attention module are obtained by performing self-attention calculation on the text to be synthesized.

[0022] The alignment unit is used to align the previous speech feature output by the previous cross attention module with the current text feature based on the current cross attention module in the decoder that corresponds to the current self attention module level, so as to obtain the current speech feature. In the decoder, except for the level corresponding to the cross attention module, the other levels are attention modules.

[0023] The synthesis unit is used to synthesize the text to be synthesized based on the speech features output by the last cross-attention module in the decoder, so as to obtain synthesized speech.

[0024] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-described speech synthesis methods.

[0025] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech synthesis method as described above.

[0026] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described speech synthesis methods.

[0027] The speech synthesis method and apparatus provided by this invention decouple the cross-attention module from the self-attention module by setting a cross-attention module at a specific level in the decoder, instead of coupling the self-attention module and the cross-attention module at every level. This reduces the coupling of semantic and acoustic information in speech synthesis, thereby improving the alignment stability of semantic and acoustic information, reducing the alignment instability caused by cross-attention, and improving the speech synthesis effect. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0029] Figure 1 This is a flowchart illustrating the speech synthesis method provided by the present invention.

[0030] Figure 2 This is a schematic diagram of feature alignment provided by the present invention.

[0031] Figure 3 This is a schematic diagram of the 1x scale cross-attention module prediction provided by the present invention.

[0032] Figure 4 It is a schematic diagram of the prediction of the 1 / 2 - scale cross - attention module provided by the present invention.

[0033] Figure 5 It is a schematic diagram of the structure of the speech synthesis device provided by the present invention.

[0034] Figure 6 It is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed implementation manners

[0035] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the scope of protection of the present invention.

[0036] Currently, during the training process of the speech synthesis model, based on the semantic information of the sample text, the sample text is speech - encoded to predict the first discrete unit (i.e., discrete Token) of the speech signal, and the speech signal label corresponding to the sample text is converted into a second discrete unit. Based on the first discrete unit and the second discrete unit, the parameters of the speech synthesis model are updated to obtain a trained speech synthesis model. However, during the training process, the discrete Token contains both semantic information and acoustic information in the speech. Currently, most speech synthesis models adopt a decoder - only architecture and do not explicitly distinguish between semantic information and acoustic information, thus leading to difficulties in aligning semantic information and acoustic information.

[0037] In addition, during the process of aligning semantic information and acoustic information in the speech synthesis model, usually only the previous historical information needs to be referred to (i.e., using the self - attention mechanism to focus on the previous historical information), but the speech synthesis model frequently focuses on text information (i.e., using the cross - attention mechanism to focus on text information), resulting in unstable speech synthesis results. For example, for the sample text "Hello", the corresponding speech signal duration is 0.5 seconds, corresponding to 25 speech frames. Among them, the pronunciation of "Ni" occupies more than ten frames. In these more than ten frames, the model only needs to focus on its own previous historical information (self - attention mechanism). After switching to the pronunciation of "Hao", the cross - attention mechanism is used. However, the traditional speech synthesis model still uses the cross - attention mechanism during the pronunciation of "Ni", resulting in an increase in the instability of speech synthesis.

[0038] In response to this, the present invention provides a speech synthesis method. Figure 1 It is a schematic diagram of the process of the speech synthesis method provided by the present invention, as Figure 1As shown, the method includes steps 110, 120 and 130, wherein the encoder and decoder described below constitute the speech synthesis model in this invention.

[0039] Step 110: Based on the current self-attention module in the encoder, extract key information from the previous text features output by the previous self-attention module to determine the current text features. The text features output by the first self-attention module are obtained by performing self-attention calculation on the text to be synthesized.

[0040] Step 120: Based on the current cross-attention module corresponding to the current self-attention module level in the decoder, the previous speech feature output by the previous cross-attention module is aligned with the current text feature to obtain the current speech feature. Except for the level corresponding to the cross-attention module, the other levels in the decoder are attention modules. The speech feature output by the first decoder is generated based on the speech data of the target speaker.

[0041] Step 130: Based on the speech features output by the last cross-attention module in the decoder, perform speech synthesis on the text to be synthesized to obtain synthesized speech.

[0042] Specifically, the encoder includes multiple self-attention modules. The current self-attention module is used to perform self-attention calculation on the previous text features output by the previous self-attention module in order to focus on the relationship between different positions in the previous text features, extract key information from it, and obtain the current text features based on the extracted key information.

[0043] In the encoder, shallow self-attention modules typically focus on information from adjacent or nearby locations in the text and perform local integration based on this information; that is, shallow attention modules capture detailed features in the text. Deep self-attention modules, on the other hand, can focus on information further back in the text and perform global integration based on this information; that is, deep self-attention modules capture global features in the text.

[0044] In addition, the decoder also includes multiple cross-attention modules. The current cross-attention module is used to perform cross-attention calculation on the previous speech features output by the previous cross-attention module in order to align the current text features output by the encoder with the previous speech features to obtain the current speech features.

[0045] Considering that self-attention modules and cross-attention modules at the same level usually use the same attention mechanism (such as dot product attention) and similar parameter settings, they can generate corresponding feature representations in the same feature space. This makes it easier to match the corresponding text features and speech features in the same feature space, thus achieving accurate alignment between text features and speech features.

[0046] In this regard, in the embodiments of the present invention, the current cross-attention module is the cross-attention module corresponding to the current self-attention module in the decoder, that is, the current text features output by the self-attention module at the current level are input into the cross-attention module at the same level, so that the cross-attention module aligns the current text features with the previous speech features.

[0047] Furthermore, considering that for the same text, there may be multiple consecutive frames corresponding to it in terms of pronunciation. For example, for the speech data "Hello", its total duration is 0.5s, the total number of frames is 25, and "你" corresponds to 12 frames. Since these 12 frames express the same semantics "你", only each frame in these 12 frames needs to be concerned, that is, self-attention calculation is performed on these 12 frames, and there is no need to adopt the cross-attention mechanism. When switching to the frames corresponding to "好", at this time, the correlation between the frames corresponding to "好" and the frames corresponding to "你" needs to be concerned, and thus cross-attention calculation needs to be performed at this time.

[0048] It can be seen that when performing speech synthesis, self-attention calculation is required for some frames in the speech data, and cross-attention calculation is required for some frames. Self-attention calculation can capture the relationships within a single speech sequence, and cross-attention calculation can capture the relationships between different speech sequences. By combining the self-attention mechanism and the cross-attention mechanism, the dependencies within and between sequences can be captured more accurately, thereby improving the alignment stability of speech information and acoustic information and generating more natural and fluent synthetic speech.

[0049] As an optional embodiment, in addition to the cross-attention module corresponding to the self-attention module of the encoder in the decoder, the remaining layers in the decoder can all be self-attention modules, and these self-attention modules are used to perform self-attention calculation on speech features. Among them, for the frames that need to capture the relationships within a single speech sequence, self-attention calculation can be performed using the self-attention module; for the frames that need to capture the relationships between different speech sequences, cross-attention calculation can be performed using the cross-attention module. That is to say, in the decoder, it is not necessary for each layer to couple the self-attention module and the cross-attention module, achieving the decoupling of the cross-attention module and the self-attention module in each layer. Furthermore, the coupling of semantic information and acoustic information can be reduced, the alignment stability of semantic information and acoustic information can be improved, the instability caused by cross-attention can be reduced, and the speech synthesis effect can be improved.

[0050] The speech synthesis method provided in this invention sets up a cross-attention module at a specific level in the decoder, instead of coupling the self-attention module and the cross-attention module at every level. This decouples the cross-attention module from the self-attention module, thereby reducing the coupling of semantic and acoustic information in speech synthesis, improving the alignment stability of semantic and acoustic information, reducing the alignment instability caused by cross-attention, and improving the speech synthesis effect.

[0051] Based on the above embodiments, the alignment granularity of each cross-attention module in the decoder decreases as the level of the self-attention module in the encoder corresponding to the aligned text features increases.

[0052] Given that current commonly used speech coding frame rates are relatively high—for example, the Vall-e model extracts speech features for encoding every 10 milliseconds—while this method can capture more details, it requires frequent attention and processing of the extracted speech features, increasing instability. Reducing the speech coding frame rate would sacrifice the detailed information of the speech features, leading to a decrease in the quality of synthesized speech, manifested as poor sound quality, poor prosody, and poor expressiveness.

[0053] To address this issue, this invention employs a cross-attention mechanism at different scales based on semantic information at different scales in the speech signal, without altering the speech coding frame rate. This aligns features at different scales, ensuring both effective detail modeling and improved stability.

[0054] Specifically, in the encoder, as the hierarchy of self-attention modules increases, the encoder gradually shifts its focus from local details to global structure and semantics. Considering that the cross-attention module in the decoder aligns the text features output by the self-attention modules in the encoder with speech features, for the text features output by deeper self-attention modules, it focuses on deeper semantic information, such as intonation and speech rate. For deeper semantic information, the alignment resolution between text features (semantic information) and speech features (acoustic information) is higher, that is, the alignment granularity is smaller.

[0055] For example, when calculating the self-attention of the text "You look beautiful today," the shallow self-attention module in the encoder focuses on the semantic information of "true" itself. Therefore, when aligning the text features and speech features corresponding to "true," a fine resolution is not required, meaning the alignment granularity is relatively large. However, when the deep self-attention module in the encoder calculates the self-attention of "true," it focuses on the semantic information of "true" within the entire text. Based on the contextual semantic information, it is known that the text is a compliment praising the other person's beauty. Therefore, "true" is emphasized throughout the text, requiring stress. Consequently, the frames corresponding to "true" need to be stressed, and when aligning the text features and speech features corresponding to "true," the frames corresponding to "true" need to be stressed, requiring a finer resolution, meaning the alignment granularity is relatively small.

[0056] Therefore, the higher the level of the self-attention module in the encoder, the smaller the alignment granularity when aligning the output text features with the acoustic features.

[0057] Figure 2 This is a schematic diagram of feature alignment provided by the present invention, such as... Figure 2 As shown, layers 5, 10, and 15 of the decoder use cross-attention modules of 1 / 4 scale, 1 / 2 scale, and 1 scale, respectively. That is, the cross-attention module in layer 5 aligns every four frames, in layer 10 it aligns every two frames, and in layer 15 it aligns every frame. In other words, the alignment granularity decreases as the cross-attention module is placed at the bottom of the layer.

[0058] Based on any of the above embodiments, based on the current cross-attention module in the decoder, the previous speech features output by the previous cross-attention module are aligned with the current text features to obtain the current speech features, including:

[0059] The number of blank frames is determined based on the alignment granularity of the current cross-attention module;

[0060] Add all blank frame features to the starting position of the previous speech feature to obtain the extended speech feature;

[0061] Align the extended speech features with the current text features to obtain the current speech features.

[0062] Considering that the decoder's autoregressive prediction features rely on previously generated speech features when generating current speech features, rather than future information, if the decoder could "see" future information, then generating current speech features based on future information would clearly not conform to autoregressive prediction.

[0063] To prevent the decoder from "seeing" future information, this embodiment of the invention determines the number of blank frames based on the alignment granularity of the current cross-attention module, and adds all blank frame features to the starting position of the previous speech feature to obtain extended speech features. Then, when aligning features based on the extended speech features with the current text features, this avoids the decoder "seeing" future information and yields the current speech features. Here, a blank frame refers to a frame that does not contain valid data or information; for example, a blank frame can be a pad frame.

[0064] If the alignment granularity is per frame, the scale of the corresponding cross-attention module is 1, and the number of blank frames required for 1 scale is 1 / 1 - 1 = 0. If the alignment granularity is every two frames, the scale of the corresponding cross-attention module is 1 / 2, and the number of blank frames required for 1 / 2 scale is 1 / (1 / 2)1 - 1 = 1. If the alignment granularity is every four frames, the scale of the corresponding cross-attention module is 1 / 4, and the number of blank frames required for 1 / 4 scale is 1 / (1 / 4)1 - 1 = 3. That is, the number of blank frames = 1 / n - 1, where n represents the scale of the cross-attention module.

[0065] Figure 3 This is a schematic diagram of the 1x scale cross-attention module prediction provided by the present invention, as shown below. Figure 3 As shown, the input speech feature sequence is Q(0,1,2,3,4,5), where “0” represents frame 0, “1” represents frame 1, and so on, and “5” represents frame 5. The 1x scale cross-attention mechanism aligns each frame, that is, it obtains the correspondence and weight between text features (K / V) and speech features (Q), and finally obtains the prediction sequence through this weight.

[0066] Figure 4 This is a schematic diagram of the cross-attention module prediction at a 1 / 2 scale provided by the present invention, as shown below. Figure 4 As shown, in the 1 / 2 scale cross-attention module, if a blank frame is not added at the beginning of the input speech feature sequence Q(0,1,2,3,4), 1 / 2 average pooling will be performed on 0 and 1 to predict 1 and 2. That is, when predicting 1, the decoder has already seen "1", which does not conform to the autoregressive prediction logic. To address this, a blank frame pad is added at the beginning of the speech feature sequence Q(0,1,2,3,4) to obtain the extended speech features (pad, 0,1,2,3,4). The extended speech features are then subjected to 1 / 2 average pooling, i.e., average pooling is performed every two frames. Finally, cross-attention is calculated with K / V to obtain the predicted sequence.

[0067] Based on any of the above embodiments, feature alignment is performed between the extended speech features and the current text features to obtain the current speech features, including:

[0068] Based on the alignment granularity of the current cross-attention module, the features of each frame in the extended speech features are grouped.

[0069] The extended speech features after grouping are aligned with the current text features to obtain the current speech features.

[0070] Specifically, after determining the alignment granularity of the current cross-attention module, the pooling ratio can be determined based on the alignment granularity. For example, if the alignment granularity is to align every two frames, then the scale of the cross-attention module is 1 / 2 of the scale, and the corresponding pooling ratio is 1 / 2, that is, average pooling is performed on every two frames. Similarly, if the alignment granularity is to align every four frames, then the scale of the cross-attention module is 1 / 4 of the scale, and the corresponding pooling ratio is 1 / 4, that is, average pooling is performed on every four frames.

[0071] After determining the pooling ratio, the extended speech features of each frame can be grouped, and feature alignment can be performed between the grouped extended speech features and the current text features to obtain the current speech features. For example... Figure 4 As shown, the pooling ratio is 1 / 2, which allows two adjacent frames to be divided into one group, that is, pad and 0 are grouped together, 1 and 2 are grouped together, and 3 and 4 are grouped together. Each group is aligned with the current text features (K / V) to obtain the predicted sequence, which is the current speech features.

[0072] Based on any of the above embodiments, the first cross-attention module input in the decoder is the target speech feature obtained after performing self-attention calculation on the speech data of the target speaker;

[0073] The positions of each cross-attention module in the decoder are determined based on prior alignment information between the text to be synthesized and the speech data.

[0074] Specifically, the target speaker refers to the person whose speech needs to be synthesized. This can also be understood as the synthesized speech that mimics the voice of the target person when synthesizing text. The target speaker's speech data is used to characterize their speech features, such as pitch, speech rate, intonation, and pronunciation habits.

[0075] Alignment prior information between the text to be synthesized and the speech data can be understood as the knowledge or information about the temporal correspondence between the text to be synthesized and the speech data acquired by the model during the learning or training phase in speech synthesis. In speech synthesis, alignment prior information is used to indicate which text content needs to correspond to which speech features. When a certain text content needs to be aligned with a certain speech feature, a cross-attention mechanism needs to be used for alignment, that is, a cross-attention module needs to be set at the corresponding position in the decoder.

[0076] For the remaining layers that do not require alignment between text content and speech features, the decoder can use a self-attention module to perform self-attention calculation on the speech features. In other words, the decoder does not need to set a cross-attention module in each layer, which reduces the number of cross-attention modules and avoids the problem of alignment instability caused by redundant cross-attention modules.

[0077] Based on any of the above embodiments, the encoder and decoder share parameters.

[0078] Specifically, parameter sharing between the encoder and decoder reduces the total number of parameters in the speech synthesis model, thereby lowering its computational complexity and improving its efficiency. Furthermore, parameter sharing allows the encoder and decoder to learn more generalized feature representations that are effective for different inputs and outputs. Therefore, the speech synthesis model can better adapt to different speech synthesis tasks, improving its generalization ability.

[0079] Based on any of the above embodiments, the present invention performs speech synthesis based on a speech synthesis model, such as... Figure 2 As shown, the speech synthesis model includes an encoder and a decoder. The encoder is composed of multiple self-attention modules stacked together. Each attention module is used to mine various semantic information in the text using a self-attention mechanism, effectively capturing the semantic relationships between words in the text sequence.

[0080] The self-attention mechanism first encodes the input text through embedding. Simultaneously, the query (Q) and key / value pairs (K / V) of the self-attention mechanism are used. The calculation process of Attention is as follows:

[0081]

[0082] As shown above, Attention outputs the weights of the parts of V that are related to Q. Since both V and Q come from the text, the self-attention mechanism focuses on the associations within the text itself. For example, in the text "I like to read in a sunny park," the words "I" and "read" can be associated through the self-attention mechanism because they are the subjects and actions of the action.

[0083] The decoder is structurally similar to the encoder, but differs in that it includes a self-attention mechanism module as well as a cross-attention module. Similar to the Attention calculation formula mentioned above, the difference lies in that the Q in the cross-attention module comes from the speech data rather than the text, while the K / V comes from the text. Therefore, the Attention output in the cross-attention module is the weight in the text that is related to the current speech features, i.e., it focuses on the alignment of speech features and text features. In the decoder, the self-attention mechanism focuses on the associations within the speech sequence itself, i.e., learning the historical information of the speech sequence (i.e., the contextual information of the speech sequence).

[0084] Considering that the learning process of the encoder-decoder is gradual, deeper layers represent deeper levels of semantic or phonetic information. For example, shallow layers of the decoder may tend to learn the alignment of phonetic information, while deeper layers may learn the alignment of stress and prosody information. Therefore, in the cross-attention module of the decoder, the final output of the encoder is not used; instead, the K and V outputs of the corresponding layers of the encoder are used. At the same time, shared encoder and decoder parameters are used to reduce the vanishing encoder gradient caused by an excessively large model.

[0085] For example, the decoder has 24 layers. Layers 5, 10, and 15 employ cross-attention modules, while the remaining layers use self-attention modules. Correspondingly, the cross-attention module in layer 5 uses the K and V aligned text and semantic features output from the self-attention module in layer 5 of the encoder; the cross-attention module in layer 10 uses the K and V aligned text and semantic features output from the self-attention module in layer 10 of the encoder; and the cross-attention module in layer 15 uses the K and V aligned text and semantic features output from the self-attention module in layer 15 of the encoder. For the self-attention modules in the remaining layers of the decoder, a self-attention mechanism is used to perform self-attention calculation on the speech features, learning the context of the speech information.

[0086] The speech synthesis apparatus provided by the present invention will be described below. The speech synthesis apparatus described below can be referred to in correspondence with the speech synthesis method described above.

[0087] Based on any of the above embodiments Figure 5 This is a schematic diagram of the speech synthesis device provided by the present invention, as shown below. Figure 5 As shown, the device includes:

[0088] The extraction unit 510 is used to extract key information from the previous text features output by the previous self-attention module based on the current self-attention module in the encoder, and determine the current text features. The text features output by the first self-attention module are obtained by performing self-attention calculation on the text to be synthesized.

[0089] Alignment unit 520 is used to align the previous speech features output by the previous cross attention module with the current text features based on the current cross attention module corresponding to the current self attention module level in the decoder, so as to obtain the current speech features. Except for the level corresponding to the cross attention module, the other levels in the decoder are attention modules.

[0090] The synthesis unit 530 is used to synthesize speech from the text to be synthesized based on the speech features output by the last cross-attention module in the decoder, so as to obtain synthesized speech.

[0091] Based on any of the above embodiments, the alignment granularity of each cross-attention module in the decoder decreases as the level of the self-attention module in the encoder corresponding to the aligned text feature increases.

[0092] Based on any of the above embodiments, based on the current cross-attention module corresponding to the current self-attention module level in the decoder, the previous speech features output by the previous cross-attention module are aligned with the current text features to obtain the current speech features, including:

[0093] The number of blank frames is determined based on the alignment granularity of the current cross-attention module;

[0094] Add all blank frame features to the starting position of the previous speech feature to obtain the extended speech feature;

[0095] Align the extended speech features with the current text features to obtain the current speech features.

[0096] Based on any of the above embodiments, feature alignment is performed between the extended speech features and the current text features to obtain the current speech features, including:

[0097] Based on the alignment granularity of the current cross-attention module, the features of each frame in the extended speech features are grouped.

[0098] The extended speech features after grouping are aligned with the current text features to obtain the current speech features.

[0099] Based on any of the above embodiments, the first cross-attention module input in the decoder is the target speech feature obtained after performing self-attention calculation on the speech data of the target speaker;

[0100] The positions of each cross-attention module in the decoder are determined based on prior alignment information between the text to be synthesized and the speech data.

[0101] Based on any of the above embodiments, the encoder and decoder share parameters.

[0102] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a speech synthesis method, which includes: extracting key information from the previous text features output by the previous self-attention module based on the current self-attention module in the encoder to determine the current text features; the text features output by the first self-attention module are obtained by performing self-attention calculation on the text to be synthesized; aligning the previous speech features output by the previous cross-attention module with the current text features based on the current cross-attention module in the decoder to obtain the current speech features; the decoder consists of attention modules at all levels except the level corresponding to the cross-attention module; and synthesizing the text to be synthesized based on the speech features output by the last cross-attention module in the decoder to obtain synthesized speech.

[0103] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0104] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the speech synthesis method provided by the above methods. The method includes: extracting key information from the previous text features output by the previous self-attention module based on the current self-attention module in the encoder to determine the current text features, wherein the text features output by the first self-attention module are obtained by performing self-attention calculation on the text to be synthesized; aligning the previous speech features output by the previous cross-attention module with the current text features based on the current cross-attention module in the decoder to obtain the current speech features, wherein the decoder consists of attention modules except for the level corresponding to the cross-attention module; and synthesizing the text to be synthesized based on the speech features output by the last cross-attention module in the decoder to obtain synthesized speech.

[0105] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the speech synthesis method provided by the above methods. This method includes: extracting key information from the previous text features output by the previous self-attention module based on the current self-attention module in the encoder to determine the current text features; the text features output by the first self-attention module are obtained by performing self-attention calculation on the text to be synthesized; aligning the previous speech features output by the previous cross-attention module with the current text features based on the current cross-attention module in the decoder to obtain the current speech features; the decoder, except for the level corresponding to the cross-attention module, consists of attention modules at the other levels; and performing speech synthesis on the text to be synthesized based on the speech features output by the last cross-attention module in the decoder to obtain synthesized speech.

[0106] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0107] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech synthesis method, characterized in that, include: Based on the current self-attention module in the encoder, key information is extracted from the previous text features output by the previous self-attention module to determine the current text features. The text features output by the first self-attention module are obtained by performing self-attention calculation on the text to be synthesized. Based on the current cross-attention module in the decoder that corresponds to the current self-attention module level, the previous speech feature output by the previous cross-attention module is aligned with the current text feature to obtain the current speech feature. In the decoder, except for the level corresponding to the cross-attention module, the other levels are attention modules. Based on the speech features output by the last cross-attention module in the decoder, the text to be synthesized is synthesized to obtain synthesized speech; The current cross-attention module, which corresponds to the current self-attention module level in the decoder, performs feature alignment between the previous speech features output by the previous cross-attention module and the current text features to obtain the current speech features, including: The number of blank frames is determined based on the alignment granularity of the current cross-attention module; Add all blank frame features to the starting position of the previous speech feature to obtain the extended speech feature; The extended speech features are aligned with the current text features to obtain the current speech features.

2. The speech synthesis method according to claim 1, characterized in that, The alignment granularity of each cross-attention module in the decoder decreases as the self-attention module level in the encoder corresponding to the aligned text features increases.

3. The speech synthesis method according to claim 1, characterized in that, The step of aligning the extended speech features with the current text features to obtain the current speech features includes: Based on the alignment granularity of the current cross-attention module, the frame features in the extended speech features are grouped. The extended speech features after grouping are aligned with the current text features to obtain the current speech features.

4. The speech synthesis method according to any one of claims 1 to 3, characterized in that, The first cross-attention module input in the decoder is the target speech feature obtained after performing self-attention calculation on the speech data of the target speaker; The position of each cross-attention module in the decoder is determined based on prior alignment information between the text to be synthesized and the speech data.

5. The speech synthesis method according to any one of claims 1 to 3, characterized in that, The encoder and the decoder share parameters.

6. A speech synthesis device, characterized in that, include: The extraction unit is used to extract key information from the previous text features output by the previous self-attention module based on the current self-attention module in the encoder, and determine the current text features. The text features output by the first self-attention module are obtained by performing self-attention calculation on the text to be synthesized. The alignment unit is used to align the previous speech feature output by the previous cross attention module with the current text feature based on the current cross attention module in the decoder that corresponds to the current self attention module level, so as to obtain the current speech feature. In the decoder, except for the level corresponding to the cross attention module, the other levels are attention modules. The synthesis unit is used to synthesize the text to be synthesized based on the speech features output by the last cross-attention module in the decoder, so as to obtain synthesized speech; The current cross-attention module, which corresponds to the current self-attention module level in the decoder, performs feature alignment between the previous speech features output by the previous cross-attention module and the current text features to obtain the current speech features, including: The number of blank frames is determined based on the alignment granularity of the current cross-attention module; Add all blank frame features to the starting position of the previous speech feature to obtain the extended speech feature; The extended speech features are aligned with the current text features to obtain the current speech features.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the speech synthesis method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the speech synthesis method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the speech synthesis method as described in any one of claims 1 to 5.

Images