Speech synthesis method and device

Abstract

The invention provides a speech synthesis method and device, and the method comprises the steps: carrying out the key information extraction of a previous text feature outputted by a previous self-attention module based on a current self-attention module in an encoder, and determining a current text feature; on the basis of a current cross attention module corresponding to the current self-attention module level in the decoder, performing feature alignment on a previous voice feature output by a previous cross attention module and the current text feature to obtain a current voice feature; and performing speech synthesis on the to-be-synthesized text based on the speech feature output by the last cross attention module in the decoder to obtain a synthesized speech. According to the method, coupling of semantic information and acoustic information in speech synthesis is reduced, the alignment stability of the semantic information and the acoustic information is improved, alignment instability caused by cross attention is reduced, and then the speech synthesis effect is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of speech synthesis, and in particular to a speech synthesis method and device. Background Art

[0002] Speech Synthesis, also known as Text-To-Speech (TTS), aims to convert input text into fluent and natural output speech. It can be used to assist visually impaired people in reading, voice navigation systems, automatic customer service systems, audio book production, etc.

[0003] At present, speech synthesis is mostly based on a speech synthesis model. During the training process, the speech synthesis model predicts the first discrete unit (i.e., discrete token) of the speech signal based on the semantic information of the sample text, and converts the speech signal label corresponding to the sample text into a second discrete unit. The parameters of the speech synthesis model are updated based on the first discrete unit and the second discrete unit to obtain a trained speech synthesis model. However, the trained speech synthesis model still has speech synthesis stability problems, such as multiple readings, missed readings, and wrong readings. Summary of the invention

[0004] The present invention provides a speech synthesis method and device to solve the defects in the prior art.

[0005] The present invention provides a speech synthesis method, comprising the following steps: Based on the current self-attention module in the encoder, the key information of the previous text feature output by the previous self-attention module is extracted to determine the current text feature. The text feature output by the first self-attention module is obtained by performing self-attention calculation on the text to be synthesized. Based on the current cross-attention module corresponding to the current self-attention module level in the decoder, feature alignment is performed on the previous speech feature output by the previous cross-attention module and the current text feature to obtain the current speech feature, wherein the remaining levels of the decoder except the level corresponding to the cross-attention module are attention modules; Based on the speech features output by the last cross-attention module in the decoder, speech synthesis is performed on the text to be synthesized to obtain synthesized speech.

[0006] According to a speech synthesis method provided by the present invention, the alignment granularity of each cross-attention module in the decoder decreases as the level of the self-attention module in the encoder corresponding to the aligned text features increases.

[0007] According to a speech synthesis method provided by the present invention, the current cross attention module corresponding to the current self-attention module level in the decoder performs feature alignment on the previous speech feature output by the previous cross attention module and the current text feature to obtain the current speech feature, including: Determining the number of blank frames based on the alignment granularity of the current cross-attention module; Adding all blank frame features to the starting position of the previous speech feature to obtain an extended speech feature; The extended speech feature is aligned with the current text feature to obtain the current speech feature.

[0008] According to a speech synthesis method provided by the present invention, the step of aligning the extended speech feature with the current text feature to obtain the current speech feature includes: Grouping each frame feature in the extended speech feature based on the alignment granularity of the current cross-attention module; Feature alignment is performed on the grouped extended speech features and the current text features to obtain the current speech features.

[0009] According to a speech synthesis method provided by the present invention, the first cross-attention module input in the decoder is the target speech feature obtained after performing self-attention calculation on the speech data of the target speaker; The position of each cross-attention module in the decoder is determined based on the alignment prior information between the text to be synthesized and the speech data.

[0010] According to a speech synthesis method provided by the present invention, the encoder and the decoder share parameters.

[0011] The present invention also provides a speech synthesis device, comprising the following modules: An extraction unit is used to extract key information from the previous text feature output by the previous self-attention module based on the current self-attention module in the encoder, and determine the current text feature. The text feature output by the first self-attention module is obtained by performing self-attention calculation on the synthesized text; An alignment unit, for aligning a previous speech feature output by a previous cross-attention module with the current text feature based on a current cross-attention module corresponding to the current self-attention module level in the decoder to obtain a current speech feature, wherein the decoder has attention modules except for the level corresponding to the cross-attention module; A synthesis unit is used to perform speech synthesis on the text to be synthesized based on the speech features output by the last cross-attention module in the decoder to obtain synthesized speech.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned speech synthesis methods is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the speech synthesis method described in any one of the above is implemented.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned speech synthesis methods.

[0015] The speech synthesis method and device provided by the present invention, by setting a cross-attention module at a specific level in the decoder, instead of coupling the self-attention module and the cross-attention module at each level, realizes the decoupling of the cross-attention module from the self-attention module, thereby reducing the coupling of semantic information and acoustic information in speech synthesis, thereby improving the alignment stability of semantic information and acoustic information, reducing the alignment instability caused by cross-attention, and improving the speech synthesis effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of the speech synthesis method provided by the present invention.

[0018] Figure 2 It is a schematic diagram of feature alignment provided by the present invention.

[0019] Figure 3 This is a schematic diagram of the prediction of the cross-attention module at 1 times the scale provided by the present invention.

[0020] Figure 4 This is a schematic diagram of the prediction of the cross-attention module at 1 / 2 scale provided by the present invention.

[0021] Figure 5 It is a structural schematic diagram of the speech synthesis device provided by the present invention.

[0022] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] At present, during the training process, the speech synthesis model performs speech encoding on the sample text based on the semantic information of the sample text, predicts the first discrete unit (i.e., discrete token) of the speech signal, and converts the speech signal label corresponding to the sample text into the second discrete unit. The parameters of the speech synthesis model are updated based on the first discrete unit and the second discrete unit to obtain the trained speech synthesis model. However, during the training process, the discrete token contains both semantic information and acoustic information in the speech. Most current speech synthesis models use a decoder-only architecture and do not explicitly distinguish between semantic information and acoustic information, which makes it difficult to align semantic information and acoustic information.

[0025] In addition, in the process of aligning semantic information and acoustic information, the speech synthesis model usually only needs to refer to previous historical information (that is, the self-attention mechanism is used to pay attention to previous historical information), but the speech synthesis model frequently pays attention to text information (that is, the cross-attention mechanism is used to pay attention to text information), resulting in unstable speech synthesis results. For example, for the sample text "hello", the corresponding speech signal duration is 0.5 seconds, and the corresponding speech frame is 25 frames. Among them, the pronunciation of "you" occupies more than ten frames. In these ten frames, the model only needs to pay attention to its previous historical information (self-attention mechanism). After switching to the pronunciation of "good", the cross-attention mechanism is used. However, the traditional speech synthesis model still uses the cross-attention mechanism when pronouncing "you", which increases the instability of speech synthesis.

[0026] To this end, the present invention provides a speech synthesis method. Figure 1 It is a flow chart of the speech synthesis method provided by the present invention, such as Figure 1 As shown, the method includes step 110, step 120 and step 130, wherein the following encoder and decoder constitute the speech synthesis model in the present invention.

[0027] Step 110: Based on the current self-attention module in the encoder, extract key information from the previous text features output by the previous self-attention module to determine the current text features. The text features output by the first self-attention module are obtained by performing self-attention calculation on the text to be synthesized.

[0028] Step 120: Based on the current cross-attention module corresponding to the current self-attention module level in the decoder, feature alignment is performed on the previous speech feature output by the previous cross-attention module and the current text feature to obtain the current speech feature. Except for the level corresponding to the cross-attention module in the decoder, the remaining levels are attention modules; the speech feature output by the first decoder is generated based on the speech data of the target speaker; Step 130: Based on the speech features output by the last cross-attention module in the decoder, speech synthesis is performed on the text to be synthesized to obtain synthesized speech.

[0029] Specifically, the encoder includes multiple self-attention modules, and the current self-attention module is used to perform self-attention calculation on the previous text feature output by the previous self-attention module to focus on the relationship between different positions in the previous text feature, extract key information therefrom, and obtain the current text feature based on the extracted key information.

[0030] In the encoder, the shallow self-attention module usually focuses on the information of adjacent or close positions in the text, and performs local integration based on this information, that is, the shallow attention module captures the detailed features in the text. The deep self-attention module can focus on the information farther in the text, and perform global integration based on this information, that is, the deep self-attention module captures the global features in the text.

[0031] In addition, the decoder also includes multiple cross-attention modules. The current cross-attention module is used to perform cross-attention calculation on the previous speech feature output by the previous cross-attention module, so as to align the current text feature output by the encoder with the previous speech feature to obtain the current speech feature.

[0032] Considering that the self-attention modules and cross-attention modules at the same level usually use the same attention mechanism (such as dot product attention) and similar parameter settings, the self-attention modules and cross-attention modules at the same level can generate corresponding feature representations in the same feature space, and thus it is easier to match the corresponding text features and speech features in the same feature space to achieve accurate alignment of text features and speech features.

[0033] In this regard, the current cross-attention module in the embodiment of the present invention is a cross-attention module at the corresponding level of the current self-attention module in the decoder, that is, the current text features output by the self-attention module at the current level are input into the cross-attention module at the same level, so that the cross-attention module performs feature alignment on the current text features and the previous speech features.

[0034] Furthermore, considering that the same word may correspond to multiple consecutive frames in pronunciation, for example, for the voice data "hello", the corresponding total duration is 0.5s, the total number of frames is 25, and "you" corresponds to 12 frames. Since the 12 frames express the same semantics "you", it is only necessary to pay attention to each of the 12 frames, that is, to perform self-attention calculation on these 12 frames, without the need to use a cross-attention mechanism. When switching to the frame corresponding to "good", it is necessary to pay attention to the correlation between the frame corresponding to "good" and the frame corresponding to "you", and then a cross-attention calculation is required.

[0035] It can be seen that when performing speech synthesis, some frames in the speech data need to perform self-attention calculations, and some frames need to perform cross-attention calculations. Self-attention calculations can capture the internal relationships of a single speech sequence, and cross-attention calculations can capture the relationships between different speech sequences. By combining the self-attention mechanism and the cross-attention mechanism, the dependencies within and between sequences can be more accurately captured, thereby improving the alignment stability of speech information and acoustic information, and generating more natural and fluent synthesized speech.

[0036] As an optional embodiment, in the decoder, except for the cross-attention module corresponding to the self-attention module of the encoder, the remaining layers in the decoder can all be self-attention modules, which are used to perform self-attention calculations on speech features. Among them, for frames that need to capture the internal relationship of a single speech sequence, the self-attention module can be used for self-attention calculations; for frames that need to capture the relationship between different speech sequences, the cross-attention module can be used for cross-attention calculations. In other words, in the decoder, not every layer needs to couple the self-attention module and the cross-attention module, which realizes the decoupling of the cross-attention module and the self-attention module in each layer, thereby reducing the coupling of semantic information and acoustic information, improving the alignment stability of semantic information and acoustic information, reducing the instability caused by cross-attention, and improving the speech synthesis effect.

[0037] The speech synthesis method provided by the embodiment of the present invention realizes the decoupling of the cross-attention module from the self-attention module by setting a cross-attention module at a specific level in the decoder instead of coupling the self-attention module and the cross-attention module at each level, thereby reducing the coupling of semantic information and acoustic information in speech synthesis, thereby improving the alignment stability of semantic information and acoustic information, reducing the alignment instability caused by cross-attention, and improving the speech synthesis effect.

[0038] Based on the above embodiment, the alignment granularity of each cross-attention module in the decoder decreases as the level of the self-attention module in the encoder corresponding to the aligned text features increases.

[0039] Considering that the currently commonly used speech coding frame rate is relatively high, for example, the vall-e model extracts speech features every 10 milliseconds for coding. Although this method can capture more details, it requires frequent attention and processing of the extracted speech features, which increases instability. If the speech coding frame rate is reduced, the detailed information of the speech features will be sacrificed, resulting in a decrease in the quality of the synthesized speech, which is manifested in poor sound quality, poor rhythmic performance, and poor expressiveness.

[0040] In order to solve this problem, the embodiment of the present invention adopts cross-attention mechanisms of different scales according to semantic information of different scales in the speech signal without changing the speech coding frame rate, and aligns features of different scales, which not only ensures the effect of detail modeling, but also improves stability.

[0041] Specifically, in the encoder, as the number of self-attention modules increases, the encoder gradually shifts its focus from local details to global structure and semantics. Considering that the cross-attention module in the decoder aligns the text features output by the self-attention module in the encoder with the speech features, for the text features output by the deep self-attention module, it pays attention to deep semantic information, such as intonation, speaking speed, etc. For deep semantic information, the alignment resolution of text features (semantic information) and speech features (acoustic information) is higher, that is, the alignment granularity is smaller.

[0042] For example, for the text "You are so beautiful today", when the shallow self-attention module in the encoder performs self-attention calculation on "true", it focuses on the semantic information of "true" itself. Therefore, when aligning the text features corresponding to "true" with the speech features, it does not require too fine a resolution, that is, the alignment granularity is large. When the deep self-attention module in the encoder performs self-attention calculation on "true", it focuses on the semantic information of "true" in the entire text. According to the contextual semantic information, the text is to praise the other person for being beautiful today. Therefore, "true" is emphasized in the entire text, that is, it needs to be stressed. Therefore, the tone of some frames corresponding to "true" needs to be emphasized. Then, when aligning the text features corresponding to "true" with the speech features, the tone of some frames corresponding to "true" needs to be emphasized, which requires a finer resolution, that is, the alignment granularity is small.

[0043] It can be seen from this that the higher the level of the self-attention module in the encoder, the smaller the corresponding alignment granularity when the text features of its output are aligned with the acoustic features.

[0044] Figure 2 is a schematic diagram of feature alignment provided by the present invention, such as Figure 2As shown in the figure, the 5th, 10th and 15th layers of the decoder use cross-attention modules with 1 / 4 scale, 1 / 2 scale and 1 scale respectively, that is, the cross-attention module corresponding to the 5th layer aligns every four frames, the 10th layer aligns every two frames, and the 15th layer aligns every frame, that is, with the lower level of the cross-attention module, the alignment granularity also decreases.

[0045] Based on any of the above embodiments, based on the current cross attention module in the decoder, feature alignment is performed on the previous speech feature output by the previous cross attention module and the current text feature to obtain the current speech feature, including: Determine the number of blank frames based on the alignment granularity of the current cross-attention module; Add all blank frame features to the starting position of the previous speech feature to obtain the extended speech feature; The extended speech feature is aligned with the current text feature to obtain the current speech feature.

[0046] Considering that the decoder's autoregressive prediction features rely on previously generated speech features when generating current speech features, rather than future information, if the decoder can "see" future information, then the decoder generates current speech features based on future information, which is obviously inconsistent with autoregressive prediction.

[0047] In order to avoid the situation where the decoder can "see" future information, the embodiment of the present invention determines the number of blank frames based on the alignment granularity of the current cross-attention module, and adds all blank frame features to the starting position of the previous speech feature to obtain the extended speech feature. Then, when aligning the extended speech feature with the current text feature, the decoder can avoid "seeing" future information and obtain the current speech feature. Among them, a blank frame refers to a frame that does not contain valid data or information, such as a blank frame can be a pad frame.

[0048] If the alignment granularity is to align every frame, the scale of the corresponding cross-attention module is 1 times the scale, and the number of blank frames required for 1 times the scale is 1 / 1-1=0. If the alignment granularity is to align every two frames, the scale of the corresponding cross-attention module is 1 / 2 times the scale, and the number of blank frames required for 1 / 2 times the scale is 1 / (1 / 2)1-1=1. If the alignment granularity is to align every four frames, the scale of the corresponding cross-attention module is 1 / 4 times the scale, and the number of blank frames required for 1 / 4 times the scale is 1 / (1 / 4)1-1=3. In other words, the number of blank frames = 1 / n-1, where n represents the scale of the cross-attention module.

[0049] Figure 3 This is a schematic diagram of the prediction of the cross attention module at 1 times the scale provided by the present invention, such as Figure 3As shown in the figure, the input speech feature sequence is Q (0, 1, 2, 3, 4, 5), "0" represents the 0th frame, "1" represents the 1st frame, and so on, "5" represents the 5th frame. The 1x scale cross attention mechanism aligns each frame, that is, the correspondence and weight between the text features (K / V) and the speech features (Q) are obtained, and the prediction sequence is finally obtained through this weight.

[0050] Figure 4 This is a schematic diagram of the cross attention module prediction at 1 / 2 scale provided by the present invention, such as Figure 4 As shown in the figure, in the 1 / 2 scale cross attention module, if a blank frame is not added at the starting position of the input speech feature sequence Q (0, 1, 2, 3, 4), 0 and 1 will be average pooled and then predicted to be 1 and 2. That is, when predicting 1, the decoder has already seen "1", which does not conform to the autoregressive prediction logic. In this regard, a blank frame pad is added to the starting position of the speech feature sequence Q (0, 1, 2, 3, 4) to obtain the extended speech feature (pad, 0, 1, 2, 3, 4), and the extended speech feature is average pooled by 1 / 2, that is, average pooling is performed every two frames, and then cross attention calculation is performed with K / V to obtain the predicted sequence.

[0051] Based on any of the above embodiments, feature alignment is performed on the extended speech feature and the current text feature to obtain the current speech feature, including: Based on the alignment granularity of the current cross-attention module, the frame features in the extended speech features are grouped; The grouped extended speech features are aligned with the current text features to obtain the current speech features.

[0052] Specifically, after determining the alignment granularity of the current cross-attention module, the pooling ratio can be determined based on the alignment granularity. For example, if the alignment granularity is to align every two frames, the scale of the cross-attention module is 1 / 2 times the scale, and the corresponding pooling ratio is 1 / 2, that is, average pooling is performed on every two frames. Similarly, if the alignment granularity is to align every four frames, the scale of the cross-attention module is 1 / 4 times the scale, and the corresponding pooling ratio is 1 / 4, that is, average pooling is performed on every four frames.

[0053] After determining the pooling ratio, the features of each frame of the extended speech feature red can be grouped, and feature alignment is performed based on the grouped extended speech features and the current text features to obtain the current speech features. Figure 4 As shown, the pooling ratio is 1 / 2, so two adjacent frames can be divided into 1 group, that is, pad and 0 are divided into one group, 1 and 2 are divided into one group, 3 and 4 are divided into one group, and each group is feature aligned with the current text feature (K / V) to obtain the predicted sequence, that is, the current speech feature.

[0054] Based on any of the above embodiments, the first cross-attention module input in the decoder is the target speech feature obtained after performing self-attention calculation on the speech data of the target speaker; The position of each cross-attention module in the decoder is determined based on the alignment prior information between the text to be synthesized and the speech data.

[0055] Specifically, the target speaker refers to the person for whom speech synthesis is required. It can also be understood that when performing speech synthesis on the synthesized text, the generated synthesized speech needs to imitate the voice of the target person. The speech data of the target speaker is used to characterize the speech characteristics of the target speaker, such as pitch, speaking speed, intonation, pronunciation habits, etc.

[0056] The alignment prior information between the text to be synthesized and the speech data can be understood as the knowledge or information about the temporal correspondence between the text to be synthesized and the speech data obtained by the model during the learning or training phase in speech synthesis. In speech synthesis, the alignment prior information is used to indicate which text content needs to correspond to which speech features. When a certain text content needs to be aligned with a certain speech feature, a cross-attention mechanism is required for alignment, that is, a cross-attention module needs to be set at the corresponding position of the decoder.

[0057] For the remaining layers that do not require alignment of text content and speech features, the decoder can use the self-attention module to perform self-attention calculations on the speech features. In other words, the decoder does not need to set up a cross-attention module at each layer, which reduces the number of cross-attention modules and avoids the problem of unstable alignment caused by redundant cross-attention modules.

[0058] Based on any of the above embodiments, encoder and decoder parameters are shared.

[0059] Specifically, the encoder and decoder parameters are shared, which can reduce the total number of parameters of the speech synthesis model, thereby reducing the computational complexity of the model and improving the model's operating efficiency. In addition, parameter sharing enables the encoder and decoder to learn more general feature representations that are valid for different inputs and outputs. Therefore, the speech synthesis model can better adapt to different speech synthesis tasks and improve generalization capabilities.

[0060] Based on any of the above embodiments, the present invention performs speech synthesis based on a speech synthesis model, such as Figure 2 As shown, the speech synthesis model includes an encoder and a decoder. The encoder is composed of multiple stacked self-attention modules. Each attention module is used to mine various semantic information in the text using the self-attention mechanism to effectively capture the semantic associations between words in the text sequence.

[0061] Among them, the self-attention mechanism first encodes the input text through embedding, and at the same time serves as the query (Q) and key / value (K / V) of the self-attention mechanism. The calculation process of Attention is: As can be seen above, Attention outputs the weight of the part of V related to Q, because both V and Q come from text, so the self-attention mechanism focuses on the association of the text itself. For example, for the text "I like to read in a sunny park", the two words "I" and "read" can be associated through the self-attention mechanism because they are the subject and behavior of the action.

[0062] The decoder is similar to the encoder in structure, except that in addition to the self-attention mechanism module, the decoder also contains a cross-attention module, which is similar to the calculation formula of Attention above. The difference is that Q in the cross-attention module comes from speech data rather than text, and K / V comes from text. Therefore, the Attention output in the cross-attention module is the weight related to the current speech feature in the text, that is, it focuses on the alignment of speech features and text features. In the decoder, the self-attention mechanism focuses on the association of the speech sequence itself, that is, learning the historical information of the speech sequence (that is, the contextual information of the speech sequence).

[0063] Considering that the learning process of the encoder and decoder is gradual, the deeper the layers, the deeper the semantic information or pronunciation information. For example, in the shallow layer of the decoder, it may tend to learn the alignment of pronunciation information, while the deep layer may have the alignment of stress and rhythm information. Therefore, in the cross-attention module of the decoder, the final output of the encoder is not used, but the K, V output of the corresponding layer of the encoder is used. At the same time, shared encoder and decoder parameters are used to reduce the vanishing encoder gradient caused by the overly large model.

[0064] For example, the decoder has 24 layers in total, and cross-attention modules are used in the 5th, 10th, and 15th layers, and self-attention modules are used in the remaining layers. Correspondingly, the cross-attention module in the 5th layer uses the K and V aligned text features and semantic features output by the self-attention module in the 5th layer of the encoder, the cross-attention module in the 10th layer uses the K and V aligned text features and semantic features output by the self-attention module in the 10th layer of the encoder, and the cross-attention module in the 15th layer uses the K and V aligned text features and semantic features output by the self-attention module in the 15th layer of the encoder. For the self-attention modules in the remaining layers of the decoder, the self-attention mechanism is used to perform self-attention calculations on the speech features to learn the context of the speech information.

[0065] The speech synthesis device provided by the present invention is described below. The speech synthesis device described below and the speech synthesis method described above can be referred to each other.

[0066] Based on any of the above embodiments, Figure 5 is a structural schematic diagram of the speech synthesis device provided by the present invention, such as Figure 5 As shown, the device comprises: An extraction unit 510 is used to extract key information from a previous text feature output by a previous self-attention module based on a current self-attention module in the encoder, and determine a current text feature, wherein the text feature output by the first self-attention module is obtained by performing self-attention calculation on the text to be synthesized; An alignment unit 520 is used to align the previous speech feature output by the previous cross-attention module with the current text feature based on the current cross-attention module corresponding to the current self-attention module level in the decoder to obtain the current speech feature. Except for the level corresponding to the cross-attention module in the decoder, the remaining levels are attention modules; The synthesis unit 530 is used to perform speech synthesis on the text to be synthesized based on the speech features output by the last cross-attention module in the decoder to obtain synthesized speech.

[0067] Based on any of the above embodiments, the alignment granularity of each cross-attention module in the decoder decreases as the level of the self-attention module in the encoder corresponding to the aligned text features increases.

[0068] Based on any of the above embodiments, based on the current cross attention module corresponding to the current self-attention module level in the decoder, feature alignment is performed on the previous speech feature output by the previous cross attention module and the current text feature to obtain the current speech feature, including: Determine the number of blank frames based on the alignment granularity of the current cross-attention module; Add all blank frame features to the starting position of the previous speech feature to obtain the extended speech feature; The extended speech feature is aligned with the current text feature to obtain the current speech feature.

[0069] Based on any of the above embodiments, feature alignment is performed on the extended speech feature and the current text feature to obtain the current speech feature, including: Based on the alignment granularity of the current cross-attention module, the frame features in the extended speech features are grouped; The grouped extended speech features are aligned with the current text features to obtain the current speech features.

[0070] Based on any of the above embodiments, the first cross-attention module input in the decoder is the target speech feature obtained after performing self-attention calculation on the speech data of the target speaker; The position of each cross-attention module in the decoder is determined based on the alignment prior information between the text to be synthesized and the speech data.

[0071] Based on any of the above embodiments, encoder and decoder parameters are shared.

[0072] Figure 6 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the speech synthesis method, which includes: based on the current self-attention module in the encoder, extracting key information from the previous text feature output by the previous self-attention module, determining the current text feature, the text feature output by the first self-attention module is obtained by performing self-attention calculation on the text to be synthesized; based on the current cross-attention module corresponding to the level of the current self-attention module in the decoder, aligning the previous speech feature output by the previous cross-attention module with the current text feature to obtain the current speech feature, except for the level corresponding to the cross-attention module in the decoder, the remaining levels are attention modules; based on the speech feature output by the last cross-attention module in the decoder, performing speech synthesis on the text to be synthesized to obtain synthesized speech.

[0073] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0074] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the speech synthesis method provided by the above-mentioned methods, which includes: based on the current self-attention module in the encoder, key information is extracted from the previous text features output by the previous self-attention module to determine the current text features, and the text features output by the first self-attention module are obtained by performing self-attention calculation on the text to be synthesized; based on the current cross-attention module corresponding to the level of the current self-attention module in the decoder, feature alignment is performed on the previous speech features output by the previous cross-attention module and the current text features to obtain the current speech features, and except for the level corresponding to the cross-attention module in the decoder, the remaining levels are attention modules; based on the speech features output by the last cross-attention module in the decoder, speech synthesis is performed on the text to be synthesized to obtain synthesized speech.

[0075] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the speech synthesis method provided by the above-mentioned methods, the method comprising: based on the current self-attention module in the encoder, extracting key information from the previous text features output by the previous self-attention module to determine the current text features, wherein the text features output by the first self-attention module are obtained by performing self-attention calculation on the text to be synthesized; based on the current cross-attention module corresponding to the level of the current self-attention module in the decoder, aligning the previous speech features output by the previous cross-attention module with the current text features to obtain the current speech features, wherein the remaining levels in the decoder except for the level corresponding to the cross-attention module are attention modules; based on the speech features output by the last cross-attention module in the decoder, performing speech synthesis on the text to be synthesized to obtain synthesized speech.

[0076] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0077] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech synthesis method, characterized in that: include: Based on the current self-attention module in the encoder, the key information of the previous text feature output by the previous self-attention module is extracted to determine the current text feature. The text feature output by the first self-attention module is obtained by performing self-attention calculation on the synthesized text. Based on the current cross-attention module corresponding to the current self-attention module level in the decoder, feature alignment is performed on the previous speech feature output by the previous cross-attention module and the current text feature to obtain the current speech feature, wherein the remaining levels of the decoder except the level corresponding to the cross-attention module are attention modules; Based on the speech features output by the last cross-attention module in the decoder, speech synthesis is performed on the text to be synthesized to obtain synthesized speech.

2. The speech synthesis method according to claim 1, characterized in that: The alignment granularity of each cross-attention module in the decoder decreases as the level of the self-attention module in the encoder corresponding to the aligned text features increases.

3. The speech synthesis method according to claim 2, characterized in that: The method of aligning the previous speech feature output by the previous cross-attention module with the current text feature based on the current cross-attention module corresponding to the current self-attention module level in the decoder to obtain the current speech feature includes: Determining the number of blank frames based on the alignment granularity of the current cross-attention module; Adding all blank frame features to the starting position of the previous speech feature to obtain an extended speech feature; The extended speech feature is aligned with the current text feature to obtain the current speech feature.

4. The speech synthesis method according to claim 3, characterized in that: The step of aligning the extended speech feature with the current text feature to obtain the current speech feature includes: Grouping each frame feature in the extended speech feature based on the alignment granularity of the current cross-attention module; The grouped extended speech features are aligned with the current text features to obtain the current speech features.

5. The speech synthesis method according to any one of claims 1 to 4, characterized in that: The first cross-attention module input in the decoder is the target speech feature obtained after self-attention calculation of the speech data of the target speaker; The position of each cross-attention module in the decoder is determined based on the alignment prior information between the text to be synthesized and the speech data.

6. The speech synthesis method according to any one of claims 1 to 4, characterized in that: The encoder and the decoder parameters are shared.

7. A speech synthesis device, characterized in that: include: An extraction unit is used to extract key information from the previous text feature output by the previous self-attention module based on the current self-attention module in the encoder, and determine the current text feature. The text feature output by the first self-attention module is obtained by performing self-attention calculation on the synthesized text; An alignment unit, for aligning a previous speech feature output by a previous cross-attention module with the current text feature based on a current cross-attention module corresponding to the current self-attention module level in the decoder to obtain a current speech feature, wherein the decoder has attention modules except for the level corresponding to the cross-attention module; A synthesis unit is used to perform speech synthesis on the text to be synthesized based on the speech features output by the last cross-attention module in the decoder to obtain synthesized speech.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the speech synthesis method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speech synthesis method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the speech synthesis method according to any one of claims 1 to 6 is implemented.

Images