A style transfer synthesis method, device and electronic device

Abstract

The present disclosure provides a style transfer synthesis method, device and electronic device. It relates to the field of artificial intelligence technology, in particular to the field of deep learning, speech synthesis, and style transfer technology, and specifically to a speech style transfer synthesis method, device and electronic device. The specific implementation scheme is: input the target text and the target audio segment into a speech synthesis model that has been pre-trained with sample text and sample audio segments; for each audio unit in the target audio segment, superimpose coarse-grained audio features and fine-grained audio features to obtain superimposed audio features of the audio unit; extract the pronunciation features of each pronunciation unit in the target text; for each pronunciation unit in the target text, fuse the pronunciation features of the pronunciation unit and the target superimposed audio features to obtain the fused features of the pronunciation unit; synthesize the audio segment based on the fused features. It is possible to synthesize an audio segment with a target style in both overall and detailed aspects.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of deep learning, speech synthesis, and style transfer technology, and specifically to a speech style transfer synthesis method, device, and electronic device. Background Art

[0002] For various practical needs, such as realizing the voice changing function provided in voice chat software and hiding the real identity of the speaker, it is necessary to synthesize an audio clip with the same voice style as the audio clip and the voice content of the text based on a given audio clip and text. Since this process can be regarded as transferring the voice style of the audio clip to the text, this process is called style transfer synthesis. Summary of the invention

[0003] The present disclosure provides a style transfer synthesis method, device and electronic device.

[0004] According to a first aspect of the present disclosure, a style transfer synthesis method is provided, comprising:

[0005] Inputting the target text and the target audio clip with the target voice style into a speech synthesis model that has been pre-trained with the sample text and the sample audio clip;

[0006] Using the style extraction sub-model of the speech synthesis model, for each audio unit in the target audio segment, superimpose the coarse-grained audio features used to characterize the target audio segment and the fine-grained audio features used to characterize the audio unit to obtain the superimposed audio features of the audio unit;

[0007] Extracting pronunciation features of each pronunciation unit in the target text through the content encoding sub-model of the speech synthesis model;

[0008] By using the content-style cross-attention sub-model of the speech synthesis model, for each pronunciation unit in the target text, the pronunciation feature of the pronunciation unit and the target superimposed audio feature are fused to obtain the fused feature of the pronunciation unit, wherein the target superimposed audio feature is a superimposed audio feature that matches the pronunciation feature;

[0009] An audio segment having the target speech style and speech content of the target text is synthesized according to the fusion features of each pronunciation unit in the target text through the spectrogram decoding submodel of the speech synthesis model.

[0010] According to a second aspect of the present disclosure, a method for training a speech synthesis model is provided, comprising:

[0011] Inputting a sample audio segment and a sample text into the original model, wherein the sample text is the speech content of the sample audio segment;

[0012] Using the original model, for each audio unit in the sample audio segment, superimpose the coarse-grained audio features used to characterize the sample audio segment and the fine-grained audio features used to characterize the audio unit to obtain superimposed audio features of the audio unit;

[0013] Extracting the pronunciation features of each pronunciation unit in the sample text through the original model;

[0014] By using the original model, for each pronunciation unit in the sample text, the pronunciation feature of the pronunciation unit and the target superimposed audio feature are fused to obtain the fused feature of the pronunciation unit, wherein the target superimposed audio feature is a superimposed audio feature that matches the pronunciation feature;

[0015] By using the original model, converting the fusion features of each pronunciation unit in the sample text into predicted spectral features;

[0016] Adjusting the model parameters of the original model according to the difference between the predicted spectral features and the real spectral features of the sample audio segment;

[0017] A new sample audio segment and a new sample text are obtained, and the step of inputting the sample audio segment and the sample text into the original model is returned to be executed until the first convergence condition is reached, and the adjusted original model is used as the speech synthesis model.

[0018] According to a third aspect of the present disclosure, a style transfer synthesis device is provided, comprising:

[0019] A first input module is used to input the target text and the target audio segment with the target voice style into a speech synthesis model pre-trained with the sample text and the sample audio segment;

[0020] A style extraction module, configured to superimpose, for each audio unit in the target audio segment, a coarse-grained audio feature for characterizing the target audio segment and a fine-grained audio feature for characterizing the audio unit, through a style extraction sub-model of the speech synthesis model, to obtain a superimposed audio feature of the audio unit;

[0021] A content encoding module, used for extracting the pronunciation features of each pronunciation unit in the target text through the content encoding sub-model of the speech synthesis model;

[0022] A content-style cross-attention module is used to fuse the pronunciation features of each pronunciation unit in the target text and the target superimposed audio features through the content-style cross-attention sub-model of the speech synthesis model to obtain the fused features of the pronunciation unit, wherein the target superimposed audio features are superimposed audio features that match the pronunciation features;

[0023] The spectrogram decoding module is used to synthesize an audio segment having the target speech style and the speech content of the target text according to the fusion features of each pronunciation unit in the target text through the spectrogram decoding submodel of the speech synthesis model.

[0024] According to a fourth aspect of the present disclosure, there is provided a training device for a speech synthesis model, comprising:

[0025] A second input module is used to input a sample audio segment and a sample text into the original model, wherein the sample text is the speech content of the sample audio segment;

[0026] A first primitive module is used to superimpose, for each audio unit in the sample audio segment, a coarse-grained audio feature for characterizing the sample audio segment and a fine-grained audio feature for characterizing the audio unit through the primitive model to obtain a superimposed audio feature of the audio unit;

[0027] A second primitive module, used for extracting the pronunciation features of each pronunciation unit in the sample text through the primitive model;

[0028] A third original module is used to fuse the pronunciation feature of each pronunciation unit in the sample text with the target superimposed audio feature through the original model to obtain a fused feature of the pronunciation unit, wherein the target superimposed audio feature is a superimposed audio feature that matches the pronunciation feature;

[0029] A fourth original module, used for converting the fusion feature of each pronunciation unit in the sample text into a predicted spectral feature through the original model;

[0030] A parameter adjustment module, used for adjusting the model parameters of the original model according to the difference between the predicted spectral features and the real spectral features of the sample audio segment;

[0031] The acquisition module is used to acquire new sample audio segments and new sample texts, and return to execute the step of inputting the sample audio segments and sample texts into the original model until the first convergence condition is reached, and the adjusted original model is used as the speech synthesis model.

[0032] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:

[0033] at least one processor; and

[0034] a memory communicatively connected to the at least one processor; wherein,

[0035] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any one of the methods of the first aspect or the second aspect.

[0036] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method according to any one of the first aspect or the second aspect above.

[0037] According to a seventh aspect provided by the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method according to any one of the above-mentioned first aspect or second aspect is implemented.

[0038] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0040] Figure 1 is a flow chart of a style transfer synthesis method provided by the present disclosure;

[0041] Figure 2 is a structural schematic diagram of a speech synthesis model used in the style transfer synthesis method provided by the present disclosure;

[0042] Figure 3a It is a schematic diagram of the structure of a style extraction sub-model in a speech synthesis model used in the style transfer synthesis method provided by the present disclosure;

[0043] Figure 3b It is a schematic diagram of the structure of a content encoding submodel in a speech synthesis model used in the style transfer synthesis method provided by the present disclosure;

[0044] Figure 3c It is a schematic diagram of the structure of a content-style cross-attention sub-model in a speech synthesis model used in the style transfer synthesis method provided by the present disclosure;

[0045] Figure 3dIt is a structural schematic diagram of a spectrogram decoding submodel in a speech synthesis model used in the style transfer synthesis method provided by the present disclosure;

[0046] Figure 4 It is a flowchart of a method for training a speech synthesis model provided by the present disclosure;

[0047] Figure 5 is a structural schematic diagram of a style transfer synthesis device provided by the present disclosure;

[0048] Figure 6 is a structural schematic diagram of a training device for a speech synthesis model provided by the present disclosure;

[0049] Figure 7 It is a block diagram of an electronic device used to implement the style transfer synthesis method or the speech synthesis model training method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0050] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0051] In order to more clearly illustrate the style transfer synthesis method provided by the present disclosure, a possible application scenario of the style transfer synthesis method provided by the present disclosure will be exemplified below. It can be understood that the following example is only one possible application scenario of the style transfer synthesis method provided by the present disclosure. In other possible embodiments, the style transfer synthesis method provided by the present disclosure can also be applied to other possible application scenarios, and the following example does not impose any limitation on this.

[0052] In order to hide the true identity of a target person, such as a player of an online game or a person being interviewed in a news interview, the words spoken by the target person can be converted into text, and then another voice style (hereinafter referred to as the target voice style) different from the voice style of the target person can be transferred to the text through style transfer synthesis. Since the voice style of the synthesized audio clip is different from that of the target person, when other people hear the synthesized audio clip, they will not associate it with the target person, thereby achieving the purpose of hiding the true identity.

[0053] In the related art, in order to realize style transfer synthesis, an encoding network is often used to encode a target audio segment with a target voice style to obtain style features, and the words spoken by the target person are converted into text and encoded to obtain content features. The style features and content features are input into a pre-trained decoding network to obtain the spectral features output by the decoding network, and then the spectral features are converted into an audio segment by a vocoder, thereby obtaining an audio segment with the target voice style and the voice content of the text.

[0054] However, this solution can only make the synthesized audio clip sound similar to the target audio frequency band as a whole, but it is quite different from the target audio clip in some details, such as speech speed, emotion, pitch, intonation, short pauses, stress, etc. In other words, the synthesized audio clip does not have the target style in details.

[0055] Based on this, the present disclosure provides a style transfer synthesis method, which can be applied to any electronic device with style transfer synthesis function, including but not limited to mobile phones, tablet computers, personal computers, servers, etc. The style transfer synthesis method provided by the present disclosure can be used as follows: Figure 1 As shown, including:

[0056] S101, inputting a target text and a target audio segment with a target voice style into a speech synthesis model that has been pre-trained with sample texts and sample audio segments.

[0057] S102, using the style extraction sub-model of the speech synthesis model, for each audio unit in the target audio segment, superimposes the coarse-grained audio features used to characterize the target audio segment and the fine-grained audio features used to characterize the audio unit to obtain the superimposed audio features of the audio unit.

[0058] S103, extracting the pronunciation features of each pronunciation unit in the target text through the content encoding sub-model of the speech synthesis model.

[0059] S104, through the content-style cross-attention sub-model of the speech synthesis model, for each pronunciation unit in the target text, the pronunciation features of the pronunciation unit and the target superimposed audio features are fused to obtain the fused features of the pronunciation unit, wherein the target added audio features are superimposed audio features that match the pronunciation features.

[0060] S105, synthesizing an audio clip having a target speech style and speech content of the target text according to the fusion features of each pronunciation unit in the target text through the spectrogram decoding sub-model of the speech synthesis model.

[0061] Select this embodiment, and use the audio features with different granularities to obtain superimposed audio features, so that the superimposed audio features can not only reflect the overall features of the target style, but also reflect the detailed features of the target audio segment. Then, using the cross-attention mechanism, fuse the audio features of each pronunciation unit with the corresponding superimposed audio features. The obtained fused features can, on the one hand, reflect the speech content included in the target text, and on the other hand, reflect the overall and detailed features of the target style. And because the audio features are fused with the corresponding superimposed audio features, the detailed features reflected by the fused features of each pronunciation unit can reflect the detailed features of the target style when pronouncing this pronunciation unit. Therefore, in the audio segment synthesized according to the fused features, not only the overall acoustic features are similar to the target style, but also the pronunciation of each pronunciation unit is similar to the target style, that is, an audio segment with the target style in both overall and detail can be synthesized.

[0062] To more clearly illustrate the steps of S101-S105 described above, the speech synthesis model provided by the present disclosure will be described first. Refer to Figure 2 , Figure 2 FIG. shows a schematic structural diagram of a speech synthesis model provided by the present disclosure, including:

[0063] A style extraction sub-model, a content encoding sub-model, a content-style cross-attention sub-model, and a spectrogram decoding sub-model.

[0064] Among them, the input of the style extraction sub-model is the target audio segment, and the output is the superimposed audio features of each audio unit in the target audio segment. Each audio unit is composed of M consecutive audio frames in the audio segment, and each audio frame belongs to only one audio unit. M can be set by the user according to actual experience or requirements, such as M = 2, 4, 5, 9, etc., and the present disclosure does not make any restrictions on this. Each audio frame is audio data of consecutive N ms in the audio segment, and the interval between two adjacent audio frames is Q ms, and Q is not greater than N. For example, N = 25, Q = 10, N = 20, Q = 20, N = 28, Q = 21, etc., and the present disclosure does not make any restrictions on this.

[0065] The input of the content encoding sub-model is the target text, and the output is the pronunciation features of each pronunciation unit in the target text. The content encoding sub-model is used to implement the steps of S103 described above.

[0066] Among them, each pronunciation unit in the target text is composed of k consecutive phonemes in the pronunciation of the target text. k can be set according to the actual needs or experience of the user. Exemplarily, when k = 1, each pronunciation unit is a phoneme in the pronunciation of the target text. Taking the target text as "hundred" and the pronunciation of the target text as Chinese pronunciation as an example, it includes the pronunciation unit "b" and the pronunciation unit "ai".

[0067] The input of the content-style cross-attention sub-model is the superimposed audio features of each audio unit in the target audio clip, and the audio features of each pronunciation unit in the target text, that is, the input of the content-style cross-attention sub-model is the output of the style extraction sub-model and the output of the content encoding sub-model. The output of the content-style cross-attention sub-model is each pronunciation unit in the target text. The content-style cross-attention sub-model is used to implement the aforementioned step S104.

[0068] The input of the spectrogram decoding submodel is each pronunciation unit in the target text, that is, the input of the spectrogram decoding submodel is the output of the content style attention submodel. The output of the spectrogram decoding submodel is a synthesized audio clip. The spectrogram decoding submodel is used to implement the above-mentioned step S105.

[0069] The implementation of the above S102-S105 will be described below in combination with the structures of each sub-model in the speech synthesis model. Figure 3a-3d , Figure 3a-3d The following is a schematic diagram of the structure of each sub-model in the speech synthesis model:

[0070] like Figure 3a As shown, the style extraction sub-model includes a wave pair feature vector (Wav2Vec) sub-network, a first branch consisting of a linear sub-network, a long short-term memory (LSTM) sub-network, and a pooling sub-network, and a second branch consisting of a long short-term memory sub-network and a pooling sub-network.

[0071] The input of the wave pair feature vector sub-network is the target audio clip, and the output is the audio features of each audio frame in the target audio clip. The number of target audio clips input to the wave pair feature vector sub-network can be one or more, and the present disclosure does not impose any restrictions on this. Exemplarily, in one possible embodiment, the number of target audio clips is two, one of which is used to reflect the overall characteristics of the target style, and the other target audio clip is used to reflect the detailed characteristics of the target style. In another possible embodiment, the number of target audio clips is four, one of which is used to reflect the overall characteristics of the target style, and the remaining three target audio clips are used to reflect the detailed characteristics of the target style.

[0072] The inputs of the first branch and the second branch are the audio features of each audio frame in the target audio segment, that is, the inputs of the first branch and the second branch are the outputs of the wave pair feature vector sub-network. If the number of target audio segments is one, the inputs of the first branch and the second branch are the same, both of which are feature vectors of each audio frame in the one target audio segment. If the number of target audio segments is multiple, the inputs of the first branch and the second branch are different, the input of the first branch is the audio features of each audio frame in the target audio segment that reflects the detailed features of the target style, and the input of the second branch is the audio features of each audio frame in the target audio segment that reflects the overall features of the target style.

[0073] Exemplarily, assuming that the target audio segment includes a first target audio segment and a second target audio segment, wherein the first target audio segment is used to reflect the detailed characteristics of the target style and the second target audio segment reflects the overall characteristics of the target style, then the input of the first branch is the audio characteristics of each audio frame in the first target audio segment, and the input of the second branch is the audio characteristics of each audio frame in the second target audio segment.

[0074] The output of the first branch is the fine-grained audio features of each audio unit in the target audio clip. The first branch averages the audio features of all audio frames included in each audio unit to obtain the fine-grained audio features used to characterize the audio unit. The first branch obtains the fine-grained audio features in this way. Since the fine-grained audio features only include the audio features of each audio frame in the audio unit, the obtained fine-grained audio features can retain more detailed features of the target style.

[0075] The output of the second branch is the coarse-grained audio feature of the target audio segment. The second branch averages the audio features of all audio frames in the target audio segment to obtain the coarse-grained audio feature used to characterize the target audio segment. The second branch obtains the coarse-grained audio feature in this way. Since the coarse-grained audio feature is the average of the audio features of all audio frames, the obtained coarse-grained feature retains more of the overall characteristics of the target style.

[0076] The style extraction sub-model adds each fine-grained audio feature output by the first branch to the coarse-grained audio feature output by the second branch to obtain a superimposed audio feature, and outputs it to the content-style cross-attention sub-model.

[0077] Exemplarily, assume that there are three audio units in total, which are recorded as audio units 1-3. The fine-grained audio feature of audio unit 1 is x1, the fine-grained audio feature of audio unit 2 is x2, and the fine-grained audio feature of audio unit 3 is x3. Assuming the coarse-grained audio feature is x4, the superimposed audio feature of audio unit 1 is x1+x4, the fine-grained audio feature of audio unit 2 is x2+x4, and the fine-grained audio feature of audio unit 3 is x3+x4. Among them, x1+x4 means adding the values ​​of x1 and x4 in each feature dimension. For example, assuming that there are three feature dimensions in total, and x1 is (a1, b1, c1), and x4 is (a4, b4, c4), the superimposed audio feature is (a1+a4, b1+b4, c1+c4). The same is true for x2+x4 and x3+x4, which will not be repeated here.

[0078] like Figure 3b As shown in FIG. 5 , the content encoding sub-model is composed of a phoneme embedding sub-network, a layer norm sub-network, and a position embedding sub-network.

[0079] The input of the phoneme embedding subnetwork is the target text, and the output is the feature vector of each pronunciation unit in the target text. The phoneme embedding subnetwork is used to encode the pronunciation of each pronunciation unit in the target text into a feature vector.

[0080] The input of the layer norm subnetwork is the feature vector of each pronunciation unit in the target text, that is, the input of the layer norm subnetwork is the output of the phoneme embedding subnetwork. The layer norm digital network is used to further extract pronunciation features from the feature vector.

[0081] The input of the position embedding subnetwork is the target text, and the output is the position encoding of each pronunciation unit in the target text. The position embedding subnetwork is used to encode the position of each pronunciation unit in the target text to obtain the position encoding of each pronunciation unit in the target text.

[0082] The content encoder combines the pronunciation features and position codes of each pronunciation unit in the target text to represent the position of the pronunciation unit to which each pronunciation feature belongs in the target text through position codes, and outputs the pronunciation features to the content-style cross-attention sub-model.

[0083] like Figure 3cAs shown, the content-style cross-attention sub-model includes a text self-attention sub-network (hereinafter referred to as the self-attention sub-network), a first add, norm and mapping sub-network, a multi-head cross-attention sub-network (hereinafter referred to as the cross-attention sub-network), and a second add, norm and mapping sub-network. The first add, norm and mapping sub-network and the second add, norm and mapping sub-network are composed of an add, norm sub-network and a feedforward neural sub-network.

[0084] The input of the self-attention network is the audio features of each pronunciation unit in the target text, that is, the input of the self-attention sub-network is the output of the content encoding sub-model. The output of the self-attention sub-network is the adjusted audio features. The self-attention sub-network is used to strengthen the audio features of relatively important pronunciation units in each pronunciation unit and weaken the audio features of relatively unimportant pronunciation units through the self-attention mechanism, so that the pronunciation features of each pronunciation unit can better reflect the characteristics of the target text.

[0085] The input of the cross-attention sub-network is the superimposed audio features of each audio unit in the target audio clip and the adjusted pronunciation features of each pronunciation unit in the target text. The output of the cross-attention sub-network is the fused features of each pronunciation unit in the target text.

[0086] The cross-attention sub-network is used to use the superimposed audio features of each audio unit as the key and value, and the adjusted pronunciation features of each pronunciation unit as the query. Through the cross-attention mechanism, for each query, the key matching the query is searched in the key, and the value corresponding to the key (i.e., the target superimposed audio feature) is fused with the query to obtain a fused feature. That is, for each pronunciation unit, the cross-attention sub-network fuses the adjusted pronunciation features of the pronunciation unit with the target superimposed audio features to obtain the fused features of the pronunciation unit.

[0087] like Figure 3d As shown, the spectrogram decoding submodel includes multiple conversion subnetworks, a pre-processing (Pre-net) subnetwork, a post-processing (Post-net) subnetwork, and a vocoder (WaveGlow Vocoder).

[0088] The input of the preprocessing subnetwork is the original Mel-spectrograms as the original spectral features, and the output of the preprocessing subnetwork is the preprocessed original spectral features. The preprocessing subnetwork is used to preprocess the original spectral features. The preprocessing network consists of multiple linear subnetworks and a linear rectifier function (ReLU) subnetwork.

[0089] In a possible embodiment, the input of the conversion subnetwork is the preprocessed original spectral features and the fused features of each pronunciation unit. The output of the conversion subnetwork is the conversion of the fused features into spectral features. In this embodiment, the conversion subnetwork is used to convert the input fused features into spectral features based on the input original spectral features.

[0090] In another possible embodiment, the input of the conversion subnetwork is the preprocessed original spectral features, coarse-grained features and fused features of each pronunciation unit, and the coarse-grained features are the output of the aforementioned lower branch. The output of the conversion subnetwork is the fused features converted into spectral features. In this embodiment, the conversion subnetwork is used to convert the input fused features into spectral features based on the input original spectral features and coarse-grained features.

[0091] It is understandable that the fusion feature is obtained by fusing the superimposed audio feature and the pronunciation feature, and the superimposed audio feature is obtained by superimposing the coarse-grained audio feature and the fine-grained audio feature, so the fusion feature can reflect the coarse-grained audio feature to a certain extent. However, as explained above, the fusion feature is obtained by a series of calculations of the coarse-grained audio feature, so it is difficult for the fusion feature to accurately reflect the coarse-grained audio feature, and therefore it is difficult for the fusion feature to accurately reflect the overall characteristics of the target style.

[0092] Therefore, inputting the coarse-grained audio features into the conversion sub-network can enable the conversion sub-network to accurately refer to the overall characteristics of the target style when converting the fusion features into spectral features, so that the converted spectral features can accurately reflect the overall characteristics of the target style, so that the subsequently synthesized audio clips have the target style as a whole.

[0093] The input of the post-processing sub-network is the converted sound spectrum feature, and the output of the post-processing sub-network is the post-processed sound spectrum feature. The post-processing sub-network is used to post-process the converted sound spectrum feature. The post-processing sub-network is composed of multiple convolutional neural sub-networks.

[0094] The input of the vocoder is the post-processed spectral features, that is, the input of the vocoder is the output of the post-processing subnetwork, and the output of the vocoder is a synthesized audio segment with the target style and the speech content of the target text. The vocoder is used to convert the input audio features into audio segments.

[0095] Corresponding to the aforementioned style transfer synthesis method, the present disclosure also provides a speech synthesis model training method for training the speech synthesis model used in the aforementioned style transfer synthesis method.

[0096] The speech synthesis model training method provided in the present disclosure can be applied to any electronic device with speech synthesis model training capability, including but not limited to servers, personal computers, etc. Moreover, the speech synthesis model provided in the present disclosure and the style transfer synthesis method provided in the present disclosure can be applied to the same device or to different devices, and the present disclosure does not impose any restrictions on this.

[0097] The speech synthesis model training method provided in this disclosure can be found in Figure 4 ,include:

[0098] S401, inputting a sample audio segment and a sample text into an original model, wherein the sample text is the speech content of the sample audio segment.

[0099] The sample text is the speech content of the sample audio segment, which means that the text obtained by speech recognition of the sample audio segment is the sample text. For example, if a sample audio segment is an audio segment recorded when Zhang San says "AAAABBBB", the sample text is "AAAABBBB".

[0100] S402: For each audio unit in the sample audio segment, the coarse-grained audio features for characterizing the sample audio segment and the fine-grained audio features for characterizing the audio unit are superimposed using the original model to obtain superimposed audio features of the audio unit.

[0101] The principle of the original model is exactly the same as the structure and principle of the aforementioned speech synthesis model, and the only difference is that the model parameters of the original model are different from those of the speech synthesis model. Therefore, please refer to the aforementioned description of the style extraction sub-model, and no further description will be given here.

[0102] S403, extracting the pronunciation features of each pronunciation unit in the sample text through the original model.

[0103] The principle of this step is the same as that of the aforementioned content coding sub-model. Please refer to the aforementioned relevant description of the content coding sub-model, which will not be repeated here.

[0104] S404, for each pronunciation unit in the sample text, the pronunciation feature of the pronunciation unit and the target superimposed audio feature are fused through the original model to obtain the fused feature of the pronunciation unit, wherein the target superimposed audio feature is the superimposed audio feature that matches the pronunciation feature.

[0105] The principle of this step is the same as that of the aforementioned content-style cross-attention sub-model. Please refer to the aforementioned description of the content-style cross-attention sub-model, and will not be repeated here.

[0106] S405, converting the fusion features of each pronunciation unit in the sample text into predicted spectral features through the original model.

[0107] The principle of this step is the same as that of the aforementioned spectrum decoding sub-model. Please refer to the aforementioned relevant description of the spectrum decoding sub-model, which will not be repeated here.

[0108] S406, adjusting the model parameters of the original model according to the difference between the predicted spectral features and the actual spectral features of the sample audio features.

[0109] It is understandable that since the sample text input to the original network is the content of the sample audio clip, the audio clip obtained by the original network transferring the semantic style of the sample audio clip to the sample text should be the sample audio clip. In other words, if the original network can accurately perform style transfer synthesis, the predicted spectral features should be the same as the real spectral features of the sample audio clip. The reason for the difference between the predicted spectral features and the real spectral features is that the original model cannot accurately perform style transfer synthesis.

[0110] Therefore, the difference between the predicted spectral features and the actual spectral features can be used to guide the adjustment of the model parameters of the original model, so that the model parameters of the original model are adjusted in the direction of reducing the difference, thereby training a speech synthesis model that can accurately perform style transfer synthesis.

[0111] S407, obtaining a new sample audio segment and a new sample text, returning to execute S401, until the first convergence condition is reached, and using the adjusted original model as the speech synthesis model.

[0112] The newly acquired sample text should be the content of the newly acquired sample audio segment, and the newly acquired sample audio segment is different from the previous sample audio segment. The first convergence condition can be set by the user according to actual needs or requirements. For example, the first convergence condition can be that the convergence of the model parameters of the original model reaches a preset convergence threshold. The first convergence condition can also be that the number of sample audio segments that have been used reaches a preset number threshold.

[0113] This embodiment is selected to supervise the original model through the real spectral features of the sample audio, and because the sample text is the speech content of the sample audio clip, the superimposed audio features extracted by the original model can better match the audio features of each pronunciation unit in the sample text, so that the trained speech synthesis model can better learn the matching relationship between audio features and pronunciation features, so as to synthesize an audio clip with a better target style in the style transfer synthesis process.

[0114] It is understandable that since the sample text is the speech content of the sample audio clip, the number of pronunciation units in the sample text should be similar to or even the same as the number of audio units in the sample audio clip. When using a speech synthesis model for style transfer synthesis, the target text is not the speech content of the target audio clip, so the number of pronunciation units in the target text may differ greatly from the number of audio units in the target audio clip.

[0115] In order to enable the speech synthesis model to accurately implement style transfer synthesis even when the number of pronunciation units and audio units differs greatly, in a possible embodiment, the target superimposed audio feature in the aforementioned S404 is a filtered audio feature that matches the pronunciation feature, and the filtered audio feature is a partial superimposed audio feature extracted from all superimposed audio features.

[0116] The extraction method is random extraction, and the number of superimposed audio features extracted can be set according to the actual needs or experience of the user, such as extracting 80% of the superimposed audio features as the filtered audio features, or extracting 90% of the superimposed audio features as the filtered audio features.

[0117] By selecting this embodiment, during the training process, some overlapping audio features can be extracted so that the audio features are only fused with the selected partially overlapping audio features through a cross-attention mechanism, so that the trained speech synthesis model can learn how to achieve style transfer synthesis when the number of pronunciation units and audio units differs greatly, that is, the speech synthesis model can accurately achieve style transfer synthesis even when the number of pronunciation units and audio units differs greatly.

[0118] In the aforementioned S407, the new sample audio segment obtained may have the same style as the previous sample audio segment (ie, an audio segment of the same person), or may have a different voice style from the previous sample audio segment (ie, an audio segment of a different person).

[0119] In a possible embodiment, obtaining a new sample segment in S407 is implemented in the following manner:

[0120] If the second convergence condition is not met, a new sample audio segment is obtained from the first sample data set; if the second convergence condition is met, a new sample audio segment is obtained from the second sample data set.

[0121] The first sample data set includes an audio clip of a first sample person, and the second sample data set includes audio clips of multiple sample persons. In this embodiment, the sample audio clip when S401 is executed for the first time is the audio clip of the first sample person.

[0122] The difficulty of achieving the second convergence condition is lower than that of the first convergence condition, that is, when the second convergence condition is achieved, the first convergence condition has not yet been achieved, and when the first convergence condition is achieved, the second convergence condition has been achieved.

[0123] By selecting this embodiment, the original model can be first trained with the audio features of the first sample person, so that the original model learns how to transfer the voice style of the first sample person to a specific text, and then the original model is trained with the audio clips of multiple sample persons, so that the original model learns how to transfer different voice styles to a specific text. Since the original model has been pre-trained before learning how to transfer different voice styles to a specific text, and has learned how to transfer the voice style of the first sample person to a specific text, only a small number of audio clips of different persons are needed to complete the training of the original model, effectively reducing the difficulty of obtaining sample audio clips.

[0124] See also Figure 5 , Figure 5 The figure is a schematic diagram of a structure of a style transfer synthesis device provided by the present disclosure, comprising:

[0125] A first input module 501 is used to input a target text and a target audio segment with a target voice style into a speech synthesis model pre-trained with sample text and sample audio segment;

[0126] A style extraction module 502 is used to superimpose the coarse-grained audio features used to characterize the target audio segment and the fine-grained audio features used to characterize the audio unit for each audio unit in the target audio segment through the style extraction sub-model of the speech synthesis model to obtain the superimposed audio features of the audio unit;

[0127] A content encoding module 503, configured to extract the pronunciation features of each pronunciation unit in the target text through the content encoding sub-model of the speech synthesis model;

[0128] A content-style cross-attention module 504 is used to fuse the pronunciation features of each pronunciation unit in the target text and the target superimposed audio features through the content-style cross-attention sub-model of the speech synthesis model to obtain a fused feature of the pronunciation unit, wherein the target superimposed audio feature is a superimposed audio feature that matches the pronunciation feature;

[0129] The spectrogram decoding module 505 is used to synthesize an audio segment having the target speech style and the speech content of the target text according to the fusion features of each pronunciation unit in the target text through the spectrogram decoding submodel of the speech synthesis model.

[0130] This embodiment is selected to obtain superimposed audio features by superimposing audio features of different granularities, so that the superimposed audio features can not only reflect the overall characteristics of the target style, but also reflect the detailed characteristics of the target audio segment. Then, by using the cross-attention mechanism, the audio features of each pronunciation unit are fused with the matching superimposed audio features. The obtained fused features can reflect the speech content included in the target text on the one hand, and the overall and detailed characteristics of the target style on the other hand. Moreover, since the audio features are fused with the matching superimposed audio features, the detailed characteristics reflected by the fused features of each pronunciation unit can reflect the detailed characteristics of reading the pronunciation unit in the target style. Therefore, in the audio segment synthesized according to the fused features, not only the overall acoustic characteristics are similar to the target style, but also the pronunciation of each pronunciation unit is similar to the target style, that is, an audio segment with the target style in both overall and detailed aspects can be synthesized.

[0131] In a possible embodiment, the style extraction module 502 superimposes, for each audio unit in the target audio segment, a coarse-grained audio feature for characterizing the target audio segment and a fine-grained audio feature for characterizing the audio unit through the style extraction sub-model of the speech synthesis model to obtain the superimposed audio feature of the audio unit, including:

[0132] Extracting, by means of a style extraction module of the speech synthesis model, average audio features of all audio frames in the target audio segment as coarse-grained audio features;

[0133] By means of the style extraction module, for each audio unit in the target audio segment, average audio features of all audio frames in the audio unit are extracted as fine-grained audio features of the audio unit;

[0134] Through the style extraction module, for each audio unit in the target audio segment, the fine-grained audio feature and the coarse-grained audio feature of the audio unit are added to obtain the superimposed audio feature of the audio unit.

[0135] In a possible embodiment, the content-style cross-attention module 504 uses the content-style cross-attention sub-model of the speech synthesis model to fuse the pronunciation features of each pronunciation unit in the target text and the target superimposed audio features to obtain the fused features of the pronunciation unit, including:

[0136] Inputting the pronunciation features of each pronunciation unit in the target text into the self-attention sub-network of the content-style cross-attention sub-model in the speech synthesis model to obtain the adjusted pronunciation features output by the sub-attention sub-network;

[0137] Through the cross-attention sub-network of the content cross-sub-model, for each pronunciation unit in the target text, the adjusted pronunciation features of the pronunciation unit and the target superimposed audio features are fused to obtain the fused features of the pronunciation unit, wherein the target superimposed audio features are superimposed audio features that match the adjusted pronunciation features.

[0138] In a possible embodiment, the spectrogram decoding module 505 synthesizes an audio segment having the target speech style and the speech content of the target text according to the fusion feature of each pronunciation unit in the target text through the spectrogram decoding submodel of the speech synthesis model, including:

[0139] Inputting the fusion feature and the coarse-grained audio feature of each pronunciation unit in the target text into the spectrogram decoding sub-model of the speech synthesis model to obtain the spectrogram feature output by the spectrogram decoding sub-network;

[0140] The spectral features are converted into an audio segment having the target voice style and the voice content being the target text.

[0141] See also Figure 6 , Figure 6 The figure is a schematic diagram of a structure of a training device for a speech synthesis model provided by an embodiment of the present invention, which may include:

[0142] The second input module 601 is used to input a sample audio segment and a sample text into the original model, wherein the sample text is the speech content of the sample audio segment;

[0143] A first original module 602 is used to superimpose the coarse-grained audio features used to characterize the sample audio segment and the fine-grained audio features used to characterize the audio unit for each audio unit in the sample audio segment through the original model to obtain a superimposed audio feature of the audio unit;

[0144] The second original module 603 is used to extract the pronunciation features of each pronunciation unit in the sample text through the original model;

[0145] The third original module 604 is used to fuse the pronunciation feature of each pronunciation unit in the sample text with the target superimposed audio feature through the original model to obtain the fused feature of the pronunciation unit, wherein the target superimposed audio feature is a superimposed audio feature matching the pronunciation feature;

[0146] A fourth original module 605, configured to convert the fusion feature of each pronunciation unit in the sample text into a predicted spectral feature through the original model;

[0147] A parameter adjustment module 606, configured to adjust the model parameters of the original model according to the difference between the predicted spectral features and the real spectral features of the sample audio segment;

[0148] The acquisition module 607 is used to obtain a new sample audio segment and a new sample text, and return to execute the step of inputting the sample audio segment and the sample text into the original model until the first convergence condition is reached, and the adjusted original model is used as the speech synthesis model.

[0149] In a possible embodiment, it further includes:

[0150] A superimposed feature extraction module, used for extracting some superimposed audio features from all superimposed audio features as filtered audio features;

[0151] The target superimposed audio feature is a filtered audio feature that matches the pronunciation feature.

[0152] In a possible embodiment, the sample audio segment is initially an audio segment of a first sample person;

[0153] The acquisition module 607 acquires a new sample audio segment, including:

[0154] If the second convergence condition is not met, obtaining a new sample audio segment from a first sample data set, where the first sample data set includes the audio segment of the first sample person;

[0155] If the second convergence condition is met, a new sample audio segment and a new sample text are obtained from a second sample data set, where the second sample data set includes audio segments of a plurality of sample persons.

[0156] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0157] It should be noted that the sample audio clips in this embodiment are from public datasets, such as LJSpeech (a public dataset) and VCTK (a public dataset).

[0158] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0159] Figure 7A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0160] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0161] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0162] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as a style transfer synthesis method or a training method for a speech synthesis model. For example, in some embodiments, the style transfer synthesis method or the training method for a speech synthesis model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the style transfer synthesis method or the training method for the speech synthesis model described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the style transfer synthesis method or the speech synthesis model training method in any other appropriate manner (for example, by means of firmware).

[0163] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0164] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0165] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0166] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0167] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0168] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0169] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0170] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A style transfer synthesis method, comprising: Inputting the target text and the target audio clip with the target voice style into a speech synthesis model that has been pre-trained with the sample text and the sample audio clip; Using the style extraction sub-model of the speech synthesis model, for each audio unit in the target audio segment, superimpose the coarse-grained audio features used to characterize the target audio segment and the fine-grained audio features used to characterize the audio unit to obtain the superimposed audio features of the audio unit; Extracting pronunciation features of each pronunciation unit in the target text through the content encoding sub-model of the speech synthesis model; By using the content-style cross-attention sub-model of the speech synthesis model, for each pronunciation unit in the target text, the pronunciation feature of the pronunciation unit and the target superimposed audio feature are fused to obtain the fused feature of the pronunciation unit, wherein the target superimposed audio feature is a superimposed audio feature that matches the pronunciation feature; An audio segment having the target speech style and the speech content of the target text is synthesized through the spectrogram decoding submodel of the speech synthesis model according to the fusion features of each pronunciation unit in the target text, specifically comprising: inputting the fusion features of each pronunciation unit in the target text and the coarse-grained audio features into the spectrogram decoding submodel of the speech synthesis model to obtain the spectrogram features output by the spectrogram decoding subnetwork; and converting the spectrogram features into an audio segment having the target speech style and the speech content of the target text.

2. The method according to claim 1, wherein: The method of superimposing, for each audio unit in the target audio segment, a coarse-grained audio feature for characterizing the target audio segment and a fine-grained audio feature for characterizing the audio unit through the style extraction sub-model of the speech synthesis model to obtain the superimposed audio feature of the audio unit includes: Extracting, by means of a style extraction module of the speech synthesis model, average audio features of all audio frames in the target audio segment as coarse-grained audio features; By means of the style extraction module, for each audio unit in the target audio segment, average audio features of all audio frames in the audio unit are extracted as fine-grained audio features of the audio unit; Through the style extraction module, for each audio unit in the target audio segment, the fine-grained audio feature and the coarse-grained audio feature of the audio unit are added to obtain the superimposed audio feature of the audio unit.

3. The method according to claim 1, wherein: The content-style cross-attention sub-model of the speech synthesis model is used to fuse the pronunciation features of each pronunciation unit in the target text and the target superimposed audio features to obtain the fusion features of the pronunciation unit, including: Inputting the pronunciation features of each pronunciation unit in the target text into the self-attention sub-network of the content-style cross-attention sub-model in the speech synthesis model to obtain the adjusted pronunciation features output by the self-attention sub-network; Through the cross-attention sub-network of the content cross-sub-model, for each pronunciation unit in the target text, the adjusted pronunciation features of the pronunciation unit and the target superimposed audio features are fused to obtain the fused features of the pronunciation unit, wherein the target superimposed audio features are superimposed audio features that match the adjusted pronunciation features.

4. A method for training a speech synthesis model, comprising: Inputting a sample audio segment and a sample text into the original model, wherein the sample text is the speech content of the sample audio segment; Using the original model, for each audio unit in the sample audio segment, superimpose the coarse-grained audio features used to characterize the sample audio segment and the fine-grained audio features used to characterize the audio unit to obtain superimposed audio features of the audio unit; Extracting the pronunciation features of each pronunciation unit in the sample text through the original model; By using the original model, for each pronunciation unit in the sample text, the pronunciation feature of the pronunciation unit and the target superimposed audio feature are fused to obtain the fused feature of the pronunciation unit, wherein the target superimposed audio feature is a superimposed audio feature that matches the pronunciation feature; The method of converting the fusion feature of each pronunciation unit in the sample text into a predicted spectral feature through the original model specifically includes: converting the fusion feature of each pronunciation unit in the sample text and the coarse-grained audio feature into a predicted spectral feature through the original model; Adjusting the model parameters of the original model according to the difference between the predicted spectral features and the real spectral features of the sample audio segment; A new sample audio segment and a new sample text are obtained, and the step of inputting the sample audio segment and the sample text into the original model is returned to be executed until the first convergence condition is reached, and the adjusted original model is used as the speech synthesis model.

5. The method according to claim 4, further comprising: Extracting some superimposed audio features from all superimposed audio features as filtered audio features; The target superimposed audio feature is a filtered audio feature that matches the pronunciation feature.

6. The method according to claim 5, wherein: The sample audio segment is initially an audio segment of a first sample person; The step of obtaining a new sample audio segment includes: If the second convergence condition is not met, obtaining a new sample audio segment from a first sample data set, where the first sample data set includes the audio segment of the first sample person; If the second convergence condition is met, a new sample audio segment and a new sample text are obtained from a second sample data set, where the second sample data set includes audio segments of a plurality of sample persons.

7. A style transfer synthesis device, comprising: A first input module is used to input the target text and the target audio segment with the target voice style into a speech synthesis model pre-trained with the sample text and the sample audio segment; A style extraction module, configured to superimpose, for each audio unit in the target audio segment, a coarse-grained audio feature for characterizing the target audio segment and a fine-grained audio feature for characterizing the audio unit, through a style extraction sub-model of the speech synthesis model, to obtain a superimposed audio feature of the audio unit; A content encoding module, used for extracting the pronunciation features of each pronunciation unit in the target text through the content encoding sub-model of the speech synthesis model; A content-style cross-attention module is used to fuse the pronunciation features of each pronunciation unit in the target text and the target superimposed audio features through the content-style cross-attention sub-model of the speech synthesis model to obtain the fused features of the pronunciation unit, wherein the target superimposed audio features are superimposed audio features that match the pronunciation features; The spectrogram decoding module is used to synthesize an audio segment having the target speech style and the speech content of the target text according to the fusion features of each pronunciation unit in the target text through the spectrogram decoding submodel of the speech synthesis model, specifically comprising: inputting the fusion features of each pronunciation unit in the target text and the coarse-grained audio features into the spectrogram decoding submodel of the speech synthesis model to obtain the spectrogram features output by the spectrogram decoding subnetwork; and converting the spectrogram features into an audio segment having the target speech style and the speech content of the target text.

8. The device according to claim 7, wherein: The style extraction module, through the style extraction sub-model of the speech synthesis model, superimposes, for each audio unit in the target audio segment, a coarse-grained audio feature for characterizing the target audio segment and a fine-grained audio feature for characterizing the audio unit to obtain a superimposed audio feature of the audio unit, including: Extracting, by means of a style extraction module of the speech synthesis model, average audio features of all audio frames in the target audio segment as coarse-grained audio features; By means of the style extraction module, for each audio unit in the target audio segment, average audio features of all audio frames in the audio unit are extracted as fine-grained audio features of the audio unit; Through the style extraction module, for each audio unit in the target audio segment, the fine-grained audio feature and the coarse-grained audio feature of the audio unit are added to obtain the superimposed audio feature of the audio unit.

9. The device according to claim 7, wherein: The content-style cross-attention module uses the content-style cross-attention sub-model of the speech synthesis model to fuse the pronunciation features of each pronunciation unit in the target text and the target superimposed audio features to obtain the fusion features of the pronunciation unit, including: Inputting the pronunciation features of each pronunciation unit in the target text into the self-attention sub-network of the content-style cross-attention sub-model in the speech synthesis model to obtain the adjusted pronunciation features output by the self-attention sub-network; Through the cross-attention sub-network of the content cross-sub-model, for each pronunciation unit in the target text, the adjusted pronunciation features of the pronunciation unit and the target superimposed audio features are fused to obtain the fused features of the pronunciation unit, wherein the target superimposed audio features are superimposed audio features that match the adjusted pronunciation features.

10. A training device for a speech synthesis model, comprising: A second input module is used to input a sample audio segment and a sample text into the original model, wherein the sample text is the speech content of the sample audio segment; A first primitive module is used to superimpose, for each audio unit in the sample audio segment, a coarse-grained audio feature for characterizing the sample audio segment and a fine-grained audio feature for characterizing the audio unit through the primitive model to obtain a superimposed audio feature of the audio unit; A second primitive module, used for extracting the pronunciation features of each pronunciation unit in the sample text through the primitive model; A third original module is used to fuse the pronunciation feature of each pronunciation unit in the sample text with the target superimposed audio feature through the original model to obtain a fused feature of the pronunciation unit, wherein the target superimposed audio feature is a superimposed audio feature that matches the pronunciation feature; The fourth original module is used to convert the fusion feature of each pronunciation unit in the sample text into a predicted spectral feature through the original model, specifically comprising: converting the fusion feature of each pronunciation unit in the sample text and the coarse-grained audio feature into a predicted spectral feature through the original model; A parameter adjustment module, used for adjusting the model parameters of the original model according to the difference between the predicted spectral features and the real spectral features of the sample audio segment; The acquisition module is used to acquire new sample audio segments and new sample texts, and return to execute the step of inputting the sample audio segments and sample texts into the original model until the first convergence condition is reached, and the adjusted original model is used as the speech synthesis model.

11. The apparatus according to claim 10, further comprising: A superimposed feature extraction module, used for extracting some superimposed audio features from all superimposed audio features as filtered audio features; The target superimposed audio feature is a filtered audio feature that matches the pronunciation feature.

12. The device according to claim 10, wherein: The sample audio segment is initially an audio segment of a first sample person; The acquisition module acquires a new sample audio segment, including: If the second convergence condition is not met, obtaining a new sample audio segment from a first sample data set, where the first sample data set includes the audio segment of the first sample person; If the second convergence condition is met, a new sample audio segment and a new sample text are obtained from a second sample data set, where the second sample data set includes audio segments of a plurality of sample persons.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-3 or 4-6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-3 or 4-6.

15. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1-3 or 4-6.

Images