Speech synthesis apparatus, method, mobile terminal, and storage medium
By adding an attention-based feature extraction module to the speech synthesis device, user-related pronunciation features are filtered out, solving the problem of inaccurate speech caused by insufficient input sample information in the existing technology, and achieving efficient speech synthesis effect.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-28
- Publication Date
- 2026-03-31
AI Technical Summary
Existing speech synthesis devices require a large amount of input sample information to achieve fluent speech. When users cannot provide enough speech samples, problems such as hoarse voice and voices that are very different from the user's own often occur.
An attention-based feature extraction module is added to the speech synthesis device. Common accent and rhythm-related pronunciation features are selected through a pre-trained model and combined with user features for speech synthesis, reducing the amount of sample information to be input.
It achieves accurate extraction of user-related accent and prosodic features with a small amount of input sample information, avoiding the problem of voices differing too much from the actual person, and improving the accuracy and efficiency of speech synthesis.
Smart Images
Figure CN116580695B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computer, in particular to a speech synthesis device, method, mobile terminal and storage medium. BACKGROUND
[0002] Speech synthesis refers to a method of converting text information into target speaker voice information. The existing speech synthesis device is divided into three modules: front end, acoustic model and vocoder. The front end converts the text information input by the user into corresponding phoneme features, the acoustic model is trained by the user's stored recording sample information, and corresponding acoustic features are generated according to the phoneme features and sent to the vocoder. The vocoder performs speech synthesis according to the acoustic features to obtain synthesized speech corresponding to the text information.
[0003] However, the existing speech synthesis device needs a large number of recording sample information to realize fluent speech. When the user cannot provide enough speech samples, the problem of hoarse voice and large difference between the voice and the user often occurs. SUMMARY
[0004] The embodiments of the present application provide a speech synthesis device, method, mobile terminal and storage medium. The speech synthesis device provided by the embodiments of the present application adds a first feature extraction module for extracting pronunciation features in recording sample information on the basis of the existing speech synthesis device. The pre-training module provided by the present application pre-trains the feature extraction model according to a large amount of general sample information, and the feature extraction model itself is constructed based on the attention mechanism, which can selectively filter out general pronunciation features such as accent and prosody from a large number of recording sample information and focus on these pronunciation features to obtain user features. The pre-trained feature extraction model has reasonable weight and other parameter settings, which can accurately extract pronunciation features such as accent and prosody related to the user from a small amount of recording sample information, avoiding the problem that the existing speech synthesis device needs a large amount of user recording voice information and the voice is too different from the user when the sample is small.
[0005] To solve the above technical problems, the present application provides a speech synthesis device, comprising:
[0006] A pre-training module is configured to acquire general sample information and pre-train feature extraction according to a pre-set feature extraction model, wherein the feature extraction model is constructed based on the attention mechanism;
[0007] A first feature extraction module is configured to acquire recording sample information and extract features from the recording sample information according to the pre-trained feature extraction model to obtain user features;
[0008] A phoneme processing module is configured to acquire phoneme features and obtain a first phoneme processing result according to the phoneme features and the user features.
[0009] The duration prediction module is used to predict the frame length of the first phoneme processing result and obtain the first duration prediction result;
[0010] The acoustic decoding module is used to generate a first acoustic feature based on the first duration prediction result and the first phoneme processing result;
[0011] The first vocode module is used to synthesize speech from the first acoustic feature using a vocoder to obtain the first audio data.
[0012] Optionally, the first feature extraction module provided in this application includes:
[0013] The preprocessing submodule is used to acquire the input sample information and pre-encode the input sample information to obtain a first pre-encoding result;
[0014] The second feature extraction submodule is used to extract features from the first precoding result according to the attention mechanism to obtain the first user features.
[0015] Optionally, the second feature extraction submodule provided in this application includes:
[0016] The third feature extraction unit is used to extract features from the first precoding result according to the attention mechanism and obtain the feature extraction result.
[0017] The encoding unit is used to perform encoding processing based on the feature extraction results to obtain the first user feature.
[0018] Optionally, the first feature extraction module provided in this application further includes:
[0019] The preprocessing submodule is also used to perform random masking on the input sample information to obtain a random masking result, and to precode the random masking result to obtain a second precoding result;
[0020] The encoding submodule is used to extract features from the second precoding result according to the attention mechanism to obtain the second user features;
[0021] The prediction submodule is used to perform mask restoration based on the second user features to obtain mask restoration information, and is also used to perform loss function calculation based on the mask restoration information and the input sample information to obtain the loss function calculation result.
[0022] Optionally, the phoneme processing module provided in this application includes:
[0023] A phoneme feature acquisition unit is used to acquire user text information corresponding to the phoneme feature, and acquire the phoneme feature based on the user text information and a pre-set reference phoneme;
[0024] The phoneme feature processing unit is used to obtain the first phoneme processing result by splicing the phoneme features and the user features.
[0025] Optionally, the speech synthesis apparatus provided in this application further includes:
[0026] The fourth feature extraction module is used to obtain the first phoneme processing result and perform feature extraction based on the first phoneme processing result according to the attention mechanism to obtain the second phoneme processing result;
[0027] The duration prediction module is also used to predict the frame length of the second phoneme processing result to obtain the second duration prediction result;
[0028] The acoustic decoding module is used to generate a second acoustic feature based on the second duration prediction result and the second phoneme processing result;
[0029] The first vocode module is used to synthesize speech from the second acoustic feature using a vocoder to obtain the second audio data from the first audio data.
[0030] Optionally, the speech synthesis apparatus provided in this application further includes:
[0031] The second vocode module is used to upsample the first duration prediction result and the first phoneme processing result through a neural vocoder, and then synthesize the third audio data.
[0032] This application also provides a speech synthesis method, including:
[0033] Obtain general sample information and perform feature extraction pre-training based on a pre-set feature extraction model, wherein the feature extraction model is built based on an attention mechanism;
[0034] The system acquires input sample information and performs feature extraction on the input sample information based on the pre-trained feature extraction model to obtain user features.
[0035] Obtain phoneme features and obtain the first phoneme processing result based on the phoneme features and the user features;
[0036] Predict the frame length of the first phoneme processing result to obtain the first duration prediction result;
[0037] A first acoustic feature is generated based on the first duration prediction result and the first phoneme processing result;
[0038] The first acoustic feature is synthesized into speech using a vocoder to obtain the first audio data.
[0039] This application also provides a mobile terminal, including:
[0040] At least one processor; and,
[0041] A memory that is communicatively connected to at least one processor; wherein,
[0042] The memory stores instructions that can be executed by at least one processor, which enables the at least one processor to implement the speech synthesis method described above.
[0043] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the speech synthesis method described above.
[0044] The pre-training module provided in this application pre-trains the feature extraction model based on a large amount of general sample information. The feature extraction model itself is built on an attention mechanism, enabling it to selectively filter out general accent and prosody-related pronunciation features from a large amount of input sample information and focus on these features to obtain user features. The pre-trained feature extraction model has reasonably set weights and other parameters, enabling it to accurately extract user-related accent and prosody-related pronunciation features from a small amount of input sample information. This avoids the problem in existing speech synthesis devices where a large amount of user-inputted speech information is required, and the voice differs significantly from the user's actual voice when the sample size is small.
[0045] The above description is merely an overview of the technical solution provided in this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are described below. Attached Figure Description
[0046] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.
[0047] Figure 1 This is an example of a sound replication model provided in this application;
[0048] Figure 2 This is one of the schematic diagrams of the speech synthesis device provided in the embodiments of this application;
[0049] Figure 3 This is the second schematic diagram of the speech synthesis device provided in the embodiments of this application;
[0050] Figure 4 This is a schematic diagram of a precoding process provided in this application;
[0051] Figure 5 This is a schematic diagram of a feature extraction process based on an attention mechanism provided in this application;
[0052] Figure 6 This is the third schematic diagram of the speech synthesis device provided in the embodiments of this application;
[0053] Figure 7 This is the fourth schematic diagram of the speech synthesis device provided in the embodiments of this application;
[0054] Figure 8 This is a schematic diagram of a mask restoration process provided in this application;
[0055] Figure 9 This is the fifth schematic diagram of the speech synthesis device provided in the embodiments of this application;
[0056] Figure 10 This is the sixth schematic diagram of the speech synthesis device provided in the embodiments of this application;
[0057] Figure 11 This is the seventh schematic diagram of the speech synthesis device provided in the embodiments of this application;
[0058] Figure 12 This is a schematic diagram of an end-to-end sound cloning module provided in this application;
[0059] Figure 13 This is one of the schematic diagrams of the speech synthesis method provided in the embodiments of this application;
[0060] Figure 14 This is the second schematic diagram of the speech synthesis method provided in the embodiments of this application;
[0061] Figure 15 This is the third schematic diagram of the speech synthesis method provided in the embodiments of this application;
[0062] Figure 16 This is the fourth schematic diagram of the speech synthesis method provided in the embodiments of this application;
[0063] Figure 17 This is the fifth schematic diagram of the speech synthesis method provided in the embodiments of this application;
[0064] Figure 18 This is the sixth schematic diagram of the speech synthesis method provided in the embodiments of this application;
[0065] Figure 19 This is the seventh schematic diagram of the speech synthesis method provided in the embodiments of this application;
[0066] Figure 20 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0067] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.
[0068] The speech synthesis device provided in this application is applied in the field of speech synthesis technology, specifically for the personalized application of speech synthesis, namely voice replication. The following is an explanation of some terms used in this application:
[0069] Speech synthesis converts input text information into the user's speech information and outputs it. Existing speech synthesis devices or systems typically consist of three modules: a front-end, an acoustic model, and a vocoder. The front-end converts the input text information into phoneme features or phoneme sequences. The acoustic model is trained based on pre-recorded speech information from the user and is used to generate acoustic features based on the phoneme features or phoneme sequences. The acoustic features are then passed through a vocoder to obtain synthesized speech that meets the user's requirements.
[0070] Voice replication or voice cloning is a personalized application of speech synthesis. Compared to traditional speech synthesis, it generates a synthetic model that closely resembles the user's timbre, pronunciation style, and other vocal characteristics based on a small amount of user-inputted recordings. This synthetic model is then used to replicate phoneme features or phoneme sequences transmitted from the front end, resulting in a voice replication result that closely resembles the user's. Specifically, the voice replication model decouples the acoustic model and the vocoder, where the acoustic model is as follows: Figure 1 As shown, two preprocessing modules preprocess phoneme features and speaker information respectively to obtain phoneme features and user features. Based on the phoneme features, user features are added through concatenation to obtain the phoneme processing result carrying user features. Compared with traditional speech synthesis devices, this method requires only a small amount of speaker information to synthesize synthesized speech that closely resembles the user's pronunciation habits and features, reducing the number of speaker information samples. However, existing voice replication methods have low user feature extraction efficiency, and acoustic features require Mel-spectrum or cepstral spectra as intermediate results, causing some information loss. Therefore, they require a considerable number, such as more than 100 user speech samples, resulting in limited applicability.
[0071] The first feature extraction module in the speech synthesis device provided in this application extracts features from the user-input sample information based on an attention mechanism. This attention mechanism model structure references the selective visual attention mechanism of humans, conditionally selecting a small amount of information from a large amount of information according to weights obtained through training, and focusing on these selected pieces of information while ignoring the rest. The process of focusing on a small amount of information is often reflected in the calculation of weight coefficients; the more important and valuable the information, the greater its weight.
[0072] The input information for the speech synthesis device provided in this application includes phoneme features or phoneme sequences, where a phoneme refers to the smallest unit of articulation in the input acoustic model. Phoneme features or phoneme sequences refer to a sequence or set obtained by arranging multiple phonemes in order.
[0073] In the speech synthesis apparatus provided in this application, first audio data is obtained by synthesizing the first acoustic feature using a vocoder. The acoustic feature refers to the collective acoustic characteristics of multiple elements in a sound, such as formant frequency, formant intensity, bandwidth, fundamental frequency representing the prosodic characteristics of speech, average speech power, and other information.
[0074] The speech synthesis device provided in this application adds a first feature extraction module to the existing speech synthesis device to extract pronunciation features from the input sample information. This first feature extraction module is based on an attention mechanism, enabling it to selectively filter out a small number of pronunciation features related to the user's accent and rhythm from a large amount of input sample information, and focus on these pronunciation features to obtain user features. Because the first feature extraction module can quickly select pronunciation features, it can obtain user features that closely resemble the user's actual speech without requiring a large amount of input sample information as training data, thus avoiding the problem of inaccurate replication results in existing speech synthesis devices when there is limited speech data.
[0075] The first embodiment of this application provides a speech synthesis device, such as... Figure 2 As shown, it includes:
[0076] The pre-training module 108 is used to acquire general sample information and perform feature extraction pre-training based on a pre-set feature extraction model, wherein the feature extraction model is built based on an attention mechanism.
[0077] The first feature extraction module 101 is used to acquire input sample information and extract features from the input sample information according to a pre-set attention mechanism to obtain user features;
[0078] Phoneme processing module 102 is used to acquire phoneme features and acquire a first phoneme processing result based on the phoneme features and the user features;
[0079] The duration prediction module 103 is used to predict the frame length of the first phoneme processing result and obtain the first duration prediction result;
[0080] Acoustic decoding module 104 is used to generate a first acoustic feature based on the first duration prediction result and the first phoneme processing result;
[0081] The first vocode module 105 is used to perform speech synthesis on the first acoustic feature through a vocoder to obtain the first audio data.
[0082] Specifically, in the speech synthesis device provided in this application, the pre-training module 108 inputs a large number of general user features into the user feature extraction model for feature extraction training. The trained user feature extraction model masters a large number of pronunciation features of different people, and the weight parameters based on the attention mechanism, after sufficient training, can also more accurately filter out a small amount of information related to pronunciation features from the sample information. The first feature extraction module 101 inputs the input sample information of the user into the pre-trained user feature extraction model to obtain user features. Among them, the input sample information can be a random one from the user's pre-recorded speech information, and the attention mechanism, through weight settings, focuses on filtering the user's accent, rhythm, and other pronunciation features in this speech information, thereby obtaining user features that are close to the user's speech.
[0083] Subsequently, the front-end connected to the speech synthesis device provided in this application converts the user-input text information into corresponding phoneme features or phoneme sequences based on common phonemes. Since the front-end of the speech synthesis device provided in this application is prior art, it will not be described in detail here. The phoneme processing module 102 combines the phoneme features and the user features obtained by the first feature extraction module through concatenation or addition, and encodes the combined phoneme features to obtain the first phoneme processing result. The duration prediction module 103 predicts the frame length of each phoneme in the combined phoneme features. The acoustic decoding module 104 decodes the encoded first phoneme processing result and the first duration prediction result obtained by the duration prediction module 103 to obtain the first acoustic feature. Since the decoding module often processes the data through decoders such as Mel decoders, the obtained first acoustic feature is often in the form of a Mel spectrum or cepstral spectrogram. Finally, the first vocoder module 105 performs speech synthesis on the first acoustic feature through independent logic or multiple vocoders to obtain the final first audio data.
[0084] Based on the above implementation methods, such as Figure 3 As shown, the user features include first user features. The first feature extraction module 101 provided in this application includes:
[0085] The preprocessing submodule 111 is used to acquire the input sample information and pre-encode the input sample information to obtain a first pre-encoding result;
[0086] The second feature extraction submodule 112 is used to extract features from the first precoding result according to the attention mechanism to obtain the first user features.
[0087] Specifically, the first feature extraction module includes a preprocessing submodule 111 and a second feature extraction submodule 112. The preprocessing submodule 111 is used to pre-encode the input sample information to obtain a first pre-encoding result. It should be emphasized that the pre-encoding processing method in the preprocessing submodule 111 is not limited in this application; for example… Figure 4 The process involves performing random masking, linear layer processing, position encoding, layer normalization, and dropout on the input sample information to obtain the first pre-encoded result after preliminary encoding processing.
[0088] The second feature extraction submodule 112 is used to further encode the first precoding result and extract user features based on an attention mechanism to obtain the first user features. The second feature extraction submodule 112 can employ methods such as... Figure 5 The processing is performed as shown. Specifically, the first precoding result, as an acoustic feature, is first input into the attention module. It is then filtered according to the weights set by the attention mechanism, focusing on information carrying user pronunciation features. Subsequently, pointwise feedforward processing and conditional layer normalization are performed to obtain the first user feature. The second feature extraction submodule can be repeated multiple times to further avoid the influence of data unrelated to user pronunciation features in the first user feature on the acoustic model construction.
[0089] Based on the above implementation method, by encoding user features, the stability of the output user features in terms of order of magnitude and feature distribution is ensured, thereby ensuring the convergence of the corresponding acoustic model construction.
[0090] Based on the above implementation methods, such as Figure 6 As shown, the second feature extraction submodule 112 provided in this application includes:
[0091] The third feature extraction unit 121 is used to extract features from the first precoding result according to the attention mechanism to obtain the feature extraction result;
[0092] The encoding unit 122 is used to perform encoding processing based on the feature extraction result to obtain the first user feature.
[0093] Specifically, the second feature extraction submodule 112 includes a third feature extraction unit 121 and an encoding unit 122. The third feature extraction unit 121 is used to obtain feature extraction results according to the attention mechanism, while the encoding unit encodes the feature extraction to obtain the first user feature.
[0094] Based on the above implementation method, by encoding user features, the stability of the output user features in terms of order of magnitude and feature distribution is ensured, thereby ensuring the convergence of the corresponding acoustic model construction.
[0095] Based on the above implementation methods, such as Figure 7 As shown, the user features also include second user features, and the first feature extraction module 101 provided in this application further includes:
[0096] The preprocessing submodule 111 is further configured to perform random masking on the input sample information to obtain a random masking result, and precode the random masking result to obtain a second precoding result;
[0097] Encoding submodule 113 is used to extract features from the second precoding result according to the attention mechanism to obtain the second user features;
[0098] The prediction submodule 114 is used to perform mask restoration based on the second user features to obtain mask restoration information, and is also used to perform loss function calculation based on the mask restoration information and the input sample information to obtain the loss function calculation result.
[0099] Specifically, the preprocessing submodule 111 also has a random mask function, such as... Figure 4 As shown, random masking is performed before precoding, for example, random masking is performed on 15% of the content in the input sample information. Then, fully connected layer processing, positional encoding, layer normalization, and selective discarding are performed. The random masking method of the preprocessing submodule provided in this application is not limited and is similar to the random masking method of other models, such as the natural language processing framework of BERT.
[0100] The encoding submodule 113 processes the second precoding result and the first precoding result in the same way, such as... Figure 5 As shown, the second user feature is obtained. The second user feature is combined with the phoneme feature and synthesized using the speech synthesis apparatus provided in this application to obtain the first audio data.
[0101] The second user feature is then fed into the prediction submodule 114 for random mask restoration. The random mask restoration method used in this application is not limited to, but can employ, but is not limited to, other methods. Figure 8The provided steps for mask restoration are as follows: User features or speaker features are processed through a fully connected layer, followed by activation function processing, layer normalization, and then another fully connected layer processing to obtain the mask restoration result. The activation function processing can use, but is not limited to, the Gaussian Error Linear Unit Activation Function (GELU). Subsequently, the information masked by the random mask is restored, and the difference between the restored information and the corresponding real data is calculated using a regression loss function, such as the L1 loss function. This assesses the effectiveness of the random masking and restoration, further improving the acoustic model's ability to capture contextual information from the random masking result.
[0102] Based on the above implementation method, by randomly masking the input sample information, the acoustic model's ability to capture contextual information in the random masking result is improved, and the robustness of the model or its insensitivity to parameter fluctuations is enhanced.
[0103] Based on the above implementation methods, such as Figure 9 As shown, the phoneme processing module 102 provided in this application includes:
[0104] The phoneme feature acquisition unit 123 is used to acquire user text information corresponding to the phoneme feature, and acquire the phoneme feature based on the user text information and a pre-set reference phoneme;
[0105] The phoneme feature processing unit 124 is used to obtain the first phoneme processing result by splicing the phoneme features and the user features.
[0106] Specifically, the phoneme processing module provided in this application obtains the phoneme features corresponding to the user text information based on the user text information and reference phonemes, and then combines the phoneme features and user features through splicing or other methods to obtain the first phoneme processing result.
[0107] Based on the above implementation method, the first phoneme processing result is obtained by concatenating user features and phoneme features. It does not require a large number of voice samples from the same user. It only requires general public samples to generate corresponding phonemes based on text information. Then, the phoneme features are adjusted in combination with user pronunciation features, so that the first phoneme processing result is close to the user's voice, thus meeting the user's voice replication needs.
[0108] Based on the above implementation methods, such as Figure 10 As shown, the speech synthesis apparatus provided in this application further includes:
[0109] The fourth feature extraction module 106 is used to obtain the first phoneme processing result and perform feature extraction based on the first phoneme processing result according to the attention mechanism to obtain the second phoneme processing result;
[0110] The duration prediction module 103 is also used to predict the frame length of the second phoneme processing result to obtain the second duration prediction result;
[0111] Acoustic decoding module 104 is used to generate a second acoustic feature based on the second duration prediction result and the second phoneme processing result;
[0112] The first vocode module 105 is used to perform speech synthesis on the second acoustic feature through a vocoder to obtain the second audio data from the first audio data.
[0113] Specifically, based on the aforementioned speech synthesis device, this application adds a fourth feature extraction module 106. Its structure and function are similar to those of the encoding submodule 113 in the first feature extraction module 101. This module further extracts features from the first phoneme processing result, extracting pronunciation features that more closely resemble the user's actual speech. This reduces the data processing load of the duration prediction module 103 and the acoustic decoding module 104 in the speech synthesis device, thereby improving the speech synthesis efficiency. Furthermore, the added fourth feature extraction module 106 ensures that the output results are stably distributed in terms of magnitude and distribution, avoiding excessive interference with model construction and ensuring the convergence of the model.
[0114] Based on the above implementation methods, such as Figure 11 As shown, the speech synthesis apparatus provided in this application further includes:
[0115] The second vocode module 107 is used to upsample the first duration prediction result and the first phoneme processing result through a neural vocoder and then synthesize the third audio data.
[0116] Specifically, based on the above embodiments, this application improves the speech synthesis device by replacing the Mel decoder of the acoustic decoding module 104 and the vocoder of the first vocode module 105 with the neural vocoder of the second vocode module 107, and performing upsampling through a single neural vocoder module. In an end-to-end structure, specifically, the speech synthesis device provided in this application combines the acoustic model with the vocoder, performing decoding and upsampling through the neural vocoder, thus avoiding information loss during the acoustic feature transmission process.
[0117] The neural vocoder in the speech synthesis device provided in this application can employ, but is not limited to, the HiFiGAN model. HiFiGAN uses a generative adversarial network as its foundation and generates a model. By introducing a multi-period discriminator and combining it with a multi-scale discriminator, HiFiGAN's ability to distinguish between synthesized and real audio is enhanced. Furthermore, the generator in HiFiGAN incorporates a residual structure, alternately using dilated convolutions and ordinary convolutions to increase the receptive field, ensuring synthesized sound quality while improving processing speed. The structures of the generator, multi-scale discriminator, and multi-period discriminator in HiFiGAN, as well as the working principle of HiFiGAN, are already publicly disclosed prior art and will not be elaborated upon here. This application utilizes neural vocoders such as HiFiGAN to directly upsample the first duration prediction result and the first phoneme processing result, replacing the Mel decoder and vocoder, thus avoiding information loss in acoustic features.
[0118] Based on the above implementation, an end-to-end structure is provided by replacing the Mel decoder and vocoder with a neural vocoder. Since it eliminates the need to process the first duration prediction result and the first phoneme processing result to obtain the intermediate Mel spectrum, the information loss caused by extracting the Mel spectrum as an intermediate variable on acoustic features is reduced, the number of acoustic model parameters is reduced, the computational cost of the acoustic model is lowered, and the efficiency of acoustic model speech synthesis is improved.
[0119] The pre-training module provided in this application pre-trains the feature extraction model based on a large amount of general sample information. The feature extraction model itself is built on an attention mechanism, enabling it to selectively filter out general accent and prosody-related pronunciation features from a large amount of input sample information and focus on these features to obtain user features. The pre-trained feature extraction model has reasonably set weights and other parameters, enabling it to accurately extract user-related accent and prosody-related pronunciation features from a small amount of input sample information. This avoids the problem in existing speech synthesis devices where a large amount of user-inputted speech information is required, and the voice differs significantly from the user's actual voice when the sample size is small.
[0120] Based on the above implementation methods, such as Figure 12 This application also provides an example of an end-to-end sound cloning module:
[0121] Users input multiple voice messages carrying their own acoustic features into the speaker feature extraction module. These acoustic features undergo random masking and precoding in the acoustic preprocessing unit before being passed to the attention unit for speaker feature extraction. Feature extraction is based on an attention mechanism, focusing on data related to the user's pronunciation features and reducing or ignoring data less relevant to pronunciation. The speaker features are then stored and processed. On one hand, the speaker features are passed to the prediction unit for masking and compared with the unmasked ground truth data using loss functions such as L1 loss to enhance the model's robustness and improve its contextual understanding. On the other hand, the end-to-end voice cloning module receives phoneme features or phoneme sequences generated by the front end based on the user's input text and concatenates them with the speaker features to obtain phoneme features carrying the speaker's characteristics. These then enter the attention module for focusing, further eliminating irrelevant data that could interfere with the overall model's convergence. The processed result is then used by the duration prediction module to predict the frame length of each phoneme. The acoustic decoder, such as the neuron decoder, directly upsamples the processed first acoustic feature to synthesize audio data carrying the user's pronunciation features.
[0122] Based on the above implementation methods, this application also provides an example of constructing a voice replication model: First, the model is pre-trained using more than 30 hours of open-source multi-person speech synthesis data; then, a small amount of user speech data, such as five sentences of about one minute, is selected to fine-tune the model. One sentence from the user speech data is selected as the input sample information, and features are extracted using an attention mechanism and saved as a resource file. When the user needs to convert text to speech, the corresponding phoneme string is obtained from the text, combined with the input sample information, and synthesized using the acoustic model provided in this application.
[0123] Based on the above embodiments, this application also provides an example of an application scenario for the speech synthesis device: In mobile map navigation or audiobook scenarios, users record five or more sentences of speech on their mobile phones and upload them to a training server. The training server performs adaptive training based on a small amount of user data. After training, a user's speech is randomly selected as an input sample for user feature extraction. The resulting new user features are used as input to the main model and saved as a separate resource file. When the user needs to listen, the navigation or audiobook with the user's voice is processed according to the resource file and played through appropriate settings, thus realizing the voice replication function.
[0124] The second embodiment of this application also provides a speech synthesis method, such as... Figure 13 As shown, it includes:
[0125] Step 149: Obtain general sample information and perform feature extraction pre-training based on a pre-set feature extraction model, wherein the feature extraction model is built based on an attention mechanism;
[0126] Step 131: Obtain the input sample information and extract features from the input sample information according to the pre-trained feature extraction model to obtain user features;
[0127] Step 132: Obtain phoneme features and obtain the first phoneme processing result based on the phoneme features and the user features;
[0128] Step 133: Predict the frame length of the first phoneme processing result to obtain the first duration prediction result;
[0129] Step 134: Generate the first acoustic feature based on the first duration prediction result and the first phoneme processing result;
[0130] Step 135: Use a vocoder to perform speech synthesis on the first acoustic feature to obtain the first audio data.
[0131] Based on the above implementation methods, such as Figure 14 As shown, the user features include the first user feature, and step 131 in the speech synthesis method includes:
[0132] Step 136: Obtain the input sample information and pre-encode the input sample information to obtain the first pre-encoding result;
[0133] Step 137: Extract features from the first precoding result according to the attention mechanism to obtain the first user features.
[0134] Based on the above implementation methods, such as Figure 15 As shown, step 137 in the speech synthesis method includes:
[0135] Step 138: Extract features from the first precoding result according to the attention mechanism to obtain the feature extraction result;
[0136] Step 139: Perform encoding processing based on the feature extraction results to obtain the first user feature.
[0137] Based on the above implementation methods, such as Figure 16 As shown, the user features also include second user features, and step 131 in the speech synthesis method further includes:
[0138] Step 140: Perform random masking on the entered sample information to obtain a random masking result, and precode the random masking result to obtain a second precoding result;
[0139] Step 141: Extract features from the second precoding result according to the attention mechanism to obtain the second user features;
[0140] Step 142: Obtain mask restoration information by performing mask restoration based on the second user features, and also calculate the loss function based on the mask restoration information and the input sample information to obtain the loss function calculation result.
[0141] Based on the above implementation methods, such as Figure 17 As shown, step 132 in the speech synthesis method includes:
[0142] Step 143: Obtain the user text information corresponding to the phoneme feature, and obtain the phoneme feature based on the user text information and the preset reference phoneme;
[0143] Step 144: Obtain the first phoneme processing result by splicing the phoneme features and the user features.
[0144] Based on the above implementation methods, such as Figure 18 As shown, in the speech synthesis method, after step 132, the following steps are also included:
[0145] Step 145: Obtain the first phoneme processing result and perform feature extraction based on the first phoneme processing result according to the attention mechanism to obtain the second phoneme processing result;
[0146] Step 146: Predict the frame length of the second phoneme processing result to obtain the second duration prediction result;
[0147] Step 147: Generate a second acoustic feature based on the second duration prediction result and the second phoneme processing result;
[0148] Step 148: Use a vocoder to perform speech synthesis on the second acoustic feature to obtain the second audio data from the first audio data.
[0149] Based on the above implementation methods, such as Figure 19 As shown, in the speech synthesis method, after step 146, the following steps are also included:
[0150] Step 150: Upsample the first duration prediction result and the first phoneme processing result using a neural vocoder, and obtain the third audio data through speech synthesis.
[0151] The third embodiment of this application relates to a mobile terminal, such as... Figure 20 As shown, it includes:
[0152] At least one processor 161; and,
[0153] Memory 162 communicatively connected to the at least one processor 161; wherein,
[0154] The memory 162 stores instructions that can be executed by the at least one processor, which are executed by the at least one processor 161 to enable the at least one processor 161 to implement the speech synthesis method described in this application.
[0155] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.
[0156] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.
[0157] The fourth embodiment of this application relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the speech synthesis method described in this application.
[0158] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0159] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.
[0160] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.
Claims
1. A speech synthesis apparatus characterized by comprising: The method comprises the following steps: a pre-training module is configured to obtain general sample information and perform feature extraction pre-training according to a pre-set feature extraction model, wherein the feature extraction model is constructed based on an attention mechanism; a first feature extraction module is configured to obtain input sample information and perform feature extraction on the input sample information according to the pre-trained feature extraction model to obtain user features; a phoneme processing module is configured to obtain phoneme features and obtain a first phoneme processing result according to the phoneme features and the user features; a duration prediction module is configured to predict the frame length of the first phoneme processing result to obtain a first duration prediction result; an acoustic decoding module is configured to generate first acoustic features according to the first duration prediction result and the first phoneme processing result; a first vocoder module is configured to perform voice synthesis on the first acoustic features through a vocoder to obtain first audio data; the user features include first user features, and the first feature extraction module comprises: a preprocessing sub-module is configured to obtain the input sample information and perform pre-encoding on the input sample information to obtain first pre-encoding results; a second feature extraction sub-module is configured to perform feature extraction on the first pre-encoding results according to the attention mechanism to obtain the first user features; the user features also include second user features, and the first feature extraction module further comprises: the preprocessing sub-module is further configured to perform random masking on the input sample information to obtain random masking results and perform pre-encoding on the random masking results to obtain second pre-encoding results; an encoding sub-module is configured to perform feature extraction on the second pre-encoding results according to the attention mechanism to obtain the second user features; a prediction sub-module is configured to perform mask restoration according to the second user features to obtain mask restoration information, and is further configured to perform loss function calculation according to the mask restoration information and the input sample information to obtain a loss function calculation result.
2. The apparatus of claim 1, wherein, The second feature extraction sub-module comprises: a third feature extraction unit is configured to perform feature extraction on the first pre-encoding results according to the attention mechanism to obtain feature extraction results; an encoding unit is configured to perform encoding processing on the feature extraction results to obtain the first user features.
3. The apparatus of claim 1, wherein, The phoneme processing module comprises: a phoneme feature acquisition unit is configured to obtain user text information corresponding to the phoneme features and obtain the phoneme features according to the user text information and pre-set reference phonemes; a phoneme feature processing unit is configured to perform splicing processing on the phoneme features and the user features to obtain the first phoneme processing result.
4. The apparatus of claim 1, wherein, Further comprising: a fourth feature extraction module is configured to obtain the first phoneme processing result and perform feature extraction on the first phoneme processing result according to the attention mechanism to obtain a second phoneme processing result; the duration prediction module is further configured to predict the frame length of the second phoneme processing result to obtain a second duration prediction result; an acoustic decoding module is configured to generate second acoustic features according to the second duration prediction result and the second phoneme processing result; The first vocoder module is configured to perform voice synthesis on the second acoustic feature by a vocoder to obtain first audio data and second audio data.
5. The apparatus of claim 1, wherein, Further comprising: The second vocoder module is configured to perform voice synthesis on the first duration prediction result and the first phoneme processing result by a neural vocoder to obtain third audio data.
6. A speech synthesis method characterized by, Comprising: Obtain general sample information, and perform feature extraction pre-training according to a pre-set feature extraction model, wherein the feature extraction model is constructed based on an attention mechanism; Obtain input sample information and perform feature extraction on the input sample information according to the pre-trained feature extraction model to obtain user features; Obtain phoneme features and obtain first phoneme processing results according to the phoneme features and the user features; Predict the frame length of the first phoneme processing result to obtain a first duration prediction result; Generate first acoustic features according to the first duration prediction result and the first phoneme processing result; Perform voice synthesis on the first acoustic features by a vocoder to obtain first audio data; Obtain the input sample information and perform pre-coding on the input sample information to obtain first pre-coding results; Perform feature extraction on the first pre-coding results according to the attention mechanism to obtain first user features; The feature extraction on the first pre-coding results according to the attention mechanism to obtain the first user features comprises: Perform feature extraction on the first pre-coding results according to the attention mechanism to obtain feature extraction results; Perform coding processing on the feature extraction results to obtain the first user features; Perform random masking on the input sample information to obtain random masking results, and perform pre-coding on the random masking results to obtain second pre-coding results; Perform feature extraction on the second pre-coding results according to the attention mechanism to obtain second user features; Perform mask restoration according to the second user features to obtain mask restoration information, and perform loss function calculation according to the mask restoration information and the input sample information to obtain loss function calculation results.
7. A mobile terminal, characterized by Comprising: At least one processor; And The memory is in communication connection with the at least one processor; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to implement the voice synthesis method in claim 6.
8. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the voice synthesis method in claim 6.
Citation Information
Patent Citations
Speech synthesis method and device, computer equipment, storage medium and product
CN114333758A
Speech synthesis method and device, equipment and storage medium
CN114708847A