Personalized speech synthesis method based on attention mechanism
Through a deep learning structure based on the attention mechanism, the problem of poor style transfer performance of voice assistants in cross-language contexts is solved, and the personalized speech synthesis effect is improved, which is suitable for speech synthesis tasks in different languages.
Patent Information
- Application Number
- CN202410502538.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-25
- Publication Date
- 2025-10-28
AI Technical Summary
Existing voice assistants do not perform well in cross-language style transfer, which makes it inconvenient for users. In addition, the training and reasoning processes of existing technologies are slow, making them difficult to apply on a large scale.
By employing a deep learning structure based on an attention mechanism, feature encoding and feature extraction are performed on text and speech data. The fusion of text encoding features and speaker style features is then used to perform speech synthesis, thereby improving the speech synthesis effect in cross-linguistic contexts.
It improves the naturalness and similarity of speech synthesis in cross-linguistic contexts, enhances the personalization effect of speech synthesis, and is applicable to speech synthesis tasks in different languages.
Smart Images
Figure CN120853545A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of speech processing, and more specifically, to a personalized speech synthesis method based on an attention mechanism. Background Technology
[0002] Currently, various smart applications typically include built-in voice assistant functionality, and users' demands for the speech synthesis quality of these assistants are constantly increasing. Personalized voice customization has gradually become a basic feature of voice assistants. However, current technologies for voice assistants still lack style transfer capabilities in cross-linguistic contexts, which causes inconvenience for users and is a problem that researchers urgently need to solve.
[0003] Regarding the issue that existing voice assistants do not perform well in style transfer in cross-linguistic contexts, the industry has not yet proposed an effective and efficient solution. Some technologies in the laboratory stage often have the disadvantage of slow training and inference processes, making them difficult to apply on a large scale. Summary of the Invention
[0004] Based on the shortcomings and deficiencies of the existing technology, this application proposes a personalized speech synthesis method based on an attention mechanism, which can improve the personalized speech synthesis effect under specific conditions.
[0005] The specific technical solution proposed in this application is as follows:
[0006] According to an example of this application, a personalized speech synthesis method based on an attention mechanism is provided, which can be used... Figure 1 This indicates, including:
[0007] Based on the character units of the text to be synthesized, feature encoding is performed on the acquired target text data to obtain text encoding features;
[0008] Based on the reference audio of the reference speaker's speech, feature extraction is performed on the acquired target audio data to obtain speaker style features;
[0009] Based on the fusion features of the text encoding features and the speaker style features, speech synthesis is performed to obtain synthesized speech data corresponding to the text to be synthesized and the reference audio.
[0010] Optionally, the step of performing feature encoding on the acquired target text data based on the character units of the text to be synthesized to obtain text encoding features includes:
[0011] The target text data is feature-encoded using a character-level encoder consisting of one-dimensional convolution and batch normalization to obtain encoded target text data; language embedding information is added to the encoded target text data to obtain text encoding features.
[0012] Text preprocessing is performed on the character units contained in the synthesized text; wherein, the text preprocessing includes one or more of the following: text cleaning, text regularization, and character-to-phoneme conversion.
[0013] Optionally, the reference audio based on the reference speaker's speech is used to extract features from the acquired target audio data to obtain speaker style features, including:
[0014] The target audio data is subjected to audio preprocessing and Mel spectrum calculation;
[0015] The audio preprocessing includes at least one of resampling processing and noise suppression processing;
[0016] The Mel spectrum calculation includes: pre-emphasizing the target audio data using a pre-emphasis filter to obtain pre-emphasized target audio data; dividing the pre-emphasized target audio data into frame signals corresponding to multiple time windows, and performing windowing processing on each frame signal; and determining the Mel spectrum of the target audio data based on each windowed frame signal.
[0017] Optionally, the step of performing speech synthesis based on the fusion features of the text encoding features and speaker style features to obtain synthesized speech data corresponding to the text to be synthesized and the reference audio includes:
[0018] The text encoding features and the speaker style features are fused using an attention mechanism.
[0019] The attention-based feature fusion uses scaled dot product attention and also includes a dynamic re-shape module to achieve dimensionality matching between features.
[0020] Optionally, the training process of the personalized speech synthesis system can be used... Figure 2 This indicates, including:
[0021] The pre-collected sample set is divided into a sample training set and a sample validation set;
[0022] The sample set includes: sample speech data and sample text data corresponding to the sample speech data;
[0023] Optionally, the training process of the personalized speech synthesis system further includes:
[0024] Using the sample speech data and sample text data in the sample validation set, the trained personalized speech synthesis system is inferred and validated. Based on the loss function between the validation audio data output by the speech synthesis system and the sample audio data, the hyperparameters of the personalized speech synthesis system are adjusted.
[0025] The personalized speech synthesis method proposed in this application includes: encoding the acquired target text data based on the character units of the text to be synthesized to obtain text encoding features; extracting features from the acquired target audio data based on reference audio of a reference speaker's speech to obtain speaker style features; and performing speech synthesis based on the fusion features of the text encoding features and speaker style features to obtain synthesized speech data corresponding to the text to be synthesized and the reference audio. The technical solution of this application employs a deep learning structure based on an attention mechanism, enabling the deep learning model to fully utilize the text encoding features and speaker style features extracted from the text and speech respectively. This method is more suitable for speech synthesis tasks where the text and speech come from different languages, and can improve the speech synthesis effect in the above situations. Attached Figure Description
[0026] Figure 1 A flowchart of the personalized speech synthesis method based on the attention mechanism of this application is shown;
[0027] Figure 2 A schematic diagram of the training process of the attention-based personalized speech synthesis system in this application is shown;
[0028] Figure 3 A model structure diagram of an embodiment built using the method of this application is shown. Detailed Implementation
[0029] The technical solutions of this application are applicable to personalized speech synthesis scenarios, especially when the reference audio and the text to be synthesized come from different languages. By employing the technical solutions of this application, the naturalness and similarity of the synthesized speech in the aforementioned scenarios can be improved.
[0030] Currently, personalized speech synthesis methods that use speaker encoders to extract reference audio features have poor synthesis results when the reference audio and the text to be synthesized come from different languages. This is mainly because in existing methods, the language features of the text to be synthesized will, to some extent, cover the speaker style information carried by the reference audio.
[0031] Based on this, this application proposes a personalized speech synthesis method. This technical solution uses a deep learning structure based on an attention mechanism, which enables the deep learning model to make full use of the text encoding features and speaker style features extracted from the text and speech respectively. It is more suitable for speech synthesis tasks when the text and speech come from different languages, and can improve the speech synthesis effect in the above situation.
[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0033] Exemplary methods
[0034] This application proposes a personalized speech synthesis method based on an attention mechanism. This method can be executed by an electronic device, which can be any device with data processing and deep learning computing capabilities, such as a computer, server, or embedded smart terminal. See also... Figure 1 As shown, the method includes:
[0035] S101. Based on the character units of the text to be synthesized, feature encoding is performed on the acquired target text data to obtain text encoding features.
[0036] Specifically, this embodiment uses a character-level encoder to encode text characters, as shown in the attached figure. Figure 3 As shown, the character-level encoder consists of one-dimensional convolution, batch normalization, ReLU activation function, Dropout, and average pooling, which can extract the required language style features from the text sequence. The language style features extracted using this method will be used together with the language embedding as the key for the cross-attention mechanism.
[0037] Furthermore, the language embedding is implemented using the lookup table embedding method of the PyTorch framework, and the required language embedding and language style features are added character by character.
[0038] S102. Reference audio based on the speaker's language, as attached. Figure 3 As shown, feature extraction is performed on the acquired target audio data to obtain speaker style features.
[0039] Specifically, to reduce the computational cost required for feature extraction, this embodiment uses Mel spectrum as the feature representation, i.e., Mel spectrum features. Mel spectrum feature extraction first involves pre-emphasizing, framing, and windowing the target speech waveform data, then applying Short Time Fourier Transform (STFT) to calculate the spectrogram, and finally applying a Mel filter bank to obtain the Mel spectrum, i.e., the Mel spectrum features.
[0040] Furthermore, as shown in the appendix Figure 3 As shown, the Mel spectral features first pass through a frame-level downsampling encoder to reduce the computational cost, and then through a speaker-style encoder with the same structure as the character-level encoder to capture and encode the speaker's vocal features, such as pitch, speech rate, and timbre. The resulting vector is used as the value for the cross-attention mechanism.
[0041] S103. Based on the fusion features of text encoding features and speaker style features, perform speech synthesis to obtain the synthesized speech data corresponding to the text to be synthesized and the reference audio.
[0042] Specifically, in this embodiment, the text encoding features and speaker style features described in S101 and S102 are fused using a scaled dot product attention mechanism to obtain the fused features for each speech frame; as illustrated in the appendix... Figure 3 As shown, the query for the scaled dot product attention above uses the encoder output; then the fusion features of each speech frame are decoded to generate the Mel spectral features corresponding to that speech frame; finally, the Mel spectral features are input into the vocoder to restore the audio waveform.
[0043] For the foregoing embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.
[0044] It should be noted that the key descriptions of the embodiments described in this application are the differences from embodiments of other methods in the art. For methods that are generally accepted or defaulted in the art, the descriptions in this specification are relatively simple.
[0045] The steps in the embodiments described in this application can be adjusted, combined, or deleted in order according to actual needs, and the technical features recorded in the embodiments can be replaced or combined.
[0046] The modules and sub-modules in the embodiments described in this application can be merged, split, or deleted according to actual needs.
[0047] It should be understood that the methods and processes disclosed in the embodiments provided in this application can be implemented in other ways. For example, the PyTorch-based embodiments described above are merely illustrative and can also be implemented using TensorFlow or other deep learning frameworks.
[0048] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A personalized speech synthesis method based on an attention mechanism, characterized in that, Includes the following steps: Step 1: Based on the character units of the text to be synthesized, perform text preprocessing and text feature encoding on the acquired target text data to obtain text encoding features; wherein, the text preprocessing includes one or more of the following: cleaning non-standard text, regularizing non-regularized text, and character text character-to-phoneme conversion; wherein, the text feature encoding includes: using a character-level encoder to perform feature encoding on the target text data to obtain encoded target text data; adding language embedding information to the encoded target text data to obtain text encoding features; Step 2: Based on the reference audio of the reference speaker's speech, feature extraction is performed on the acquired target audio data to obtain speaker style features. The feature extraction includes audio preprocessing, Mel spectrum calculation, and audio feature extraction. The audio preprocessing includes at least one of resampling audio with mismatched sampling rates and noise suppression processing for audio with strong background noise. The Mel spectrum calculation includes pre-emphasizing and windowing the target audio data using a pre-emphasis filter, and determining the Mel spectrum of the target audio data for each windowed frame signal. The feature extraction includes downsampling the Mel spectrum of the target audio data using a downsampling encoder to obtain a downsampled Mel spectrum; and extracting speaker style features from the downsampled Mel spectrum using a frame-level encoder. Step 3: Using a public or private dataset, perform data preprocessing and model training according to the methods described in Steps 1 to 2 to obtain the trained speech synthesis model; Step 4: Based on the fusion features of the text encoding features and the speaker style features, use the speech synthesis model to perform speech synthesis to obtain the synthesized speech data corresponding to the text to be synthesized and the reference audio. The fusion feature is obtained by performing attention-based feature fusion on the text encoding feature and the speaker style feature, and the attention-based feature fusion uses scaled dot product attention. Step 5: Test and evaluate the model trained in Step 4 using subjective and objective evaluation methods; the subjective evaluation method uses the naturalness subjective opinion score MOS-P, and the objective evaluation method uses the similarity subjective opinion score SMOS.
2. The personalized speech synthesis method according to claim 1, characterized in that, In the text feature encoding described in step 1, not only is the target text data included, but also language embedding information. Specifically, the language embedding information is a language-corresponding encoding generated using the lookup table embedding method of the PyTorch framework. Then, the language embedding information is added character by character to the character-level encoding output of the text to be synthesized to obtain the text feature encoding.
3. The personalized speech synthesis method according to claim 1, characterized in that, In the feature fusion process described in step 3, the scaled dot product attention used also includes a dynamic re-shape module to achieve dimensionality matching between features. Specifically, the text encoding features described in step 1 are used as the values of the attention mechanism, the speaker style features described in step 2 are used as the keys of the attention mechanism, and the output vector of the phoneme encoder is used as the query of the attention mechanism. After that, the output of the attention mechanism is combined with the output of the phoneme encoder and sent to the variance adapter for further processing.
4. The personalized speech synthesis method according to claim 1, characterized in that, The personalized speech synthesis system further includes, before training: dividing a pre-collected sample set into a sample training set and a sample validation set; wherein, the sample set includes: sample speech data and sample text data corresponding to the sample speech data.
5. The personalized speech synthesis method according to claim 1, characterized in that, The training process of the personalized speech synthesis system includes: using sample speech data and sample text data in the sample verification set as described in claim 4 to perform inference verification on the trained personalized speech synthesis system, and adjusting the hyperparameters of the personalized speech synthesis system according to the loss function between the verification audio data output by the speech synthesis system and the sample audio data; wherein, the loss function includes L1 loss, duration loss, pitch loss, energy loss, and flow loss between the synthesized speech and the language in the training set.