Speech synthesis method and device, electronic equipment, storage medium and program product
By constructing a timbre clone sub-model and a local base sub-model using low-rank adaptive technology, and combining this with unsupervised end-to-end learning, the problem of personalized speech synthesis for users is solved, achieving efficient and personalized speech synthesis results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2026-03-31
AI Technical Summary
Existing technologies struggle to generate personalized synthetic speech, and model training requires significant resources and time, is complex to operate, and results in a poor user experience.
A low-rank adaptive technique is used to construct a timbre clone sub-model, which is combined with a local base sub-model. The target acoustic model is trained through unsupervised end-to-end learning to learn the timbre features of the target user and generate personalized speech.
It reduces the amount of data processing required for model training, lowers computational resource and time costs, enables personalized speech synthesis for users, and improves user experience.
Smart Images
Figure CN121768356A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of audio processing technology, specifically to a method, apparatus, electronic device, storage medium, and program product for speech synthesis. Background Technology
[0002] In speech synthesis scenarios (such as singing synthesis), statistical models are typically trained using a large amount of training data to obtain a well-trained speech synthesis model. This model learns the characteristics and patterns of human speech, and then speech synthesis can be performed using the well-trained speech synthesis model.
[0003] However, this approach often makes it difficult to generate personalized synthetic speech, and model training typically requires significant resources, such as computational resources and time costs. Summary of the Invention
[0004] The purpose of this disclosure is to provide a method, apparatus, electronic device, storage medium, and program product for speech synthesis, in order to solve the problem of difficulty in generating personalized synthesized speech while taking into account the model training cost.
[0005] On one hand, this disclosure provides a speech synthesis method, the method comprising:
[0006] Obtain the data to be processed;
[0007] The data to be processed is input into the speech synthesis model to obtain the synthesized speech of the target user. The speech synthesis model is obtained by combining the trained timbre clone sub-model with the local base sub-model. The timbre clone sub-model is constructed based on low-rank adaptive technology and trained based on the audio sample data of the target user. It is used to learn the timbre features of the target user.
[0008] In one implementation, the data to be processed includes at least one of the following: text to be processed and audio of the target user.
[0009] In one implementation, the timbre clone model is obtained by training using the following steps:
[0010] Obtain training data;
[0011] The initial timbre clone sub-model is combined with the pre-trained cloud-based basic sub-model to obtain the initial acoustic model; the initial acoustic model is built based on unsupervised end-to-end learning.
[0012] Based on the training data, the initial acoustic model is trained to obtain the trained target acoustic model;
[0013] The timbre clone sub-model in the target model is identified as the trained timbre clone sub-model.
[0014] In one implementation, an initial acoustic model is trained based on training data to obtain a trained target acoustic model, including:
[0015] The training data is input into the initial acoustic model to obtain the predicted Mel spectrogram;
[0016] Based on the target Mel spectrogram of the target user and the predicted Mel spectrogram, determine the prediction loss;
[0017] Based on the predicted loss, the parameters of the timbre clone sub-model in the target model are adjusted until a well-trained target acoustic model is obtained.
[0018] In one implementation, the training data includes audio sample data of the target user, audio sample data of multiple non-target users, and text sample data;
[0019] The training data is input into the initial acoustic model to obtain the predicted Mel spectrogram, including:
[0020] Feature extraction is performed on audio sample data from non-target users to obtain audio features of non-target users;
[0021] Feature extraction is performed based on text sample data and audio sample data of the target user to obtain the audio features of the target user.
[0022] Based on the audio features of non-target users, the audio features of target users, and text sample data, predictions are made to obtain the predicted Mel spectrogram.
[0023] In one implementation, feature extraction is performed on audio sample data from non-target users to obtain audio features of non-target users, including:
[0024] Extract initial audio features from audio sample data of non-target users in the audio dimension;
[0025] Phoneme position encoding is performed on the initial audio features to obtain the first phoneme features;
[0026] The first phoneme feature is encoded to obtain the second phoneme feature;
[0027] The first phoneme feature and the second phoneme feature are fused together to obtain the fused feature;
[0028] The fused features are decoded to obtain audio features of non-target users.
[0029] In one aspect, this disclosure provides a speech synthesis apparatus, comprising:
[0030] The acquisition unit is used to acquire data to be processed.
[0031] The acquisition unit is used to input the data to be processed into the speech synthesis model to obtain the synthesized speech of the target user. The speech synthesis model is obtained by combining the trained timbre cloning sub-model with the local base sub-model. The timbre cloning sub-model is constructed based on low-rank adaptive technology and trained based on the audio sample data of the target user, and is used to learn the timbre features of the target user.
[0032] In one implementation, the data to be processed includes at least one of the following: text to be processed and audio of the target user.
[0033] In one implementation, the obtaining unit is further configured to: train and obtain a timbre clone sub-model using the following steps:
[0034] Obtain training data;
[0035] The initial timbre clone sub-model is combined with the pre-trained cloud-based basic sub-model to obtain the initial acoustic model; the initial acoustic model is built based on unsupervised end-to-end learning.
[0036] Based on the training data, the initial acoustic model is trained to obtain the trained target acoustic model;
[0037] The timbre clone sub-model in the target model is identified as the trained timbre clone sub-model.
[0038] In one embodiment, the obtaining unit is further configured to:
[0039] The training data is input into the initial acoustic model to obtain the predicted Mel spectrogram;
[0040] Based on the target Mel spectrogram of the target user and the predicted Mel spectrogram, determine the prediction loss;
[0041] Based on the predicted loss, the parameters of the timbre clone sub-model in the target model are adjusted until a well-trained target acoustic model is obtained.
[0042] In one implementation, the training data includes audio sample data of the target user, audio sample data of multiple non-target users, and text sample data;
[0043] The obtaining unit is also used for:
[0044] Feature extraction is performed on audio sample data from non-target users to obtain audio features of non-target users;
[0045] Feature extraction is performed based on text sample data and audio sample data of the target user to obtain the audio features of the target user.
[0046] Based on the audio features of non-target users, the audio features of target users, and text sample data, predictions are made to obtain the predicted Mel spectrogram.
[0047] In one embodiment, the obtaining unit is further configured to:
[0048] Extract initial audio features from audio sample data of non-target users in the audio dimension;
[0049] Phoneme position encoding is performed on the initial audio features to obtain the first phoneme features;
[0050] The first phoneme feature is encoded to obtain the second phoneme feature;
[0051] The first phoneme feature and the second phoneme feature are fused together to obtain the fused feature;
[0052] The fused features are decoded to obtain audio features of non-target users.
[0053] In one aspect, this disclosure provides an electronic device, including:
[0054] Processor; and
[0055] The memory stores computer instructions that cause the processor to perform the steps of the methods provided in the various alternative implementations of any of the above-described speech synthesis methods.
[0056] In one aspect, this disclosure provides a computer-readable storage medium storing computer instructions for causing a computer to perform the steps of the methods provided in various alternative implementations of any of the above-described speech synthesis methods.
[0057] On one hand, this disclosure provides a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the steps of the method provided in any of the above-described alternative implementations of speech synthesis.
[0058] The speech synthesis method in this embodiment includes constructing and training a timbre clone sub-model based on low-rank adaptive technology, combining the trained timbre clone sub-model with a local base sub-model to obtain a speech synthesis model, and generating synthesized speech for the target user based on the data to be processed using the speech synthesis model. In this way, constructing and training the model based on low-rank adaptive technology reduces the amount of training data processing and allows the model to learn the timbre features of the target user for speech synthesis, achieving personalized speech synthesis for the user. Attached Figure Description
[0059] Figure 1 This is a logical diagram illustrating model training and application in an embodiment of this disclosure.
[0060] Figure 2 This is a flowchart of a model training method according to an embodiment of this disclosure.
[0061] Figure 3 This is a schematic diagram of the structure of an acoustic model in an embodiment of this disclosure.
[0062] Figure 4 This is a schematic diagram of the structure of a speech synthesis module in an embodiment of this disclosure.
[0063] Figure 5 This is a flowchart of a speech synthesis method according to an embodiment of the present disclosure.
[0064] Figure 6 This is a structural block diagram of a speech synthesis apparatus according to an embodiment of the present disclosure.
[0065] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. Detailed Implementation
[0066] The technical solutions of this disclosure will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure. Furthermore, the technical features involved in the different embodiments of this disclosure described below can be combined with each other as long as they do not conflict with each other.
[0067] In related technologies, training speech synthesis models typically requires a large amount of training data. In certain speech synthesis scenarios, such as singing synthesis, a large amount of hard-to-obtain singing data is also needed as training data, making data collection difficult. Furthermore, complex preprocessing operations such as aligning lyrics are usually required, which consumes a lot of computing resources and time. Since the model parameters are usually numerous, the model training process also consumes a lot of computing resources and time. Moreover, the singing voices synthesized by speech synthesis models are usually generic and difficult to achieve personalized speech synthesis for specific users, resulting in a large difference between the user's voice and the user's voice, leading to a poor user experience. Furthermore, speech synthesis usually requires professionals to follow complex operating procedures, which are cumbersome and increase the cost of speech synthesis.
[0068] Based on the deficiencies of the aforementioned related technologies, this disclosure provides a method, apparatus, electronic device, storage medium, and program product for speech synthesis, aiming to solve the problem of difficulty in generating personalized synthesized speech while taking into account model training costs.
[0069] This disclosure provides a speech synthesis method that can be applied to electronic devices. This disclosure does not limit the type of electronic device; it can be any suitable type of device, such as terminal devices and servers, etc. This disclosure will not elaborate further on this.
[0070] In this embodiment of the disclosure, there are two stages. The first stage is to train a model for the target user to obtain a trained speech synthesis model for the target user. The second stage is to generate the synthesized speech of the target user through the trained speech synthesis model.
[0071] The following is combined Figure 1 The logical flow of model training and application is explained. (See also...) Figure 1 The diagram shown is a logical illustration of model training and application. Figure 1 This includes model training equipment and speech synthesis equipment. Both the model training equipment and the speech synthesis equipment can be electronic devices. Optionally, the model training equipment can be a server, such as a cloud server. The speech synthesis equipment can be a terminal device, such as a mobile phone. The model training equipment and the speech synthesis equipment can be the same device or different devices.
[0072] The model training equipment is used to train the initial acoustic model to obtain a trained acoustic model. For ease of explanation, the acoustic models before and after training are referred to as the initial acoustic model and the target acoustic model, respectively. The acoustic model includes a cloud-based base sub-model and a timbre clone sub-model. Optionally, the cloud-based base sub-model can be a pre-trained large-scale language model.
[0073] The speech synthesis device is used to acquire a timbre clone sub-model from the target acoustic model, and combine this timbre clone sub-model with a local base sub-model to obtain a speech synthesis model. Then, based on the speech synthesis model and the data to be processed, it generates synthesized speech for the target user. The local base sub-model can be a pre-trained large-scale language model. Both the local base sub-model and the cloud-based base sub-model can be the same pre-trained model.
[0074] Both the acoustic model and the speech synthesis model are built using unsupervised end-to-end learning and low-rank adaptation techniques. The training data for the acoustic model includes at least audio sample data from the target user. The timbre cloning sub-model is built using low-rank adaptation (LoRA) techniques to learn the timbre features of the target user.
[0075] LoRA, a low-rank adaptive technique for Large Language Models (LLMs), is a commonly used method for efficient parameter fine-tuning. The essence of LoRA is to approximate the incremental parameters obtained from full parameter fine-tuning of an LLM with fewer training parameters, thus achieving efficient fine-tuning with less GPU memory usage. In its implementation, after freezing the weights of the local base sub-model, the trainable low-rank decomposition matrix is injected into each model of the corresponding parameter architecture, significantly reducing the number of trainable parameters in downstream tasks. In speech synthesis inference applications, for LoRA models, the weights of the original pre-trained model (i.e., the cloud-based base sub-model) can be directly merged with the weights of the trained LoRA model, thus incurring no additional overhead during inference. Furthermore, adding a LoRA model to a large speech model allows the large speech model to process the data and generate initial output results, which can then be fine-tuned using the LoRA model to achieve a "cloning" of the target person's (i.e., the target user's) voice style. This allows for the transfer of the voice style of an unseen speaker or singer, enabling fast and highly realistic speech synthesis.
[0076] Furthermore, before model training, the training data can undergo at least one of the following common preprocessing operations: silence removal, volume normalization, segmentation and padding, and frequency domain transformation. After obtaining the trained target acoustic model, the timbre clone sub-model within the target acoustic model can be quantized before being sent to the speech synthesis device. For example, quantization can be performed using 8-bit integers (INT8). The quantized timbre clone sub-model is reusable and has a compressed storage space, reducing storage resource consumption.
[0077] For example, the embodiments disclosed herein can be applied to karaoke applications, music creation applications, voice assistant applications, and game development scenarios to process the user's voice and create better singing effects.
[0078] The following is combined Figure 1 and Figure 2 The method for model training in the first stage described above will be explained. (See also...) Figure 2 The diagram shown is a flowchart of a model training method according to an embodiment of this disclosure. The specific implementation process of this method is as follows:
[0079] Step 201: Obtain training data.
[0080] The training data includes audio sample data from the target user, audio sample data from multiple non-target users, and text sample data. For example, the audio duration of the audio sample data can range from 30 seconds to 5 minutes.
[0081] In one implementation, an audio acquisition device is used to collect sound from both the target user and non-target users, obtaining audio sample data from the target user and audio sample data from multiple non-target users. For example, data acquisition can be performed through one or more speakers of the device.
[0082] It should be noted that, under related technologies, in singing synthesis scenarios, the speech synthesis model used for singing synthesis usually needs to be trained based on singing data. Obviously, singing data is usually expensive and difficult to obtain, making data collection difficult and costly. Furthermore, under related technologies, complex data preprocessing such as lyric alignment is usually required for singing training data, which consumes a lot of computing resources and time, and is prone to causing speech synthesis errors. However, in this embodiment, the model is built based on unsupervised end-to-end learning, which reduces the limitation on data resources. The audio sample data can be any voice of the user (e.g., it can be just the user's speech without singing). Each speaker's information is available, and rich categories of speech features can be extracted from the source audio. Singing data does not need to be collected, reducing the difficulty and cost of data collection. Moreover, complex data preprocessing operations such as lyric alignment are not required, reducing the consumption of computing resources and time.
[0083] Step 202: Combine the initial timbre clone sub-model with the pre-trained cloud-based basic sub-model to obtain the initial acoustic model.
[0084] The following is combined Figure 3 An example illustrating the structure of the acoustic model is provided. (See also...) Figure 3 The diagram shown is a structural schematic of an acoustic model. Figure 3 In this system, the acoustic model comprises a cloud-based foundational sub-model and a timbre cloning sub-model. The cloud-based foundational sub-model includes an audio feature processing module, a text feature processing module, and a speech synthesis module; it is built upon an attention-based model.
[0085] Step 203: Train the initial acoustic model based on the training data to obtain the trained target acoustic model.
[0086] In one implementation, step 203 may be performed using the following steps:
[0087] S2031: Input the training data into the initial acoustic model to obtain the predicted Mel spectrogram.
[0088] In one implementation, when executing S2031, the following steps may be taken:
[0089] S2031-1: Extract features from audio sample data of non-target users to obtain audio features of non-target users.
[0090] In one implementation, audio sample data of non-target users is input into an audio feature processing module to obtain audio features of non-target users.
[0091] Figure 3 The audio feature processing module includes: a feature extraction module, a first phoneme position encoder, a feature encoder, a mixed attention submodule, and a feature decoder.
[0092] In one implementation, when executing S2031-1, the following steps may be taken:
[0093] S2031-11: Extract initial audio features in the audio dimension from audio sample data of non-target users.
[0094] In one implementation, the initial audio features of audio sample data from non-target users are extracted in the audio dimension using a feature extraction module.
[0095] The audio dimensions include at least one of the following: pitch, harmony, cepstral prominence, intensity, formants, and octave. Initial audio features include features related to the classification of vocal patterns in singing. Initial audio features can also be referred to as identification (ID) features.
[0096] S2031-12: Perform phoneme position encoding on the initial audio features to obtain the first phoneme features.
[0097] In one implementation, the initial audio features are phoneme position encoded using a phoneme position encoder to obtain the first phoneme feature. As an example, the initial audio features are encoded using a multi-level frame-level feature encoder, and the selected frame-level features are: pitch (F0), harmonic-to-noise ratio (HNR), cepstral peak prominence (CPP), root mean square (RMS), formants (F1-F4), and octave.
[0098] F0 is the fundamental frequency, also known as the first formant, which is the lowest frequency component in a speech signal and usually corresponds to the pitch.
[0099] S2031-13: Encode the features of the first phoneme to obtain the features of the second phoneme.
[0100] In one implementation, a feature encoder encodes the first phoneme features to obtain the second phoneme features. As an example, the feature encoder extracts a set of representative features from the first phoneme features, maps these features to an 8-dimensional space to adjust the acoustic model, and performs masking based on their filling.
[0101] The feature encoder consists of 8 blocks; each block contains 3 stacked 1x1 convolutional layers (channels: [4,8], [8,64], [64,8]), with Batch Normalization (BN) and Exponential Linear Units (ELU) active.
[0102] S2031-14: The first phoneme feature and the second phoneme feature are fused to obtain the fused feature.
[0103] In one implementation, a fused feature is obtained by fusing the first phoneme feature and the second phoneme feature through a hybrid attention submodule.
[0104] Since the outputs of the phoneme position encoder and the feature encoder are generally considered to be the most basic and original audio features, they are connected to a hybrid attention submodule, which then fuses the first and second phoneme features. Because mixed human voices may have been used during model training, this method decouples the initial audio features from the timbre.
[0105] S2031-15: Decode the fused features to obtain audio features of non-target users.
[0106] In one implementation, the fused features are decoded using a feature decoder to obtain audio features of the non-target user.
[0107] In this way, feature extraction and optimization can be performed on audio sample data from non-target users to obtain optimized audio features from non-target users.
[0108] S2031-2: Extract features from text sample data and target user audio sample data to obtain target user audio features.
[0109] In one implementation, text sample data is input into a text feature processing module, and audio sample data of the target user is input into a timbre cloning sub-model to obtain the target user audio features output by the user timbre injection module.
[0110] Figure 3 In this model, the timbre cloning sub-model comprises multiple LoRA modules: LoRA-part1, LoRA-part2, LoRA-part3, LoRA-part4, and LoRA-part5. The text feature processing module includes multiple sub-modules: a text encoder, a second phoneme position encoder, a positional attention sub-module, a decoder, a U-net network, and a compression sub-module. The output of each LoRA module in the timbre cloning sub-model is injected into the corresponding sub-module in the text feature processing module, thereby incorporating the timbre features of the target user.
[0111] The Text-Encoder extracts text features from the text sample data and can be constructed using six sub-convolutions. The Second Phoneme Position Encoder (PSE) associates the text features with the acoustic features. It can be constructed using a modeling approach similar to the Tacotron model to compress model parameters and reduce the complexity of subsequent feature processing. Positional attention extracts features from the output of the PSE to obtain positional encoded features, thus concentrating the effective text content into a spatial dimension that is then combined with the subsequent acoustic features.
[0112] See Figure 3 As shown, the first Phoneme Position Encoding and Positional Attention in the audio feature processing module are connected to the decoder. Through this connection, partial features corresponding to the current frame in the output of the first Phoneme Position Encoding are obtained, and these partial features are horizontally projected. Then, a window of random size is decoded based on the horizontally projected data and the random positions of the position-encoded features. The decoder includes two 1x1 convolutions with ELU activation and the output of a sigmoid function. Each feature can have a dedicated feature decoder. Unet is used for feature space scaling and consists of three 1x1 convolutional layers with ELU activation. Compress is used to prepare for subsequent data processing and contains four 1x1 convolutions with ELU activation and max pooling.
[0113] S2031-3: Based on the audio features of non-target users, the audio features of target users, and text sample data, make predictions to obtain the predicted Mel spectrogram.
[0114] In one implementation, non-target user audio features, target user audio features, and text sample data are input into a speech synthesis module to obtain a predicted Mel spectrogram.
[0115] The speech synthesis module forms the backbone network of the initial acoustic model. The audio feature processing module and the user timbre injection module are used for feature extraction and optimization, serving as inputs to the speech synthesis module. The predicted Mel spectrogram can be the Mel spectrogram of the predicted timbre.
[0116] The following is combined Figure 4 The speech synthesis module is explained. (See attached document.) Figure 4 The diagram shown is a structural schematic of a speech synthesis module. Figure 4 In this system, the speech synthesis module includes a spectrogram prediction network and a vocoder. The spectrogram prediction network is an encoder-attention-decoder network, and both its input and output data can be sequences. For example, text sample data is a sequence of characters, and the predicted Mel spectrogram is a sequence of frames.
[0117] The spectrogram prediction network includes an encoder, a location-sensitive attention mechanism, and a decoder. Text sample data is input into the encoder to obtain a character sequence. Non-target user audio features, target user audio features, and the character sequence are input into the location-sensitive attention mechanism to obtain location-sensitive attention features. The decoder then uses these location-sensitive attention features to obtain a predicted Mel spectrogram. The vocoder can be a modified version of a deep neural network (WaveNet) used to generate a temporal waveform, i.e., a sound wave, or synthesized speech, based on the predicted Mel spectrogram. Optionally, the vocoder can be omitted during model training and used only for speech synthesis during model inference.
[0118] In this way, by extracting the audio features of the target user, the problem of excessive differences between synthesized speech and target user speech can be effectively solved, and personalized speech synthesis for the target user can be achieved.
[0119] S2032: Determine the prediction loss based on the target Mel spectrogram of the target user and the predicted Mel spectrogram.
[0120] In one implementation, the prediction loss can be a normalized Mel-rate, and the following formula can be used to determine the prediction loss:
[0121]
[0122] Where L is the prediction loss, X mel For the target Mel spectrum, X′ mel To predict the Mel spectrum, min(X′) mel ) represents the minimum value among multiple predicted Mel spectrograms of the same batch of training data, max(X′) mel ) represents the maximum value among multiple predicted Mel spectrograms of the same batch of training data. The coefficient of exp is used to control the effect of singing perturbation during the generation process.
[0123] S2033: Based on the prediction loss, adjust the parameters of the timbre clone sub-model in the target model until a well-trained target acoustic model is obtained.
[0124] In this way, only some model parameters (i.e., the parameters of the timbre cloning sub-model) need to be fine-tuned, which reduces the complexity of model training, improves the efficiency of model training, and allows the timbre characteristics of the target user to be learned through the timbre cloning sub-model so that a singing voice similar to the user's timbre can be synthesized in the future.
[0125] Step 204: Identify the timbre clone sub-model in the target model as the trained timbre clone sub-model.
[0126] In this embodiment of the disclosure, after obtaining the trained timbre clone sub-model, the trained timbre clone sub-model can be sent to the speech synthesis device to generate a speech synthesis model.
[0127] In one implementation, the speech synthesis device receives a trained timbre clone sub-model from a model training device; the timbre clone sub-model is then combined with a pre-trained local base sub-model to obtain a speech synthesis model. The timbre clone sub-model is constructed based on low-rank adaptive technology and trained on audio sample data of the target user, and is used to learn the timbre features of the target user.
[0128] After generating the speech synthesis model, speech synthesis can be performed based on the text to be processed using the speech synthesis model. The following section will combine... Figure 5 The methods of speech synthesis are explained; please refer to [link / reference]. Figure 5 The diagram shown is a flowchart of a speech synthesis method according to an embodiment of this disclosure. The specific implementation process of the method includes steps 501-502.
[0129] Step 501: Obtain the data to be processed.
[0130] The data to be processed includes at least one of the following: the text to be processed and the target user's audio.
[0131] In one implementation, when the model fine-tuning conditions are met, the text to be processed and the target user's audio are obtained as the data to be processed.
[0132] Optionally, the model fine-tuning condition can be the first or the first n speech synthesiss after the speech synthesis model is generated, where n is a positive integer, or it can be the fine-tuning instruction received from the user.
[0133] In this way, when initially using a speech synthesis model or when it is desired to improve the quality of speech synthesis to better reflect the characteristics of the target user's voice, the text to be processed and the target user's audio can be obtained. In subsequent steps, the target user's audio can be combined for speech synthesis to improve the performance of speech synthesis and obtain synthesized speech that is more in line with the characteristics of the target user's voice.
[0134] Step 502: Input the data to be processed into the speech synthesis model to obtain the synthesized speech of the target user.
[0135] For example, if the data to be processed is only text, the text can be input into the speech synthesis model to obtain the synthesized speech of the target user. If the data to be processed is only audio from the target user, the text of the target user's audio can be extracted as the text to be processed, and both the target user's audio and the text to be processed can be input into the speech synthesis model to obtain the synthesized speech of the target user.
[0136] Furthermore, the data to be processed can also be empty, that is, no text to be processed can be input into the speech synthesis model. In this case, the speech synthesis model will generate random text features without any text to be processed for reference, and synthesize speech based on the random text features. The effect is usually a random melody humming, which may not have any practical significance, but it is still loved by users as a random humming effect.
[0137] In this embodiment, the timbre cloning sub-model is trained using a combination of cloud and mobile devices. This eliminates the need to adjust all model parameters of the overall model, requiring only minor adjustments to some parameters. Furthermore, complex preprocessing operations such as lyric alignment are not required during vocal synthesis, reducing computational, storage, and time costs, as well as the error rate of speech synthesis. Moreover, building the timbre cloning sub-model based on LoRA technology reduces model parameters while simultaneously learning the target user's timbre. The resulting speech synthesis model can transfer the unseen speaker's or singer's vocal style, reducing the similarity difference between synthesized speech and the user's timbre characteristics, and increasing the similarity to the target user's singing. This enables personalized speech synthesis with high-speed simulation for the target user. Training can be performed solely on speech data without dedicated resources, achieving frame-level replication of the speaker / singer's style and precise replication in pitch and speaker similarity. It is also suitable for speech synthesis scenarios with rich content such as melody, rhythm, and chords while preserving the user's timbre information. Furthermore, the operation process of speech synthesis is simple, requiring no professional personnel, making it convenient for ordinary users and improving the user experience.
[0138] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points shall be provided for users to choose to authorize or refuse.
[0139] Based on the same inventive concept, this disclosure also provides a speech synthesis apparatus. Since the principle of the above-mentioned apparatus and device in solving the problem is similar to that of a speech synthesis method, the implementation of the above-mentioned apparatus can refer to the implementation of the method, and repeated details will not be elaborated further. This apparatus can be applied to electronic devices. This disclosure does not limit the type of electronic device; it can be any suitable type of device, such as terminal devices and servers, etc., which will not be elaborated further in this disclosure. The apparatus embodiment can be implemented by software, or by hardware, or a combination of hardware and software. Taking software implementation as an example, as a logically defined apparatus, it is formed by the processor of the electronic device reading the corresponding computer program instructions from the non-volatile memory into memory and running them.
[0140] See Figure 6 The diagram shown is a structural block diagram of a speech synthesis apparatus according to an embodiment of this disclosure. In some embodiments, the speech synthesis apparatus of this disclosure includes:
[0141] Acquisition unit 601 is used to acquire data to be processed;
[0142] The acquisition unit 602 is used to input the data to be processed into the speech synthesis model to obtain the synthesized speech of the target user. The speech synthesis model is obtained by combining the trained timbre cloning sub-model with the local base sub-model. The timbre cloning sub-model is constructed based on low-rank adaptive technology and trained based on the audio sample data of the target user, and is used to learn the timbre features of the target user.
[0143] In one implementation, the data to be processed includes at least one of the following: text to be processed and audio of the target user.
[0144] In one embodiment, the obtaining unit 602 is further configured to: train and obtain a timbre clone sub-model using the following steps:
[0145] Obtain training data;
[0146] The initial timbre clone sub-model is combined with the pre-trained cloud-based basic sub-model to obtain the initial acoustic model; the initial acoustic model is built based on unsupervised end-to-end learning.
[0147] Based on the training data, the initial acoustic model is trained to obtain the trained target acoustic model;
[0148] The timbre clone sub-model in the target model is identified as the trained timbre clone sub-model.
[0149] In one embodiment, the obtaining unit 602 is further configured to:
[0150] The training data is input into the initial acoustic model to obtain the predicted Mel spectrogram;
[0151] Based on the target Mel spectrogram of the target user and the predicted Mel spectrogram, determine the prediction loss;
[0152] Based on the predicted loss, the parameters of the timbre clone sub-model in the target model are adjusted until a well-trained target acoustic model is obtained.
[0153] In one implementation, the training data includes audio sample data of the target user, audio sample data of multiple non-target users, and text sample data;
[0154] The obtaining unit 602 is also used for:
[0155] Feature extraction is performed on audio sample data from non-target users to obtain audio features of non-target users;
[0156] Feature extraction is performed based on text sample data and audio sample data of the target user to obtain the audio features of the target user.
[0157] Based on the audio features of non-target users, the audio features of target users, and text sample data, predictions are made to obtain the predicted Mel spectrogram.
[0158] In one embodiment, the obtaining unit 602 is further configured to:
[0159] Extract initial audio features from audio sample data of non-target users in the audio dimension;
[0160] Phoneme position encoding is performed on the initial audio features to obtain the first phoneme features;
[0161] The first phoneme feature is encoded to obtain the second phoneme feature;
[0162] The first phoneme feature and the second phoneme feature are fused together to obtain the fused feature;
[0163] The fused features are decoded to obtain audio features of non-target users.
[0164] The speech synthesis method in this embodiment includes constructing and training a timbre clone sub-model based on low-rank adaptive technology, combining the trained timbre clone sub-model with a local base sub-model to obtain a speech synthesis model, and generating synthesized speech for the target user based on the data to be processed using the speech synthesis model. In this way, constructing and training the model based on low-rank adaptive technology reduces the amount of training data processing and allows the model to learn the timbre features of the target user for speech synthesis, achieving personalized speech synthesis for the user.
[0165] In this embodiment of the disclosure, an electronic device is also provided, including:
[0166] Processor; and
[0167] The memory stores computer instructions that cause the processor to execute the methods of any of the above-described embodiments.
[0168] In this embodiment of the disclosure, a computer-readable storage medium is provided, storing computer instructions for causing a computer to perform the methods of any of the above embodiments.
[0169] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the method described in any of the above embodiments.
[0170] Figure 7 A schematic diagram of the structure of an electronic device 7000 is shown. (See also...) Figure 7As shown, the electronic device 7000 includes a processor 7010 and a memory 7020, and optionally may also include a power supply 7030, a display unit 7040, and an input unit 7050.
[0171] The processor 7010 is the control center of the electronic device 7000. It connects various components through various interfaces and lines, and performs various functions of the electronic device 7000 by running or executing software programs and / or data stored in the memory 7020, thereby performing overall monitoring of the electronic device 7000.
[0172] In this embodiment of the present disclosure, the processor 7010 executes the steps in the above embodiments when it calls the computer program stored in the memory 7020.
[0173] Optionally, the processor 7010 may include one or more processing units; preferably, the processor 7010 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 7010. In some embodiments, the processor and memory may be implemented on a single chip; in some embodiments, they may also be implemented separately on independent chips.
[0174] The memory 7020 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, various applications, etc.; the data storage area may store data created based on the use of the electronic device 7000, etc. In addition, the memory 7020 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device, etc.
[0175] Electronic device 7000 also includes a power supply 7030 (such as a battery) that supplies power to various components. The power supply can be logically connected to processor 7010 through a power management system, thereby enabling the management of charging, discharging, and power consumption.
[0176] The display unit 7040 can be used to display information input by the user or information provided to the user, as well as various menus of the electronic device 7000. In this embodiment, it is mainly used to display the display interface of various applications in the electronic device 7000, as well as text, images, and other objects displayed on the display interface. The display unit 7040 may include a display panel 7041. The display panel 7041 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.
[0177] The input unit 7050 can be used to receive information such as numbers or characters input by the user. The input unit 7050 may include a touch panel 7051 and other input devices 7052. The touch panel 7051, also known as a touch screen, can collect touch operations on or near the touch panel 7051 (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 7051).
[0178] Specifically, the touch panel 7051 can detect user touch operations and the signals generated by these operations, converting them into touch point coordinates and sending them to the processor 7010. It can also receive and execute commands from the processor 7010. Furthermore, the touch panel 7051 can be implemented using various types of sensors, including resistive, capacitive, infrared, and surface acoustic wave sensors. Other input devices 7052 can include, but are not limited to, one or more of the following: a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick.
[0179] Of course, the touch panel 7051 can cover the display panel 7041. When the touch panel 7051 detects a touch operation on or near it, it transmits the information to the processor 7010 to determine the type of touch event. Subsequently, the processor 7010 provides corresponding visual output on the display panel 7041 based on the type of touch event. Although in Figure 7 In this embodiment, the touch panel 7051 and the display panel 7041 are two separate components to realize the input and output functions of the electronic device 7000. However, in some embodiments, the touch panel 7051 and the display panel 7041 can be integrated to realize the input and output functions of the electronic device 7000.
[0180] The electronic device 7000 may also include one or more sensors, such as a pressure sensor, a gravity acceleration sensor, a proximity light sensor, etc. Of course, depending on the specific application, the electronic device 7000 may also include other components such as a camera. Since these components are not the focus of this disclosure, therefore... Figure 7 It is not shown in the text and will not be described in detail here.
[0181] Those skilled in the art will understand that Figure 7 This is merely an example of an electronic device and does not constitute a limitation on the electronic device. It may include more or fewer components than shown, or a combination of certain components, or different components.
[0182] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this disclosure, the functions of each module (or unit) can be implemented in one or more software or hardware components.
Claims
1. A method of speech generation, characterized by, The method comprises: obtaining to-be-processed data; inputting the to-be-processed data into a speech synthesis model to obtain synthesized speech of a target user; the speech synthesis model is obtained by combining a trained timbre cloning sub-model and a local basic sub-model; the timbre cloning sub-model is constructed based on low-rank adaptive technology and trained based on audio sample data of the target user to learn timbre characteristics of the target user.
2. The method of claim 1, wherein, The to-be-processed data comprises at least one of to-be-processed text and target user audio of the target user.
3. The method according to claim 1 or 2, characterized in that, The timbre cloning sub-model is trained by the following steps: obtaining training data; combining an initial timbre cloning sub-model with a pre-trained cloud basic sub-model to obtain an initial acoustic model; the initial acoustic model is constructed based on unsupervised end-to-end learning; training the initial acoustic model according to the training data to obtain a trained target acoustic model; determining the timbre cloning sub-model in the target model as the trained timbre cloning sub-model.
4. The method of claim 3, wherein, The training of the initial acoustic model according to the training data to obtain a trained target acoustic model comprises: inputting the training data into the initial acoustic model to obtain a predicted mel spectrogram; determining a prediction loss according to a target mel spectrogram of the target user and the predicted mel spectrogram; adjusting parameters of the timbre cloning sub-model in the target model according to the prediction loss until the trained target acoustic model is obtained.
5. The method of claim 4, wherein, The training data comprises audio sample data of the target user, audio sample data of a plurality of non-target users, and text sample data; The inputting of the training data into the initial acoustic model to obtain a predicted mel spectrogram comprises: performing feature extraction on the audio sample data of the non-target users to obtain non-target user audio features; performing feature extraction on the text sample data and the audio sample data of the target user to obtain target user audio features; performing prediction on the non-target user audio features, the target user audio features, and the text sample data to obtain the predicted mel spectrogram.
6. The method of claim 5, wherein, The feature extraction on the audio sample data of the non-target users to obtain non-target user audio features comprises: extracting initial audio features of the audio sample data of the non-target users in an audio dimension; performing phoneme position coding on the initial audio features to obtain first phoneme features; performing feature coding on the first phoneme features to obtain second phoneme features; fusing the first phoneme features and the second phoneme features to obtain fused features; decoding the fused features to obtain non-target user audio features.
7. An apparatus for speech synthesis, characterized by The device comprises: an obtaining unit, configured to obtain to-be-processed data; The obtaining unit is configured to input the to-be-processed data into a speech synthesis model to obtain synthesized speech of the target user, wherein the speech synthesis model is obtained by combining a trained timbre cloning sub-model and a local basic sub-model, the timbre cloning sub-model is constructed based on a low-rank adaptive technology and is trained based on audio sample data of the target user to learn timbre characteristics of the target user.
8. An electronic device, comprising: The method comprises: a processor; and a memory storing computer instructions for causing the processor to perform the method according to any one of claims 1 to 6. The computer instructions are configured to cause a computer to perform the method according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable code or the non-volatile computer-readable storage medium carrying the computer-readable code, when executed in a processor of an electronic device, causes the processor in the electronic device to perform the method according to any one of claims 1 to 6.
10. A computer program product, characterised in that,