Audio synthesis method, device, electronic equipment and storage medium
By extracting and splicing user emotion and speaker feature vectors in the audio synthesis method, and combining them with prediction alignment information, the problem of poor audio synthesis quality and effect in the existing technology is solved, and high-quality audio synthesis is achieved.
Patent Information
- Application Number
- CN202111398062.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-23
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-11-23
AI Technical Summary
Existing audio synthesis methods struggle to effectively combine user emotional characteristics and speaker features, resulting in poor quality and effectiveness of synthesized audio.
By extracting the phoneme feature vectors of the text to be synthesized, predicting alignment information, and combining user emotion and speaker feature vectors with the phoneme feature vectors, the target audio is synthesized using the predicted alignment information. Convolutional neural networks and time-delay neural networks are used for feature extraction and recognition.
It improves the quality and effect of synthesized audio, giving it the user's vocal characteristics and emotional features that match the text, thereby enhancing the accuracy of audio synthesis and the user experience.
Smart Images

Figure CN114220415B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing technology, in particular to the field of speech technology, and specifically to an audio synthesis method, device, electronic device and storage medium. Background Art
[0002] The audio synthesis method in the related art mainly performs emotion conversion on the synthesized audio or audio features of a specific emotion of a target speaker, or performs timbre conversion on the synthesized audio or audio features of a target emotion of a specific speaker, so as to achieve audio synthesis. Summary of the invention
[0003] The present disclosure provides an audio synthesis method, device, electronic device and storage medium.
[0004] According to one aspect of the present disclosure, there is provided an audio synthesis method, comprising:
[0005] Get the text to be synthesized;
[0006] Extracting a plurality of first phoneme feature vectors from the text to be synthesized;
[0007] Predicting alignment information of a plurality of first phoneme feature vectors to obtain predicted alignment information;
[0008] Collecting audio in which the user's emotion is a specific emotion as reference audio, and extracting an emotion feature vector and a speaker feature vector from the reference audio;
[0009] splicing the emotion feature vector and the speaker feature vector onto each first phoneme feature vector to obtain a plurality of second phoneme feature vectors;
[0010] The target audio is synthesized based on the predicted alignment information and the plurality of second phoneme feature vectors.
[0011] According to another aspect of the present disclosure, there is provided an audio synthesis device, comprising:
[0012] A collection module, used to obtain the text to be synthesized;
[0013] An extraction module, used for extracting a plurality of first phoneme feature vectors from the text to be synthesized;
[0014] A calculation module, used for predicting the alignment information of the plurality of first phoneme feature vectors to obtain predicted alignment information;
[0015] The extraction module is further used to collect audio in which the user's emotion is a specific emotion as reference audio, and extract an emotion feature vector and a speaker feature vector from the reference audio;
[0016] The calculation module is further used to concatenate the emotion feature vector and the speaker feature vector onto each first phoneme feature vector to obtain a plurality of second phoneme feature vectors;
[0017] A synthesis module is used to synthesize the target audio based on the predicted alignment information and the multiple second phoneme feature vectors.
[0018] According to another aspect of the present disclosure, there is provided an electronic device, comprising:
[0019] at least one processor; and
[0020] a memory communicatively connected to the at least one processor; wherein,
[0021] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any of the above methods.
[0022] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the above methods.
[0023] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements any of the above methods when executed by a processor.
[0024] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description.
[0025] In the above method of the present invention, by splicing the emotion feature vector and speaker feature vector extracted from the reference audio onto each first phoneme feature vector, the final synthesized target audio has the user's timbre characteristics and emotion characteristics consistent with the text, thereby improving the quality of the final synthesized audio, and synthesizing the target audio based on the predicted alignment information and multiple second phoneme feature vectors, further improving the synthesis effect and quality of the target audio. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0027] Figure 1 is a flowchart of an audio synthesis method provided according to an embodiment of the present disclosure;
[0028] Figure 2It is a flowchart of an audio synthesis model training method provided according to an embodiment of the present disclosure;
[0029] Figure 3 is a flowchart of another audio synthesis model training method provided according to an embodiment of the present disclosure;
[0030] Figure 4 is a flowchart of a specific method for audio synthesis provided according to an embodiment of the present disclosure;
[0031] Figure 5 is a structural schematic diagram of an audio synthesis device provided according to an embodiment of the present disclosure;
[0032] Figure 6 The block diagram is a block diagram of an electronic device used to implement the audio synthesis method of the embodiment of the present disclosure. DETAILED DESCRIPTION
[0033] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0034] In order to improve the quality and speed of synthesized audio, such as Figure 1 As shown, an embodiment of the present disclosure provides an audio synthesis method, the method comprising:
[0035] Step 101, obtaining the text to be synthesized.
[0036] The text of the audio that the user needs to synthesize is obtained as the text to be synthesized.
[0037] Step 102: extract a plurality of first phoneme feature vectors from the text to be synthesized.
[0038] A plurality of first phoneme feature vectors are extracted from the text to be synthesized.
[0039] Step 103: predict the alignment information of the plurality of first phoneme feature vectors to obtain predicted alignment information.
[0040] The duration prediction module in the audio synthesis model is used to predict the alignment information of multiple first phoneme feature vectors to obtain predicted alignment information, where the alignment information refers to the number of frames of each first phoneme feature vector. The duration prediction module is composed of multiple CNNs (convolutional neural network layers) and fully connected layers.
[0041] For example, the three extracted first phoneme feature vectors are [a1, a2, a3], and the predicted alignment information predicted for these three first phoneme feature vectors is [2, 3, 1], which proves that the number of frames predicted by the first first phoneme feature vector is 2, the number of frames predicted by the second first phoneme feature vector is 3, and the number of frames predicted by the third first phoneme feature vector is 1.
[0042] Step 104 , collecting audio in which the user's emotion is a specific emotion as reference audio, and extracting an emotion feature vector and a speaker feature vector from the reference audio.
[0043] The user's audio is collected as reference audio, and the emotion of the reference audio is a specific emotion;
[0044] For example, if the user needs to synthesize an audio with a happy mood, then collect an audio with the user's happy mood as the reference audio. If the audio with the user's happy mood has been collected in advance, it can be used directly without re-collecting.
[0045] Extract audio features of the reference audio from the reference audio, and use the emotion recognition module in the audio synthesis model to extract an emotion feature vector from the audio features of the reference audio. The emotion recognition module consists of multiple CNNs, a pooling layer, a fully connected layer, and softmax (logistic regression);
[0046] The voiceprint recognition module in the audio synthesis model is used to extract a speaker feature vector from the audio features of the reference audio. The voiceprint recognition module consists of multiple TDNNs (time-delay neural networks), a pooling layer, a fully connected layer and softmax.
[0047] Step 105: concatenate the emotion feature vector and the speaker feature vector onto each first phoneme feature vector to obtain a plurality of second phoneme feature vectors.
[0048] The emotion feature vector and the speaker feature vector are concatenated onto each first phoneme feature vector, so that the obtained multiple second phoneme feature vectors carry speaker features and emotion features, and the number of the obtained second phoneme feature vectors is the same as the number of the first phoneme feature vectors.
[0049] Step 106: synthesize target audio based on the predicted alignment information and the plurality of second phoneme feature vectors.
[0050] In the above method provided in this embodiment, the emotion feature vector and speaker feature vector extracted from the reference audio are spliced on each first phoneme feature vector so that the final synthesized target audio has the user's timbre characteristics and emotion features consistent with the text, rather than being achieved by converting the speaker features or the emotion features. This improves the quality of the final synthesized audio, and synthesizing the target audio based on the predicted alignment information and multiple second phoneme feature vectors further improves the target audio synthesis effect and the quality of the target audio.
[0051] In step 102, a plurality of first phoneme feature vectors are extracted from the text to be synthesized. In one embodiment, a plurality of third phoneme feature vectors and position codes corresponding to the third phoneme feature vectors are extracted from the text to be synthesized.
[0052] A plurality of third phoneme feature vectors and position codes corresponding to each third phoneme feature vector are extracted from the text to be synthesized. The plurality of third phoneme feature vectors are vectors of fixed length, and each position code represents position information of the corresponding third phoneme feature vector.
[0053] The plurality of first phoneme feature vectors are extracted from the plurality of third phoneme feature vectors according to the position codes.
[0054] The AM-encoder (acoustic model encoder) module in the audio synthesis model is used to further extract multiple first phoneme feature vectors from multiple third phoneme feature vectors according to multiple position codes. The first phoneme feature vector carries more information than the third phoneme feature vector and carries position information in multiple position codes, which improves the accuracy of subsequent recognition. The AM-encoder module is composed of multiple blocks, each of which contains a self-attention layer and a fully connected layer. By further extracting multiple first phoneme feature vectors from multiple third phoneme feature vectors according to multiple position codes, phoneme features carrying more information are further extracted, so that the final synthesized target audio is more in line with user requirements and has higher quality.
[0055] In step 106, a target audio is synthesized based on the predicted alignment information and the plurality of second phoneme feature vectors. In one embodiment, the plurality of second phoneme feature vectors are adjusted based on the predicted alignment information to obtain a plurality of adjusted second phoneme feature vectors.
[0056] Adjusting the corresponding plurality of second phoneme feature vectors according to the predicted alignment information, where the number of the adjusted second phoneme feature vectors may be greater than the number of the second phoneme feature vectors before the adjustment;
[0057] For example, the three extracted second phoneme feature vectors are [a1, a2, a3], and the predicted alignment information predicted for these three second phoneme feature vectors is [2, 3, 1], then the adjusted multiple second phoneme feature vectors are [a1, a1, a2, a2, a2, a3].
[0058] The target audio is synthesized based on the adjusted plurality of second phoneme feature vectors.
[0059] The adjusted multiple third phoneme feature vectors increase the duration information, and the target audio is synthesized based on the adjusted multiple second phoneme feature vectors, which further improves the target audio synthesis effect and the quality of the target audio.
[0060] Based on the adjusted multiple second phoneme feature vectors, synthesizing the target audio, in one embodiment, converting the adjusted multiple second phoneme feature vectors into predicted Mel-spectrograms;
[0061] The AM-decoder (acoustic model decoder) module in the audio synthesis model is used to predict the Mel-spectra of the adjusted multiple second phoneme feature vectors to obtain the predicted Mel-spectra. The AM-decoder module is similar to the AM-encoder module and is also composed of multiple blocks. Each block includes a self-attention layer and a fully connected layer. However, after the corresponding multiple second phoneme feature vectors are adjusted according to the predicted alignment information, the current output of the self-attention layer in the AM-decoder module is no longer related to each third phoneme feature vector of the input, but is only related to the second phoneme feature vector of the current input phoneme, the second phoneme feature vector of the previous phoneme of the current input, and the second phoneme feature vector of the next phoneme of the current input.
[0062] The target audio is synthesized according to the predicted Mel-spectrogram.
[0063] An embodiment of the present disclosure provides a method for training an audio synthesis model, the method comprising:
[0064] Step 201 : collecting a training audio set, wherein the training audio set includes a plurality of training audios and text content, emotion labels, and speaker labels corresponding to each training audio.
[0065] A large number of audios with emotions from different speakers are collected to form a training audio set. Each audio is annotated with its text content, emotion label and speaker label, which are used to train the audio synthesis model, so that the model can improve various recognition and synthesis capabilities after training.
[0066] Step 202: According to each training audio sample, a training audio sample having the same emotion label and speaker label as the training audio sample is selected from the training audio set as a reference audio of the training audio sample.
[0067] Step 203: Calculate the alignment information loss value according to the audio alignment information and the predicted alignment information corresponding to each training audio sample.
[0068] Step 204: Calculate the spectrum loss value according to the audio spectrum features and the predicted spectrum features corresponding to each training audio sample.
[0069] Step 205 , calculating an emotion loss value according to the emotion recognition result of the reference audio corresponding to each training audio sample, the predicted emotion recognition result of each training audio sample and the emotion label of the reference audio.
[0070] Step 206: Calculate the loss value corresponding to each training audio sample according to the alignment information loss value, spectrum loss value and emotion loss value corresponding to the training audio sample.
[0071] Step 207: Update the parameters of the audio synthesis model according to the loss values corresponding to the training audio samples.
[0072] The loss value corresponding to the training audio sample is calculated by the alignment information loss value, spectrum loss value and emotion loss value corresponding to the training audio sample. Therefore, by using the loss value corresponding to the training audio sample to update the parameters of the audio synthesis model, the ability of the audio synthesis model to extract phoneme features, speaker features and emotion features can be greatly improved, and the audio synthesis model's ability to recognize speakers and emotions in audio features can be improved.
[0073] In one possible implementation manner, the audio alignment information and the audio spectrum features are obtained in the following manner:
[0074] Corresponding audio alignment information and audio spectrum features are extracted from each of the training audio samples.
[0075] In one possible implementation, the predicted alignment information is obtained in the following manner:
[0076] Extracting a plurality of first phoneme feature vectors from the text content of each training audio sample;
[0077] Alignment information of a plurality of first phoneme feature vectors corresponding to each training audio sample is predicted to obtain predicted alignment information.
[0078] The duration prediction module in the audio synthesis model is used to predict the alignment information of multiple first phoneme feature vectors corresponding to each training audio sample to obtain predicted alignment information, where the alignment information refers to the number of frames of each first phoneme feature vector.
[0079] In one possible implementation, the emotion recognition result of the reference audio is obtained by:
[0080] Extracting emotion feature vectors and speaker feature vectors from the reference audio corresponding to each training audio sample;
[0081] Extracting audio features of the reference audio from the reference audio corresponding to each training audio sample, and extracting an emotion feature vector from the audio features of the reference audio corresponding to each training audio sample using the emotion recognition module in the audio synthesis model;
[0082] The voiceprint recognition module in the audio synthesis model is used to extract a speaker feature vector from the audio features of the reference audio corresponding to each training audio sample.
[0083] An emotion recognition result of the reference audio is calculated based on the emotion feature vector.
[0084] The emotion recognition module in the audio synthesis model is used to identify the emotion feature vector and obtain the emotion recognition result of the reference audio.
[0085] In one possible implementation, the predicted spectrum feature is obtained by:
[0086] The emotion feature vector and the speaker feature vector corresponding to each training audio sample are concatenated onto each first phoneme feature vector corresponding to the training audio sample to obtain a plurality of second phoneme feature vectors.
[0087] The emotion feature vector and speaker feature vector corresponding to each training audio sample are concatenated onto each first phoneme feature vector corresponding to the training audio sample, so that the obtained multiple second phoneme feature vectors corresponding to the training audio sample have speaker features and emotion features.
[0088] Based on the prediction alignment information corresponding to each training audio sample, the multiple second phoneme feature vectors corresponding to the training audio sample are adjusted to obtain the adjusted multiple second phoneme feature vectors corresponding to each training audio sample.
[0089] The multiple second phoneme feature vectors corresponding to each training audio sample are adjusted according to the prediction alignment information corresponding to the training audio sample, and the number of the adjusted second phoneme feature vectors may be greater than the number of the second phoneme feature vectors before the adjustment.
[0090] The adjusted plurality of second phoneme feature vectors corresponding to each training audio sample are converted into predicted Mel-spectrum features to obtain the predicted spectrum features.
[0091] In one possible implementation, the predicted emotion recognition result is obtained by:
[0092] The predicted emotion recognition result is calculated based on the predicted Mel spectrum corresponding to each training audio sample.
[0093] Emotion recognition is performed on the predicted Mel spectrum corresponding to each training audio sample to obtain the predicted emotion recognition result.
[0094] Extracting a plurality of first phoneme feature vectors from the text content of each training audio sample, and in one embodiment, extracting a plurality of third phoneme feature vectors and position codes corresponding to each third phoneme feature vector from the text content of each training audio sample;
[0095] A plurality of third phoneme feature vectors and position codes corresponding to each third phoneme feature vector are extracted from the text content of each training audio sample, wherein the plurality of third phoneme feature vectors are vectors of fixed length, and each position code represents position information of the corresponding third phoneme feature vector.
[0096] The multiple first phoneme feature vectors are extracted from the multiple third phoneme feature vectors corresponding to each training audio sample according to the multiple position codes corresponding to the training audio sample.
[0097] The AM-encoder (acoustic model encoder) module in the audio synthesis model is used to further extract multiple first phoneme feature vectors from multiple third phoneme feature vectors corresponding to each training audio sample according to multiple position codes corresponding to the training audio sample. The first phoneme feature vector carries more information than the third phoneme feature vector.
[0098] In step 205, an emotion loss value is calculated according to the reference audio emotion recognition result corresponding to each training audio sample, the predicted emotion recognition result and the emotion label of the reference audio. In one embodiment, a first sub-emotion loss value is calculated according to the reference audio emotion recognition result corresponding to each training audio sample and the predicted emotion recognition result of the training audio sample;
[0099] Calculate a second sub-emotion loss value according to the reference audio emotion recognition result corresponding to each training audio sample and the emotion label of the reference audio;
[0100] The emotion loss value is calculated according to the first sub-emotion loss value and the second sub-emotion loss value corresponding to each training audio sample.
[0101] The emotion loss value not only considers the difference between the reference audio emotion recognition result and the predicted Mel audio emotion recognition result, but also the difference between the reference audio emotion recognition result and the real emotion label of the reference audio. The loss value is calculated according to the final calculated emotion loss value and used to update the parameters of the audio synthesis model, which can further improve the audio synthesis model's emotion recognition ability and ability to extract emotion features.
[0102] In step 207, the parameters of the audio synthesis model are updated according to the loss values corresponding to the respective training audio samples. In one embodiment, the total loss value is calculated according to the loss values corresponding to the respective training audio samples.
[0103] Add up the loss values corresponding to each training audio sample and calculate the total loss value.
[0104] The parameters of the audio synthesis model are updated according to the total overall loss value.
[0105] like Figure 2 As shown, in step 207, after the parameters of the audio synthesis model are updated according to the loss values corresponding to each of the training audio samples, in one embodiment, step 301 determines whether the number of training times for the audio synthesis model is equal to a preset number of times;
[0106] Step 302: If yes, stop training the audio synthesis model;
[0107] Step 303: If not, continue training the audio synthesis model.
[0108] According to the estimation, a preset number of training times is set for the audio synthesis model. After multiple training audio samples and corresponding reference audios are selected for the first training, they will not be re-selected for subsequent training. The multiple training audio samples and corresponding reference audios selected for the first training will be directly used. Therefore, steps 203 to 207 are executed once for a training of the audio synthesis model. After each training, it is determined whether the number of training times for the audio synthesis model has reached the preset number. If not, the training is continued. If reached, the training of the audio synthesis model is completed and further training is stopped. As long as the preset number of training times for the audio synthesis model is set according to the estimation, the automatic training of the audio synthesis model can be realized without manual participation, thus saving a lot of manpower costs.
[0109] like Figure 3 As shown, after the parameters of the audio synthesis model are updated according to the overall loss total value, in one possible implementation, step 401 records the overall loss total value as a historical overall loss total value;
[0110] Step 402, recalculating the total overall loss value according to the plurality of training audio samples and the corresponding reference audio;
[0111] Step 403, determining whether the difference between the recalculated total loss value and the historical total loss value is less than or equal to a preset threshold;
[0112] Step 404: If yes, stop training the audio synthesis model;
[0113] Step 405: If not, continue to record the recalculated total loss value as the historical total loss value and continue to train the audio synthesis model.
[0114] The total overall loss value is recorded as the historical total overall loss value. After multiple training audio samples and corresponding reference audios are selected for the first training, the subsequent training will no longer be reselected, and the multiple training audio samples and corresponding reference audios selected for the first training will be directly used. Therefore, the subsequent training only needs to execute steps 203 to 207, recalculate the total overall loss value, and determine whether the difference between the recalculated total overall loss value and the historical total overall loss value is less than or equal to the preset threshold. If so, the training is completed. If not, the recalculated total overall loss value is recorded as the new historical total overall loss value, and steps 203 to 207 are re-executed to recalculate the total overall loss value until the difference between the total overall loss value and the historical total overall loss value is less than or equal to the preset threshold. Setting the preset threshold not only realizes the automated training of the audio synthesis model without manual participation, thus saving a lot of manpower costs, but also prevents the audio synthesis model from failing to meet the performance requirements due to insufficient training times or overfitting, thereby improving the stability of the audio synthesis model.
[0115] like Figure 4 As shown, an embodiment of the present disclosure provides a specific method for audio synthesis, the method comprising:
[0116] Extracting a plurality of third phoneme feature vectors and position codes corresponding to the third phoneme feature vectors from the text to be synthesized;
[0117] Extracting a plurality of first phoneme feature vectors from a plurality of third phoneme feature vectors according to position coding using an AM-encoder module;
[0118] Using a duration prediction module to predict alignment information of a plurality of first phoneme feature vectors, to obtain predicted alignment information;
[0119] Extract emotion feature vectors from reference audio using emotion recognition module;
[0120] Use the voiceprint recognition module to extract the speaker feature vector from the reference audio;
[0121] splicing the emotion feature vector and the speaker feature vector onto each first phoneme feature vector to obtain a plurality of second phoneme feature vectors;
[0122] Performing LR (logistic regression) on the plurality of second phoneme feature vectors based on the predicted alignment information to obtain a plurality of adjusted second phoneme feature vectors;
[0123] The adjusted plurality of second phoneme feature vectors are converted into predicted Mel-spectrograms by an AM-decoder module through a linear layer;
[0124] The target audio is synthesized using a vocoder based on the predicted Mel-spectrogram.
[0125] An embodiment of the present disclosure provides an audio synthesis device, such as Figure 5 As shown, the device comprises:
[0126] The acquisition module 10 is used to obtain the text to be synthesized;
[0127] An extraction module 20, configured to extract a plurality of first phoneme feature vectors from the text to be synthesized;
[0128] A calculation module 30, configured to predict alignment information of a plurality of first phoneme feature vectors to obtain predicted alignment information;
[0129] The extraction module 20 is further used to collect audio in which the user's emotion is a specific emotion as a reference audio, and extract an emotion feature vector and a speaker feature vector from the reference audio;
[0130] The calculation module 30 is further used to concatenate the emotion feature vector and the speaker feature vector onto each first phoneme feature vector to obtain a plurality of second phoneme feature vectors;
[0131] The synthesis module 40 is used to synthesize the target audio based on the predicted alignment information and the multiple second phoneme feature vectors.
[0132] The extraction module 20 is further used to extract a plurality of third phoneme feature vectors and position codes corresponding to the third phoneme feature vectors from the text to be synthesized;
[0133] The extraction module 20 is further configured to extract the plurality of first phoneme feature vectors from the plurality of third phoneme feature vectors according to the position codes.
[0134] The calculation module 30 is further configured to adjust the plurality of second phoneme feature vectors based on the predicted alignment information to obtain a plurality of adjusted second phoneme feature vectors;
[0135] The synthesis module 40 is further configured to synthesize the target audio based on the adjusted plurality of second phoneme feature vectors.
[0136] Wherein, the calculation module 30 is further used to convert the adjusted plurality of second phoneme feature vectors into predicted Mel-spectrograms;
[0137] The synthesis module 40 is further configured to synthesize the target audio according to the predicted Mel spectrum.
[0138] The acquisition module 10 is further used to acquire a training audio sample set, wherein the training audio sample set includes a plurality of training audio samples and text content, emotion labels and speaker labels corresponding to each training audio sample;
[0139] The extraction module 20 is further configured to select, from the training audio sample set, a training audio sample having the same emotion label and speaker label as the training audio sample as a reference audio of the training audio sample, according to each training audio sample;
[0140] The calculation module 30 is further used to calculate the alignment information loss value according to the audio alignment information and the predicted alignment information corresponding to each training audio sample;
[0141] The calculation module 30 is further used to calculate the spectrum loss value according to the audio spectrum features and the predicted spectrum features corresponding to each training audio sample;
[0142] The calculation module 30 is further used to calculate the emotion loss value according to the emotion recognition result of the reference audio corresponding to each training audio sample, the predicted emotion recognition result of each training audio sample and the emotion label of the reference audio;
[0143] The calculation module 30 is further used to calculate the loss value corresponding to the training audio sample according to the alignment information loss value, spectrum loss value and emotion loss value corresponding to each training audio sample;
[0144] The calculation module 30 is further used to update the parameters of the audio synthesis model according to the loss value corresponding to each of the training audio samples.
[0145] The extraction module 20 is further configured to extract corresponding audio alignment information and audio spectrum features from each training audio sample.
[0146] Wherein, the extraction module 20 is further used to extract a plurality of first phoneme feature vectors from the text content of each training audio sample;
[0147] The calculation module 30 is further used to predict the alignment information of multiple first phoneme feature vectors corresponding to each training audio sample to obtain predicted alignment information.
[0148] Wherein, the extraction module 20 is further used to extract an emotion feature vector and a speaker feature vector from the reference audio corresponding to each training audio sample;
[0149] The calculation module 30 is further used to calculate the emotion recognition result of the reference audio according to the emotion feature vector.
[0150] The calculation module 30 is further used to concatenate the emotion feature vector and the speaker feature vector corresponding to each training audio sample onto each first phoneme feature vector corresponding to the training audio sample to obtain a plurality of second phoneme feature vectors;
[0151] The calculation module 30 is further used to adjust the multiple second phoneme feature vectors corresponding to each training audio sample based on the prediction alignment information corresponding to the training audio sample to obtain the adjusted multiple second phoneme feature vectors corresponding to each training audio sample;
[0152] The calculation module 30 is further used to convert the adjusted plurality of second phoneme feature vectors corresponding to each training audio sample into a predicted Mel spectrum to obtain the predicted spectrum feature.
[0153] The calculation module 30 is further configured to calculate the predicted emotion recognition result according to the predicted Mel spectrum corresponding to each training audio sample.
[0154] The extraction module 20 is further configured to extract a plurality of third phoneme feature vectors and position codes corresponding to the third phoneme feature vectors from the text content of each training audio sample;
[0155] The extraction module 20 is further configured to extract the plurality of first phoneme feature vectors from the plurality of third phoneme feature vectors corresponding to each training audio sample according to the plurality of position codes corresponding to the training audio sample.
[0156] The calculation module 30 is further used to calculate the first sub-emotion loss value according to the reference audio emotion recognition result corresponding to each training audio sample and the predicted emotion recognition result of the training audio sample;
[0157] The calculation module 30 is further used to calculate the second sub-emotion loss value according to the reference audio emotion recognition result corresponding to each training audio sample and the emotion label of the reference audio;
[0158] The calculation module 30 is further used to calculate the emotion loss value according to the first sub-emotion loss value and the second sub-emotion loss value corresponding to each training audio sample.
[0159] The calculation module 30 is further used to calculate the total loss value according to the loss values corresponding to each of the training audio samples;
[0160] The calculation module 30 is further used to update the parameters of the audio synthesis model according to the total value of the overall loss.
[0161] The calculation module 30 is further used to determine whether the number of training times for the audio synthesis model reaches a preset number;
[0162] The calculation module 30 is further configured to stop training the audio synthesis model if yes;
[0163] The calculation module 30 is further used to continue training the audio synthesis model if the answer is no.
[0164] Wherein, the calculation module 30 is further used to record the total overall loss value as the total overall loss value of history;
[0165] The calculation module 30 is further used to recalculate the total overall loss value based on the multiple training audio samples and the corresponding reference audio;
[0166] The calculation module 30 is further used to determine whether the difference between the recalculated total loss value and the historical total loss value is less than or equal to a preset threshold;
[0167] The calculation module 30 is further configured to stop training the audio synthesis model if yes;
[0168] The calculation module 30 is further used to, if not, continue to record the recalculated total overall loss value as the historical total overall loss value and train the audio synthesis model.
[0169] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0170] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0171] Figure 6A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0172] like Figure 6 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0173] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0174] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as an audio synthesis method. For example, in some embodiments, the audio synthesis method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the audio synthesis method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the audio synthesis method in any other appropriate manner (e.g., by means of firmware).
[0175] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0176] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0177] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0178] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, audio input, or tactile input).
[0179] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0180] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0181] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0182] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. An audio synthesis method, include: Get the text to be synthesized; Extracting a plurality of first phoneme feature vectors from the text to be synthesized; Predicting alignment information of a plurality of first phoneme feature vectors to obtain predicted alignment information; Collecting audio in which the user's emotion is a specific emotion as reference audio, and extracting an emotion feature vector and a speaker feature vector from the reference audio; splicing the emotion feature vector and the speaker feature vector onto each first phoneme feature vector to obtain a plurality of second phoneme feature vectors; The target audio is synthesized based on the predicted alignment information and the plurality of second phoneme feature vectors.
2. The method according to claim 1, wherein a plurality of first phoneme feature vectors are extracted from the text to be synthesized, include: Extracting a plurality of third phoneme feature vectors and position codes corresponding to the third phoneme feature vectors from the text to be synthesized; The plurality of first phoneme feature vectors are extracted from the plurality of third phoneme feature vectors according to the position codes.
3. The method according to claim 1, wherein the target audio is synthesized based on the predicted alignment information and the plurality of second phoneme feature vectors. include: Adjusting the plurality of second phoneme feature vectors based on the predicted alignment information to obtain a plurality of adjusted second phoneme feature vectors; The target audio is synthesized based on the adjusted plurality of second phoneme feature vectors.
4. The method according to claim 3, wherein the target audio is synthesized based on the adjusted plurality of second phoneme feature vectors. include: Converting the adjusted plurality of second phoneme feature vectors into predicted Mel-spectrograms; The target audio is synthesized according to the predicted Mel-spectrogram.
5. A training method for an audio synthesis model, include: Collecting a training audio sample set, wherein the training audio sample set includes a plurality of training audio samples and text content, emotion labels, and speaker labels corresponding to each training audio sample; According to each training audio sample, a training audio sample having the same emotion label and speaker label as the training audio sample is selected from the training audio sample set as a reference audio of the training audio sample; Calculate the alignment information loss value according to the audio alignment information and the predicted alignment information corresponding to each training audio sample; The spectrum loss value is calculated based on the audio spectrum features and predicted spectrum features corresponding to each training audio sample; Calculate the emotion loss value according to the emotion recognition result of the reference audio corresponding to each training audio sample, the predicted emotion recognition result of each training audio sample, and the emotion label of the reference audio; The loss value corresponding to each training audio sample is calculated according to the alignment information loss value, spectrum loss value and emotion loss value corresponding to the training audio sample; The parameters of the audio synthesis model are updated according to the loss values corresponding to the training audio samples.
6. The method according to claim 5, obtaining the audio alignment information and the audio spectrum features by: Corresponding audio alignment information and audio spectrum features are extracted from each of the training audio samples.
7. The method according to claim 5, wherein the prediction alignment information is obtained by: Extracting a plurality of first phoneme feature vectors from the text content of each training audio sample; Alignment information of a plurality of first phoneme feature vectors corresponding to each training audio sample is predicted to obtain predicted alignment information.
8. The method according to claim 5, obtaining the emotion recognition result of the reference audio by: Extracting emotion feature vectors and speaker feature vectors from the reference audio corresponding to each training audio sample; An emotion recognition result of the reference audio is calculated based on the emotion feature vector.
9. The method according to claim 7, wherein the predicted spectrum characteristics are obtained by: splicing the emotion feature vector and the speaker feature vector corresponding to each training audio sample onto each first phoneme feature vector corresponding to the training audio sample to obtain a plurality of second phoneme feature vectors; Adjusting the multiple second phoneme feature vectors corresponding to each training audio sample based on the prediction alignment information corresponding to the training audio sample to obtain the adjusted multiple second phoneme feature vectors corresponding to each training audio sample; The adjusted plurality of second phoneme feature vectors corresponding to each training audio sample are converted into predicted Mel-spectrum features to obtain the predicted spectrum features.
10. The method according to claim 9, obtaining the predicted emotion recognition result by: The predicted emotion recognition result is calculated based on the predicted Mel spectrum corresponding to each training audio sample.
11. The method according to claim 7, wherein a plurality of first phoneme feature vectors are extracted from the text content of each training audio sample. include: Extracting a plurality of third phoneme feature vectors and position codes corresponding to the third phoneme feature vectors from the text content of each training audio sample; The multiple first phoneme feature vectors are extracted from the multiple third phoneme feature vectors corresponding to each training audio sample according to the multiple position codes corresponding to the training audio sample.
12. The method according to claim 5, wherein the emotion loss value is calculated based on the emotion recognition result of the reference audio corresponding to each training audio sample, the predicted emotion recognition result of each training audio sample and the emotion label of the reference audio, include: Calculate a first sub-emotion loss value according to a reference audio emotion recognition result corresponding to each training audio sample and a predicted emotion recognition result of the training audio sample; Calculate a second sub-emotion loss value according to the reference audio emotion recognition result corresponding to each training audio sample and the emotion label of the reference audio; The emotion loss value is calculated according to the first sub-emotion loss value and the second sub-emotion loss value corresponding to each training audio sample.
13. The method according to claim 5, wherein the parameters of the audio synthesis model are updated according to the loss values corresponding to the training audio samples. include: Calculating the total loss value according to the loss values corresponding to the training audio samples; The parameters of the audio synthesis model are updated according to the total overall loss value.
14. The method according to claim 5, further comprising: updating the parameters of the audio synthesis model according to the loss values corresponding to the training audio samples; include: Determine whether the number of training times for the audio synthesis model reaches a preset number; If yes, stop training the audio synthesis model; If not, continue training the audio synthesis model.
15. The method according to claim 13, after updating the parameters of the audio synthesis model according to the overall loss total value, further include: Recording the total overall loss value as the historical total overall loss value; Recalculate the total overall loss value based on the multiple training audio samples and the corresponding reference audio; Determine whether the difference between the recalculated total overall loss value and the historical total overall loss value is less than or equal to a preset threshold; If yes, stop training the audio synthesis model; If not, continue to record the recalculated total overall loss value as the historical total overall loss value and continue to train the audio synthesis model.
16. An audio synthesis device, include: A collection module, used to obtain the text to be synthesized; An extraction module, used for extracting a plurality of first phoneme feature vectors from the text to be synthesized; A calculation module, used for predicting the alignment information of the plurality of first phoneme feature vectors to obtain predicted alignment information; The extraction module is further used to collect audio in which the user's emotion is a specific emotion as reference audio, and extract an emotion feature vector and a speaker feature vector from the reference audio; The calculation module is further used to concatenate the emotion feature vector and the speaker feature vector onto each first phoneme feature vector to obtain a plurality of second phoneme feature vectors; A synthesis module is used to synthesize the target audio based on the predicted alignment information and the multiple second phoneme feature vectors.
17. An electronic device, include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 15.
18. A non-transitory computer-readable storage medium storing computer instructions, in, The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 15.
19. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 15.
Citation Information
Patent Citations
Training method and system for RNN transducer model, and device
CN110689879A
Speech synthesis model training method and speech synthesis method
CN112634856A
Singing sound synthesis
CN112951198A
Voice conversion method and device, server and storage medium
CN113178200A
Cartoon character animation generation method and device, equipment and storage medium
CN113379875A
