Speech synthesis method and device for newly added speaker, equipment and storage medium

By performing low-rank matrix decomposition on the speech synthesis model, the problem of difficulty in feature capture when data is insufficient in traditional models is solved, the quality and naturalness of speech synthesis are improved, and the generalization ability and robustness of the model are enhanced.

CN119091850APending Publication Date: 2024-12-06PING AN TECH (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411138472.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Traditional speech synthesis models require a large amount of data when capturing the characteristics of a specific speaker, resulting in a decrease in speech quality and naturalness when data is insufficient, and parameter selective fine-tuning is difficult to generalize to new text or context.

Method used

The basic model of speech synthesis based on the Adaspeech network structure is adopted, and the one-dimensional convolution weight matrix of the FFT Block module is decomposed into two low-rank matrices through low-rank matrices, and low-dimensional network training is carried out to reduce the computational amount and data dependence.

Benefits of technology

It effectively reduces the computational volume and training parameters of the fine-tuning process, improves the generalization ability of the model, solves the speech quality and naturalness problems caused by insufficient data, and improves the robustness in different contexts.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091850A_ABST
    Figure CN119091850A_ABST
Patent Text Reader

Abstract

The invention discloses a speech synthesis method, device and equipment for a new speaker and a storage medium, and the method comprises the steps: decomposing a part of parameters in a speech synthesis basic model into a low-dimensional matrix through low-rank matrix decomposition based on the speech synthesis basic model and a low-rank matrix decomposition theory, replacing the conventional gradient updating with the training of a low-dimensional network, and improving the speech synthesis efficiency. The method greatly reduces the calculation amount and training parameters in the fine tuning process, reduces the dependence on the data volume, can be applied to different speech synthesis models, has good generalization ability, and solves the problems that in the prior art, a traditional speech synthesis model usually needs a large amount of data to capture the features of a specific speaker, and only a small amount of data is available. The quality and the naturalness of the synthesized speech are reduced due to the fact that the model is difficult to learn enough features; and by adopting parameter selective fine tuning, the naturalness and expressive force of the synthesized speech are still possibly limited, and a small data volume possibly causes model over-fitting and cannot be generalized to new texts or contexts which are not seen.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of speech synthesis technology, and in particular to a speech synthesis method, device, equipment and storage medium for adding a new speaker. Background Art

[0002] The rapid development of deep learning technology, especially in the field of speech synthesis, has achieved remarkable breakthroughs. Modern speech synthesis systems, also known as text-to-speech (TTS) systems, are able to generate human voices that sound very natural and smooth. These systems are usually based on deep neural networks that can learn complex mappings from text to speech. In recent years, the performance of speech synthesis models has greatly improved. Given several hours of high-quality single-speaker or multi-speaker recordings, speech synthesis models can now synthesize high-quality human voices.

[0003] However, when training a new speaker model, a large amount of audio training data is still required to fine-tune the model. Especially in the insurance customer service scenario, recording data for customer service agents takes a lot of time and is difficult to coordinate. This makes it difficult to promote and apply speech synthesis models on a large scale in the insurance customer service field, and there is an urgent need for a method to fine-tune the model with a small amount of data to solve the problem of requiring a large amount of data recording.

[0004] In the field of speech synthesis, fine-tuning models to adapt to the voices of new speakers with a small amount of data is a challenging task. Traditional speech synthesis models usually require a large amount of data to capture the characteristics of a specific speaker. When only a small amount of data is available, it is difficult for the model to learn enough features, resulting in a decrease in the quality and naturalness of the synthesized speech. To address the problem of insufficient data, researchers have tried to fine-tune only the parameters in the model that are most relevant to the speaker's characteristics. For example, only the vocal tract model or the timbre control layer may be adjusted instead of the entire network. Even with selective parameter fine-tuning, the naturalness and expressiveness of the synthesized speech may still be limited. Small amounts of data may cause the model to overfit and fail to generalize to new unseen texts or contexts. The fine-tuned model may perform well in specific situations, but may not be robust enough when faced with different speaking styles, emotional expressions, or noisy environments. Summary of the invention

[0005] The present application provides a speech synthesis method, apparatus, device and storage medium for adding a new speaker, which solves the technical problem that traditional speech synthesis models in the prior art usually require a large amount of data to capture the characteristics of a specific speaker. When only a small amount of data is available, it is difficult for the model to learn enough features, resulting in a decrease in the quality and naturalness of the synthesized speech; and when using selective parameter fine-tuning, the naturalness and expressiveness of the synthesized speech may still be limited, and the small amount of data may cause the model to overfit and fail to generalize to new texts or contexts that have not been seen.

[0006] In view of this, the first aspect of the present application provides a method, the method comprising:

[0007] Step S1, obtaining a basic model for speech synthesis containing multiple speakers, wherein the basic model for speech synthesis is obtained by pre-training based on an Adaspeech network structure, wherein the Adaspeech network structure contains an acoustic condition model AcousticConditionModel and a MEL decoder Mel Decoder, and both the acoustic condition model AcousticConditionModel and the MEL decoder Mel Decoder are based on a FFTBlock module;

[0008] Step S2, based on the low-order matrix decomposition theory, decompose the one-dimensional convolution weight matrix W in the FFT Block module into two low-order matrices W1 and W2;

[0009] Step S3, obtaining first training data of the newly added speaker;

[0010] Step S4, in the fine-tuning stage, the acoustic condition model based on the FFT Block module in the speech synthesis basic model and the two low-order matrices W1 and W2 in the MEL decoder are trained by using the first training data of the newly added speaker to obtain updated low-order matrices W1 and W2;

[0011] Step S5: In the inference stage, the speech synthesis basic model and the updated low-order matrices W1 and W2 are used to perform speech synthesis output for the newly added speech.

[0012] Optionally, before step S1, the step further includes:

[0013] Step S0: Generate a basic model for speech synthesis based on the Adaspeech network structure pre-training by using second training data containing multiple speakers.

[0014] Optionally, the step S0 specifically includes:

[0015] Obtaining second training data including multiple speakers;

[0016] extracting phoneme information of the second training data;

[0017] Extract hidden states from phoneme information through the Acoustic Condition Model;

[0018] The hidden state and the Mel Spectrum MelSpectrum of the corresponding audio are forcibly aligned through a variable information adapter VarianceAdaptor;

[0019] The forced aligned hidden state is mapped to a Mel spectrum through the MEL decoder to generate a basic model for speech synthesis.

[0020] Optionally, the FFT Block module is composed of a multi-head attention mechanism and a one-dimensional convolution weight matrix, and the multi-head attention mechanism and the one-dimensional convolution weight matrix are connected using residual connections and layer normalization connections.

[0021] Optionally, the step S2 specifically includes:

[0022] Based on the low-alias matrix decomposition theory, the one-dimensional convolution weight matrix W in the FFT Block module is decomposed into two low-alias matrices W1 and W2, wherein the dimension of the low-alias matrix W1 is d*r, and the dimension of the low-alias matrix W2 is r*k, where r is much smaller than d and k.

[0023] A second aspect of the present application provides a device, the device comprising:

[0024] A first acquisition unit is used to acquire a basic model for speech synthesis including multiple speakers, wherein the basic model for speech synthesis is obtained by pre-training based on an Adaspeech network structure, wherein the Adaspeech network structure includes an acoustic condition model Acoustic Condition Model and a MEL decoder Mel Decoder, and both the acoustic condition model AcousticConditionModel and the MEL decoder Mel Decoder are based on a FFTBlock module;

[0025] A decomposition unit, used for decomposing the one-dimensional convolution weight matrix W in the FFTBlock module into two low-nothing matrices W1 and W2 based on the low-nothing matrix decomposition theory;

[0026] A second acquisition unit, used to acquire first training data of a newly added speaker;

[0027] A fine-tuning unit, used for training the acoustic condition model based on the FFTBlock module in the speech synthesis basic model and two low-order matrices W1 and W2 in the MEL decoder Mel Decoder in the fine-tuning stage through the first training data of the newly added speaker to obtain updated low-order matrices W1 and W2;

[0028] The inference unit is used to perform speech synthesis output of the newly added speech by using the speech synthesis basic model and the updated low-order matrices W1 and W2 during the inference stage.

[0029] Optionally, it also includes:

[0030] The pre-training unit is used to generate a basic model for speech synthesis based on the Adaspeech network structure through pre-training using second training data containing multiple speakers.

[0031] Optionally, the pre-training unit is specifically used for:

[0032] Obtaining second training data including multiple speakers;

[0033] extracting phoneme information of the second training data;

[0034] Extract hidden states from phoneme information through the Acoustic Condition Model;

[0035] The hidden state and the Mel Spectrum MelSpectrum of the corresponding audio are forcibly aligned through a variable information adapter VarianceAdaptor;

[0036] The forced aligned hidden state is mapped to a Mel spectrum through the MEL decoder to generate a basic model for speech synthesis.

[0037] Optionally, the FFT Block module is composed of a multi-head attention mechanism and a one-dimensional convolution weight matrix, and the multi-head attention mechanism and the one-dimensional convolution weight matrix are connected using residual connections and layer normalization connections.

[0038] Optionally, the decomposition unit is specifically used for:

[0039] Based on the low-alias matrix decomposition theory, the one-dimensional convolution weight matrix W in the FFT Block module is decomposed into two low-alias matrices W1 and W2, wherein the dimension of the low-alias matrix W1 is d*r, and the dimension of the low-alias matrix W2 is r*k, where r is much smaller than d and k.

[0040] A third aspect of the present application provides a speech synthesis device for adding a new speaker, the device comprising a processor and a memory:

[0041] The memory is used to store program codes and transmit the program codes to the processor;

[0042] The processor is used to execute the steps of the method for speech synthesis for adding a speaker as described in the first aspect according to the instructions in the program code.

[0043] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the method described in the first aspect above.

[0044] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0045] In the present application, a speech synthesis method, apparatus, device and storage medium for adding a new speaker are provided. Based on a pre-trained speech synthesis basic model and low-rank matrix decomposition theory, some parameters in the speech synthesis basic model are decomposed into low-dimensional matrices through low-rank matrix decomposition, and the traditional gradient update is replaced by the training of a low-dimensional network, which greatly reduces the amount of calculation and training parameters in the fine-tuning process, reduces the dependence on the amount of data, can be applied to different speech synthesis models, has good generalization ability, and solves the technical problem that traditional speech synthesis models in the prior art usually require a large amount of data to capture the characteristics of a specific speaker. When only a small amount of data is available, it is difficult for the model to learn enough features, resulting in a decrease in the quality and naturalness of the synthesized speech; and when using selective parameter fine-tuning, the naturalness and expressiveness of the synthesized speech may still be limited, and the small amount of data may cause the model to overfit and be unable to generalize to new texts or contexts that have not been seen. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A method flow chart of a speech synthesis method for adding a new speaker in an embodiment of the present application;

[0047] Figure 2 This is a structural schematic diagram of a speech synthesis device for adding a new speaker in an embodiment of the present application;

[0048] Figure 3 This is a structural schematic diagram of a speech synthesis device for adding a new speaker in an embodiment of the present application;

[0049] Figure 4 This is the overall architecture diagram of the Adaspeech network structure in the embodiment of the present application;

[0050] Figure 5This is a first overall architecture diagram of the FFT Block module in an embodiment of the present application;

[0051] Figure 6 This is a second overall architecture diagram of the FFT Block module in the embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0053] The present application designs a speech synthesis method, apparatus, device and storage medium for adding a new speaker, which solves the technical problem that traditional speech synthesis models in the prior art usually require a large amount of data to capture the characteristics of a specific speaker. When only a small amount of data is available, it is difficult for the model to learn enough features, resulting in a decrease in the quality and naturalness of the synthesized speech; and when using selective parameter fine-tuning, the naturalness and expressiveness of the synthesized speech may still be limited, and the small amount of data may cause the model to overfit and fail to generalize to new texts or contexts that have not been seen.

[0054] For easier understanding, see Figure 1 , Figure 1 Flowchart of the method for speech synthesis of adding a new speaker in the embodiment of the present application. Figure 1 As shown, specifically:

[0055] Step S1, obtaining a basic speech synthesis model including multiple speakers, wherein the basic speech synthesis model is pre-trained based on an Adaspeech network structure, wherein the Adaspeech network structure includes an acoustic condition model AcousticCondition Model and a MEL decoder Mel Decoder, and both the acoustic condition model Acoustic Condition Model and the MEL decoder Mel Decoder are based on an FFT Block module;

[0056] It should be noted that the overall architecture of the Adaspeech network structure is as follows Figure 4As shown in the figure, fastspeech2 is used as the basic structure of the model, which includes: Phoneme Encoder, Mel Decoder, VarianceAdaptor. On this basis, two additional components are added to solve the goal of sound customization: AcousticConditionModel and conditional layernormalization.

[0057] Both Acoustic ConditionModel and Mel Decoder are based on Figure 5 The FFTBlock module shown in the figure consists of a multi-head attention mechanism and a one-dimensional convolution, in which residual connections and layer normalization (LN) are utilized.

[0058] Step S2, based on the low-order matrix decomposition theory, decompose the one-dimensional convolution weight matrix W in the FFT Block module into two low-order matrices W1 and W2;

[0059] It should be noted that based on the low-inclusive matrix decomposition theory, we believe that the one-dimensional convolution weight matrix W in the FFT Block module in the basic model of speech synthesis is a non-full-inclusive matrix with a dimension of d*k;

[0060] Decompose W=W1W2 into two low-insignificance matrices W1 and W2, with dimensions d*r and r*k respectively, where r is much smaller than d and k.

[0061] Then the weight update process of the one-dimensional convolution weight matrix W in the FFT Block module is:

[0062] Wt=Wt-1+△Wt, △Wt is the gradient;

[0063] Can be improved to:

[0064] Wt=Wt-1+W1W2.

[0065] The traditional gradient update process is replaced by a two-layer network, which greatly reduces the amount of calculation and training parameters in the fine-tuning process and reduces the dependence on the amount of data.

[0066] Step S3, obtaining first training data of the newly added speaker;

[0067] Step S4, in the fine-tuning stage, the acoustic condition model based on the FFT Block module in the speech synthesis basic model and the two low-order matrices W1 and W2 in the MEL decoder are trained by adding the first training data of the speaker to obtain updated low-order matrices W1 and W2;

[0068] It should be noted that, in the fine-tuning stage, the weight parameters of the basic model of speech synthesis are loaded and the weight parameters are fixed unchanged.

[0069] The one-dimensional convolution weight matrix W of all FFT Block modules in the basic model of speech synthesis is decomposed into low-order matrix, such as Figure 6 shown.

[0070] The first training data of the newly added speaker is used for training according to the training method of the basic model for speech synthesis, but only the weight parameters of the low-order matrices W1 and W2 obtained after the low-order matrix decomposition are updated.

[0071] Step S5: In the inference stage, the speech synthesis basic model and the updated low-order matrices W1 and W2 are used to perform speech synthesis output for the newly added speech.

[0072] It should be noted that in the inference stage, the weight W of the pre-trained speech synthesis basic model and the updated low-order matrices W1 and W2 are used together for inference calculation, that is, for the input x and output y, y=Wx+W1*W2*x is obtained.

[0073] Furthermore, before step S1, the following steps are also included:

[0074] Step S0: Generate a basic model for speech synthesis based on the Adaspeech network structure pre-training by using second training data containing multiple speakers.

[0075] Furthermore, step S0 specifically includes:

[0076] Obtaining second training data including multiple speakers;

[0077] extracting phoneme information of second training data;

[0078] Extract hidden states from phoneme information through the Acoustic ConditionModel;

[0079] The hidden state and the Mel Spectrum of the corresponding audio are forced to be aligned through the Variance Adaptor;

[0080] The forced aligned hidden state is mapped to a Mel spectrum through the MEL decoder to generate a basic model for speech synthesis.

[0081] It should be noted that if Figure 4 The overall architecture diagram of the Adaspeech network structure shown in the figure, the process of training the second training data containing multiple speakers is specifically as follows:

[0082] Extract hidden states from phoneme information through the Acoustic ConditionModel;

[0083] The hidden state and the Mel Spectrum of the corresponding audio are forced to be aligned through the Variance Adaptor;

[0084] The forced aligned hidden state is mapped to a Mel spectrum through the MEL decoder to generate a basic model for speech synthesis.

[0085] Furthermore, the FFT Block module consists of a multi-head attention mechanism and a one-dimensional convolution weight matrix, and the multi-head attention mechanism and the one-dimensional convolution weight matrix are connected using residual connections and layer normalization.

[0086] Furthermore, step S2 specifically includes:

[0087] Based on the low-alias matrix decomposition theory, the one-dimensional convolution weight matrix W in the FFT Block module is decomposed into two low-alias matrices W1 and W2, where the dimension of the low-alias matrix W1 is d*r, and the dimension of the low-alias matrix W2 is r*k, where r is much smaller than d and k.

[0088] It should be noted that based on the low-inclusive matrix decomposition theory, we believe that the one-dimensional convolution weight matrix W in the FFT Block module in the basic model of speech synthesis is a non-full-inclusive matrix with a dimension of d*k;

[0089] Decompose W=W1W2 into two low-insignificance matrices W1 and W2, with dimensions d*r and r*k respectively, where r is much smaller than d and k.

[0090] Then the weight update process of the one-dimensional convolution weight matrix W in the FFT Block module is:

[0091] Wt=Wt-1+△Wt, △Wt is the gradient;

[0092] Can be improved to:

[0093] Wt=Wt-1+W1W2.

[0094] The traditional gradient update process is replaced by a two-layer network, which greatly reduces the amount of calculation and training parameters in the fine-tuning process and reduces the dependence on the amount of data.

[0095] See also Figure 2 , Figure 2 Schematic diagram of the structure of the speech synthesis device for adding a speaker in the embodiment of the present application. Figure 2 As shown, specifically:

[0096] A first acquisition unit 201 is used to acquire a basic speech synthesis model including multiple speakers, where the basic speech synthesis model is pre-trained based on an Adaspeech network structure, where the Adaspeech network structure includes an acoustic condition model Acoustic Condition Model and a MEL decoder Mel Decoder, both of which are based on a FFTBlock module;

[0097] It should be noted that the overall architecture of the Adaspeech network structure is as follows Figure 4 As shown in the figure, fastspeech2 is used as the basic structure of the model, which includes: Phoneme Encoder, Mel Decoder, VarianceAdaptor. On this basis, two additional components are added to solve the goal of sound customization: AcousticConditionModel and conditional layernormalization.

[0098] Both Acoustic ConditionModel and Mel Decoder are based on Figure 5 The FFTBlock module shown in the figure consists of a multi-head attention mechanism and a one-dimensional convolution, in which residual connections and layer normalization (LN) are utilized.

[0099] A decomposition unit 202 is used to decompose the one-dimensional convolution weight matrix W in the FFTBlock module into two low-nothing matrices W1 and W2 based on the low-nothing matrix decomposition theory;

[0100] It should be noted that based on the low-inclusive matrix decomposition theory, we believe that the one-dimensional convolution weight matrix W in the FFTBlock module in the basic model of speech synthesis is a non-full-inclusive matrix with a dimension of d*k;

[0101] Decompose W=W1W2 into two low-insignificance matrices W1 and W2, with dimensions d*r and r*k respectively, where r is much smaller than d and k.

[0102] Then the weight update process of the one-dimensional convolution weight matrix W in the FFT Block module is:

[0103] Wt=Wt-1+△Wt, △Wt is the gradient;

[0104] Can be improved to:

[0105] Wt=Wt-1+W1W2.

[0106] The traditional gradient update process is replaced by a two-layer network, which greatly reduces the amount of calculation and training parameters in the fine-tuning process and reduces the dependence on the amount of data.

[0107] A second acquisition unit 203, used to acquire first training data of a newly added speaker;

[0108] The fine-tuning unit 204 is used to train the acoustic condition model Acoustic ConditionModel based on the FFT Block module in the speech synthesis basic model and the two low-order matrices W1 and W2 in the MEL decoder MelDecoder by adding the first training data of the speaker in the fine-tuning stage to obtain updated low-order matrices W1 and W2;

[0109] It should be noted that in the fine-tuning stage, the weight parameters of the speech synthesis basic model are loaded and fixed. The one-dimensional convolution weight matrix W of all FFTBlock modules in the speech synthesis basic model is decomposed into a low-order matrix. Using the first training data of the newly added speaker, training is performed according to the training method of the speech synthesis basic model, but only the weight parameters of the low-order matrix W1 and W2 obtained after the low-order matrix decomposition are updated.

[0110] The inference unit 205 is used to perform speech synthesis output of the newly added speech using the speech synthesis basic model and the updated low-order matrices W1 and W2 during the inference stage.

[0111] It should be noted that in the inference stage, the weight W of the pre-trained speech synthesis basic model and the updated low-order matrices W1 and W2 are used together for inference calculation, that is, for the input x and output y, y=Wx+W1*W2*x is obtained.

[0112] Furthermore, it also includes:

[0113] The pre-training unit is used to generate a basic model for speech synthesis based on the Adaspeech network structure through pre-training using second training data containing multiple speakers.

[0114] Furthermore, the pre-training unit is specifically used for:

[0115] Obtaining second training data including multiple speakers;

[0116] extracting phoneme information of second training data;

[0117] Extract hidden states from phoneme information through the Acoustic ConditionModel;

[0118] The hidden state and the Mel Spectrum of the corresponding audio are forced to be aligned through the Variance Adaptor;

[0119] The forced aligned hidden state is mapped to a Mel spectrum through the MEL decoder to generate a basic model for speech synthesis.

[0120] It should be noted that if Figure 4 The overall architecture diagram of the Adaspeech network structure shown in the figure, the process of training the second training data containing multiple speakers is specifically as follows:

[0121] Extract hidden states from phoneme information through the Acoustic ConditionModel;

[0122] The hidden state and the Mel Spectrum of the corresponding audio are forced to be aligned through the Variance Adaptor;

[0123] The forced aligned hidden state is mapped to a Mel spectrum through the MEL decoder to generate a basic model for speech synthesis.

[0124] Furthermore, the FFTBlock module consists of a multi-head attention mechanism and a one-dimensional convolution weight matrix, and the multi-head attention mechanism and the one-dimensional convolution weight matrix are connected using residual connections and layer normalization.

[0125] Furthermore, the decomposition unit is specifically used for:

[0126] Based on the low-alias matrix decomposition theory, the one-dimensional convolution weight matrix W in the FFT Block module is decomposed into two low-alias matrices W1 and W2, where the dimension of the low-alias matrix W1 is d*r, and the dimension of the low-alias matrix W2 is r*k, where r is much smaller than d and k.

[0127] It should be noted that based on the low-inclusive matrix decomposition theory, we believe that the one-dimensional convolution weight matrix W in the FFTBlock module in the basic model of speech synthesis is a non-full-inclusive matrix with a dimension of d*k;

[0128] Decompose W=W1W2 into two low-insignificance matrices W1 and W2, with dimensions d*r and r*k respectively, where r is much smaller than d and k.

[0129] Then the weight update process of the one-dimensional convolution weight matrix W in the FFT Block module is:

[0130] Wt=Wt-1+△Wt, △Wt is the gradient;

[0131] Can be improved to:

[0132] Wt=Wt-1+W1W2.

[0133] The traditional gradient update process is replaced by a two-layer network, which greatly reduces the amount of calculation and training parameters in the fine-tuning process and reduces the dependence on the amount of data.

[0134] The present application also provides another speech synthesis device for adding a new speaker, such as Figure 3 For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA), a sales terminal (English full name: Point of Sales, English abbreviation: POS), a car computer, etc., taking the terminal as a mobile phone as an example:

[0135] Figure 3 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal provided in an embodiment of the present application. Figure 3 The mobile phone includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art can understand that Figure 3 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0136] Combine the following Figure 3 A detailed introduction to the various components of the mobile phone:

[0137] The RF circuit 1010 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 1080 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 1010 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (full name: Low Noise Amplifier, English abbreviation: LNA), a duplexer, etc. In addition, the RF circuit 1010 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communications can use any communication standard or protocol, including but not limited to Global System of Mobile communications (Global System of Mobile communication, English abbreviation: GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, English abbreviation: CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (Long Term Evolution, English abbreviation: LTE), e-mail, Short Messaging Service (SMS), etc.

[0138] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1020. The memory 1020 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0139] The input unit 1030 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1031 or near the touch panel 1031 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 1080, and can receive and execute commands sent by the processor 1080. In addition, the touch panel 1031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1031, the input unit 1030 may also include other input devices 1032. Specifically, the other input devices 1032 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, etc.

[0140] The display unit 1040 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1040 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of a liquid crystal display (full name in English: Liquid Crystal Display, English abbreviation: LCD), an organic light-emitting diode (full name in English: Organic Light-Emitting Diode, English abbreviation: OLED), etc. Further, the touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides a corresponding visual output on the display panel 1041 according to the type of touch event. Although in Figure 3 In the embodiment, the touch panel 1031 and the display panel 1041 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.

[0141] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0142] The audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the mobile phone. The audio circuit 1060 can transmit the received audio data to the speaker 1061 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data, and then the audio data is output to the processor 1080 for processing, and then sent to another mobile phone through the RF circuit 1010, or the audio data is output to the memory 1020 for further processing.

[0143] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 1070. It provides users with wireless broadband Internet access. Figure 3 A WiFi module 1070 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0144] The processor 1080 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 1020 and calling data stored in the memory 1020, it executes various functions of the mobile phone and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 1080 may include one or more processing units; preferably, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1080.

[0145] The mobile phone also includes a power supply 1090 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1080 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions.

[0146] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0147] In the embodiment of the present application, the processor 1080 included in the terminal also has the following functions:

[0148] Step S1, obtaining a basic speech synthesis model including multiple speakers, wherein the basic speech synthesis model is pre-trained based on an Adaspeech network structure, wherein the Adaspeech network structure includes an acoustic condition model AcousticCondition Model and a MEL decoder Mel Decoder, and both the acoustic condition model Acoustic Condition Model and the MEL decoder Mel Decoder are based on an FFT Block module;

[0149] It should be noted that the overall architecture of the Adaspeech network structure is as follows Figure 4 As shown in the figure, fastspeech2 is used as the basic structure of the model, which includes: Phoneme Encoder, Mel Decoder, VarianceAdaptor. On this basis, two additional components are added to solve the goal of sound customization: AcousticConditionModel and conditional layernormalization.

[0150] Both Acoustic ConditionModel and Mel Decoder are based on Figure 5 The FFTBlock module shown in the figure consists of a multi-head attention mechanism and a one-dimensional convolution, in which residual connections and layer normalization (LN) are utilized.

[0151] Step S2, based on the low-order matrix decomposition theory, decompose the one-dimensional convolution weight matrix W in the FFT Block module into two low-order matrices W1 and W2;

[0152] It should be noted that based on the low-inclusive matrix decomposition theory, we believe that the one-dimensional convolution weight matrix W in the FFT Block module in the basic model of speech synthesis is a non-full-inclusive matrix with a dimension of d*k;

[0153] Decompose W=W1W2 into two low-insignificance matrices W1 and W2, with dimensions d*r and r*k respectively, where r is much smaller than d and k.

[0154] Then the weight update process of the one-dimensional convolution weight matrix W in the FFT Block module is:

[0155] Wt=Wt-1+△Wt, △Wt is the gradient;

[0156] Can be improved to:

[0157] Wt=Wt-1+W1W2.

[0158] The traditional gradient update process is replaced by a two-layer network, which greatly reduces the amount of calculation and training parameters in the fine-tuning process and reduces the dependence on the amount of data.

[0159] Step S3, obtaining first training data of the newly added speaker;

[0160] Step S4, in the fine-tuning stage, the acoustic condition model based on the FFT Block module in the speech synthesis basic model and the two low-order matrices W1 and W2 in the MEL decoder are trained by adding the first training data of the speaker to obtain updated low-order matrices W1 and W2;

[0161] It should be noted that in the fine-tuning stage, the weight parameters of the speech synthesis basic model are loaded and fixed. The one-dimensional convolution weight matrix W of all FFT Block modules in the speech synthesis basic model is decomposed into a low-order matrix. Using the first training data of the newly added speaker, training is performed according to the training method of the speech synthesis basic model, but only the weight parameters of the low-order matrix W1 and W2 obtained after the low-order matrix decomposition are updated.

[0162] Step S5: In the inference stage, the speech synthesis basic model and the updated low-order matrices W1 and W2 are used to perform speech synthesis output for the newly added speech.

[0163] It should be noted that in the inference stage, the weight W of the pre-trained speech synthesis basic model and the updated low-order matrices W1 and W2 are used together for inference calculation, that is, for the input x and output y, y=Wx+W1*W2*x is obtained.

[0164] Furthermore, before step S1, the following steps are also included:

[0165] Step S0: Generate a basic model for speech synthesis based on the Adaspeech network structure pre-training by using second training data containing multiple speakers.

[0166] Furthermore, step S0 specifically includes:

[0167] Obtaining second training data including multiple speakers;

[0168] extracting phoneme information of second training data;

[0169] Extract hidden states from phoneme information through the Acoustic ConditionModel;

[0170] The hidden state and the Mel Spectrum of the corresponding audio are forced to be aligned through the Variance Adaptor;

[0171] The forced aligned hidden state is mapped to a Mel spectrum through the MEL decoder to generate a basic model for speech synthesis.

[0172] It should be noted that if Figure 4 The overall architecture diagram of the Adaspeech network structure shown in the figure, the process of training the second training data containing multiple speakers is specifically as follows:

[0173] Extract hidden states from phoneme information through the Acoustic ConditionModel;

[0174] The hidden state and the Mel Spectrum of the corresponding audio are forced to be aligned through the Variance Adaptor;

[0175] The forced aligned hidden state is mapped to a Mel spectrum through the MEL decoder to generate a basic model for speech synthesis.

[0176] Furthermore, the FFTBlock module consists of a multi-head attention mechanism and a one-dimensional convolution weight matrix, and the multi-head attention mechanism and the one-dimensional convolution weight matrix are connected using residual connections and layer normalization.

[0177] Furthermore, step S2 specifically includes:

[0178] Based on the low-alias matrix decomposition theory, the one-dimensional convolution weight matrix W in the FFT Block module is decomposed into two low-alias matrices W1 and W2, where the dimension of the low-alias matrix W1 is d*r, and the dimension of the low-alias matrix W2 is r*k, where r is much smaller than d and k.

[0179] It should be noted that based on the low-inclusive matrix decomposition theory, we believe that the one-dimensional convolution weight matrix W in the FFT Block module in the basic model of speech synthesis is a non-full-inclusive matrix with a dimension of d*k;

[0180] Decompose W=W1W2 into two low-insignificance matrices W1 and W2, with dimensions d*r and r*k respectively, where r is much smaller than d and k.

[0181] Then the weight update process of the one-dimensional convolution weight matrix W in the FFT Block module is:

[0182] Wt=Wt-1+△Wt, △Wt is the gradient;

[0183] Can be improved to:

[0184] Wt=Wt-1+W1W2.

[0185] The traditional gradient update process is replaced by a two-layer network, which greatly reduces the amount of calculation and training parameters in the fine-tuning process and reduces the dependence on the amount of data.

[0186] The embodiment of the present application further provides a computer-readable storage medium for storing program code, wherein the program code is used to execute any implementation of the speech synthesis method for adding a speaker described in the aforementioned embodiments.

[0187] In an embodiment of the present application, a speech synthesis method, apparatus, device and storage medium for adding a new speaker are provided. Based on a basic model of speech synthesis and low-rank matrix decomposition theory, some parameters in the basic model of speech synthesis are decomposed into low-dimensional matrices through low-rank matrix decomposition, and the traditional gradient update is replaced by the training of a low-dimensional network, which greatly reduces the amount of calculation and training parameters in the fine-tuning process, reduces the dependence on the amount of data, can be applied to different speech synthesis models, has good generalization ability, and solves the technical problem that traditional speech synthesis models in the prior art usually require a large amount of data to capture the characteristics of a specific speaker. When only a small amount of data is available, it is difficult for the model to learn enough features, resulting in a decrease in the quality and naturalness of the synthesized speech; and when selective parameter fine-tuning is used, the naturalness and expressiveness of the synthesized speech may still be limited, and the small amount of data may cause the model to overfit and be unable to generalize to new texts or contexts that have not been seen.

[0188] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0189] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0190] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0191] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0192] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0193] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0194] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0195] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A speech synthesis method for adding a new speaker, characterized in that: include: Step S1, obtaining a basic model for speech synthesis containing multiple speakers, wherein the basic model for speech synthesis is obtained by pre-training based on an Adaspeech network structure, wherein the Adaspeech network structure contains an acoustic condition model AcousticConditionModel and a MEL decoder Mel Decoder, and both the acoustic condition model AcousticConditionModel and the MEL decoder Mel Decoder are based on a FFTBlock module; Step S2, based on the low-order matrix decomposition theory, decompose the one-dimensional convolution weight matrix W in the FFT Block module into two low-order matrices W1 and W2; Step S3, obtaining first training data of the newly added speaker; Step S4, in the fine-tuning stage, the acoustic condition model based on the FFT Block module in the speech synthesis basic model and the two low-order matrices W1 and W2 in the MEL decoder are trained by using the first training data of the newly added speaker to obtain updated low-order matrices W1 and W2; Step S5: In the inference stage, the speech synthesis basic model and the updated low-order matrices W1 and W2 are used to perform speech synthesis output for the newly added speech.

2. The speech synthesis method for adding a new speaker according to claim 1, characterized in that: Before step S1, the following steps are also included: Step S0: Generate a basic model for speech synthesis based on the Adaspeech network structure pre-training by using second training data containing multiple speakers.

3. The speech synthesis method for adding a new speaker according to claim 2, characterized in that: The step S0 specifically includes: Acquire second training data including multiple speakers; extracting phoneme information of the second training data; Extract hidden states from phoneme information through the Acoustic ConditionModel; The hidden state and the Mel Spectrum of the corresponding audio are forcibly aligned through a variable information adapter VarianceAdaptor; The forced aligned hidden state is mapped to a Mel Spectrum through the MEL decoder to generate a basic model for speech synthesis.

4. The speech synthesis method for adding a new speaker according to claim 1, characterized in that: The FFTBlock module consists of a multi-head attention mechanism and a one-dimensional convolution weight matrix, and the multi-head attention mechanism and the one-dimensional convolution weight matrix are connected using residual connections and layer normalization connections.

5. The speech synthesis method for adding a new speaker according to claim 1, characterized in that: The step S2 specifically includes: Based on the low-alias matrix decomposition theory, the one-dimensional convolution weight matrix W in the FFT Block module is decomposed into two low-alias matrices W1 and W2, wherein the dimension of the low-alias matrix W1 is d*r, and the dimension of the low-alias matrix W2 is r*k, where r is much smaller than d and k.

6. A speech synthesis device for adding a new speaker, characterized in that: include: A first acquisition unit is used to acquire a basic model for speech synthesis including multiple speakers, wherein the basic model for speech synthesis is obtained by pre-training based on an Adaspeech network structure, wherein the Adaspeech network structure includes an acoustic condition model Acoustic Condition Model and a MEL decoder Mel Decoder, and both the acoustic condition model AcousticConditionModel and the MEL decoder Mel Decoder are based on a FFTBlock module; A decomposition unit, used for decomposing the one-dimensional convolution weight matrix W in the FFTBlock module into two low-nothing matrices W1 and W2 based on the low-nothing matrix decomposition theory; A second acquisition unit, used to acquire first training data of a newly added speaker; A fine-tuning unit, used for training the acoustic condition model based on the FFTBlock module in the speech synthesis basic model and two low-order matrices W1 and W2 in the MEL decoder Mel Decoder in the fine-tuning stage through the first training data of the newly added speaker to obtain updated low-order matrices W1 and W2; The inference unit is used to perform speech synthesis output of the newly added speech by using the speech synthesis basic model and the updated low-order matrices W1 and W2 during the inference stage.

7. The speech synthesis device for adding a new speaker according to claim 6, characterized in that: Also includes: The pre-training unit is used to generate a basic model for speech synthesis based on the Adaspeech network structure through pre-training using second training data containing multiple speakers.

8. The speech synthesis device for adding a new speaker according to claim 7, characterized in that: The pre-training unit is specifically used for: Acquire second training data including multiple speakers; extracting phoneme information of the second training data; Extract hidden states from phoneme information through the Acoustic ConditionModel; The hidden state and the Mel Spectrum of the corresponding audio are forcibly aligned through a variable information adapter VarianceAdaptor; The forced aligned hidden state is mapped to a Mel Spectrum through the MEL decoder to generate a basic model for speech synthesis.

9. A speech synthesis device for adding a new speaker, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the speech synthesis method for adding a new speaker according to any one of claims 1-5 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and the program codes are used to execute the speech synthesis method for adding a new speaker according to any one of claims 1 to 5.