Model training method based on semi-supervised knowledge distillation and speech synthesis method
By using a semi-supervised knowledge distillation method, the student model is trained using the phoneme duration and audio information of the teacher model, which solves the problem of inaccurate phoneme duration information in speech synthesis and improves the accuracy and quality of speech synthesis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- EEASY TECH CO LTD
- Filing Date
- 2023-01-18
- Publication Date
- 2026-05-12
AI Technical Summary
Existing knowledge distillation-based speech synthesis models suffer from poor training performance and low speech synthesis accuracy due to the lack of phoneme duration annotations.
A semi-supervised knowledge distillation method is adopted. The linear spectrum of the samples is encoded by the initial teacher model, and monotonic alignment search training is performed to obtain the teacher's predicted phoneme duration and audio. This information is then used to train the initial student model to improve the accuracy of phoneme alignment.
It achieves accurate alignment of the target student model at the frame level, improving training accuracy and speech synthesis quality.
Smart Images
Figure CN116092469B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of speech processing technology, and in particular to a model training method and a speech synthesis method based on semi-supervised knowledge distillation. Background Technology
[0002] Currently, in the field of speech synthesis, due to the limitation of its inference speed, most acoustic models based on knowledge distillation adopt non-autoregressive models. However, most of the datasets available for non-autoregressive models do not have phoneme duration information labeled. Therefore, during the training process, it is necessary to obtain phoneme duration information from the teacher model or by using auxiliary tools, and then train the prediction of Mel spectrum based on the phoneme duration information.
[0003] However, the target Mel spectrum distilled by the teacher model loses a significant amount of information, and the teacher model cannot use real labels, resulting in a mismatch between the predicted target Mel spectrum and the actual Mel spectrum in terms of duration. Using this target Mel spectrum to train the student model reduces training effectiveness. Phoneme duration information obtained through auxiliary tools is time-domain information, while Mel spectrum is frequency-domain information. During audio framing, phoneme alignment is not possible, leading to inaccurate phoneme duration information and affecting the training effect of the student model. Therefore, the student model trained using these techniques exhibits poor performance and low accuracy in speech synthesis. Summary of the Invention
[0004] This invention aims to address at least one of the technical problems existing in the prior art. To this end, this invention proposes a model training method and a speech synthesis method based on semi-supervised knowledge distillation, which can improve the training effect of student models and enhance the quality of Mel spectra synthesized by the student models.
[0005] In a first aspect, embodiments of the present invention provide a model training method based on semi-supervised knowledge distillation, comprising:
[0006] Obtain the sample phoneme sequence and sample linear spectrum, and encode the sample linear spectrum using the initial teacher model to obtain the sample teacher coding sequence;
[0007] The initial teacher model is trained by monotonic alignment search based on the sample phoneme sequence and the sample teacher encoding sequence to obtain the target teacher model;
[0008] The sample phoneme sequence is input into the target teacher model to obtain the teacher's predicted phoneme duration and the teacher's predicted audio.
[0009] The sample phoneme sequence is input into the initial student model, and the initial student model is trained according to the teacher's predicted phoneme duration and the teacher's predicted audio to obtain the target student model. The target student model is used to synthesize target speech based on the input target phoneme sequence.
[0010] According to some embodiments of the present invention, before obtaining the sample phoneme sequence and the sample linear spectrum, the method further includes:
[0011] Obtain the sample text sequence and the corresponding sample audio, and convert the sample text sequence into the sample phoneme sequence;
[0012] Determine the start and end silence durations for each of the sample audio files;
[0013] The head silence duration and tail silence duration of each sample audio are adjusted to a preset silence duration by cropping or silence compensation;
[0014] Based on the adjusted head silence duration and adjusted tail silence duration of the sample audio, a silence identifier is added to the corresponding position in the sample phoneme sequence.
[0015] According to some embodiments of the present invention, the student model includes a text encoder, a duration predictor, a length adjuster, and a spectrum decoder. The step of inputting the sample phoneme sequence into the initial student model and training the initial student model based on the teacher-predicted phoneme duration and the teacher-predicted audio includes:
[0016] The text encoder converts the sample phoneme sequence into an initial student coding sequence, and the duration predictor determines the student predicted phoneme duration of the initial student coding sequence.
[0017] The teacher-predicted phoneme duration and the initial student coding sequence are input into the length regulator. The length regulator increases the feature number of each dimension of the initial student coding sequence according to the teacher-predicted phoneme duration to obtain the sample student coding sequence.
[0018] The sample student coding sequence is decoded using the spectrum decoder to obtain the student predicted Mel spectrum;
[0019] The initial student model is trained based on the teacher's predicted phoneme duration, the teacher's predicted audio, the student's predicted phoneme duration, and the student's predicted Mel spectrum.
[0020] According to some embodiments of the present invention, the text encoder and the spectrum decoder are dilated convolutional residual modules, and the duration predictor is a convolutional residual module.
[0021] According to some embodiments of the present invention, training the initial student model based on the teacher-predicted phoneme duration, the teacher-predicted audio, the student-predicted phoneme duration, and the student-predicted Mel spectrum includes:
[0022] Perform Mel spectrum transformation on the teacher's predicted audio to obtain the teacher's predicted Mel spectrum;
[0023] The duration loss value is calculated based on the teacher's predicted phoneme duration, the student's predicted phoneme duration, and a preset duration loss function;
[0024] The Mel spectrum loss value is calculated based on the teacher's predicted Mel spectrum, the student's predicted Mel spectrum, and the preset Mel spectrum loss function;
[0025] The teacher's predicted audio is subjected to speech recognition and speech quality assessment. The loss weight value of the target teacher model is determined based on the results of speech recognition and speech quality assessment.
[0026] The total training loss is determined based on the duration loss, the Mel loss, and the loss weight. When the total training loss satisfies the preset convergence condition, the initial student model is determined to be trained into the target student model.
[0027] According to some embodiments of the present invention, the duration loss function is a smoothed absolute value loss function, and the Mel spectrum loss function includes a structural similarity loss function and an absolute value loss function.
[0028] According to some embodiments of the present invention, the step of performing speech recognition and speech quality assessment on the teacher's predicted audio, and determining the loss weight value of the target teacher model based on the results of speech recognition and speech quality assessment, includes:
[0029] The teacher's predicted audio is input into a preset MOSNet network to obtain a speech quality assessment result, which is used to characterize the intelligibility of the teacher's predicted audio.
[0030] If the value of the speech quality assessment result is greater than the preset intelligibility threshold, the first weighting factor is set to 1; otherwise, the first weighting factor is set to 0.
[0031] The teacher's predicted audio is input into a preset U2 network to decode the predicted audio recognition phoneme. When the predicted audio recognition phoneme is the same as the sample phoneme sequence, the second weight factor is set to 1; otherwise, the second weight factor is set to 0.
[0032] The product of the first weighting factor and the second weighting factor is determined as the loss weight value.
[0033] Secondly, embodiments of the present invention provide a speech synthesis method, including:
[0034] Obtain the target text information to be synthesized, and convert the target text information into a target phoneme sequence;
[0035] The target phoneme sequence is input into the target student model, and the target speech is synthesized through the target student model, wherein the target student model is trained by the model training method based on semi-supervised knowledge distillation as described in the first aspect.
[0036] Thirdly, embodiments of the present invention provide an electronic device, including at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, the instructions being executed by the at least one control processor to enable the at least one control processor to perform a model training method based on semi-supervised knowledge distillation as described in the first aspect, or to perform a speech synthesis method as described in the second aspect.
[0037] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer-executable instructions, the computer-executable instructions being used to execute the model training method based on semi-supervised knowledge distillation as described in the first aspect above, or to execute the speech synthesis method as described in the second aspect.
[0038] The model training method based on semi-supervised knowledge distillation according to embodiments of the present invention has at least the following beneficial effects: acquiring sample phoneme sequences and sample linear spectra; encoding the sample linear spectra using an initial teacher model to obtain a sample teacher encoding sequence; training the initial teacher model using monotonic alignment search based on the sample phoneme sequences and the sample teacher encoding sequence to obtain a target teacher model; inputting the sample phoneme sequences into the target teacher model to obtain teacher predicted phoneme durations and teacher predicted audio; inputting the sample phoneme sequences into an initial student model, training the initial student model based on the teacher predicted phoneme durations and teacher predicted audio to obtain a target student model, wherein the target student model is used to synthesize target speech based on the input target phoneme sequences. According to the technical solution of the embodiments of the present invention, the target teacher model can achieve alignment at the frame level, the phoneme alignment method is more accurate, the accuracy of phoneme duration information is improved when training the initial student model, and training accuracy and training effect are improved. Attached Figure Description
[0039] Figure 1 This is a flowchart of a model training method based on semi-supervised knowledge distillation provided in an embodiment of the present invention;
[0040] Figure 2 This is a flowchart of sample preprocessing provided in another embodiment of the present invention;
[0041] Figure 3 This is a flowchart of the training process for the initial student model provided in another embodiment of the present invention;
[0042] Figure 4 This is a schematic diagram of training an initial student model provided in another embodiment of the present invention;
[0043] Figure 5 This is a flowchart of training an initial student model based on a target teacher model, provided in another embodiment of the present invention;
[0044] Figure 6 This is a schematic diagram of training an initial student model based on a target teacher model, provided in another embodiment of the present invention;
[0045] Figure 7 This is a flowchart for determining the loss weight value provided in another embodiment of the present invention;
[0046] Figure 8 This is a flowchart of a speech synthesis method provided in another embodiment of the present invention;
[0047] Figure 9 This is a structural diagram of an electronic device provided in another embodiment of the present invention. Detailed Implementation
[0048] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0049] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.
[0050] In the description of this invention, "several" means one or more, "multiple" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.
[0051] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.
[0052] This invention provides a model training method and a semantic synthesis method based on semi-supervised knowledge distillation. The model training method based on semi-supervised knowledge distillation includes: acquiring sample phoneme sequences and sample linear spectra; encoding the sample linear spectra using an initial teacher model to obtain a sample teacher encoding sequence; training the initial teacher model using monotonic alignment search based on the sample phoneme sequences and the sample teacher encoding sequence to obtain a target teacher model; inputting the sample phoneme sequences into the target teacher model to obtain teacher predicted phoneme durations and teacher predicted audio; inputting the sample phoneme sequences into an initial student model, and training the initial student model based on the teacher predicted phoneme durations and the teacher predicted audio to obtain a target student model. The target student model is used to synthesize target speech based on the input target phoneme sequences. According to the technical solution of this invention, the target teacher model can achieve alignment at the frame level, the phoneme alignment method is more accurate, the accuracy of phoneme duration information is improved when training the initial student model, and training accuracy and training effect are improved.
[0053] The control method of the present invention will be further described below with reference to the accompanying drawings.
[0054] Reference Figure 1 , Figure 1 A flowchart of a model training method based on semi-supervised knowledge distillation provided in this embodiment of the invention. The model training method based on semi-supervised knowledge distillation includes:
[0055] Step S110: Obtain the sample phoneme sequence and sample linear spectrum, and encode the sample linear spectrum using the initial teacher model to obtain the sample teacher coding sequence;
[0056] Step S120: Train the initial teacher model using monotonic alignment search based on the sample phoneme sequence and the sample teacher coding sequence to obtain the target teacher model;
[0057] Step S130: Input the sample phoneme sequence into the target teacher model to obtain the teacher's predicted phoneme duration and the teacher's predicted audio.
[0058] Step S140: Input the sample phoneme sequence into the initial student model, train the initial student model according to the teacher's predicted phoneme duration and the teacher's predicted audio, and obtain the target student model. The target student model is used to synthesize target speech according to the input target phoneme sequence.
[0059] It should be noted that there can be multiple sample phoneme sequences and sample linear spectra, and there is a one-to-one correspondence between the sample phoneme sequences and sample linear spectra to ensure accuracy during monotonic alignment search. The sample linear spectra are used to assist in the training of the initial teacher model. That is, the input to the initial teacher model during training is the sample phoneme sequence and the sample linear spectra. The sample linear spectra can be encoded by a posterior encoder to obtain a frame-level sample teacher encoding sequence. The initial teacher model learns the alignment between the sample phoneme sequence and the frame-level sample teacher encoding sequence through monotonic alignment search, thus achieving frame-level alignment during audio framing. When the target teacher model is obtained and used for inference, the input is only the sample phoneme sequence. The target teacher model outputs the teacher's predicted audio, and based on the alignment method obtained during training, it ensures that the teacher's predicted Mel spectrum obtained by converting the teacher's predicted audio is aligned with the sample phoneme sequence, thereby ensuring the accuracy of the predicted phoneme duration information and providing a data foundation for the training of the student model.
[0060] It should be noted that the initial teacher model can be a VI TS model. VI TS is an end-to-end non-autoregressive model based on variational autoencoders. The variational autoencoder of the VI TS model has a built-in normalized flow structure, so it can better distinguish between the training and inference stages in the process. During training, the sample phoneme sequence and sample linear spectrum are used to output the teacher's predicted audio. After training, inference only requires the input of the sample phoneme sequence.
[0061] It should be noted that after training the target teacher model, the target teacher model can be switched to inference mode to train the initial student model. The initial student model is usually small in scale. Since the target teacher model trained in this embodiment can achieve frame-level alignment in its output, the target teacher model can obtain teacher prediction duration and teacher prediction audio with more accurate alignment. The alignment method of the training data provided for the initial student model is more accurate, the accuracy loss during the training of the initial student model is smaller, the trained target student model is more accurate, and the speech synthesized using the trained target student model has higher quality.
[0062] Additionally, in one embodiment, reference is made to Figure 2 In execution Figure 1 Before step S110 shown, the following steps may also be included, but are not limited to:
[0063] Step S210: Obtain the sample text sequence and the corresponding sample audio, and convert the sample text sequence into a sample phoneme sequence;
[0064] Step S220: Determine the start and end silence durations for each sample audio.
[0065] Step S230: Adjust the head silent duration and tail silent duration of each sample audio to a preset silent duration through cropping or silence compensation.
[0066] Step S240: Based on the adjusted head silent duration and adjusted tail silent duration of the sample audio, add silence identifiers at the corresponding positions in the sample phoneme sequence.
[0067] It should be noted that the conversion from the sample text sequence to the sample phoneme sequence can be obtained by looking up a pronunciation dictionary or through a G2P model (Grapheme to Phoneme). Those skilled in the art are familiar with how to perform the conversion between text and phonemes, and thus will not be elaborated here.
[0068] It should be noted that due to the relationship of the recording speaker, there will be certain differences in the head and tail silent durations of different sample audios. If the head and tail silent durations are not unified, it will affect the alignment training of the initial teacher model. Based on this, in this embodiment, the head silent duration and tail silent duration of each sample audio are adjusted to a unified preset silent duration to ensure that the head and tail silent durations of each sample audio are the same, ensuring alignment during training.
[0069] Exemplarily, the sample text is "The weather is good", and the converted phoneme sequence is "t ian1 q i4 h ao3". The corresponding sample audio is a passage with the content "The weather is good". The silent part before the word "tian" in the sample audio is the head silence. When the head silent duration is greater than the preset silent duration, crop the head silent duration so that the cropped head silent duration is equal to the preset silent duration; when the head silent duration is less than the preset silent duration, increase the length of the head silent duration through silence compensation so that the compensated head silent duration is equal to the preset silent duration; the silent part after the word "hao" in the sample audio is the tail silence, and the processing method of the tail silent duration refers to the processing method of the head silent duration, which will not be repeated here. After the processing is completed, add silence identifiers before and after the sample phoneme sequence. For example, the sample phoneme sequence is "sp t ian1 q i4 h ao3 sp", where sp is the silence identifier, enabling the teacher model to identify the silent part through the silence identifier and improving the efficiency of alignment learning.
[0070] In addition, in one embodiment, the student model includes a text encoder, a duration predictor, a length regulator, and a spectral decoder. Referring to Figure 3 , Figure 1 Step S140 of the embodiment shown also includes but is not limited to the following steps:
[0071] Step S310: The sample phoneme sequence is converted into an initial student coding sequence by a text encoder, and the duration of the predicted student phonemes in the initial student coding sequence is determined by a duration predictor.
[0072] Step S320: Input the teacher's predicted phoneme duration and the initial student coding sequence into the length regulator. The length regulator increases the number of features in each dimension of the initial student coding sequence according to the teacher's predicted phoneme duration to obtain the sample student coding sequence.
[0073] Step S330: Decode the sample student coding sequence using a spectrum decoder to obtain the student predicted Mel spectrum;
[0074] Step S340: Train the initial student model based on the teacher's predicted phoneme duration, the teacher's predicted audio, the student's predicted phoneme duration, and the student's predicted Mel spectrum.
[0075] It should be noted that, referring to Figure 4 The initial student model consists of a text encoder, a duration predictor, a length adjuster, and a spectrum decoder. The text encoder converts the sample phoneme sequence into an initial student coding sequence. The duration predictor then predicts the duration of the student's predicted phonemes from the initial student coding sequence. In training mode, the teacher's predicted phoneme duration of the sample phoneme sequence can be input into the length adjuster. The length adjuster increases the number of features in each dimension of the initial student coding sequence based on the teacher's predicted phoneme duration to obtain the sample student coding sequence. The spectrum decoder then decodes the sample student coding sequence to obtain the student's predicted Mel spectrum, ensuring that the length of the sample student coding sequence for the student's predicted Mel spectrum matches the length of the sample phoneme sequence, effectively improving the training accuracy of the target student model.
[0076] The following example illustrates the working principle of the length adjuster. For instance, when the input sample phoneme sequence is "n i3", the initial student encoding sequence output by the text encoder is [[1,2,3],[2,3,4]], which is a 2×3 sequence with a length of 2 and a duration dimension of 3. When the teacher predicts the phoneme duration as [2,3], the first dimension feature [1,2,3] is increased twice, and the second dimension feature [2,3,4] is increased three times, resulting in the sample student encoding sequence [[1,2,3],[1,2,3],[2,3,4],[2,3,4],[2,3,4]], which is a sample student encoding sequence with a duration of 5 and a duration dimension of 3.
[0077] It should be noted that after training the target student model is completed, refer to Figure 4In the inference mode of the target student model, it is no longer necessary to input the teacher's predicted phoneme duration. The target phoneme sequence is directly input into the target student model. The initial student coding sequence is obtained through the text encoder. The duration predictor predicts the duration of the initial student coding sequence to obtain the target predicted duration. The target predicted duration and the initial student coding sequence are then input into the length regulator. The length regulator copies the initial student coding sequence by the corresponding factor to obtain the sample student coding sequence. The sample student coding sequence is input into the spectrum decoder to obtain the target Mel spectrum. Finally, the target Mel spectrum is converted into speech to obtain the target synthesized speech.
[0078] Additionally, in one embodiment, reference is made to Figure 4 The text encoder and spectrum decoder are dilated convolutional residual modules, and the duration predictor is a convolutional residual module.
[0079] It should be noted that the initial student model in this embodiment can adopt a dilated convolution-based structure with the SpeedySpeech model as the backbone. Using the dilated convolution residual module can enable the text encoder and the spectral encoder to obtain a larger receptive field and improve the encoding accuracy.
[0080] Additionally, in one embodiment, reference is made to Figure 5 , Figure 3 Step S340 of the illustrated embodiment also includes, but is not limited to, the following steps:
[0081] Step S510: Perform Mel spectrum transformation on the teacher's predicted audio to obtain the teacher's predicted Mel spectrum;
[0082] Step S520: Calculate the duration loss value based on the teacher's predicted phoneme duration, the student's predicted phoneme duration, and the preset duration loss function;
[0083] Step S530: Calculate the Mel spectrum loss value based on the teacher's predicted Mel spectrum, the student's predicted Mel spectrum, and the preset Mel spectrum loss function;
[0084] Step S540: Perform speech recognition and speech quality assessment on the teacher's predicted audio, and determine the loss weight value of the target teacher model based on the results of speech recognition and speech quality assessment.
[0085] Step S550: Determine the total training loss value based on the duration loss value, the Mel loss value, and the loss weight value. When the total training loss value meets the preset convergence condition, the initial student model is determined to be trained into the target student model.
[0086] It should be noted that, referring to Figure 6When training the initial student model, the target teacher model is in inference mode. The sample phoneme sequence is simultaneously input into both the target teacher model and the initial student model. According to the description in the above embodiment, the target teacher model outputs the teacher's predicted phoneme duration and the corresponding teacher's predicted audio in inference mode. Figure 4 As shown, the initial student model, in training mode, outputs the student's predicted phoneme duration and predicted Mel spectrum after receiving a sample phoneme sequence as input. Based on this, to achieve knowledge distillation-based training, the teacher's predicted audio needs to undergo Mel spectrum transformation to obtain the teacher's predicted Mel spectrum. Then, the initial student model is trained using the target teacher model, for example... Figure 6 As shown, the duration loss value is calculated using the teacher's predicted phoneme duration and the student's predicted phoneme duration, and the Mel-spectral loss value is calculated using the teacher's predicted Mel-spectral and the student's predicted Mel-spectral.
[0087] It should be noted that the accuracy of the teacher prediction audio output by the target teacher model has a direct impact on the training effect of the initial student model. Therefore, in addition to calculating the duration loss value and the Mel spectrum loss value, this embodiment further calculates the loss weight value of the target teacher model. For example, speech recognition and speech quality assessment can be performed on the teacher prediction audio. The accuracy of the teacher prediction audio can be determined by the results of speech recognition and speech quality assessment. Different loss weight values are assigned to accurate and inaccurate teacher prediction audio. The loss weight value can be used to characterize the contribution of the teacher prediction audio to this training.
[0088] For example, the total training loss can be determined by the following formula: L total =α×(L D (D t D s )+(L M (M t M s ), where α is the loss weight value, L D () represents the duration loss function, D t To help teachers predict phoneme duration, D s To predict phoneme duration for students, L M () represents the Mel spectrum loss function, M t For teachers to predict Mel spectrum, M s The model predicts Mel spectrum for students. As those skilled in the art will know, convergence conditions are set during training. When the total training loss value meets the convergence conditions, the model can be considered to have completed training. This embodiment does not limit the specific settings of the convergence conditions.
[0089] It should be noted that after the target student model is trained, the speech synthesis still requires training of the vocoder in subsequent processes. Since the target student model is trained with reference to the teacher prediction Mel spectrum converted from the teacher prediction audio synthesized by the target teacher model, the teacher prediction audio obtained in this embodiment can also be used for subsequent vocoder training, which can reduce the difference in training data between the target student model and the vocoder. The vocoder training is not an improvement made in this embodiment, and will not be elaborated on here.
[0090] In another embodiment, the duration loss function is a smoothed absolute value loss function, and the Mel spectrum loss function includes a structural similarity loss function and an absolute value loss function.
[0091] It should be noted that the duration loss function in this embodiment adopts the smoothed absolute value loss function, which can better measure the difference between the teacher's predicted phoneme duration and the student's predicted phoneme duration. The specific formula is as follows: L D (D t D s ) = Smoothing L1(D t D s ), where Smoothing L1 is a common smoothing absolute value loss function, L D () represents the duration loss function, D t To help teachers predict phoneme duration, D s Predict phoneme durations for students.
[0092] It should be noted that the Mel spectrum loss function in this embodiment can adopt common structural similarity loss functions and absolute value loss functions, which can effectively measure the difference between the teacher's predicted Mel spectrum and the student's predicted Mel spectrum. The specific formula is as follows: L M (M t M s ) = SSIM(M t M s )+L1(M t M s ), where L M () represents the Mel spectrum loss function, M t For teachers to predict Mel spectrum, M s For predicting Mel spectrum for students, SSIM() is a common structural similarity loss function, and L1() is a common absolute value loss function.
[0093] Additionally, in one embodiment, reference is made to Figure 7 , Figure 5 Step S540 of the illustrated embodiment also includes, but is not limited to, the following steps:
[0094] Step S710: Input the teacher's predicted audio into the preset MOSNet network to obtain the speech quality assessment result. The speech quality assessment result is used to characterize the intelligibility of the teacher's predicted audio pronunciation.
[0095] Step S720: When the value of the speech quality assessment result is greater than the preset intelligibility threshold, the first weight factor is set to 1; otherwise, the first weight factor is set to 0.
[0096] Step S730: The teacher's predicted audio input is decoded into the preset U2 network to identify the predicted audio phonemes. If the predicted audio phonemes are the same as the sample phoneme sequence, the second weight factor is set to 1; otherwise, the second weight factor is set to 0.
[0097] Step S740: The product of the first weight factor and the second weight factor is determined as the loss weight value.
[0098] It should be noted that a speech quality assessment network can be used to evaluate the intelligibility of pronunciation. This embodiment uses the MOSNet network, which consists of a CNN network and a BLSTM network. The output value of the MOSNet network model can be set between 1 and 5. The higher the value of the MOSNet network model output, the higher the intelligibility of the teacher's predicted audio. When the output value is greater than a certain threshold, the first weight factor is set to 1; otherwise, it is set to 0, as shown in the following formula:
[0099] Of course, the first weighting factor α1 can be set to a soft weight, such as 0.95 or 0.05, etc. Understandably, the intelligibility threshold can be set according to actual needs. For example, in this embodiment, the intelligibility threshold is set to 3.3. It can also be differentiated according to language, such as setting 3.3 for Chinese and 3.1 for English. This embodiment does not impose any restrictions on this.
[0100] It should be noted that speech recognition networks can be used to evaluate pronunciation accuracy, such as the U2 network composed of a ConformerEncoder + CTC Decoder + Attention Decoder structure. This embodiment does not impose too many restrictions on the structure of the U2 network. It is worth noting that when decoding the teacher's predicted audio using the U2 network, this embodiment uses CTC decoding because Attention decoding considers the correlation between output contexts, allowing for some pronunciation errors, which can lead to some bias in the evaluation of pronunciation accuracy. The CTC decoding used in this embodiment does not consider the correlation between output contexts, ensuring accurate evaluation of pronunciation accuracy. Furthermore, the U2 model output is Chinese character or byte-pair encoding (BPE), so the speech recognition output needs to be converted to phonemes via G2P before comparison. When the predicted audio phonemes match the sample phoneme sequence, the second weighting factor α2 is set to 1; otherwise, it is set to 0, satisfying the following expression:
[0101]
[0102] It should be noted that since the output of the target teacher model is the teacher's predicted audio, this embodiment adds two auxiliary networks, namely a speech recognition network and a speech quality assessment network, to improve the accuracy of knowledge distillation.
[0103] Additionally, refer to Figure 8 An embodiment of the present invention also provides a speech synthesis method, including but not limited to the following steps:
[0104] Step S810: Obtain the target text information to be synthesized and convert the target text information into a target phoneme sequence;
[0105] Step S820: Input the target phoneme sequence into the target student model and synthesize the target speech through the target student model. The target student model is trained by the model training method based on semi-supervised knowledge distillation as described above.
[0106] It should be noted that when the target student model trained using the semi-supervised knowledge distillation-based model training method described above is used for inference, since the target student model achieves frame-level phoneme alignment during training, it can ensure the accuracy of the phoneme duration of the predicted Mel spectrum when performing speech synthesis on the target phoneme sequence, thereby producing a better quality target speech and improving the effect of speech synthesis.
[0107] like Figure 9 As shown, Figure 9 This is a structural diagram of an electronic device provided in one embodiment of the present invention. The present invention also provides an electronic device, comprising:
[0108] The processor 901 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.
[0109] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 to execute the model training method based on semi-supervised knowledge distillation of the embodiments of this application.
[0110] The input / output interface 903 is used to implement information input and output;
[0111] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).
[0112] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);
[0113] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.
[0114] This application embodiment also provides a storage medium, which is a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the above-described model training method based on semi-supervised knowledge distillation, or implements the above-described speech synthesis method.
[0115] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate, and may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0116] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.
[0117] The above provides a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.
Claims
1. A model training method based on semi-supervised knowledge distillation, characterized in that, include: Obtain the sample phoneme sequence and sample linear spectrum, and encode the sample linear spectrum using the initial teacher model to obtain the sample teacher coding sequence; The initial teacher model is trained by monotonic alignment search based on the sample phoneme sequence and the sample teacher encoding sequence to obtain the target teacher model; The sample phoneme sequence is input into the target teacher model to obtain the teacher's predicted phoneme duration and the teacher's predicted audio. The sample phoneme sequence is input into the initial student model, and the initial student model is trained according to the teacher's predicted phoneme duration and the teacher's predicted audio to obtain the target student model. The target student model is used to synthesize target speech according to the input target phoneme sequence. The student model includes a text encoder, a duration predictor, a length adjuster, and a spectrum decoder. The process of inputting the sample phoneme sequence into the initial student model and training the initial student model based on the teacher-predicted phoneme duration and the teacher-predicted audio includes: The text encoder converts the sample phoneme sequence into an initial student coding sequence, and the duration predictor determines the student predicted phoneme duration of the initial student coding sequence. The teacher-predicted phoneme duration and the initial student coding sequence are input into the length regulator. The length regulator increases the feature number of each dimension of the initial student coding sequence according to the teacher-predicted phoneme duration to obtain the sample student coding sequence. The sample student coding sequence is decoded using the spectrum decoder to obtain the student predicted Mel spectrum; The initial student model is trained based on the teacher's predicted phoneme duration, the teacher's predicted audio, the student's predicted phoneme duration, and the student's predicted Mel spectrum.
2. The model training method based on semi-supervised knowledge distillation according to claim 1, characterized in that, Before obtaining the sample phoneme sequence and sample linear spectrum, the method further includes: Obtain the sample text sequence and the corresponding sample audio, and convert the sample text sequence into the sample phoneme sequence; Determine the start and end silence durations for each of the sample audio files; The head silence duration and tail silence duration of each sample audio are adjusted to a preset silence duration by cropping or silence compensation; Based on the adjusted head silence duration and adjusted tail silence duration of the sample audio, a silence identifier is added to the corresponding position in the sample phoneme sequence.
3. The model training method based on semi-supervised knowledge distillation according to claim 1, characterized in that, The text encoder and the spectrum decoder are dilated convolutional residual modules, and the duration predictor is a convolutional residual module.
4. The model training method based on semi-supervised knowledge distillation according to claim 1, characterized in that, The step of training the initial student model based on the teacher's predicted phoneme duration, the teacher's predicted audio, the student's predicted phoneme duration, and the student's predicted Mel spectrum includes: Perform Mel spectrum transformation on the teacher's predicted audio to obtain the teacher's predicted Mel spectrum; The duration loss value is calculated based on the teacher's predicted phoneme duration, the student's predicted phoneme duration, and a preset duration loss function; The Mel spectrum loss value is calculated based on the teacher's predicted Mel spectrum, the student's predicted Mel spectrum, and the preset Mel spectrum loss function; The teacher's predicted audio is subjected to speech recognition and speech quality assessment. The loss weight value of the target teacher model is determined based on the results of speech recognition and speech quality assessment. The total training loss is determined based on the duration loss value, the Mel spectrum loss value, and the loss weight value. When the total training loss value meets the preset convergence condition, the initial student model is determined to be trained into the target student model.
5. The model training method based on semi-supervised knowledge distillation according to claim 4, characterized in that, The duration loss function is a smoothed absolute value loss function, and the Mel spectrum loss function includes a structural similarity loss function and an absolute value loss function.
6. The model training method based on semi-supervised knowledge distillation according to claim 4, characterized in that, The step of performing speech recognition and speech quality assessment on the teacher's predicted audio, and determining the loss weight value of the target teacher model based on the results of speech recognition and speech quality assessment, includes: The teacher's predicted audio is input into a preset MOSNet network to obtain a speech quality assessment result, which is used to characterize the intelligibility of the teacher's predicted audio. If the value of the speech quality assessment result is greater than the preset intelligibility threshold, the first weighting factor is set to 1; otherwise, the first weighting factor is set to 0. The teacher's predicted audio is input into a preset U2 network to decode the predicted audio recognition phoneme. When the predicted audio recognition phoneme is the same as the sample phoneme sequence, the second weight factor is set to 1; otherwise, the second weight factor is set to 0. The product of the first weighting factor and the second weighting factor is determined as the loss weight value.
7. A speech synthesis method, characterized in that, include: Obtain the target text information to be synthesized, and convert the target text information into a target phoneme sequence; The target phoneme sequence is input into the target student model, and the target speech is synthesized through the target student model, wherein the target student model is trained by the model training method based on semi-supervised knowledge distillation as described in any one of claims 1 to 6.
8. An electronic device, characterized in that, It includes at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, the instructions being executed by the at least one control processor to enable the at least one control processor to perform the model training method based on semi-supervised knowledge distillation as described in any one of claims 1 to 6, or to perform the speech synthesis method as described in claim 7.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the model training method based on semi-supervised knowledge distillation as described in any one of claims 1 to 6, or to perform the speech synthesis method as described in claim 7.