Voice restoration methods, devices, equipment and storage media

Abstract

This invention provides a speech restoration method, apparatus, device, and storage medium. The method includes: aligning the encoded features of the speech to be restored acquired at the current moment with target text features to obtain target features; the target text features are the text features corresponding to the speech at the previous moment output by the decoder of a speech recognition model; determining the posterior probability of each text corresponding to the speech to be restored based on the target features; determining the text recognition result of the speech to be restored based on each posterior probability; inputting the target information into the text restoration module of the speech restoration model to obtain a first latent feature output by the text restoration module, the first latent feature including text latent features, the target information including each posterior probability and the text recognition result; and restoring the speech to be restored based on the text latent features. Embodiments of this invention can improve the intelligibility of the restored speech.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and in particular to a speech restoration method, apparatus, device, and storage medium. Background Technology

[0002] With the development of end-to-end text-to-speech (TTS) technology, it has been widely used in various scenarios. In call and face-to-face scenarios, some users suffer from speech impairment due to illness, accidents, or other reasons, which may lead to unclear pronunciation or inaccurate speech. Therefore, it is necessary to restore the speech of these users.

[0003] In existing technologies, the usual approach is to first use an Automatic Speech Recognition (ASR) model to identify the text corresponding to the speech to be repaired, and then input the text into a speech synthesis model to generate the repaired speech.

[0004] However, in the above methods, when there are many missing parts or unclear pronunciation in the speech to be repaired, the ASR model may misidentify text in the recognition result, resulting in low intelligibility of the generated repaired speech. Summary of the Invention

[0005] This invention provides a speech restoration method, apparatus, device, and storage medium to address the shortcomings of low intelligibility of restored speech in the prior art, thereby improving the intelligibility of restored speech.

[0006] This invention provides a voice restoration method, comprising:

[0007] Align the encoded features of the speech to be repaired acquired at the current moment with the target text features to obtain the target features; the target text features are the text features corresponding to the speech at the previous moment output by the decoder of the speech recognition model.

[0008] Based on the target features, the posterior probability of each text corresponding to the speech to be repaired is determined;

[0009] Based on the posterior probabilities, the text recognition result of the speech to be repaired is determined;

[0010] The target information is input into the text restoration module of the speech restoration model to obtain the first latent feature output by the text restoration module. The first latent feature includes text latent features. The target information includes each of the posterior probabilities and the text recognition result.

[0011] The speech to be repaired is repaired based on the latent features of the text.

[0012] According to a speech restoration method provided by the present invention, the step of restoring the speech to be restored based on the latent features of the text includes:

[0013] The first latent feature is input into the prosody restoration module of the speech restoration model to obtain the prosody latent feature output by the prosody restoration module.

[0014] The speech to be repaired is repaired based on the latent text features and the latent prosodic features.

[0015] According to a speech restoration method provided by the present invention, the target information further includes timbre features corresponding to the speech to be restored, and the first potential feature further includes timbre potential features;

[0016] The repair of the speech to be repaired based on the text latent features and the prosodic latent features includes:

[0017] The text latent features, timbre latent features, and prosodic latent features are input into the acoustic detail restoration module of the speech restoration model to obtain the acoustic detail latent features output by the acoustic detail restoration module.

[0018] The speech to be repaired is repaired based on the textual latent features, the prosodic latent features, and the acoustic detail latent features.

[0019] According to a speech restoration method provided by the present invention, the step of restoring the speech to be restored based on the text latent features, the prosodic latent features, and the acoustic detail latent features includes:

[0020] The textual latent features, the prosodic latent features, and the acoustic detail latent features are weighted and summed to obtain the weighted latent features;

[0021] The weighted latent features are input into the vocoder to obtain the repaired speech output by the vocoder.

[0022] According to a speech restoration method provided by the present invention, the step of aligning the encoded features of the speech to be restored acquired at the current moment with the target text features to obtain target features includes:

[0023] The speech features corresponding to the speech to be repaired are input into the encoder of the speech recognition model to obtain the encoded features output by the encoder.

[0024] The encoded features and target text features are aligned based on the blank space to obtain the target features;

[0025] The step of determining the posterior probability of each text corresponding to the speech to be repaired based on the target features includes:

[0026] The target features are input into the joint network of the speech recognition model to obtain the posterior probabilities output by the joint network.

[0027] The speech recognition model is trained based on normal speech samples and speech samples to be repaired.

[0028] According to a speech restoration method provided by the present invention, the speech restoration model is trained based on the following method:

[0029] Normal speech samples are input into the factor vector quantization codebook module to obtain the text codebook, prosodic codebook and acoustic detail codebook output by the factor vector quantization codebook module;

[0030] The normal speech sample is input into the initial speech restoration model to obtain the predicted text latent features output by the initial text restoration module of the initial speech restoration model, the predicted prosodic latent features output by the initial prosodic restoration module of the initial speech restoration model, and the predicted acoustic detail latent features output by the initial acoustic detail restoration module of the initial speech restoration model.

[0031] Based on the text codebook, the prosodic codebook, the acoustic detail codebook, the predicted text latent features, the predicted prosodic latent features, and the predicted acoustic detail latent features, loss information is determined;

[0032] Based on the loss information, the parameters of the initial text restoration module, the initial prosody restoration module, and the initial acoustic detail restoration module are adjusted to obtain the speech restoration model.

[0033] According to a speech restoration method provided by the present invention, the text restoration module includes an autoregressive model, and the prosodic restoration module and the acoustic detail restoration module include non-autoregressive models.

[0034] The present invention also provides a voice restoration device, comprising:

[0035] The alignment module is used to align the encoded features of the speech to be repaired acquired at the current moment with the target text features to obtain the target features; the target text features are the text features corresponding to the speech at the previous moment output by the decoder of the speech recognition model.

[0036] The determination module is used to determine the posterior probability of each text corresponding to the speech to be repaired based on the target features;

[0037] The determining module is further configured to determine the text recognition result of the speech to be repaired based on each of the posterior probabilities;

[0038] An input module is used to input target information into the text restoration module of the speech restoration model to obtain the first latent feature output by the text restoration module. The first latent feature includes text latent features, and the target information includes each of the posterior probabilities and the text recognition result.

[0039] The repair module is used to repair the speech to be repaired based on the latent features of the text.

[0040] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described voice restoration methods.

[0041] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech restoration method as described above.

[0042] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described voice restoration methods.

[0043] The speech restoration method, apparatus, device, and storage medium provided by this invention align the encoded features of the speech to be restored acquired at the current moment with the target text features to obtain the target features. Based on the target features, the posterior probabilities of each text corresponding to the speech to be restored are determined. Based on each posterior probability, the text recognition result of the speech to be restored is determined. The posterior probabilities and the text recognition result are input into the text restoration module of the speech restoration model to obtain the first latent features output by the text restoration module. These first latent features include text latent features. Based on the text latent features, the speech to be restored is restored. Since the text latent features can be obtained by inputting the posterior probabilities and the text recognition result into the text restoration module of the speech restoration model, even if a recognition error occurs in the text recognition result, the text recognition result can be corrected based on the predicted text latent features. Therefore, after restoring the speech to be restored based on the text latent features, the lost semantics in the text recognition result can be restored, improving the intelligibility of the restored speech. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0045] Figure 1This is a flowchart illustrating the voice restoration method provided in an embodiment of the present invention.

[0046] Figure 2 A block diagram of a voice restoration method provided in an embodiment of the present invention.

[0047] Figure 3 This is a schematic diagram of the voice restoration device provided in an embodiment of the present invention.

[0048] Figure 4 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0050] In end-to-end scenarios such as mobile phone calls or face-to-face interactions, some users may experience communication barriers due to unclear pronunciation or inaccurate articulation. Therefore, accurate speech restoration is crucial. Existing technologies employ a non-real-time cascaded architecture for speech restoration. Specifically, the speech to be restored is first input into an ASR (Automatic Speech Recognition) model for recognition, yielding the corresponding text. This text is then input into a speech synthesis model to generate normal speech. However, this approach is susceptible to errors when the speech contains a large amount of unclear speech. The ASR model's output text may contain errors, resulting in the restored speech lacking semantic information and significantly reducing its intelligibility.

[0051] To address the aforementioned problems, this invention provides a speech restoration method. In this method, in addition to determining the text recognition result of the speech to be restored, the posterior probabilities of each text corresponding to the speech to be restored are also added. The posterior probabilities of each text corresponding to the speech to be restored and the text recognition result are input into the text restoration module of the speech restoration model to obtain latent text features. Thus, even if a recognition error occurs in the text recognition result, the text recognition result can be corrected based on the predicted latent text features. Therefore, after restoring the speech to be restored based on the latent text features, the lost semantics in the text recognition result can be restored, improving the intelligibility and accuracy of the obtained restored speech.

[0052] The following is combined Figure 1 and Figure 2 The voice restoration method provided by the embodiments of the present invention is described below. The embodiments of the present invention are applicable to scenarios where user voice restoration is required during end-to-end communication, such as mobile phone calls, or face-to-face interactions. The executing entity of this method can be an electronic device such as a terminal device, computer, server, server cluster, or a specially designed voice restoration device, or a voice restoration device installed in the electronic device. This voice restoration device can be implemented through software, hardware, or a combination of both.

[0053] Figure 1 This is a flowchart illustrating the voice restoration method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0054] Step 101: Align the encoded features of the speech to be repaired acquired at the current moment with the target text features to obtain the target features; the target text features are the text features corresponding to the speech at the previous moment output by the decoder of the speech recognition model.

[0055] In this step, the speech to be repaired may include speech that is semantically unclear or hinders normal communication with others due to inaccurate pronunciation or unclear enunciation. Fbank features are extracted from the speech to be repaired acquired at the current moment. and the Fbank feature The encoded features are obtained by inputting them into the encoder of the ASR (Automatic Speech Recognition) model. .

[0056] Furthermore, the encoded features can be... The text features corresponding to the speech at the previous time step u-1, output by the decoder of the speech recognition model. Alignment can be performed, for example, by padding with blanks, to obtain the target features. Aligning the encoded features with the text features corresponding to the previous speech output from the speech recognition model's decoder ensures that the speech and text features correspond in the time dimension. Therefore, it guarantees that the output of the speech recognition model is input into the backend speech restoration model in real time, ensuring the real-time performance of speech restoration, especially in end-to-end call scenarios, meeting the requirements of real-time communication.

[0057] Step 102: Based on the target features, determine the posterior probability of each text corresponding to the speech to be repaired.

[0058] In this step, after obtaining the target features, these features can be input into the joint network of the speech recognition model to obtain the posterior probabilities of each text corresponding to the speech to be repaired at the current time, as output by the joint network. Wherein, the posterior probability The probability values ​​used to characterize the speech to be repaired as each text are used, where the posterior probability of each text can be understood as the softmax result of the speech to be repaired.

[0059] Step 103: Based on each posterior probability, determine the text recognition result of the speech to be repaired.

[0060] In this step, the text recognition result This can be understood as the most likely recognition result of the speech to be repaired, determined based on each posterior probability. For example, the text corresponding to the highest posterior probability among all probabilities can be determined as the text recognition result of the speech to be repaired. Here, the text recognition result can be understood as the hard recognition result (argmax) of the speech to be repaired.

[0061] It should be understood that, because the encoded features and target text features are aligned using a blank method for audio and text alignment, the posterior probability... and text recognition results The output will also include blank information to ensure the quality of the ASR model's output. and It can be input into the backend speech repair model in real time.

[0062] Step 104: Input the target information into the text restoration module of the speech restoration model to obtain the first latent feature output by the text restoration module. The first latent feature includes text latent features, and the target information includes each posterior probability and the text recognition result.

[0063] In this step, the speech restoration model can be, for example, a Generative Pre-trained Transformer (GPT), and the text restoration module within the speech restoration model can be an Autoregressive (AR) model. The posterior probabilities and text recognition results are input into the text restoration module, which can predict the first latent features corresponding to the next frame of speech through autoregression. These first latent features include text latent features. These text latent features include hidden or implicit text features, which can be used to correct erroneous text in the text recognition results.

[0064] Step 105: Repair the speech to be repaired based on the latent features of the text.

[0065] In this step, latent text features are input into a vocoder. Based on these features, missing or unclear content in the speech to be repaired is corrected, resulting in repaired speech. Since latent text features include some hidden text content, errors or semantic gaps in the text recognition results can be corrected. Therefore, after repairing the speech based on the corrected text, the quality of the repaired speech can be improved.

[0066] The speech restoration method provided in this invention aligns the encoded features of the speech to be restored at the current moment with the target text features to obtain the target features. Based on the target features, the posterior probabilities of each text corresponding to the speech to be restored are determined. Based on each posterior probability, the text recognition result of the speech to be restored is determined. The posterior probabilities and the text recognition result are input into the text restoration module of the speech restoration model to obtain the first latent features output by the text restoration module. These first latent features include text latent features. Based on these text latent features, the speech to be restored is restored. Since the text latent features can be obtained by inputting the posterior probabilities and the text recognition result into the text restoration module of the speech restoration model, even if a recognition error occurs in the text recognition result, the text recognition result can be corrected based on the predicted text latent features. Therefore, after restoring the speech to be restored based on the text latent features, the lost semantics in the text recognition result can be restored, improving the intelligibility of the restored speech.

[0067] Furthermore, in existing technologies, the method of recognizing the speech to be repaired as text and then converting the text into synthesized speech loses the prosody of the original speech. To address this issue, in this embodiment, when repairing the speech based on text latent features, the first latent feature can be input into the prosody repair module of the speech repair model to obtain the prosody latent features output by the prosody repair module, and the speech to be repaired can be repaired based on the text latent features and the prosody latent features.

[0068] Specifically, since blanks can be used to align encoded features and target text features, the text contains blanks, which are then included in the predicted text latent features. Based on the blanks in the text latent features, the pause times and durations in the speech to be repaired can be predicted, thereby predicting the prosody of the speech to be repaired. In summary, after inputting the first latent features into the non-autoregressive (NAR) prosody repair module, the prosodic latent features in the next frame of speech can be predicted based on the blanks in the text latent features included in the first latent features.

[0069] Furthermore, the obtained latent text features and prosodic features can be weighted and summed, and the result of the weighted summation can be input into the vocoder in real time in a windowed manner to repair the speech to be repaired, thereby obtaining the repaired speech output by the vocoder.

[0070] In this embodiment, the first latent feature can be input into the prosody repair module. After obtaining the prosody latent feature output by the prosody repair module, the text latent feature and the prosody latent feature are used to repair the speech to be repaired. Since the prosody latent feature of the speech to be repaired can also be predicted, the repaired speech will have prosody-related content added, so that the repaired speech will not lose the prosody in the original speech to be repaired, thus improving the user experience.

[0071] For example, based on the above embodiments, the target information also includes the timbre features corresponding to the speech to be repaired, and the first latent feature also includes the timbre latent feature. When repairing the speech to be repaired based on the text latent feature and the prosodic latent feature, the text latent feature, the timbre latent feature and the prosodic latent feature can be input into the acoustic detail repair module of the speech repair model to obtain the acoustic detail latent feature output by the acoustic detail repair module, and the speech to be repaired can be repaired based on the text latent feature, the prosodic latent feature and the acoustic detail latent feature.

[0072] Specifically, Fbank features are extracted from the speech to be repaired. Then, Fbank features The input is fed into the timbre extraction model, which outputs the timbre features corresponding to the speech to be repaired. These timbre features are then... Each posterior probability and the text recognition result are used as target information. After being input into the text repair module, the first latent feature output includes not only text latent features but also timbre latent features.

[0073] The first latent feature is input into the prosody restoration module, which outputs both the prosody latent feature and the timbre latent feature. The text latent feature, timbre latent feature, and prosody latent feature are then input into the acoustic detail restoration module of the speech restoration model. Through a NAR-type acoustic detail restoration module, acoustic details other than text and prosody in the speech to be restored can be predicted, thus obtaining acoustic detail latent features. These acoustic details include timbre and emotion, among others.

[0074] Furthermore, the speech to be repaired can be based on the three dimensions of the obtained textual latent features, prosodic latent features, and acoustic detail latent features.

[0075] In this embodiment, since the target information also includes the timbre features corresponding to the speech to be repaired, and the first latent feature also includes timbre latent features, the predicted acoustic detail latent features will also contain timbre-related content. This ensures that the generated repaired speech also contains the timbre, guaranteeing that the timbre of the repaired speech and the speech to be repaired are consistent, i.e., the speaker is consistent. Furthermore, based on the text latent features and prosodic latent features, acoustic detail latent features can be added to repair the speech to be repaired. This allows the repaired speech to preserve the emotional content and other elements of the original speech to be repaired to the greatest extent possible, improving the repair quality and enhancing the user experience.

[0076] For example, based on the above embodiments, when repairing the speech to be repaired based on text latent features, prosodic latent features and acoustic detail latent features, the text latent features, prosodic latent features and acoustic detail latent features can be weighted and summed to obtain weighted latent features. After obtaining weighted latent features, the weighted latent features are input into the vocoder to obtain the repaired speech output by the vocoder.

[0077] Specifically, the weights of text latent features, prosodic latent features, and acoustic detail latent features can be pre-set. These weights can be used to characterize the importance of each feature in speech restoration; for example, the higher the importance, the greater the corresponding weight. For instance, the weight of text latent features is higher than that of acoustic detail latent features, and the weight of acoustic detail latent features is higher than that of prosodic latent features, and so on.

[0078] By weighting and summing the textual latent features, prosodic latent features, and acoustic detail latent features according to their respective weights, we can obtain the weighted latent features. Then, the weighted latent features are input into the vocoder in real time in a chunked manner to obtain the repaired speech output by the vocoder.

[0079] In this embodiment, by weighting and summing the textual latent features, prosodic latent features, and acoustic detail latent features according to their respective weights, and then performing speech restoration based on the weighted latent features, the restored speech can be more targeted. Furthermore, the restored speech obtained in this manner preserves the timbre, prosody, and emotional information of the original speech, greatly improving the quality of the restored speech, resulting in a better listening experience for the user.

[0080] For example, based on the above embodiments, when aligning the encoded features of the speech to be repaired and the target text features obtained at the current moment to obtain the target features, it can be done in the following way: input the speech features corresponding to the speech to be repaired into the encoder of the speech recognition model to obtain the encoded features output by the encoder, and align the encoded features and the target text features based on the blank to obtain the target features.

[0081] Correspondingly, when determining the posterior probabilities of each text corresponding to the speech to be repaired based on the target features, the target features can be input into the joint network of the speech recognition model to obtain the posterior probabilities output by the joint network. The speech recognition model is trained based on normal speech samples and speech samples to be repaired.

[0082] Specifically, to ensure the real-time performance of speech restoration, a Transducer-based speech recognition model can be used. This model includes an encoder, a decoder, and a JointNet network. The corresponding speech features, Fbank features, are extracted from the speech to be restored. Next, the speech features are input into the encoder to obtain the encoded features output by the encoder. .

[0083] Furthermore, encoded features can be further enhanced by padding with blanks. The target text features corresponding to the speech at the previous time step u-1 output by the decoder. Alignment is performed to obtain the target features.

[0084] After obtaining the target features, these features can be input into the JointNet of the speech recognition model to obtain the posterior probabilities of each text corresponding to the speech to be repaired at the current time, as output by the JointNet. Wherein, the posterior probability The probability values ​​used to characterize the speech to be repaired as each text are used, where the posterior probability of each text can be understood as the softmax result of the speech to be repaired.

[0085] In training the speech recognition model, multiple frames of normal speech samples and multiple frames of speech samples to be repaired can be acquired. These samples are then labeled, and the model is trained based on the labeled information to obtain a trained speech recognition model. It should be understood that the aforementioned normal speech samples include speech that can be recognized normally without repair, or speech with clear pronunciation or without lost semantic information. Speech samples to be repaired include speech that needs repair due to unclear pronunciation or missing semantics.

[0086] In this embodiment, by aligning the encoded features with the target text features corresponding to the previous moment's speech output by the decoder of the speech recognition model, the speech and text features can correspond in the time dimension. Therefore, it can be ensured that the output of the speech recognition model is input into the backend speech restoration model in real time, guaranteeing the real-time performance of speech restoration, especially in end-to-end call scenarios, meeting the requirements of real-time communication. Furthermore, since the speech recognition model is trained using a mixture of normal speech samples and speech samples to be restored, it can correctly recognize speech samples with semantic deficiencies due to unclear pronunciation or other reasons, while ensuring normal speech recognition.

[0087] For example, based on the above embodiments, the speech restoration models involved in the foregoing embodiments are trained in the following manner:

[0088] Normal speech samples are input into the factorized vector quantization codebook module to obtain the text codebook, prosodic codebook, and acoustic detail codebook output by the factorized vector quantization codebook module. Normal speech samples are then input into the initial speech restoration model to obtain the predicted text latent features output by the initial text restoration module, the predicted prosodic latent features output by the initial prosodic restoration module, and the predicted acoustic detail latent features output by the initial acoustic detail restoration module. Based on the text codebook, prosodic codebook, acoustic detail codebook, predicted text latent features, predicted prosodic latent features, and predicted acoustic detail latent features, loss information is determined. Based on the loss information, the parameters of the initial text restoration module, initial prosodic restoration module, and initial acoustic detail restoration module are adjusted to obtain the speech restoration model.

[0089] Specifically, during the training process of the speech restoration model, normal speech samples are used for training, and the Fbank audio features of the t-th frame corresponding to the normal speech samples can be obtained. In the Factorized Vector Quantization (FACodec) codebook module, the text codebook output by the FACodec module can be obtained based on formula (1). Rhythmic Codebook Harmony and Acoustic Details Codebook :

[0090] (1)

[0091] The FACodec module is pre-trained with large datasets and keeps its parameters fixed. Its main purpose is to decouple the Fbank audio features from content, prosody, and acoustic details, facilitating the progressive learning of speech restoration models based on autoregression and non-autoregression.

[0092] In addition, by inputting normal speech samples into the initial text restoration module of the initial speech restoration model, the latent features of the predicted text output by the initial text restoration module can be obtained. And will predict the latent features of the text. The input to the initial prosodic restoration module in the initial speech restoration model yields the predicted prosodic latent features output by the initial prosodic restoration module. This will predict the latent features of the text. and predict prosodic latent features By inputting the initial acoustic detail restoration module into the initial speech restoration model, we can obtain the predicted acoustic detail latent features output by the initial acoustic detail restoration module. Predicting latent features of text Predicting latent prosodic features and predict acoustic details and potential features By using the output layer of the initial speech restoration model, the posterior probability of the text corresponding to the normal speech sample in frame t can be obtained based on formula (2). Prosodic posterior probability Harmony Acoustic details posterior probability :

[0093] (2)

[0094] Based on the text posterior probability Prosodic posterior probability Harmony Acoustic details posterior probability and text codebook Rhythmic Codebook Harmony and Acoustic Details Codebook According to formula (3), the cross-entropy (CE) loss information Loss can be obtained:

[0095] (3)

[0096] Furthermore, the parameters of the initial text restoration module, the initial prosody restoration module, and the initial acoustic detail restoration module can be adjusted based on the loss information. By repeatedly executing the above process until the obtained model converges or the number of repetitions reaches the preset number, the final model is determined as the speech restoration model.

[0097] In this embodiment, normal speech samples are used for training when training the speech restoration model, so that the predicted prosody and acoustic details are biased towards the normal situation, thereby realizing the conversion of the speech to be restored into normal content, normal prosody and normal acoustic details.

[0098] The following section will describe in detail the speech restoration method provided in this embodiment of the invention, using the ASR model as the speech recognition model and the GPT model as the speech restoration model.

[0099] Figure 2 A block diagram of the voice restoration method provided in the embodiments of the present invention, as shown below. Figure 2 As shown, the speech restoration method provided in this embodiment of the invention performs real-time sound restoration of the speech to be restored through progressive learning. The speech restoration process includes four modules: an ASR module, a GPT module, a FACodec module, and a vocoder module. In the ASR module, to ensure the real-time performance of the speech restoration, a Transducer-based model can be used, which includes an encoder, a decoder, and a JointNet network.

[0100] During real-time repair, the Fbank features of the t-th frame of the speech to be repaired are acquired at the current time. Fbank features In the encoder of the input ASR module, the encoded features of the encoder output can be obtained through formula (4). :

[0101] (4)

[0102] Then encode the features The target text features corresponding to the speech at the previous time step u-1 output by the decoder. After alignment and obtaining the target features, the target features are input into the joint network. The posterior probabilities of each text corresponding to the speech to be repaired at the current time are obtained by formula (5). :

[0103] (5)

[0104] Additionally, the target features need to be input into the decoder to obtain the decoded features of the t-th frame of the speech to be repaired. Among these features, the posterior probability of each text... This can be understood as the softmax result of the speech to be repaired.

[0105] After determining the posterior probability of each text Then, the text recognition result of the speech to be repaired can be determined by formula (6). The text recognition result can be understood as the hard recognition result (argmax) of the speech to be repaired:

[0106] (6)

[0107] In particular, because the Transducer model uses blank for audio-text alignment, the posterior probability... and text recognition results The data will include blank information to ensure that the ASR results are sent to the backend GPT module in real time. Furthermore, during training, the ASR module uses a mix of samples to be repaired and normal speech samples for training, ensuring both normal and correct recognition of the speech to be repaired.

[0108] Additionally, the Fbank features of the speech to be repaired can be used. The input is fed into the timbre extraction model (xvector) to obtain the timbre features output by the timbre extraction model. This timbre feature ensures that the timbre of the final generated repaired speech is the same as that of the speech to be repaired, meaning it is from the same speaker.

[0109] The GPT module includes both autoregressive and non-autoregressive GPT networks, whose main function is to model text, prosody, and content sequentially. Specifically, the text restoration module uses an autoregressive network, while the prosody restoration and acoustic detail restoration modules use non-autoregressive networks. The input to the text restoration module can be the posterior probability. Text recognition results and timbre characteristics As inputs containing semantically relevant information, the content latent (CL) features of the next frame are predicted using an autoregressive approach through formulas (7) and (8):

[0110] (7)

[0111] (8)

[0112] Among them, posterior probability is added to the input. It can guarantee that when the text recognition result In cases of recognition errors, soft information is used to smooth the speech and prevent it from affecting the intelligibility of the generated restored speech. Additionally, timbre features are incorporated. It can ensure that the generated repaired speech is consistent with the target speaker, including timbre features. It can also be understood as voiceprint characteristics.

[0113] For non-autoregressive GPT networks, latent text features can be... The prosody latent (PL) output by the non-autoregressive prosody repair module can be obtained through formula (9):

[0114] (9)

[0115] latent features of text and rhythmic potential features In the input non-autoregressive acoustic detail restoration module, the acoustic detail features output by the acoustic detail restoration module can be obtained through formula (10). :

[0116] (10)

[0117] In obtaining latent text features Potential characteristics of rhythm Potential features of harmony and acoustic details Then, the three are weighted and summed to obtain the weighted latent features, which are then fed into the vocoder in real time according to the method of windowing (chunks). The repaired speech (wav) is obtained through formula (11):

[0118] (11)

[0119] The speech restoration method provided by this invention, based on a real-time combination of Transducer, GPT, and Vocoder, not only ensures the real-time performance of speech restoration but also significantly improves the intelligibility of the speech to be restored. The Transducer is a naturally autoregressive structure, the GPT employs both autoregressive and non-autoregressive structures, and the Vocoder uses a chunked approach to guarantee real-time performance. Furthermore, the input to the GPT can utilize soft recognition results based on posterior probability, hard recognition results based on text recognition results, and timbre information at the frame level, ensuring the preservation of timbre, pauses, and emotional information in the original speech to be restored, thus enhancing the user experience. During the training phase of the GPT model, only normal speech samples are used. In this way, the prosody and acoustic details predicted by GPT are biased towards normal speech, achieving a conversion of the speech to be restored into normal content, prosody, and acoustic details. Additionally, the GPT module, based on both autoregressive and non-autoregressive approaches, can progressively learn the content, prosody, and acoustic details of the soft and hard recognition results and timbre information, gradually constructing complete speech information and resulting in a better listening experience.

[0120] The speech restoration device provided by the present invention is described below. The speech restoration device described below and the speech restoration method described above can be referred to in correspondence.

[0121] Figure 3 This is a schematic diagram of the voice restoration device provided in an embodiment of the present invention, with reference to... Figure 3 As shown, the voice restoration device 300 includes:

[0122] Alignment module 11 is used to align the encoded features of the speech to be repaired acquired at the current moment with the target text features to obtain the target features; the target text features are the text features corresponding to the speech at the previous moment output by the decoder of the speech recognition model.

[0123] The determining module 12 is used to determine the posterior probability of each text corresponding to the speech to be repaired based on the target features;

[0124] The determining module 12 is further configured to determine the text recognition result of the speech to be repaired based on each of the posterior probabilities;

[0125] Input module 13 is used to input target information into the text repair module of the speech repair model to obtain the first latent feature output by the text repair module. The first latent feature includes text latent features, and the target information includes each of the posterior probabilities and the text recognition result.

[0126] Repair module 14 is used to repair the speech to be repaired based on the latent features of the text.

[0127] In one example embodiment, the repair module 14 is specifically used for:

[0128] The first latent feature is input into the prosody restoration module of the speech restoration model to obtain the prosody latent feature output by the prosody restoration module.

[0129] The speech to be repaired is repaired based on the latent text features and the latent prosodic features.

[0130] In one example embodiment, the target information further includes the timbre features corresponding to the speech to be repaired, and the first potential feature further includes timbre potential features;

[0131] Repair module 14 is specifically used for:

[0132] The text latent features, timbre latent features, and prosodic latent features are input into the acoustic detail restoration module of the speech restoration model to obtain the acoustic detail latent features output by the acoustic detail restoration module.

[0133] The speech to be repaired is repaired based on the textual latent features, the prosodic latent features, and the acoustic detail latent features.

[0134] In one example embodiment, the repair module 14 is specifically used for:

[0135] The textual latent features, the prosodic latent features, and the acoustic detail latent features are weighted and summed to obtain the weighted latent features;

[0136] The weighted latent features are input into the vocoder to obtain the repaired speech output by the vocoder.

[0137] In one example embodiment, the alignment module 11 is specifically used for:

[0138] The speech features corresponding to the speech to be repaired are input into the encoder of the speech recognition model to obtain the encoded features output by the encoder.

[0139] The encoded features and target text features are aligned based on the blank space to obtain the target features;

[0140] Module 12 is specifically used for:

[0141] The target features are input into the joint network of the speech recognition model to obtain the posterior probabilities output by the joint network.

[0142] The speech recognition model is trained based on normal speech samples and speech samples to be repaired.

[0143] In one example embodiment, the speech restoration model is trained in the following manner:

[0144] Normal speech samples are input into the factor vector quantization codebook module to obtain the text codebook, prosodic codebook and acoustic detail codebook output by the factor vector quantization codebook module;

[0145] The normal speech sample is input into the initial speech restoration model to obtain the predicted text latent features output by the initial text restoration module of the initial speech restoration model, the predicted prosodic latent features output by the initial prosodic restoration module of the initial speech restoration model, and the predicted acoustic detail latent features output by the initial acoustic detail restoration module of the initial speech restoration model.

[0146] Based on the text codebook, the prosodic codebook, the acoustic detail codebook, the predicted text latent features, the predicted prosodic latent features, and the predicted acoustic detail latent features, loss information is determined;

[0147] Based on the loss information, the parameters of the initial text restoration module, the initial prosody restoration module, and the initial acoustic detail restoration module are adjusted to obtain the speech restoration model.

[0148] In one example embodiment, the text restoration module includes an autoregressive model, while the prosodic restoration module and the acoustic detail restoration module include non-autoregressive models.

[0149] The apparatus of this embodiment can be used to execute the method of any embodiment in the speech restoration method side embodiment. Its specific implementation process and technical effects are similar to those in the speech restoration method side embodiment. For details, please refer to the detailed description in the speech restoration method side embodiment, which will not be repeated here.

[0150] Figure 4 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a speech restoration method, which includes: aligning the encoded features of the speech to be restored acquired at the current moment with the target text features to obtain target features; the target text features are the text features corresponding to the speech at the previous moment output by the decoder of the speech recognition model; determining the posterior probability of each text corresponding to the speech to be restored based on the target features; determining the text recognition result of the speech to be restored based on each posterior probability; inputting the target information into the text restoration module of the speech restoration model to obtain a first latent feature output by the text restoration module, the first latent feature including text latent features, the target information including each posterior probability and the text recognition result; and restoring the speech to be restored based on the text latent features.

[0151] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0152] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the speech restoration method provided by the above methods. The method includes: aligning the encoded features of the speech to be restored acquired at the current moment with the target text features to obtain target features; the target text features are the text features corresponding to the speech at the previous moment output by the decoder of the speech recognition model; determining the posterior probability of each text corresponding to the speech to be restored based on the target features; determining the text recognition result of the speech to be restored based on each posterior probability; inputting the target information into the text restoration module of the speech restoration model to obtain a first latent feature output by the text restoration module, the first latent feature including text latent features, the target information including each posterior probability and the text recognition result; and restoring the speech to be restored based on the text latent features.

[0153] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the speech restoration method provided by the above methods. The method includes: aligning the encoded features of the speech to be restored acquired at the current moment with target text features to obtain target features; the target text features being the text features corresponding to the speech at the previous moment output by the decoder of a speech recognition model; determining the posterior probability of each text corresponding to the speech to be restored based on the target features; determining the text recognition result of the speech to be restored based on each of the posterior probabilities; inputting the target information into the text restoration module of the speech restoration model to obtain a first latent feature output by the text restoration module, the first latent feature including text latent features, the target information including each of the posterior probabilities and the text recognition result; and restoring the speech to be restored based on the text latent features.

[0154] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0155] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A voice restoration method, characterized in that, include: Align the encoded features of the speech to be repaired acquired at the current moment with the features of the target text to obtain the target features; The target text features are the text features corresponding to the speech output by the decoder of the speech recognition model at the previous moment. Based on the target features, the posterior probability of each text corresponding to the speech to be repaired is determined; Based on the posterior probabilities, the text recognition result of the speech to be repaired is determined; The target information is input into the text restoration module of the speech restoration model to obtain the first latent feature output by the text restoration module. The first latent feature includes text latent features. The target information includes each of the posterior probabilities and the text recognition result. The speech to be repaired is repaired based on the latent features of the text.

2. The voice restoration method according to claim 1, characterized in that, The repair of the speech to be repaired based on the latent features of the text includes: The first latent feature is input into the prosody restoration module of the speech restoration model to obtain the prosody latent feature output by the prosody restoration module. The speech to be repaired is repaired based on the latent text features and the latent prosodic features.

3. The voice restoration method according to claim 2, characterized in that, The target information also includes the timbre features corresponding to the speech to be repaired, and the first potential feature also includes timbre potential features; The repair of the speech to be repaired based on the text latent features and the prosodic latent features includes: The text latent features, timbre latent features, and prosodic latent features are input into the acoustic detail restoration module of the speech restoration model to obtain the acoustic detail latent features output by the acoustic detail restoration module. The speech to be repaired is repaired based on the textual latent features, the prosodic latent features, and the acoustic detail latent features.

4. The voice restoration method according to claim 3, characterized in that, The repair of the speech to be repaired based on the textual latent features, the prosodic latent features, and the acoustic detail latent features includes: The textual latent features, the prosodic latent features, and the acoustic detail latent features are weighted and summed to obtain the weighted latent features; The weighted latent features are input into the vocoder to obtain the repaired speech output by the vocoder.

5. The speech restoration method according to any one of claims 1-4, characterized in that, The step of aligning the encoded features of the speech to be repaired acquired at the current moment with the target text features to obtain the target features includes: The speech features corresponding to the speech to be repaired are input into the encoder of the speech recognition model to obtain the encoded features output by the encoder. The encoded features and target text features are aligned based on the blank space to obtain the target features; The step of determining the posterior probability of each text corresponding to the speech to be repaired based on the target features includes: The target features are input into the joint network of the speech recognition model to obtain the posterior probabilities output by the joint network. The speech recognition model is trained based on normal speech samples and speech samples to be repaired.

6. The speech restoration method according to any one of claims 1-4, characterized in that, The speech restoration model was trained in the following manner: Normal speech samples are input into the factor vector quantization codebook module to obtain the text codebook, prosodic codebook and acoustic detail codebook output by the factor vector quantization codebook module; The normal speech sample is input into the initial speech restoration model to obtain the predicted text latent features output by the initial text restoration module of the initial speech restoration model, the predicted prosodic latent features output by the initial prosodic restoration module of the initial speech restoration model, and the predicted acoustic detail latent features output by the initial acoustic detail restoration module of the initial speech restoration model. Based on the text codebook, the prosodic codebook, the acoustic detail codebook, the predicted text latent features, the predicted prosodic latent features, and the predicted acoustic detail latent features, loss information is determined; Based on the loss information, the parameters of the initial text restoration module, the initial prosody restoration module, and the initial acoustic detail restoration module are adjusted to obtain the speech restoration model.

7. The voice restoration method according to claim 3, characterized in that, The text restoration module includes an autoregressive model, while the prosody restoration module and the acoustic detail restoration module include non-autoregressive models.

8. A voice restoration device, characterized in that, include: The alignment module is used to align the encoded features of the speech to be repaired acquired at the current moment with the features of the target text to obtain the target features; The target text features are the text features corresponding to the speech output by the decoder of the speech recognition model at the previous moment. The determination module is used to determine the posterior probability of each text corresponding to the speech to be repaired based on the target features; The determining module is further configured to determine the text recognition result of the speech to be repaired based on each of the posterior probabilities; An input module is used to input target information into the text restoration module of the speech restoration model to obtain the first latent feature output by the text restoration module. The first latent feature includes text latent features, and the target information includes each of the posterior probabilities and the text recognition result. The repair module is used to repair the speech to be repaired based on the latent features of the text.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the voice restoration method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the voice restoration method as described in any one of claims 1 to 7.

Images