Voice restoration method and device, equipment and storage medium

By aligning the coding features of the repaired speech with the target text features, combining posterior probability and text recognition results, potential text features are generated, and repaired in combination with prosthetic and acoustic details, the low intelligibility of speech repair in the existing technology is solved. Problem, achieving higher voice repair quality and user experience.

CN118968965AActive Publication Date: 2024-11-15ANHUI IFLYTEK UNIVERSAL LANGUAGE TECH CO LTD

Patent Information

Application Number
CN202411169095.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-11-15
Estimated Expiration
2044-08-23

AI Technical Summary

Technical Problem

In the prior art, the speech intelligibility after speech repair is low, especially when there are broken parts or unclear pronunciations in the speech to be repaired, the identification error of the ASR model leads to the loss of generated speech semantic information, which reduces the Intelligible.

Method used

By aligning the coding features of the speech to be repaired with the target text features, determining the posterior probability and text recognition results, input the text repair module of the speech repair model to generate potential text features, and repair them in combination with pronunciation and acoustic detail features, using autoregression The non-autoregressive model is weighted and summed, and finally the voice is generated and repaired by the vocoder.

Benefits of technology

It improves the intelligibility and accuracy of repaired speech, fixes semantic loss in text recognition results, retains the rhythm and tone of the original speech, and improves the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968965A_ABST
    Figure CN118968965A_ABST
Patent Text Reader

Abstract

The invention provides a voice restoration method and device, equipment and a storage medium, and the method comprises the steps: enabling a coding feature of a to-be-restored voice obtained at a current moment to be aligned with a target text feature, and obtaining a target feature; the target text feature is a text feature corresponding to a voice at a previous moment output by a decoder of a voice recognition model; based on the target feature, determining a posterior probability of each text corresponding to the to-be-restored voice; based on each posterior probability, determining a text recognition result of the to-be-restored voice; target information is input into a text repairing module of a voice repairing model, a first potential feature output by the text repairing module is obtained, the first potential feature comprises text potential features, and the target information comprises the posterior probabilities and the text recognition result; and repairing the to-be-repaired voice based on the text potential features. According to the embodiment of the invention, the intelligibility of the restored voice can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to a speech repair method, device, equipment and storage medium. Background Art

[0002] With the development of end-to-end text-to-speech (TTS) technology, it has been widely used in different scenarios. In calls and face-to-face scenarios, some users suffer from speech function loss due to illness, accidents or other reasons, which leads to unclear articulation or inaccurate pronunciation. Therefore, the speech of these users needs to be repaired.

[0003] In the prior art, an Automatic Speech Recognition (ASR) model is usually used to first identify the text corresponding to the speech to be repaired, and then the text is input into a speech synthesis model to generate the repaired speech.

[0004] However, in the above method, when there are many incomplete parts or unclear words in the speech to be repaired, the ASR model will have incorrect recognition text in the recognition results, resulting in low intelligibility of the generated repaired speech. Summary of the invention

[0005] The present invention provides a speech repair method, device, equipment and storage medium, which are used to solve the defect of low intelligibility of the repaired speech in the prior art and improve the intelligibility of the repaired speech.

[0006] The present invention provides a voice repair method, comprising: Align the encoding features of the speech to be repaired obtained at the current moment with the target text features to obtain the target features; the target text features are the text features corresponding to the speech at the previous moment output by the decoder of the speech recognition model; Based on the target feature, determining the posterior probability of each text corresponding to the speech to be repaired; Based on each of the posterior probabilities, determining a text recognition result of the speech to be repaired; Inputting the target information into a text repair module of a speech repair model to obtain a first potential feature output by the text repair module, wherein the first potential feature includes a text potential feature, and the target information includes each of the posterior probabilities and the text recognition result; The speech to be repaired is repaired based on the text potential features.

[0007] According to a speech repair method provided by the present invention, the speech to be repaired is repaired based on the text potential features, comprising: Inputting the first latent feature into a prosody restoration module of the speech restoration model to obtain a prosody latent feature output by the prosody restoration module; The speech to be repaired is repaired based on the text latent features and the prosody latent features.

[0008] According to a speech restoration method provided by the present invention, the target information further includes a timbre feature corresponding to the speech to be restored, and the first latent feature further includes a timbre latent feature; The step of repairing the speech to be repaired based on the text potential feature and the prosody potential feature includes: Inputting the text latent feature, the timbre latent feature and the rhythm latent feature into the acoustic detail restoration module of the speech restoration model to obtain the acoustic detail latent feature output by the acoustic detail restoration module; The speech to be repaired is repaired based on the text latent features, the prosody latent features and the acoustic detail latent features.

[0009] According to a speech restoration method provided by the present invention, the speech to be restored is restored based on the text potential features, the prosody potential features and the acoustic detail potential features, comprising: Performing weighted summation on the text latent feature, the rhythmic latent feature and the acoustic detail latent feature to obtain a weighted latent feature; The weighted latent features are input into a vocoder to obtain the restored speech output by the vocoder.

[0010] According to a speech repair method provided by the present invention, aligning the encoding features of the speech to be repaired acquired at the current moment with the target text features to obtain the target features includes: Inputting the speech features corresponding to the speech to be repaired into the encoder of the speech recognition model to obtain the encoding features output by the encoder; Aligning the encoding feature with the target text feature based on blank to obtain the target feature; The determining, based on the target feature, the posterior probability of each text corresponding to the speech to be repaired includes: Inputting the target feature into the joint network of the speech recognition model to obtain the posterior probabilities output by the joint network; The speech recognition model is trained based on normal speech samples and speech samples to be repaired.

[0011] According to a speech restoration method provided by the present invention, the speech restoration model is trained based on the following method: Inputting a normal speech sample into a factor vector quantization codebook module to obtain a text codebook, a prosody codebook and an acoustic detail codebook output by the factor vector quantization codebook module; Inputting the normal speech sample into an initial speech repair model, obtaining predicted text potential features output by an initial text repair module of the initial speech repair model, predicted prosody potential features output by an initial prosody repair module of the initial speech repair model, and predicted acoustic detail potential features output by an initial acoustic detail repair module of the initial speech repair model; Determining loss information based on the text codebook, the prosody codebook, the acoustic detail codebook, the predicted text latent features, the predicted prosody latent features, and the predicted acoustic detail latent features; The parameters of the initial text repair module, the initial prosody repair module, and the initial acoustic detail repair module are adjusted based on the loss information to obtain the speech repair model.

[0012] According to a speech repair method provided by the present invention, the text repair module includes an autoregressive model, and the prosody repair module and the acoustic detail repair module include non-autoregressive models.

[0013] The present invention also provides a speech repair device, comprising: An alignment module is used to align the encoding features of the speech to be repaired obtained at the current moment with the target text features to obtain the target features; the target text features are the text features corresponding to the speech at the previous moment output by the decoder of the speech recognition model; A determination module, used to determine the posterior probability of each text corresponding to the speech to be repaired based on the target feature; The determination module is further used to determine the text recognition result of the speech to be repaired based on each of the posterior probabilities; An input module, used for inputting target information into a text repair module of a speech repair model, to obtain a first potential feature output by the text repair module, wherein the first potential feature includes a text potential feature, and the target information includes each of the posterior probabilities and the text recognition result; A repair module is used to repair the speech to be repaired based on the potential features of the text.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned speech repair methods when executing the computer program.

[0015] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the speech repair method described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the speech repair method described above is implemented.

[0017] The speech repair method, device, equipment and storage medium provided by the present invention align the coding features of the speech to be repaired obtained at the current moment with the target text features, obtain the target features, determine the posterior probabilities of each text corresponding to the speech to be repaired based on the target features, and determine the text recognition results of the speech to be repaired based on each posterior probability, input each posterior probability and the text recognition results into the text repair module of the speech repair model, obtain the first potential features output by the text repair module, and the first potential features include text potential features, and repair the speech to be repaired based on the text potential features. Since each posterior probability and the text recognition results are input into the text repair module of the speech repair model, the text potential features can be obtained, so that even if the text recognition result has a recognition error, the text recognition result can be corrected based on the predicted text potential features. Therefore, after the speech to be repaired is repaired based on the text potential features, the lost semantics in the text recognition result can be repaired, thereby improving the intelligibility of the repaired speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A flowchart of a voice repair method provided by an embodiment of the present invention.

[0020] Figure 2 A block diagram of a speech repair method provided by an embodiment of the present invention.

[0021] Figure 3 A schematic diagram of the structure of a speech repair device provided in an embodiment of the present invention.

[0022] Figure 4 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] In end-to-end scenarios such as mobile phone calls or face-to-face communication between users, some users may encounter communication barriers due to unclear articulation or inaccurate pronunciation. Therefore, it is very important to accurately repair the voices of these users. In the prior art, a non-real-time cascade architecture can be used for voice repair. Specifically, the voice to be repaired can be first input into the ASR model for recognition to obtain the text corresponding to the voice, and then the text can be input into the speech synthesis model to generate the corresponding normal voice. However, in the above method, if the voice to be repaired contains a large amount of unclear voice, the recognition text output by the ASR model will be wrong, resulting in the repaired voice generated based on the recognition text will not have semantic information, so that the intelligibility of the repaired voice is greatly reduced.

[0025] In view of the above problems, an embodiment of the present invention provides a speech repair method. In the method, on the basis of determining the text recognition result of the speech to be repaired, the posterior probability of each text corresponding to the speech to be repaired is also added, and the posterior probability of each text corresponding to the speech to be repaired and the text recognition result are input into the text repair module of the speech repair model, so that the text latent features can be obtained. In this way, even if the text recognition result has a recognition error, the text recognition result can be corrected based on the predicted text latent features. Therefore, after the speech to be repaired is repaired based on the text latent features, the lost semantics in the text recognition result can be repaired, thereby improving the intelligibility and accuracy of the repaired speech.

[0026] Combine the following Figure 1 and Figure 2 The voice repair method provided by the embodiment of the present invention is described. The embodiment of the present invention can be applicable to scenarios where the user's voice needs to be repaired during end-to-end communication such as mobile phone calls or face-to-face communication. The executor of the method can be an electronic device such as a terminal device, a computer, a server, a server cluster, or a specially designed voice repair device, or a voice repair device set in the electronic device, which can be implemented by software, hardware, or a combination of both.

[0027] Figure 1 A flow chart of a voice repair method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes: Step 101: Align the encoding features of the speech to be repaired obtained at the current moment with the target text features to obtain the target features; the target text features are the text features corresponding to the speech at the previous moment output by the decoder of the speech recognition model.

[0028] In this step, the speech to be repaired may include speech that is unclear in meaning or affects normal communication with others due to inaccurate pronunciation or unclear articulation. Extract Fbank features from the speech to be repaired obtained at the current moment , and the Fbank feature Input the encoder of the speech recognition model (ASR model) to obtain the encoding features of the encoder output .

[0029] Furthermore, the encoding features can be The text features corresponding to the speech at the previous moment u-1 output by the decoder of the speech recognition model Alignment can be performed, for example, by filling blanks to obtain the target features. After aligning the coded features with the text features corresponding to the speech at the previous moment output by the decoder of the speech recognition model, the speech and text features can correspond in the time dimension. Therefore, the results output by the speech recognition model can be input into the back-end speech restoration model in real time, which can ensure the real-time nature of speech restoration, especially in the end-to-end call scenario, which can meet the needs of real-time calls.

[0030] Step 102: Based on the target features, determine the posterior probability of each text corresponding to the speech to be repaired.

[0031] In this step, after obtaining the target feature, the target feature can be input into the joint network of the speech recognition model, so as to obtain the posterior probability of each text corresponding to the current moment of the speech to be repaired output by the joint network. Among them, the posterior probability It is used to characterize the probability value of the speech to be repaired being each text, where the posterior probability of each text can be understood as the soft recognition result (softmax) of the speech to be repaired.

[0032] Step 103: Based on each posterior probability, determine the text recognition result of the speech to be repaired.

[0033] In this step, the text recognition results It can be understood as the most likely recognition result of the speech to be repaired determined based on each posterior probability. Exemplarily, the text corresponding to the largest posterior probability among the posterior probabilities can be determined as the text recognition result of the speech to be repaired. Among them, the text recognition result can be understood as the hard recognition result (argmax) of the speech to be repaired.

[0034] It should be understood that since the encoding features and the target text features are aligned with blank for audio and text, the posterior probability And text recognition results Blank information will also be included in the ASR model to ensure the output results and It can be input into the back-end speech restoration model in real time.

[0035] Step 104: Input the target information into the text repair module of the speech repair model to obtain the first latent feature output by the text repair module, where the first latent feature includes the text latent feature, and the target information includes various posterior probabilities and text recognition results.

[0036] In this step, the speech repair model can be, for example, a Generative Pre-trained Transformer (GPT), and the text repair module in the speech repair model can be an autoregressive (AR) model. The posterior probabilities and text recognition results are input into the text repair module, and the first potential feature corresponding to the next frame of speech can be predicted by autoregression, and the first potential feature includes text potential features. Among them, the text potential features include some hidden or implicit text features, and the erroneous text in the text recognition results can be corrected through the text potential features.

[0037] Step 105: Repair the speech to be repaired based on the text potential features.

[0038] In this step, the text potential features can be input into the vocoder, and based on the text potential features, the missing content or unclearly pronounced content in the speech to be repaired can be repaired to obtain the repaired speech. Since the text potential features include some hidden text content, the parts of the text recognition results that are incorrectly recognized or semantically missing can be corrected, so that after the speech to be repaired is repaired based on the corrected text, the repair quality of the repaired speech can be improved.

[0039] The speech repair method provided by the embodiment of the present invention aligns the coding features of the speech to be repaired obtained at the current moment with the target text features, obtains the target features, determines the posterior probabilities of each text corresponding to the speech to be repaired based on the target features, and determines the text recognition results of the speech to be repaired based on each posterior probability, inputs each posterior probability and the text recognition results into the text repair module of the speech repair model, obtains the first potential features output by the text repair module, and the first potential features include text potential features, and repairs the speech to be repaired based on the text potential features. Since each posterior probability and the text recognition results are input into the text repair module of the speech repair model, the text potential features can be obtained, so that even if the text recognition result has a recognition error, the text recognition result can be corrected based on the predicted text potential features. Therefore, after the speech to be repaired is repaired based on the text potential features, the lost semantics in the text recognition result can be repaired, thereby improving the intelligibility of the repaired speech.

[0040] Furthermore, in the prior art, after recognizing the speech to be repaired as text, the text is converted into a synthesized speech, which will lose the rhythm of the original speech to be repaired. To address this problem, in this embodiment, when repairing the speech to be repaired based on the text latent feature, the first latent feature can be input into the rhythm repair module of the speech repair model to obtain the rhythm latent feature output by the rhythm repair module, and the speech to be repaired is repaired based on the text latent feature and the rhythm latent feature.

[0041] Specifically, since blank can be used to align the encoding features and the target text features, blank is included in the text, so blank is also included in the predicted text latent features. Based on blank in the text latent features, the pause moment and pause duration in the speech to be repaired can be predicted, thereby predicting the rhythm of the speech to be repaired. In summary, after the first latent feature is input into the non-autoregressive (NAR) rhythm repair module, the rhythmic latent features in the next frame of speech can be predicted based on blank in the text latent features contained in the first latent feature.

[0042] Furthermore, the obtained text latent features and prosody latent features can be weighted and summed, and the weighted summation result can be input into the vocoder in real time in a windowed manner to repair the speech to be repaired, thereby obtaining the repaired speech output by the vocoder.

[0043] In this embodiment, the first latent feature can be input into the rhythm repair module. After the rhythm latent feature output by the rhythm repair module is obtained, the speech to be repaired is repaired based on the text latent feature and the rhythm latent feature. Since the rhythm latent feature of the speech to be repaired can also be predicted, the rhythm-related content will be added to the repaired speech obtained after the repair, so that the repaired speech will not lose the rhythm of the original speech to be repaired, thereby improving the user experience.

[0044] Exemplarily, on the basis of the above embodiment, the target information also includes the timbre features corresponding to the speech to be repaired, and the first latent feature also includes the timbre latent feature. When the speech to be repaired is repaired based on the text latent feature and the prosody latent feature, the text latent feature, the timbre latent feature and the prosody latent feature can be input into the acoustic detail repair module of the speech repair model to obtain the acoustic detail latent feature output by the acoustic detail repair module, and the speech to be repaired is repaired based on the text latent feature, the prosody latent feature and the acoustic detail latent feature.

[0045] Specifically, Fbank features are extracted from the speech to be repaired. After that, the Fbank feature Input into the timbre extraction model, the timbre features corresponding to the speech to be repaired output by the timbre extraction model will be obtained. , each posterior probability and text recognition result are taken as target information and input into the text repair module. Then, the first latent feature output includes not only the text latent feature but also the timbre latent feature.

[0046] The first latent feature is input into the prosody restoration module, which outputs the timbre latent feature in addition to the prosody latent feature. The text latent feature, timbre latent feature and prosody latent feature are input into the acoustic detail restoration module of the speech restoration model. Through the NAR type acoustic detail restoration module, other acoustic details in the speech to be restored except for the text and prosody can be predicted, thereby obtaining the acoustic detail latent feature. Among them, the acoustic details include timbre and emotion.

[0047] Furthermore, the speech to be repaired can be repaired based on the three dimensions of the obtained text latent features, prosody latent features and acoustic detail latent features.

[0048] In this embodiment, since the target information also includes the timbre features corresponding to the speech to be repaired, and the first latent features also include the timbre latent features, the predicted acoustic detail latent features will also include the timbre-related content, so that the generated repaired speech can also include the timbre, ensuring that the timbre of the repaired speech is consistent with that of the speech to be repaired, that is, the speaker is consistent. In addition, on the basis of the text latent features and the rhythmic latent features, the acoustic detail latent features can also be added to repair the speech to be repaired, so that the repaired speech can guarantee the emotions and other contents in the original speech to be repaired to the greatest extent, thereby improving the repair quality of the repaired speech and enhancing the user experience.

[0049] Exemplarily, based on the above embodiments, when repairing the speech to be repaired based on textual latent features, prosodic latent features and acoustic detail latent features, the textual latent features, prosodic latent features and acoustic detail latent features may be weightedly summed to obtain the weighted latent features, which are then input into a vocoder to obtain the repaired speech output by the vocoder.

[0050] Specifically, the weights corresponding to the text latent features, prosody latent features, and acoustic detail latent features can be preset, and the weights can be used to characterize the importance of each feature in speech restoration. For example, the higher the importance, the greater the corresponding weight. For example, the weight of the text latent feature is higher than the weight of the acoustic detail latent feature, and the weight of the acoustic detail latent feature is higher than the weight of the prosody latent feature, etc.

[0051] After weighted summing up the text latent features, prosody latent features and acoustic detail latent features according to their respective weights, a weighted latent feature can be obtained, and then the weighted latent feature is input into the vocoder in real time in a windowed (chunk) manner to obtain the restored speech output by the vocoder.

[0052] In this embodiment, since the text potential features, rhythm potential features and acoustic detail potential features can be weighted and summed according to their corresponding weights, and then the voice restoration is performed based on the weighted potential features, the restored voice can be made more targeted. In addition, the restored voice obtained after restoration in the above manner can ensure the timbre, rhythm and emotion of the voice to be restored, which greatly improves the quality of the restored voice, makes the user's listening experience better, and improves the user's experience.

[0053] Exemplarily, on the basis of the above-mentioned embodiment, when aligning the coding features of the speech to be repaired obtained at the current moment with the target text features to obtain the target features, it can be done in the following way: input the speech features corresponding to the speech to be repaired into the encoder of the speech recognition model to obtain the coding features output by the encoder, and align the coding features with the target text features based on blank to obtain the target features.

[0054] Correspondingly, when determining the posterior probabilities of each text corresponding to the speech to be repaired based on the target feature, the target feature can be input into the joint network of the speech recognition model to obtain the posterior probabilities output by the joint network. The speech recognition model is trained based on normal speech samples and speech samples to be repaired.

[0055] Specifically, in order to ensure the real-time performance of speech restoration, the speech recognition model can adopt a Transducer-based model. The speech recognition model includes an encoder, a decoder, and a joint network. When extracting the corresponding speech feature Fbank from the speech to be restored, After that, the speech features are input into the encoder to obtain the encoded features output by the encoder. .

[0056] Furthermore, the encoded features can be filled with blanks. The target text features corresponding to the speech at the previous moment u-1 output by the decoder Align to obtain the target features.

[0057] After obtaining the target feature, the target feature can be input into the joint network (JointNet) of the speech recognition model to obtain the posterior probability of each text corresponding to the current moment of the speech to be repaired output by the joint network (JointNet) Among them, the posterior probability It is used to characterize the probability value of the speech to be repaired being each text, where the posterior probability of each text can be understood as the soft recognition result (softmax) of the speech to be repaired.

[0058] Among them, when training the speech recognition model, multiple frames of normal speech samples and multiple frames of speech samples to be repaired can be obtained, and the normal speech samples and the speech samples to be repaired are respectively annotated, so as to perform model training based on the annotated information to obtain a trained speech recognition model. It should be understood that the above-mentioned normal speech samples include speech that can be recognized normally without repair, or can also be understood as speech with clear pronunciation or no loss of semantic information. The speech samples to be repaired include speech that needs to be repaired due to unclear pronunciation or missing semantics.

[0059] In this embodiment, after aligning the coding features and the target text features corresponding to the speech at the previous moment output by the decoder of the speech recognition model, the speech and text features can be made to correspond in the time dimension. Therefore, the results output by the speech recognition model can be input into the back-end speech repair model in real time, and the real-time nature of speech repair can be guaranteed, especially in the end-to-end call scenario, which can meet the needs of real-time calls. In addition, since the speech recognition model is obtained by mixed training based on normal speech samples and speech samples to be repaired, it can ensure normal speech recognition while also correctly recognizing the speech to be repaired that has semantic loss due to unclear pronunciation and other reasons.

[0060] Exemplarily, based on the above embodiments, the speech restoration model involved in the above embodiments is trained based on the following method: A normal speech sample is input into a factor vector quantization codebook module to obtain a text codebook, a rhythm codebook and an acoustic detail codebook output by the factor vector quantization codebook module; a normal speech sample is input into an initial speech repair model to obtain predicted text potential features output by an initial text repair module of the initial speech repair model, predicted rhythm potential features output by an initial rhythm repair module of the initial speech repair model, and predicted acoustic detail potential features output by an initial acoustic detail repair module of the initial speech repair model; loss information is determined based on the text codebook, the rhythm codebook, the acoustic detail codebook, the predicted text potential features, the predicted rhythm potential features and the predicted acoustic detail potential features; parameters of the initial text repair module, the initial rhythm repair module and the initial acoustic detail repair module are adjusted based on the loss information to obtain a speech repair model.

[0061] Specifically, in the training process of the speech restoration model, normal speech samples are used for training, and the Fbank audio features of the tth frame corresponding to the normal speech samples can be Input the Factorized Vector Quantization codebook module (FACodec), and the text codebook output by the FACodec module can be obtained based on formula (1): , Prosody Codebook Harmony Detail Codebook : (1)

[0062] Among them, the FACodec module is pre-trained with big data and keeps the parameters fixed. Its main purpose is to decouple the content, rhythm and acoustic details of the Fbank audio features to facilitate the progressive learning of autoregressive and non-autoregressive speech restoration models.

[0063] In addition, by inputting the normal speech sample into the initial text repair module in the initial speech repair model, the predicted text potential features output by the initial text repair module can be obtained. , and predict the text potential features Input into the initial prosody restoration module in the initial speech restoration model, and the predicted prosody potential features output by the initial prosody restoration module can be obtained. , predicting the potential features of the text and predicting prosodic latent features By inputting the initial acoustic detail restoration module in the initial speech restoration model, we can obtain the predicted acoustic detail potential features output by the initial acoustic detail restoration module. . Predict the latent features of the text , predicting prosodic latent features and predict acoustic detail latent features Through the output layer of the initial speech restoration model, the text posterior probability corresponding to the normal speech sample of the tth frame can be obtained based on formula (2): , prosody posterior probability Harmony Detail Posterior Probability : (2)

[0064] According to the posterior probability of the text , prosody posterior probability Harmony Detail Posterior Probability , and text codebook , Prosody Codebook Harmony Detail Codebook According to formula (3), we can get the cross-entropy (CE) loss information Loss: (3)

[0065] Furthermore, the parameters of the initial text repair module, the initial prosody repair module, and the initial acoustic detail repair module can be adjusted based on the loss information. By repeatedly executing the above process until the obtained model converges or the number of repetitions reaches a preset number, the final model is determined as the speech repair model.

[0066] In this embodiment, when training the speech restoration model, normal speech samples are used for training, so that the final predicted prosody and acoustic details are biased towards normal conditions, thereby achieving the conversion of the speech to be restored into normal content, normal prosody and normal acoustic details.

[0067] Below, taking the speech recognition model as an ASR model and the speech restoration model as a GPT model as an example, the speech restoration method provided by an embodiment of the present invention is described in detail.

[0068] Figure 2 A block diagram of a voice repair method provided by an embodiment of the present invention, such as Figure 2 As shown, the speech restoration method provided by the embodiment of the present invention performs real-time sound restoration on the speech to be restored by progressive learning. The speech restoration process includes four modules: ASR module, GPT module, FACodec module and vocoder module. Among them, in the ASR module, in order to ensure the real-time performance of speech restoration, a Transducer-based model can be used, which includes an encoder, a decoder and a joint network.

[0069] In the real-time repair process, the Fbank feature of the t-th frame of the speech to be repaired obtained at the current moment is , the Fbank feature Input into the encoder of the ASR module, the encoding features of the encoder output can be obtained through formula (4) : (4)

[0070] Then the encoded features The target text features corresponding to the speech at the previous moment u-1 output by the decoder After alignment and obtaining the target features, the target features are input into the joint network, and the posterior probability of each text corresponding to the current moment of the speech to be repaired output by the joint network is obtained by formula (5): : (5)

[0071] In addition, the target features need to be input into the decoder to obtain the decoded features of the tth frame of the speech to be repaired output by the decoder. It can be understood as the soft recognition result (softmax) of the speech to be repaired.

[0072] After determining the posterior probability of each text After that, the text recognition result of the speech to be repaired can be determined by formula (6): , where the text recognition result can be understood as the hard recognition result (argmax) of the speech to be repaired: (6)

[0073] Among them, since the Transducer model uses blank for audio-text alignment, the posterior probability And text recognition results The blank information will be included in the ASR, so as to ensure that the ASR results can be sent to the back-end GPT module in real time. In addition, during the training process, the ASR module uses the speech samples to be repaired and the normal speech samples for mixed training, which can ensure normal recognition while also correctly recognizing the speech to be repaired.

[0074] In addition, the Fbank features of the speech to be repaired can be Input into the timbre extraction model (xvector) to obtain the timbre features output by the timbre extraction model , through this timbre feature, it can be ensured that the timbre of the finally generated repaired speech is the same as that of the speech to be repaired, that is, it is the same speaker.

[0075] The GPT module contains two types of GPT networks, autoregressive and non-autoregressive. Their main function is to model text, rhythm, and content in sequence. The text repair module is an autoregressive network, while the rhythm repair module and acoustic detail repair module are non-autoregressive networks. For the input of the text repair module, the posterior probability , text recognition results and timbre characteristics As the input of semantically relevant information, the text latent feature (CL) corresponding to the next frame is predicted by autoregression through formulas (7) and (8): (7)

[0076] (8)

[0077] Among them, adding the posterior probability to the input Can ensure that the text recognition results In the case of recognition errors, soft information is used for smoothing to avoid affecting the intelligibility of the generated repaired speech. In addition, timbre features are added It can ensure that the generated repaired speech is consistent with the target speaker, among which the timbre feature It can also be understood as a voiceprint feature.

[0078] For non-autoregressive GPT networks, the text potential features can be Input into the non-autoregressive prosody restoration module, the prosody latent feature (PL) output by the prosody restoration module can be obtained through formula (9): (9)

[0079] The latent features of the text and rhythmic latent features Input into the non-autoregressive acoustic detail restoration module, the acoustic detail features output by the acoustic detail restoration module can be obtained through formula (10): : (10)

[0080] After obtaining the latent features of the text , rhythmic latent features Harmonic detail latent features After that, the weighted sum of the three is taken to obtain the weighted latent feature, which is then sent to the vocoder in real time in a windowed (chunk) manner, and the repaired speech (wav) is obtained through formula (11): (11)

[0081] The voice restoration method provided by the embodiment of the present invention, based on the real-time combination of Transducer, GPT and Vocoder, can not only ensure the real-time performance of voice restoration, but also greatly improve the intelligibility of the voice to be restored. Among them, Transducer is a natural autoregressive structure, GPT adopts autoregressive and non-autoregressive structures, and Vocoder adopts a chunk method to ensure real-time performance. In addition, for the input of GPT, the soft recognition results of the posterior probability, the hard recognition results of the text recognition results and the information at the timbre information frame level can be used to ensure the timbre, pauses and emotions of the original voice to be restored, and improve the user experience. Among them, in the training stage of the GPT model, only normal voice samples are used. In this way, the rhythm and acoustic details predicted by GPT are biased towards normal voice, and the conversion of the voice to be restored to normal content, rhythm and acoustic details is realized. In addition, based on the autoregressive and non-autoregressive methods in the GPT module, the soft recognition results and hard recognition results as well as the learning content, rhythm and acoustic details of the timbre information can be layered, and the complete voice information can be gradually constructed, so that the listening experience is better.

[0082] The speech repair device provided by the present invention is described below. The speech repair device described below and the speech repair method described above can be referred to each other.

[0083] Figure 3 The structure diagram of the speech repair device provided by the embodiment of the present invention is shown in FIG. Figure 3 As shown, the speech repair device 300 includes: The alignment module 11 is used to align the coding features of the speech to be repaired obtained at the current moment with the target text features to obtain the target features; the target text features are the text features corresponding to the speech at the previous moment output by the decoder of the speech recognition model; A determination module 12, configured to determine the posterior probability of each text corresponding to the speech to be repaired based on the target feature; The determination module 12 is further configured to determine a text recognition result of the speech to be repaired based on each of the posterior probabilities; An input module 13 is used to input the target information into the text repair module of the speech repair model to obtain the first potential feature output by the text repair module, wherein the first potential feature includes the text potential feature, and the target information includes each of the posterior probabilities and the text recognition result; The repair module 14 is used to repair the speech to be repaired based on the text potential features.

[0084] In an exemplary embodiment, the repair module 14 is specifically configured to: Inputting the first latent feature into a prosody restoration module of the speech restoration model to obtain a prosody latent feature output by the prosody restoration module; The speech to be repaired is repaired based on the text latent features and the prosody latent features.

[0085] In an exemplary embodiment, the target information further includes a timbre feature corresponding to the speech to be repaired, and the first latent feature further includes a timbre latent feature; The repair module 14 is specifically used for: Inputting the text latent feature, the timbre latent feature and the rhythm latent feature into the acoustic detail restoration module of the speech restoration model to obtain the acoustic detail latent feature output by the acoustic detail restoration module; The speech to be repaired is repaired based on the text latent features, the prosody latent features and the acoustic detail latent features.

[0086] In an exemplary embodiment, the repair module 14 is specifically configured to: Performing weighted summation on the text latent feature, the rhythmic latent feature and the acoustic detail latent feature to obtain a weighted latent feature; The weighted latent features are input into a vocoder to obtain the restored speech output by the vocoder.

[0087] In an exemplary embodiment, the alignment module 11 is specifically configured to: Inputting the speech features corresponding to the speech to be repaired into the encoder of the speech recognition model to obtain the encoding features output by the encoder; Aligning the encoding feature with the target text feature based on blank to obtain the target feature; The determination module 12 is specifically used for: Inputting the target feature into the joint network of the speech recognition model to obtain the posterior probabilities output by the joint network; The speech recognition model is trained based on normal speech samples and speech samples to be repaired.

[0088] In an exemplary embodiment, the speech restoration model is trained based on the following method: Inputting a normal speech sample into a factor vector quantization codebook module to obtain a text codebook, a prosody codebook and an acoustic detail codebook output by the factor vector quantization codebook module; Inputting the normal speech sample into an initial speech repair model, obtaining predicted text potential features output by an initial text repair module of the initial speech repair model, predicted prosody potential features output by an initial prosody repair module of the initial speech repair model, and predicted acoustic detail potential features output by an initial acoustic detail repair module of the initial speech repair model; Determining loss information based on the text codebook, the prosody codebook, the acoustic detail codebook, the predicted text latent features, the predicted prosody latent features, and the predicted acoustic detail latent features; The parameters of the initial text repair module, the initial prosody repair module, and the initial acoustic detail repair module are adjusted based on the loss information to obtain the speech repair model.

[0089] In an example embodiment, the text repair module includes an autoregressive model, and the prosody repair module and the acoustic detail repair module include non-autoregressive models.

[0090] The device of this embodiment can be used to execute the method of any embodiment in the embodiment of the voice repair method side. Its specific implementation process and technical effects are similar to those in the embodiment of the voice repair method side. For details, please refer to the detailed description in the embodiment of the voice repair method side, which will not be repeated here.

[0091] Figure 4 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the speech repair method, which includes: aligning the encoding features of the speech to be repaired obtained at the current moment with the target text features to obtain the target features; the target text features are the text features corresponding to the speech at the previous moment output by the decoder of the speech recognition model; based on the target features, determining the posterior probabilities of each text corresponding to the speech to be repaired; based on each of the posterior probabilities, determining the text recognition result of the speech to be repaired; inputting the target information into the text repair module of the speech repair model to obtain the first potential feature output by the text repair module, the first potential feature includes the text potential feature, the target information includes each of the posterior probabilities and the text recognition result; based on the text potential feature, repairing the speech to be repaired.

[0092] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0093] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the speech repair method provided by the above-mentioned methods, which includes: aligning the encoding features of the speech to be repaired obtained at the current moment with the target text features to obtain the target features; the target text features are the text features corresponding to the speech at the previous moment output by the decoder of the speech recognition model; based on the target features, determining the posterior probabilities of each text corresponding to the speech to be repaired; based on each of the posterior probabilities, determining the text recognition result of the speech to be repaired; inputting the target information into the text repair module of the speech repair model to obtain the first potential feature output by the text repair module, the first potential feature including the text potential feature, the target information including each of the posterior probabilities and the text recognition result; based on the text potential feature, repairing the speech to be repaired.

[0094] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the speech repair method provided by the above-mentioned methods, the method comprising: aligning the encoding features of the speech to be repaired obtained at the current moment with the target text features to obtain the target features; the target text features are the text features corresponding to the speech at the previous moment output by the decoder of the speech recognition model; based on the target features, determining the posterior probabilities of each text corresponding to the speech to be repaired; based on each of the posterior probabilities, determining the text recognition result of the speech to be repaired; inputting the target information into the text repair module of the speech repair model to obtain the first potential feature output by the text repair module, the first potential feature including the text potential feature, the target information including each of the posterior probabilities and the text recognition result; based on the text potential feature, repairing the speech to be repaired.

[0095] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0096] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for voice repair, It is characterized in that include: Align the encoding features of the speech to be repaired obtained at the current moment with the target text features to obtain the target features; The target text feature is the text feature corresponding to the speech at the previous moment output by the decoder of the speech recognition model; Based on the target feature, determining the posterior probability of each text corresponding to the speech to be repaired; Based on each of the posterior probabilities, determining a text recognition result of the speech to be repaired; Inputting the target information into a text repair module of a speech repair model to obtain a first potential feature output by the text repair module, wherein the first potential feature includes a text potential feature, and the target information includes each of the posterior probabilities and the text recognition result; The speech to be repaired is repaired based on the text potential features.

2. The voice repair method according to claim 1, It is characterized in that The step of repairing the speech to be repaired based on the text potential features includes: Inputting the first latent feature into a prosody restoration module of the speech restoration model to obtain a prosody latent feature output by the prosody restoration module; The speech to be repaired is repaired based on the text latent features and the prosody latent features.

3. The voice repair method according to claim 2, It is characterized in that The target information also includes the timbre feature corresponding to the speech to be repaired, and the first latent feature also includes the timbre latent feature; The step of repairing the speech to be repaired based on the text potential feature and the prosody potential feature includes: Inputting the text latent feature, the timbre latent feature and the rhythm latent feature into the acoustic detail restoration module of the speech restoration model to obtain the acoustic detail latent feature output by the acoustic detail restoration module; The speech to be repaired is repaired based on the text latent features, the prosody latent features and the acoustic detail latent features.

4. The voice repair method according to claim 3, It is characterized in that The repairing of the speech to be repaired based on the text potential feature, the prosody potential feature and the acoustic detail potential feature comprises: Performing weighted summation on the text latent feature, the rhythmic latent feature and the acoustic detail latent feature to obtain a weighted latent feature; The weighted latent features are input into a vocoder to obtain the restored speech output by the vocoder.

5. The voice repair method according to any one of claims 1 to 4, It is characterized in that The step of aligning the encoding features of the speech to be repaired obtained at the current moment with the target text features to obtain the target features includes: Inputting the speech features corresponding to the speech to be repaired into the encoder of the speech recognition model to obtain the encoding features output by the encoder; Aligning the encoding feature with the target text feature based on blank to obtain the target feature; The determining, based on the target feature, the posterior probability of each text corresponding to the speech to be repaired includes: Inputting the target feature into the joint network of the speech recognition model to obtain the posterior probabilities output by the joint network; The speech recognition model is trained based on normal speech samples and speech samples to be repaired.

6. The voice repair method according to any one of claims 1 to 4, It is characterized in that The speech restoration model is trained based on the following method: Inputting a normal speech sample into a factor vector quantization codebook module to obtain a text codebook, a prosody codebook and an acoustic detail codebook output by the factor vector quantization codebook module; Inputting the normal speech sample into an initial speech repair model, obtaining predicted text potential features output by an initial text repair module of the initial speech repair model, predicted prosody potential features output by an initial prosody repair module of the initial speech repair model, and predicted acoustic detail potential features output by an initial acoustic detail repair module of the initial speech repair model; Determining loss information based on the text codebook, the prosody codebook, the acoustic detail codebook, the predicted text latent features, the predicted prosody latent features, and the predicted acoustic detail latent features; The parameters of the initial text repair module, the initial prosody repair module, and the initial acoustic detail repair module are adjusted based on the loss information to obtain the speech repair model.

7. The voice repair method according to claim 3, It is characterized in that The text repair module includes an autoregressive model, and the prosody repair module and the acoustic detail repair module include non-autoregressive models.

8. A voice repair device, It is characterized in that include: An alignment module is used to align the encoding features of the speech to be repaired obtained at the current moment with the target text features to obtain the target features; The target text feature is the text feature corresponding to the speech at the previous moment output by the decoder of the speech recognition model; A determination module, used to determine the posterior probability of each text corresponding to the speech to be repaired based on the target feature; The determination module is further used to determine the text recognition result of the speech to be repaired based on each of the posterior probabilities; An input module, used for inputting target information into a text repair module of a speech repair model, to obtain a first potential feature output by the text repair module, wherein the first potential feature includes a text potential feature, and the target information includes each of the posterior probabilities and the text recognition result; A repair module is used to repair the speech to be repaired based on the potential features of the text.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the speech repair method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the speech restoration method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Voice synthesis method and device thereof, electronic equipment, storage medium and program product

    CN114005428A

  • Pathological voice repairing method based on voice conversion

    CN116312469A

  • Automatic audio processing and repairing data preprocessing method and system

    CN118072735A

  • Method for performing audio inpainting on a speech signal and apparatus for performing audio inpainting on a speech signal

    EP3113180A1

  • KR20240044865A

Cited By

  • Voice restoration method and device, electronic equipment and storage medium

    CN119741909A

  • Audio data processing method and device based on artificial intelligence and computer

    CN119993206A