Voice restoration method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of voice processing, and provides a voice restoration method and device, electronic equipment and a storage medium, and the method comprises the steps: determining a to-be-restored pathological voice, and recognizing a phoneme sequence of the pathological voice; based on a phoneme correction model, correcting the phoneme sequence to obtain a corrected text, the phoneme correction model being obtained by training based on a sample damaged phoneme sequence and a standard text corresponding to the sample damaged phoneme sequence; and performing speech synthesis on the corrected text to obtain a restored speech of the pathological speech. According to the voice restoration method and device, the electronic equipment and the storage medium provided by the embodiment of the invention, the tone quality is improved, and the semantic information in the pathological voice is further reserved and restored, so that the intelligibility and communication efficiency of the restored voice are comprehensively improved.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to a speech repair method, device, electronic equipment and storage medium. Background Art

[0002] Pathological speech repair is an important emerging direction in speech disorder research in recent years, aiming to improve the intelligibility and comprehensibility of pathological speech. Pathological speech is usually caused by lesions of the vocal organs or neurological diseases, and is commonly seen in vocal cord injuries, neurodegenerative diseases such as Parkinson's disease, or sequelae of stroke.

[0003] Traditional pathological speech quality improvement technology mainly relies on voice conversion methods to improve the naturalness of pathological speech and make it closer to the voice of healthy people. This method mainly changes the patient's timbre or voice characteristics to optimize the sound quality and clarity of the speech perceived by the audience to a certain extent, thereby improving the user's communication experience.

[0004] Existing speech conversion methods have some effect in improving sound quality, but still have significant defects in dealing with semantic damage in pathological speech. Summary of the invention

[0005] The present invention provides a speech repair method, device, electronic device and storage medium, which are used to solve the defects of the prior art in the problem of semantic damage of pathological speech.

[0006] The present invention provides a voice repair method, comprising: Determining a pathological speech to be repaired, and identifying a phoneme sequence of the pathological speech; Based on a phoneme correction model, the phoneme sequence is corrected to obtain a corrected text, wherein the phoneme correction model is trained based on a sample damaged phoneme sequence and its corresponding standard text; The corrected text is subjected to speech synthesis to obtain a repaired speech of the pathological speech.

[0007] According to the speech repair method provided by the present invention, the phoneme sequence is corrected based on the phoneme correction model to obtain the corrected text, including: Based on the phoneme correction model, applying the semantic correlation between the phonemes in the phoneme sequence, context encoding the phoneme sequence to obtain phoneme context features; The phoneme context feature is decoded to obtain the corrected text.

[0008] According to the speech repair method provided by the present invention, decoding the phoneme context feature to obtain the corrected text includes: Based on the positional relationship between the phonemes in the phoneme sequence, position encoding is performed on the phoneme sequence to obtain positional features of each phoneme; Decoding is performed based on the phoneme context features and the position features of each phoneme to obtain the corrected text.

[0009] According to the speech repair method provided by the present invention, the speech synthesis is performed on the corrected text to obtain the repaired speech of the pathological speech, including: Based on the speech synthesis model, extract the speaker characteristics of the pathological speech, and based on the speaker characteristics, perform speech synthesis on the corrected text to obtain the repaired speech; The speech synthesis model is trained based on sample noisy speech and sample clean speech of the same speaker, as well as sample text corresponding to the sample noisy speech.

[0010] According to the speech repair method provided by the present invention, the training step of the speech synthesis model includes: Extracting speaker features of the sample noisy speech; Performing speech synthesis based on the speaker characteristics and the sample text corresponding to the sample noisy speech to obtain predicted synthesized speech; Based on the difference between the predicted synthesized speech and the sample clean speech, the parameters of the pre-trained initial model are iterated to obtain the speech synthesis model.

[0011] According to the speech repair method provided by the present invention, the identifying the phoneme sequence of the pathological speech comprises: Based on a trained phoneme recognition model, the phoneme sequence of the pathological speech is recognized, wherein the phoneme recognition model is based on a pre-trained speech model and a phoneme classification layer, and is trained using sample speech data and its corresponding annotated phoneme sequence.

[0012] According to the speech repair method provided by the present invention, the training step of the phoneme recognition model includes: Based on the feature extraction layer of the pre-trained speech model, extract the acoustic features of the sample speech data, and based on the phoneme classification layer, perform phoneme recognition on the acoustic features to obtain a predicted phoneme sequence; Freezing the feature extraction layer of the pre-trained speech model, and performing a first-stage training on the phoneme classification layer based on the difference between the predicted phoneme sequence and the annotated phoneme sequence; The feature extraction layer of the pre-trained speech model is unfrozen, and based on the difference between the predicted phoneme sequence and the annotated phoneme sequence, the pre-trained speech model and the phoneme classification layer are subjected to a second phase of joint training, and the trained pre-trained speech model and phoneme classification layer are determined to be the phoneme recognition model.

[0013] The present invention also provides a speech repair device, comprising: A phoneme recognition unit, used to determine the pathological speech to be repaired and recognize the phoneme sequence of the pathological speech; A phoneme correction unit, configured to correct the phoneme sequence based on a phoneme correction model to obtain a corrected text, wherein the phoneme correction model is trained based on a sample damaged phoneme sequence and its corresponding standard text; The speech synthesis unit is used to perform speech synthesis on the correction text to obtain the repaired speech of the pathological speech.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned speech repair methods when executing the computer program.

[0015] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the speech repair method described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the speech repair method described above is implemented.

[0017] The speech repair method, device, electronic device and storage medium provided by the present invention can identify the phoneme sequence of pathological speech, which is usually a damaged phoneme sequence with damaged semantics due to the loss of content information and the integrity of the sentence. The phoneme correction model can correct the damaged phoneme sequence to obtain a corrected text containing complete semantic information, and then perform speech synthesis on the corrected text to obtain the repaired speech of the pathological speech. Through a more comprehensive pathological speech repair technology, not only the sound quality is improved, but also the semantic information in the pathological speech is further retained and repaired, thereby comprehensively improving the intelligibility and communication efficiency of the repaired speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 This is one of the flow charts of the voice repair method provided by the present invention.

[0020] Figure 2A schematic flow chart of an implementation of step 120 in the voice repair method provided by the present invention.

[0021] Figure 3 It is a schematic diagram of the training steps of the speech synthesis model provided by the present invention.

[0022] Figure 4 It is a schematic diagram of the training steps of the phoneme recognition model provided by the present invention.

[0023] Figure 5 This is the second flow chart of the voice repair method provided by the present invention.

[0024] Figure 6 It is a structural schematic diagram of the speech repair device provided by the present invention.

[0025] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] In the related art, speech conversion methods for improving the quality of pathological speech include the application of timbre conversion algorithms, frequency feature adjustment, and speech enhancement technology. These methods usually map the pathological speech to the target timbre space, and try to remove the roughness, ambiguity and other problems in the pathological speech to generate a more fluent and natural speech effect. This type of solution has achieved certain success in speech generation, especially in terms of improving sound quality, reducing noise, and improving speech clarity.

[0028] Although existing speech conversion methods have a certain effect on improving sound quality, they still have significant defects in dealing with the problem of semantic damage in pathological speech. The problem of pathological speech is not only manifested in abnormal timbre, but also involves the loss of content information and the damage to the integrity of the sentence. When this type of speech is generated, the effectiveness of communication is often reduced due to the lack of repair at the semantic level. Specifically, many syllables and words in pathological speech may be inaccurately pronounced or missing, making it difficult for the audience to understand the full meaning of the sentence, and the existing timbre conversion technology is powerless to do this. This limitation makes it difficult to achieve the expected communication effect in actual communication even after the pathological speech is converted.

[0029] In view of the above problems, an embodiment of the present invention proposes a speech repair method, which aims to solve the semantic repair problem neglected in the existing methods. In this method, the phoneme sequence of the pathological speech is first identified. The phoneme sequence of the pathological speech is usually a damaged phoneme sequence with damaged semantics due to the loss of content information and the damage of sentence integrity. The phoneme correction model can correct the damaged phoneme sequence to obtain a corrected text containing complete semantic information, and then perform speech synthesis on the corrected text to obtain the repaired speech of the pathological speech. Through a more comprehensive pathological speech repair technology, not only the sound quality is improved, but also the semantic information in the pathological speech is further retained and repaired, thereby comprehensively improving the intelligibility and communication efficiency of the repaired speech.

[0030] The embodiments of the present invention can be applied to scenarios where voice repair is required, such as repairing pathological voices caused by lesions of vocal organs or nervous system diseases. The execution subject of the method can be an electronic device such as a terminal device, a computer, a server, a server cluster, or a specially designed voice repair device, or a voice repair device set in the electronic device, which can be implemented by software, hardware, or a combination of the two.

[0031] Figure 1 It is one of the flow charts of the voice repair method provided by the present invention, such as Figure 1 As shown, the method comprises the following steps: Step 110: determine the pathological speech to be repaired and identify the phoneme sequence of the pathological speech.

[0032] Specifically, the pathological voice to be repaired is the pathological voice that needs voice repair. The pathological voice can be obtained from the patient or a relevant medical institution, and the pathological voice can be a recording file or a file in other audio formats.

[0033] Phonemes are the smallest units of pronunciation in speech, which represent a specific sound in speech. For example, the English word "cat" contains three phonemes: / k / , / æ / , / t / . The Chinese word "sheng" consists of two phonemes, namely the initial consonant "sh" and the final vowel "eng".

[0034] Then, phoneme recognition technology, such as an automatic speech recognition system, is used to convert the pathological speech into a phoneme sequence. The acoustic features of the pathological speech can be extracted first, such as Mel-frequency cepstral coefficients (MFCC), linear prediction cepstral coefficients (LPCC), etc. Acoustic features can reflect the spectral information and dynamic characteristics of speech. The extracted acoustic features are input into the phoneme recognition tool to recognize the phoneme sequence, and the language model and the context information of the phoneme sequence can be combined to improve the recognition accuracy.

[0035] It is understandable that in pathological speech, due to the pathology or dysfunction of the vocal organs, the pronunciation of phonemes may be affected, resulting in differences between the phoneme sequence and normal speech. For example, the phoneme sequence of pathological speech may have characteristics such as abnormal pronunciation, phoneme substitution, phoneme omission, phoneme addition, and intonation change, that is, the phoneme sequence identified in this step is a damaged phoneme sequence.

[0036] Step 120, based on the phoneme correction model, the phoneme sequence is corrected to obtain a corrected text, and the phoneme correction model is trained based on the sample damaged phoneme sequence and its corresponding standard text.

[0037] Specifically, in order to repair the loss of content information and the integrity of sentences in the pathological speech, after obtaining the phoneme sequence of the pathological speech, the damaged phoneme sequence can be corrected to obtain a corrected text.

[0038] Phoneme sequence correction can be achieved through a phoneme correction model. The phoneme correction model is a machine learning model designed specifically to correct phoneme sequence errors in pathological or impaired speech. It identifies and corrects incorrect phonemes in pathological speech by learning the mapping relationship between a large number of sample impaired phoneme sequences and their corresponding standard texts.

[0039] The phoneme sequence of pathological speech is input into a trained phoneme correction model, which corrects the phoneme sequence to obtain a corrected phoneme sequence. The corrected phoneme sequence should be closer to the standard text, and then the corrected phoneme sequence is converted into text form to obtain and output the corrected text.

[0040] Before executing step 120, a large amount of sample pathological speech data containing damaged phonemes can be collected to ensure that the data covers different types of speech disorders and diverse degrees of damage. Phoneme recognition is performed for each segment of sample pathological speech, and the sample pathological speech is converted into a damaged phoneme sequence, i.e., a sample damaged phoneme sequence. At the same time, the corresponding normal text, i.e., the standard text, is obtained for each segment of sample pathological speech. The sample damaged phoneme sequence and the standard text are aligned to establish a one-to-one correspondence, providing input and target output for model training.

[0041] The phoneme correction model is based on the phoneme-to-text converter structure. A phoneme-to-text converter consisting of a multi-layer Transformer module is constructed, where the encoder and decoder are multi-layer Transformer modules of the same structure, and the hyperparameters of the model are set, such as the embedding dimension, the number of attention heads, the feedforward network dimension, and the dropout rate.

[0042] The model is iteratively trained using the training set to minimize the loss function. During the training process, the model learns how to predict the correct standard text based on the damaged phoneme sequence. The performance of the model is evaluated on the validation set, and indicators such as character error rate (CER) and word error rate (WER) are calculated. Based on the validation results, the model structure and hyperparameters are adjusted, and the training process is repeated to improve performance. The trained model is saved as a phoneme correction model for use in the phoneme correction task.

[0043] Step 130, performing speech synthesis on the corrected text to obtain a repaired speech of the pathological speech.

[0044] Specifically, the corrected text is a text representation that is closer to the standard text after being processed by the phoneme correction model. The corrected text corrects the phoneme errors in the original pathological speech, making the text content more accurate and clear.

[0045] Speech synthesis for corrected text refers to converting the corrected text into repaired speech through speech synthesis technology, that is, a repaired version of pathological speech that sounds more natural and closer to normal speech. Speech synthesis for corrected text can be based on a text-to-speech (TTS) system or a specially designed pathological speech repair system to ensure that the generated repaired speech meets the expected quality standards.

[0046] The parameters of the speech synthesis system, such as speech rate, intonation, volume, etc., can be pre-set or pre-trained using specific features of pathological speech to further improve the quality of the repaired speech.

[0047] The corrected text is input into a configured speech synthesis system for speech synthesis to obtain the synthesized speech, i.e., the repaired speech of the pathological speech.

[0048] The method provided by the embodiment of the present invention can identify the phoneme sequence of pathological speech. The phoneme sequence of pathological speech is usually a damaged phoneme sequence with damaged semantics due to the loss of content information and the damage of sentence integrity. The phoneme correction model can correct the damaged phoneme sequence to obtain a corrected text containing complete semantic information, and then perform speech synthesis on the corrected text to obtain the repaired speech of the pathological speech. Through a more comprehensive pathological speech repair technology, not only the sound quality is improved, but also the semantic information in the pathological speech is further retained and repaired, thereby comprehensively improving the intelligibility and communication efficiency of the repaired speech.

[0049] Based on any of the above embodiments, Figure 2 The flowchart of the implementation of step 120 in the voice repair method provided by the present invention is as follows: Figure 2 As shown, in step 120, based on the phoneme correction model, the phoneme sequence is corrected to obtain a corrected text, which specifically includes: Step 121, based on the phoneme correction model, applying the semantic correlation between the phonemes in the phoneme sequence, context encoding the phoneme sequence to obtain the phoneme context feature; Step 122, decode the phoneme context features to obtain corrected text.

[0050] Specifically, the phoneme correction model can be used to identify and correct damaged phoneme sequences in pathological speech based on an encoder-decoder structure. The phoneme correction model predicts and corrects incorrect phonemes by analyzing the contextual information of the phoneme sequence and using the semantic correlation between phonemes. The phoneme sequence is a series of phonemes identified from pathological speech, which are arranged in a specific order to form the acoustic representation of pathological speech.

[0051] The phoneme correction model contains an encoder for context encoding of phoneme sequences. The encoder considers the context information of each phoneme in the phoneme sequence, that is, the semantic relevance between them and other phonemes. This relevance can be reflected by the weights or feature representations learned by the model. The phoneme context feature output by the encoder is a feature representation that includes the semantic relevance and context information between phonemes. It helps the model to more accurately identify and correct incorrect phonemes.

[0052] The phoneme correction model also contains a decoder to decode the phoneme context features. Decoding refers to the process of converting phoneme context features into text. The decoder predicts the most likely phoneme sequence, i.e. the corrected phoneme sequence, based on the information in the feature representation. Then, the corrected phoneme sequence is converted into text to obtain the corrected text.

[0053] The corrected text obtained in this way is a text representation that is closer to the standard text after being processed by the phoneme correction model. It corrects the errors in the original phoneme sequence, making the text content more accurate and clear.

[0054] The method provided in the embodiment of the present invention analyzes the context information of the phoneme sequence through a phoneme correction model based on an encoder-decoder structure, and utilizes the semantic correlation between phonemes to predict and correct erroneous phonemes, thereby obtaining a more accurate corrected text, which can further improve the comprehensibility and semantic integrity of pathological speech.

[0055] Based on any of the above embodiments, decoding the phoneme context features in step 122 to obtain the corrected text specifically includes: Step 122-1, based on the positional relationship between the phonemes in the phoneme sequence, position encoding is performed on the phoneme sequence to obtain positional features of each phoneme; Step 122-2, decoding is performed based on the phoneme context features and the position features of each phoneme to obtain a corrected text.

[0056] Specifically, in order to further improve the accuracy of phoneme correction, not only the phoneme context features but also the position features of each phoneme are taken into account during decoding. The position feature of each phoneme refers to the feature obtained based on the positional relationship between each phoneme in the phoneme sequence. It reflects the relative position or absolute position of the phoneme in the sequence, which helps the model understand the sequential relationship between phonemes.

[0057] Position encoding of phoneme sequences can be achieved in a variety of ways, such as using position embeddings or position vectors. The purpose of position encoding is to assign a unique position feature to each phoneme in the sequence, which reflects the position information of the phoneme in the sequence.

[0058] The obtained positional features of each phoneme are a feature set corresponding to the phoneme sequence, wherein each feature contains the positional information of the corresponding phoneme.

[0059] Then, decoding is performed based on the phoneme context features and the position features of each phoneme to obtain the corrected text. The phoneme context features and the position features of each phoneme are used as input to the decoder. The decoder will comprehensively consider the context information and position information of the phonemes, use this information to predict the most likely phoneme sequence, and gradually convert the input features into textual output, namely the corrected text. The corrected text corrects the errors in the original phoneme sequence and retains the correct order and semantic relationship between phonemes.

[0060] During the decoding process, the phoneme context features and position features can be fused or combined to more accurately predict the phoneme sequence. Feature fusion can be achieved by concatenation, addition or other feature interaction methods.

[0061] The method provided in the embodiment of the present invention can further improve the accuracy of phoneme correction by decoding based on phoneme context features and position features of each phoneme to obtain a corrected text.

[0062] Preferably, the phoneme correction model may further include an embedding layer for constructing respective embedding layers for phonemes and texts, and converting discrete symbols into continuous vector representations.

[0063] Based on any of the above embodiments, step 130 performs speech synthesis on the corrected text to obtain the repaired speech of the pathological speech, which specifically includes: Step 131, based on the speech synthesis model, extract the speaker characteristics of the pathological speech, and based on the speaker characteristics, perform speech synthesis on the correction text to obtain the repaired speech; The speech synthesis model is trained based on sample noisy speech and sample clean speech of the same speaker, as well as sample text corresponding to the sample noisy speech.

[0064] Specifically, speech synthesis is performed on the corrected text to obtain the repaired speech of the pathological speech, which can be achieved through a trained speech synthesis model. The speech synthesis model extracts speaker features from the pathological speech. This can be achieved by using speaker recognition technology or speech feature extraction algorithms. The extracted features should be able to reflect the identity and speech characteristics of the speaker. Speaker features include but are not limited to pitch, speech rate, timbre, etc. In speech synthesis, the use of speaker features can ensure that the generated speech is consistent with the original speaker's voice.

[0065] The speech synthesis model then generates a speech waveform corresponding to the corrected text based on the input corrected text and the extracted speaker features, while maintaining consistency with the original speaker's voice. The clean speech generated by the speech synthesis model that corresponds to the corrected text is the repaired speech, which can clearly convey the text content and sound like the original speaker's voice.

[0066] Before executing step 131, the speech synthesis model has been trained using sample noisy speech, sample clean speech and corresponding sample text of the same speaker. During the training process, the model learns how to extract useful speech information from noisy speech and generate clean speech based on the text content.

[0067] Collect a large amount of clean speech data, covering a variety of speakers, genders, ages, and accents to ensure data diversity as sample clean speech.

[0068] Sample noisy speech refers to speech samples that contain noise. Such noise may come from factors such as the environment, equipment, or abnormal vocal organs. In this scenario, sample noisy speech is used to train the speech synthesis model so that it can adapt to and reduce the impact of noise on speech synthesis.

[0069] Collect noise data of hoarseness caused by abnormal vocal organs, covering pathological characteristics of different degrees and types; mix clean speech data with noise of different types and intensities to generate more noisy speech data as sample noisy speech. Here, the sample clean speech and the sample noisy speech belong to the same speaker.

[0070] The sample text corresponding to the sample noisy speech is the text content corresponding to the sample noisy speech. When training a speech synthesis model, the sample text is used to guide the model how to convert text into speech.

[0071] By using speech with different noises (sample noisy speech) and clean speaker speech (sample clean speech) for training, the part of the speaker representation extracted has the ability to reduce noise, which can offset the hoarse noise caused by abnormal vocal organs in pathological speech.

[0072] The method provided in the embodiment of the present invention generates high-quality repaired speech using a speech synthesis model based on the corrected text and speaker characteristics of the pathological speech, which can further improve the audibility and comprehensibility of the pathological speech.

[0073] Based on any of the above embodiments, Figure 3 Schematic diagram of the training steps of the speech synthesis model provided by the present invention, such as Figure 3 As shown, the training steps of the speech synthesis model include: Step 310, extracting speaker features of the sample noisy speech; Step 320, performing speech synthesis based on speaker characteristics and sample text corresponding to the sample noisy speech to obtain predicted synthesized speech; Step 330, based on the difference between the predicted synthesized speech and the sample clean speech, iterate the parameters of the pre-trained initial model to obtain a speech synthesis model.

[0074] Specifically, the pre-trained initial model may be an initial speech synthesis model that has been pre-trained with a large amount of clean speech data, and may achieve zero-sample speech synthesis.

[0075] Preprocess the sample noisy speech, including noise reduction and echo removal, to improve the accuracy of subsequent feature extraction. Use professional audio editing software or algorithms to extract speaker features from the preprocessed noisy speech. Speaker features can include the speaker's age, gender, timbre, speaking speed, etc. For speaker feature extraction, spectrum analysis tools can be used to observe the spectrum distribution of the audio in order to more accurately locate the human voice signal. By inverting and superimposing the human voice and accompaniment in the left and right channels, the accompaniment is eliminated to extract the pure human voice and then analyze the speaker features. In addition, a bandpass filter or a notch filter can be applied to further purify the human voice signal and remove noise and interference.

[0076] Analyze the sample text corresponding to the sample noisy speech, including text content, grammatical structure, vocabulary selection, etc. Map the speaker characteristics with the sample text to generate a speech synthesis model input containing the speaker characteristics. Use a speech synthesis algorithm (such as speech synthesis technology based on hidden Markov model (HMM) and speech synthesis technology based on deep learning) to convert the mapped input into a speech signal to obtain the predicted synthesized speech.

[0077] The difference between the predicted synthesized speech and the sample clean speech is calculated using speech quality assessment indicators (such as speech similarity, clarity, naturalness, fluency, etc.). Based on the difference calculation results, the parameters of the pre-trained initial model are adjusted to improve the accuracy of speech synthesis. The above difference calculation and parameter adjustment process is repeated until the difference between the predicted synthesized speech and the sample clean speech reaches a preset threshold or the number of iterations reaches a preset upper limit, at which time the final speech synthesis model is obtained.

[0078] The noisy speech data is used as training data, and the target speech is the corresponding clean speech, which is used to train the noise reduction capability of the model and extract speaker representation. The speech synthesis model trained based on the method of the embodiment of the present invention can extract robust speaker features from the noisy speech.

[0079] Preferably, a noise reduction mechanism can also be integrated into the speech synthesis model, such as introducing a denoising autoencoder to enhance the noise reduction capability.

[0080] Based on any of the above embodiments, the step 110 of identifying the phoneme sequence of the pathological speech includes: Step 111, based on the trained phoneme recognition model, the phoneme sequence of the pathological speech is recognized, the phoneme recognition model is based on the pre-trained speech model and the phoneme classification layer, and is trained using the sample speech data and its corresponding annotated phoneme sequence.

[0081] Specifically, the recognition of the phoneme sequence of pathological speech can be achieved through a trained phoneme recognition model. The phoneme recognition model is built based on the pre-trained speech model and the phoneme classification layer. The pre-trained speech model has been trained on a large amount of speech data and can capture the basic features of the speech signal, such as spectrum, pitch, and intensity. These features provide the basis for subsequent phoneme classification.

[0082] The phoneme classification layer is responsible for mapping the features extracted by the pre-trained speech model to specific phoneme categories. The phoneme classification layer can be implemented using structures such as a fully connected network, a convolutional neural network (CNN), or a recurrent neural network (RNN).

[0083] Obtain a large amount of diverse voice data, including recordings in different languages, accents, and pronunciations, and perform noise reduction, silence removal, and normalization on the collected voice data to improve data quality, thereby obtaining sample voice data.

[0084] Use professional annotation tools to annotate the preprocessed sample speech data at the phoneme level to generate the corresponding phoneme sequence, i.e., the annotated phoneme sequence.

[0085] During the training process, the model learns the mapping relationship between the sample speech data and its annotated phoneme sequence, and iterates the parameters multiple times until the performance of the model on the validation set reaches a stable state or the preset number of iterations reaches an upper limit. In this way, an accurate phoneme recognition model can be trained, which can convert the input speech signal into the corresponding phoneme sequence, providing a basis for subsequent phoneme correction and speech synthesis tasks.

[0086] Based on any of the above embodiments, Figure 4 Schematic diagram of the training steps of the phoneme recognition model provided by the present invention, such as Figure 4 As shown, the training steps of the phoneme recognition model include: Step 410, extracting acoustic features of the sample speech data based on the feature extraction layer of the pre-trained speech model, and performing phoneme recognition on the acoustic features based on the phoneme classification layer to obtain a predicted phoneme sequence; Step 420, freezing the feature extraction layer of the pre-trained speech model, and performing the first phase training on the phoneme classification layer based on the difference between the predicted phoneme sequence and the annotated phoneme sequence; Step 430, unfreeze the feature extraction layer of the pre-trained speech model, perform a second phase of joint training on the pre-trained speech model and the phoneme classification layer based on the difference between the predicted phoneme sequence and the annotated phoneme sequence, and determine that the trained pre-trained speech model and phoneme classification layer are a phoneme recognition model.

[0087] Specifically, select a self-supervised pre-trained speech model. This type of model is trained on large-scale unlabeled speech data and can learn rich speech feature representations. Use a deep learning framework to load the pre-trained self-supervised model, including its parameters and network structure. Add one or more fully connected layers on top of the pre-trained speech model as a phoneme classification layer. The number of neurons in the last layer is equal to the total number of phoneme categories, and the activation function can select Softmax to output the probability distribution of each phoneme.

[0088] The feature extraction layer of the pre-trained speech model is used to process the sample speech data and extract the acoustic features of the speech signal. These features usually include spectral features, time domain features, etc., which can reflect the basic properties and structure of the speech signal.

[0089] The extracted acoustic features are input into the phoneme classification layer for phoneme recognition. The phoneme classification layer outputs a predicted phoneme sequence based on the input features and pre-trained weights. In this embodiment, a segmented training strategy is adopted.

[0090] First stage training: Freeze the feature extraction layer of the pre-trained speech model, that is, do not update its parameters. Doing so can maintain the stability of the feature extraction layer and avoid introducing too much noise during training. Based on the difference between the predicted phoneme sequence and the annotated phoneme sequence, use an appropriate loss function (such as cross entropy loss) to train the phoneme classification layer. Update the parameters of the phoneme classification layer through the back propagation algorithm to minimize the loss function. The purpose of the first stage training is to enable the phoneme classification layer to gradually adapt to the input acoustic features and initially have the ability to recognize phonemes.

[0091] Second stage training: Unfreeze the feature extraction layer of the pre-trained speech model, allowing its parameters to be updated during the training process. Still based on the difference between the predicted phoneme sequence and the annotated phoneme sequence, the pre-trained speech model and the phoneme classification layer are jointly trained. At this point, the entire model (including the feature extraction layer and the phoneme classification layer) will be updated according to the gradient of the loss function. The purpose of the second stage training is to further optimize the performance of the entire model, so that the feature extraction layer can better extract acoustic features related to phoneme recognition, while enabling the phoneme classification layer to more accurately recognize phonemes.

[0092] Evaluate the model performance on the validation set and calculate indicators such as accuracy and recall to ensure the effectiveness of the model. According to the validation results, adjust the model structure and hyperparameters and repeat the training process to improve performance. Save the trained model as a phoneme recognition model for subsequent phoneme recognition tasks.

[0093] The method provided by the embodiment of the present invention can train an accurate phoneme recognition model, which can extract the acoustic features of sample speech data using the feature extraction layer of the pre-trained speech model and perform phoneme recognition through the phoneme classification layer. Through staged training, the performance of the model is gradually optimized to have stronger generalization ability and accuracy.

[0094] Based on any of the above embodiments, Figure 5 FIG. 2 is a flow chart of the voice repair method provided by the present invention. Figure 5 As shown, the method includes: S1, determining the pathological speech to be repaired.

[0095] S2, based on the trained phoneme recognition model, recognize the phoneme sequence of pathological speech. The phoneme recognition model is based on the pre-trained speech model and the phoneme classification layer, and is trained using sample speech data and its corresponding annotated phoneme sequence. The training steps of the phoneme recognition model include: Based on the feature extraction layer of the pre-trained speech model, the acoustic features of the sample speech data are extracted, and based on the phoneme classification layer, phoneme recognition is performed on the acoustic features to obtain a predicted phoneme sequence; the feature extraction layer of the pre-trained speech model is frozen, and based on the difference between the predicted phoneme sequence and the annotated phoneme sequence, the phoneme classification layer is trained in the first stage; the feature extraction layer of the pre-trained speech model is unfrozen, and based on the difference between the predicted phoneme sequence and the annotated phoneme sequence, the pre-trained speech model and the phoneme classification layer are jointly trained in the second stage, and the pre-trained speech model and the phoneme classification layer that have been trained are determined to be a phoneme recognition model.

[0096] S3, based on the phoneme correction model, correct the phoneme sequence to obtain the corrected text. The phoneme correction model is trained based on the sample damaged phoneme sequence and its corresponding standard text. Specifically includes: Based on the phoneme correction model, the semantic correlation between the phonemes in the phoneme sequence is used to contextually encode the phoneme sequence to obtain the phoneme context features; based on the positional relationship between the phonemes in the phoneme sequence, the phoneme sequence is positionally encoded to obtain the positional features of each phoneme; decoding is performed based on the phoneme context features and the positional features of each phoneme to obtain the corrected text.

[0097] S4, based on the speech synthesis model, extract the speaker features of the pathological speech, and based on the speaker features, perform speech synthesis on the correction text to obtain the repaired speech. The speech synthesis model is trained based on the sample noisy speech and sample clean speech of the same speaker, as well as the sample text corresponding to the sample noisy speech. The training steps of the speech synthesis model specifically include: The speaker features of the sample noisy speech are extracted; based on the speaker features and the sample text corresponding to the sample noisy speech, speech synthesis is performed to obtain predicted synthesized speech; based on the difference between the predicted synthesized speech and the sample clean speech, the parameters of the pre-trained initial model are iterated to obtain a speech synthesis model.

[0098] The speech repair device provided by the present invention is described below. The speech repair device described below and the speech repair method described above can be referred to each other.

[0099] Based on any of the above embodiments, Figure 6 is a schematic diagram of the structure of the speech repair device provided by the present invention, such as Figure 6 As shown, the voice repair device includes: The phoneme recognition unit 610 is used to determine the pathological speech to be repaired and recognize the phoneme sequence of the pathological speech; A phoneme correction unit 620, configured to correct the phoneme sequence based on a phoneme correction model to obtain a corrected text, wherein the phoneme correction model is trained based on a sample damaged phoneme sequence and its corresponding standard text; The speech synthesis unit 630 is used to perform speech synthesis on the correction text to obtain the repaired speech of the pathological speech.

[0100] The speech repair device provided by the embodiment of the present invention can identify the phoneme sequence of pathological speech. The phoneme sequence of pathological speech is usually a damaged phoneme sequence with damaged semantics due to the loss of content information and the damage of sentence integrity. The phoneme correction model can correct the damaged phoneme sequence to obtain a corrected text containing complete semantic information, and then perform speech synthesis on the corrected text to obtain the repaired speech of the pathological speech. Through a more comprehensive pathological speech repair technology, not only the sound quality is improved, but also the semantic information in the pathological speech is further retained and repaired, thereby comprehensively improving the intelligibility and communication efficiency of the repaired speech.

[0101] Based on any of the above embodiments, the phoneme correction unit is specifically used for: Based on the phoneme correction model, applying the semantic correlation between the phonemes in the phoneme sequence, context encoding the phoneme sequence to obtain phoneme context features; The phoneme context feature is decoded to obtain the corrected text.

[0102] Based on any of the above embodiments, the phoneme correction unit is specifically used for: Based on the positional relationship between the phonemes in the phoneme sequence, position encoding is performed on the phoneme sequence to obtain positional features of each phoneme; Decoding is performed based on the phoneme context features and the position features of each phoneme to obtain the corrected text.

[0103] Based on any of the above embodiments, the speech synthesis unit is specifically used for: Based on the speech synthesis model, extract the speaker characteristics of the pathological speech, and based on the speaker characteristics, perform speech synthesis on the corrected text to obtain the repaired speech; The speech synthesis model is trained based on sample noisy speech and sample clean speech of the same speaker, as well as sample text corresponding to the sample noisy speech.

[0104] Based on any of the above embodiments, it also includes a speech synthesis model training unit, which is used to: Extracting speaker features of the sample noisy speech; Performing speech synthesis based on the speaker characteristics and the sample text corresponding to the sample noisy speech to obtain predicted synthesized speech; Based on the difference between the predicted synthesized speech and the sample clean speech, the parameters of the pre-trained initial model are iterated to obtain the speech synthesis model.

[0105] Based on any of the above embodiments, the phoneme recognition unit is specifically used for: Based on a trained phoneme recognition model, the phoneme sequence of the pathological speech is recognized, wherein the phoneme recognition model is based on a pre-trained speech model and a phoneme classification layer, and is trained using sample speech data and its corresponding annotated phoneme sequence.

[0106] Based on any of the above embodiments, it also includes a phoneme recognition model training unit, which is used to: Based on the feature extraction layer of the pre-trained speech model, extract the acoustic features of the sample speech data, and based on the phoneme classification layer, perform phoneme recognition on the acoustic features to obtain a predicted phoneme sequence; Freezing the feature extraction layer of the pre-trained speech model, and performing a first-stage training on the phoneme classification layer based on the difference between the predicted phoneme sequence and the annotated phoneme sequence; The feature extraction layer of the pre-trained speech model is unfrozen, and based on the difference between the predicted phoneme sequence and the annotated phoneme sequence, the pre-trained speech model and the phoneme classification layer are subjected to a second phase of joint training, and the trained pre-trained speech model and phoneme classification layer are determined to be the phoneme recognition model.

[0107] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the speech repair method, which includes: determining the pathological speech to be repaired and identifying the phoneme sequence of the pathological speech; based on the phoneme correction model, correcting the phoneme sequence to obtain a corrected text, wherein the phoneme correction model is obtained by training based on the sample damaged phoneme sequence and its corresponding standard text; performing speech synthesis on the corrected text to obtain the repaired speech of the pathological speech.

[0108] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0109] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the speech repair method provided by the above-mentioned methods, which includes: determining the pathological speech to be repaired and identifying the phoneme sequence of the pathological speech; based on a phoneme correction model, correcting the phoneme sequence to obtain a corrected text, and the phoneme correction model is trained based on a sample damaged phoneme sequence and its corresponding standard text; performing speech synthesis on the corrected text to obtain the repaired speech of the pathological speech.

[0110] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the speech repair method provided by the above-mentioned methods, the method comprising: determining the pathological speech to be repaired, and identifying the phoneme sequence of the pathological speech; based on a phoneme correction model, correcting the phoneme sequence to obtain a corrected text, the phoneme correction model being trained based on a sample damaged phoneme sequence and its corresponding standard text; performing speech synthesis on the corrected text to obtain the repaired speech of the pathological speech.

[0111] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0112] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A voice repair method, characterized in that: include: Determining a pathological speech to be repaired, and identifying a phoneme sequence of the pathological speech; Based on a phoneme correction model, the phoneme sequence is corrected to obtain a corrected text, wherein the phoneme correction model is trained based on a sample damaged phoneme sequence and its corresponding standard text; The corrected text is subjected to speech synthesis to obtain a repaired speech of the pathological speech.

2. The voice repair method according to claim 1, characterized in that: The method of correcting the phoneme sequence based on the phoneme correction model to obtain a corrected text includes: Based on the phoneme correction model, applying the semantic correlation between the phonemes in the phoneme sequence, context encoding the phoneme sequence to obtain phoneme context features; The phoneme context feature is decoded to obtain the corrected text.

3. The voice repair method according to claim 2, characterized in that: The decoding of the phoneme context feature to obtain the corrected text includes: Based on the positional relationship between the phonemes in the phoneme sequence, position encoding is performed on the phoneme sequence to obtain positional features of each phoneme; Decoding is performed based on the phoneme context features and the position features of each phoneme to obtain the corrected text.

4. The method for voice restoration according to claim 1, characterized in that: The performing speech synthesis on the corrected text to obtain the repaired speech of the pathological speech includes: Based on the speech synthesis model, extract the speaker characteristics of the pathological speech, and based on the speaker characteristics, perform speech synthesis on the corrected text to obtain the repaired speech; The speech synthesis model is trained based on sample noisy speech and sample clean speech of the same speaker, as well as sample text corresponding to the sample noisy speech.

5. The method for voice restoration according to claim 4, characterized in that: The training steps of the speech synthesis model include: Extracting speaker features of the sample noisy speech; Performing speech synthesis based on the speaker characteristics and the sample text corresponding to the sample noisy speech to obtain predicted synthesized speech; Based on the difference between the predicted synthesized speech and the sample clean speech, the parameters of the pre-trained initial model are iterated to obtain the speech synthesis model.

6. The speech repair method according to any one of claims 1 to 5, characterized in that: The identifying of the phoneme sequence of the pathological speech comprises: Based on a trained phoneme recognition model, the phoneme sequence of the pathological speech is recognized, wherein the phoneme recognition model is based on a pre-trained speech model and a phoneme classification layer, and is trained using sample speech data and its corresponding annotated phoneme sequence.

7. The method for voice restoration according to claim 6, characterized in that: The training steps of the phoneme recognition model include: Based on the feature extraction layer of the pre-trained speech model, extract the acoustic features of the sample speech data, and based on the phoneme classification layer, perform phoneme recognition on the acoustic features to obtain a predicted phoneme sequence; Freezing the feature extraction layer of the pre-trained speech model, and performing a first-stage training on the phoneme classification layer based on the difference between the predicted phoneme sequence and the annotated phoneme sequence; The feature extraction layer of the pre-trained speech model is unfrozen, and based on the difference between the predicted phoneme sequence and the annotated phoneme sequence, the pre-trained speech model and the phoneme classification layer are subjected to a second phase of joint training, and the trained pre-trained speech model and phoneme classification layer are determined to be the phoneme recognition model.

8. A speech repair device, characterized in that: include: A phoneme recognition unit, used to determine the pathological speech to be repaired and recognize the phoneme sequence of the pathological speech; A phoneme correction unit, configured to correct the phoneme sequence based on a phoneme correction model to obtain a corrected text, wherein the phoneme correction model is trained based on a sample damaged phoneme sequence and its corresponding standard text; The speech synthesis unit is used to perform speech synthesis on the correction text to obtain the repaired speech of the pathological speech.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the speech repair method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speech restoration method according to any one of claims 1 to 7 is implemented.

Images