Speech repair method and apparatus, electronic device, and storage medium

Abstract

The present application relates to the technical field of speech processing, and provides a speech repair method and device, electronic equipment and storage medium, the method comprising: determining a pathological speech to be repaired, and identifying a phoneme sequence of the pathological speech; correcting the phoneme sequence based on a phoneme correction model to obtain corrected text, the phoneme correction model being trained based on sample damaged phoneme sequences and corresponding standard texts; and performing speech synthesis on the corrected text to obtain a repaired speech of the pathological speech. The speech repair method, device, electronic equipment and storage medium provided by the present application not only improve the sound quality, but also further retain and repair the semantic information in the pathological speech, thereby comprehensively improving the intelligibility and communication efficiency of the repaired speech.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and in particular to a speech restoration method, apparatus, electronic device, and storage medium. Background Technology

[0002] Pathological speech restoration is an important emerging direction in speech disorder research in recent years, aiming to improve the intelligibility and comprehensibility of speech by repairing pathological speech. Pathological speech is usually caused by lesions of the vocal organs or diseases of the nervous system, and is commonly seen in vocal cord damage, neurodegenerative diseases such as Parkinson's disease, or sequelae of stroke.

[0003] Traditional techniques for improving the quality of pathological speech primarily rely on speech conversion methods to enhance the naturalness of pathological speech, making it closer to the speech of healthy individuals. This method mainly works by altering the patient's timbre or vocal characteristics, optimizing the perceived sound quality and clarity to a certain extent, thereby improving the user's communication experience.

[0004] Existing speech conversion methods have some effect on improving sound quality, but they still have significant shortcomings in dealing with semantic damage in pathological speech. Summary of the Invention

[0005] This invention provides a speech restoration method, apparatus, electronic device, and storage medium to address the shortcomings of existing technologies in dealing with semantic damage to pathological speech.

[0006] This invention provides a voice restoration method, comprising:

[0007] Identify the pathological speech to be repaired and recognize the phoneme sequence of the pathological speech;

[0008] Based on the phoneme correction model, the phoneme sequence is corrected to obtain the corrected text. The phoneme correction model is trained based on the sample damaged phoneme sequence and its corresponding standard text.

[0009] The corrected text is processed by speech synthesis to obtain the restored speech of the pathological speech.

[0010] According to the speech restoration method provided by the present invention, the step of correcting the phoneme sequence based on the phoneme correction model to obtain the corrected text includes:

[0011] Based on the phoneme correction model, the semantic correlation between each phoneme in the phoneme sequence is applied to perform context encoding on the phoneme sequence to obtain phoneme context features;

[0012] The corrected text is obtained by decoding the phoneme context features.

[0013] According to the speech restoration method provided by the present invention, the step of decoding the phoneme context features to obtain the corrected text includes:

[0014] Based on the positional relationship between each phoneme in the phoneme sequence, the phoneme sequence is positionally encoded to obtain the positional features of each phoneme;

[0015] The corrected text is obtained by decoding based on the phoneme context features and the position features of each phoneme.

[0016] According to the speech restoration method provided by the present invention, the step of performing speech synthesis on the corrected text to obtain the restored speech of the pathological speech includes:

[0017] Based on the speech synthesis model, the speaker features of the pathological speech are extracted, and based on the speaker features, the corrected text is synthesized to obtain the restored speech.

[0018] The speech synthesis model is trained based on noisy and clean speech samples from the same speaker, as well as the sample text corresponding to the noisy speech samples.

[0019] According to the speech restoration method provided by the present invention, the training steps of the speech synthesis model include:

[0020] Extract speaker features from the noisy speech samples;

[0021] Based on the speaker features and the sample text corresponding to the noisy sample speech, speech synthesis is performed to obtain predicted synthesized speech;

[0022] Based on the difference between the predicted synthesized speech and the clean speech sample, the parameters of the pre-trained initial model are iterated to obtain the speech synthesis model.

[0023] According to the speech restoration method provided by the present invention, the step of recognizing the phoneme sequence of the pathological speech includes:

[0024] Based on a trained phoneme recognition model, the phoneme sequence of the pathological speech is identified. The phoneme recognition model is trained using sample speech data and its corresponding labeled phoneme sequences, based on a pre-trained speech model and a phoneme classification layer.

[0025] According to the speech restoration method provided by the present invention, the training steps of the phoneme recognition model include:

[0026] Based on the feature extraction layer of the pre-trained speech model, acoustic features of the sample speech data are extracted, and phoneme recognition is performed on the acoustic features based on the phoneme classification layer to obtain a predicted phoneme sequence.

[0027] Freeze the feature extraction layer of the pre-trained speech model, and perform the first stage training of the phoneme classification layer based on the difference between the predicted phoneme sequence and the labeled phoneme sequence;

[0028] Unfreeze the feature extraction layer of the pre-trained speech model, and based on the difference between the predicted phoneme sequence and the labeled phoneme sequence, perform a second stage of joint training on the pre-trained speech model and the phoneme classification layer to determine the pre-trained speech model and the phoneme classification layer as the phoneme recognition model.

[0029] The present invention also provides a voice restoration device, comprising:

[0030] A phoneme recognition unit is used to determine the pathological speech to be repaired and to identify the phoneme sequence of the pathological speech.

[0031] A phoneme correction unit is used to correct the phoneme sequence based on a phoneme correction model to obtain corrected text. The phoneme correction model is trained based on the sample damaged phoneme sequence and its corresponding standard text.

[0032] A speech synthesis unit is used to synthesize the corrected text into a restored speech of the pathological speech.

[0033] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described voice restoration methods.

[0034] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech restoration method as described above.

[0035] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described voice restoration methods.

[0036] The speech restoration method, apparatus, electronic device, and storage medium provided by this invention identify the phoneme sequence of pathological speech. Due to the loss of content information and impaired sentence integrity, the phoneme sequence of pathological speech is typically a semantically damaged phoneme sequence. A phoneme correction model can correct the damaged phoneme sequence to obtain corrected text containing complete semantic information. Subsequently, speech synthesis is performed on the corrected text to obtain restored speech of the pathological speech. Through a more comprehensive pathological speech restoration technology, not only is the sound quality improved, but the semantic information in the pathological speech is further preserved and restored, thereby comprehensively improving the intelligibility and communication efficiency of the restored speech. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1 This is one of the flowcharts of the voice restoration method provided by the present invention.

[0039] Figure 2 A flowchart illustrating the implementation of step 120 in the voice restoration method provided by this invention.

[0040] Figure 3 This is a schematic diagram of the training steps of the speech synthesis model provided by the present invention.

[0041] Figure 4 This is a schematic diagram of the training steps of the phoneme recognition model provided by the present invention.

[0042] Figure 5 This is the second flowchart of the voice restoration method provided by the present invention.

[0043] Figure 6 This is a schematic diagram of the voice restoration device provided by the present invention.

[0044] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0046] Among related technologies, speech conversion methods for improving the quality of pathological speech include the application of timbre conversion algorithms, frequency feature adjustment, and speech enhancement techniques. These methods typically map the pathological speech into the target timbre space, attempting to remove roughness and ambiguity in the pathological speech to generate a smoother and more natural speech effect. This type of approach has achieved some success in speech generation, particularly in improving sound quality, reducing noise, and enhancing speech clarity.

[0047] While existing speech-to-speech methods have shown some effectiveness in improving sound quality, they still have significant limitations in addressing the semantic impairments in pathological speech. The problems with pathological speech are not merely abnormal timbre, but also involve the loss of content information and impaired sentence integrity. During the generation of such speech, the lack of semantic restoration often reduces the effectiveness of communication. Specifically, many syllables and words in pathological speech may be pronounced inaccurately or missing, making it difficult for listeners to understand the full meaning of the sentences, a problem that existing timbre conversion technologies cannot address. This limitation means that even after conversion, pathological speech may still fail to achieve the desired communication effect in actual conversation.

[0048] To address the aforementioned problems, this invention proposes a speech restoration method aimed at resolving the semantic restoration issue neglected in existing methods. This method first identifies the phoneme sequence of pathological speech. Due to the loss of content information and impaired sentence integrity, the phoneme sequence of pathological speech is typically a semantically damaged phoneme sequence. A phoneme correction model can correct the damaged phoneme sequence to obtain corrected text containing complete semantic information. Subsequently, speech synthesis is performed on the corrected text to obtain the restored speech of the pathological speech. Through a more comprehensive pathological speech restoration technology, not only is the sound quality improved, but the semantic information in the pathological speech is further preserved and restored, thereby comprehensively improving the intelligibility and communication efficiency of the restored speech.

[0049] This invention can be applied to scenarios requiring speech restoration, such as the restoration of pathological speech caused by lesions of the vocal organs or diseases of the nervous system. The subject executing this method can be an electronic device such as a terminal device, computer, server, server cluster, or a specially designed speech restoration device, or a speech restoration device installed in such an electronic device. This speech restoration device can be implemented through software, hardware, or a combination of both.

[0050] Figure 1 This is one of the flowcharts illustrating the speech restoration method provided by the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0051] Step 110: Identify the pathological speech to be repaired and recognize the phoneme sequence of the pathological speech.

[0052] Specifically, the pathological speech to be restored refers to the pathological speech that requires speech restoration. Pathological speech can be obtained from the patient or relevant medical institutions, and can be audio recordings or other audio format files.

[0053] A phoneme is the smallest unit of pronunciation in speech, representing a specific sound in speech. For example, the English word "cat" contains three phonemes: / k / , / æ / , / t / . The Chinese character "声" includes two phonemes, namely the initial consonant "sh" and the final "eng".

[0054] Immediately, phoneme recognition technology, such as an automatic speech recognition system, is used to convert pathological speech into a phoneme sequence. First, the acoustic features of the pathological speech can be extracted, such as Mel Frequency Cepstral Coefficients (MFCC), Linear Prediction Cepstral Coefficients (LPCC), etc. The acoustic features can reflect the spectral information and dynamic characteristics of the speech. The extracted acoustic features are input into a phoneme recognition tool for phoneme sequence recognition, and the language model and context information of the phoneme sequence can be combined to improve the recognition accuracy.

[0055] It is understandable that in pathological speech, due to lesions or dysfunctions of the vocal organs, the pronunciation of phonemes may be affected, resulting in differences between the phoneme sequence and normal speech. For example, the phoneme sequence of pathological speech may have characteristics such as abnormal pronunciation, phoneme substitution, phoneme omission, phoneme addition, intonation changes, etc., that is, the phoneme sequence obtained in this step is a damaged phoneme sequence.

[0056] Step 120: Based on the phoneme correction model, the phoneme sequence is corrected to obtain a corrected text. The phoneme correction model is trained based on sample damaged phoneme sequences and their corresponding standard texts.

[0057] Specifically, in order to repair the loss of content information and the integrity of sentences in pathological speech, after obtaining the phoneme sequence of the pathological speech, the damaged phoneme sequence can be corrected to obtain a corrected text.

[0058] For phoneme sequence correction, it can be achieved through a phoneme correction model. The phoneme correction model is a machine learning model specifically designed to correct phoneme sequence errors in pathological speech or damaged speech. It learns the mapping relationship between a large number of sample damaged phoneme sequences and their corresponding standard texts to identify and correct the incorrect phonemes in pathological speech.

[0059] The phoneme sequence of the pathological speech is input into the trained phoneme correction model, and the phoneme correction model corrects the phoneme sequence to obtain a corrected phoneme sequence. The corrected phoneme sequence should be closer to the standard text, and then the corrected phoneme sequence is converted into text form to obtain and output the corrected text.

[0060] Before performing step 120, a large amount of sample pathological speech data containing damaged phonemes can be collected to ensure that the data covers different types of speech disorders and diverse degrees of impairment. Phoneme recognition is performed on each sample pathological speech, converting it into a sequence of damaged phonemes, i.e., the sample damaged phoneme sequence. Simultaneously, the corresponding normal text, i.e., standard text, is obtained for each sample pathological speech. The sample damaged phoneme sequence and the standard text are aligned to establish a one-to-one correspondence, providing input and target output for model training.

[0061] The phoneme correction model is based on a phoneme-to-text converter structure. A phoneme-to-text converter consisting of multi-layer Transformer modules is constructed, where the encoder and decoder are multi-layer Transformer modules with the same structure. Hyperparameters of the model are set, such as embedding dimension, number of attention heads, feedforward network dimension, and Dropout rate.

[0062] The model is iteratively trained using the training set to minimize the loss function. During training, the model learns how to predict the correct standard text based on the damaged phoneme sequence. The model's performance is evaluated on the validation set, calculating metrics such as character error rate (CER) and word error rate (WER). Based on the validation results, the model structure and hyperparameters are adjusted, and the training process is repeated to improve performance. The trained model is then saved as a phoneme correction model for use in phoneme correction tasks.

[0063] Step 130: Perform speech synthesis on the corrected text to obtain the restored speech of the pathological speech.

[0064] Specifically, corrected text is a text representation that is closer to standard text, obtained after processing by a phoneme correction model. Corrected text corrects phoneme errors in the original pathological speech, making the text content more accurate and clear.

[0065] Speech synthesis of corrective text refers to the process of converting corrective text into restored speech using speech synthesis technology—that is, a pathological speech restoration version that sounds more natural and closer to normal speech. Speech synthesis of corrective text can be performed using text-based speech synthesis (TTS) systems or specially designed pathological speech restoration systems to ensure that the generated restored speech meets the desired quality standards.

[0066] The parameters of a speech synthesis system, such as speech rate, tone, and volume, can be preset or obtained through pre-training using specific features of pathological speech, in order to further improve the quality of the restored speech.

[0067] The corrected text is input into the configured speech synthesis system, and speech synthesis is performed to obtain the synthesized speech, which is the restored speech of the pathological speech.

[0068] The method provided in this invention identifies the phoneme sequence of pathological speech. Due to the loss of content information and impaired sentence integrity, the phoneme sequence of pathological speech is typically a semantically damaged phoneme sequence. A phoneme correction model can correct the damaged phoneme sequence to obtain corrected text containing complete semantic information. The corrected text is then used for speech synthesis to obtain restored pathological speech. Through a more comprehensive pathological speech restoration technology, not only is the sound quality improved, but the semantic information in the pathological speech is also further preserved and restored, thereby comprehensively improving the intelligibility and communication efficiency of the restored speech.

[0069] Based on any of the above embodiments Figure 2 A flowchart illustrating the implementation of step 120 in the speech restoration method provided by this invention is shown below. Figure 2 As shown, step 120 involves correcting the phoneme sequence based on a phoneme correction model to obtain corrected text, specifically including:

[0070] Step 121: Based on the phoneme correction model, apply the semantic correlation between each phoneme in the phoneme sequence to perform context encoding on the phoneme sequence to obtain phoneme context features;

[0071] Step 122: Decode the phoneme context features to obtain the corrected text.

[0072] Specifically, phoneme correction models, based on encoder-decoder structures, are used to identify and correct damaged phoneme sequences in pathological speech. These models predict and correct erroneous phonemes by analyzing the contextual information of the phoneme sequences and utilizing the semantic correlations between phonemes. A phoneme sequence is a series of phonemes identified from pathological speech, arranged in a specific order to constitute the acoustic representation of the pathological speech.

[0073] The phoneme correction model internally includes an encoder for contextual encoding of the phoneme sequence. The encoder considers the contextual information of each phoneme in the sequence, that is, its semantic relevance to other phonemes. This relevance can be represented by weights or feature representations learned by the model. The phoneme contextual features output by the encoder are feature representations that include the semantic relevance and contextual information between phonemes. This helps the model more accurately identify and correct erroneous phonemes.

[0074] The phoneme correction model also includes a decoder that decodes the phoneme context features. Decoding refers to the process of converting phoneme context features into text. Based on the information in the feature representation, the decoder predicts the most likely phoneme sequence, i.e., the corrected phoneme sequence. Then, the corrected phoneme sequence is converted into text form to obtain the corrected text.

[0075] The resulting corrected text is a representation of the standard text that has been processed by the phoneme correction model. It corrects errors in the original phoneme sequence, making the text content more accurate and clear.

[0076] The method provided in this invention analyzes the contextual information of phoneme sequences using a phoneme correction model based on an encoder-decoder structure, and uses the semantic correlation between phonemes to predict and correct erroneous phonemes, thereby obtaining more accurate corrected text and further improving the comprehensibility and semantic integrity of pathological speech.

[0077] Based on any of the above embodiments, step 122, which decodes the phoneme context features to obtain the corrected text, specifically includes:

[0078] Step 122-1: Based on the positional relationship between each phoneme in the phoneme sequence, perform positional encoding on the phoneme sequence to obtain the positional features of each phoneme;

[0079] Step 122-2: Decode the text based on the phoneme context features and the positional features of each phoneme to obtain the corrected text.

[0080] Specifically, to further improve the accuracy of phoneme correction, decoding considers not only the contextual features of the phonemes but also the positional features of each phoneme. The positional features of each phoneme refer to features derived from the positional relationships between phonemes in a phoneme sequence. They reflect the relative or absolute position of the phonemes in the sequence, helping the model understand the sequential relationships between phonemes.

[0081] Positional encoding of phoneme sequences can be achieved in various ways, such as using position embeddings or position vectors. The purpose of positional encoding is to assign a unique positional feature to each phoneme in the sequence, which reflects the phoneme's position information within the sequence.

[0082] The obtained positional features of each phoneme are a set of features corresponding to the phoneme sequence, where each feature contains the positional information of the corresponding phoneme.

[0083] Then, decoding is performed based on the phoneme context features and the positional features of each phoneme to obtain the corrected text. The phoneme context features and the positional features of each phoneme are used as input to the decoder. The decoder comprehensively considers the contextual and positional information of the phonemes, uses this information to predict the most likely phoneme sequence, and gradually converts the input features into text-based output, i.e., the corrected text. The corrected text corrects the errors in the original phoneme sequence while preserving the correct order and semantic relationships between the phonemes.

[0084] During decoding, phoneme context features and positional features can be fused or combined to more accurately predict phoneme sequences. Feature fusion can be achieved through concatenation, addition, or other feature interaction methods.

[0085] The method provided in this invention, by decoding based on phoneme context features and the positional features of each phoneme, obtains corrected text, which can further improve the accuracy of phoneme correction.

[0086] Preferably, the phoneme correction model may further include an embedding layer for constructing separate embedding layers for phonemes and text, converting discrete symbols into continuous vector representations.

[0087] Based on any of the above embodiments, step 130 performs speech synthesis on the corrected text to obtain the restored speech of the pathological speech, specifically including:

[0088] Step 131: Based on the speech synthesis model, extract the speaker features of the pathological speech, and based on the speaker features, perform speech synthesis on the corrected text to obtain the restored speech;

[0089] The speech synthesis model is trained based on noisy and clean speech samples from the same speaker, as well as the corresponding sample text of the noisy speech samples.

[0090] Specifically, speech synthesis of the corrected text to obtain the restored speech from the pathological speech can be achieved through a trained speech synthesis model. The speech synthesis model extracts speaker features from the pathological speech. This can be achieved using speaker recognition technology or speech feature extraction algorithms. The extracted features should reflect the speaker's identity and vocal characteristics. Speaker features include, but are not limited to, pitch, speech rate, and timbre. In speech synthesis, using speaker features ensures that the generated speech is consistent with the original speaker's voice.

[0091] The speech synthesis model then generates a speech waveform corresponding to the corrected text based on the input corrected text and extracted speaker features, while maintaining consistency with the original speaker's voice. The clean speech generated by the speech synthesis model that corresponds to the corrected text is the restored speech. The restored speech can clearly convey the text content and sounds like the original speaker's voice.

[0092] Prior to step 131, the speech synthesis model had been trained using noisy sample speech, clean sample speech, and corresponding sample text from the same speaker. During training, the model learned how to extract useful speech information from noisy speech and generate clean speech based on the text content.

[0093] We collected a large amount of clean speech data, covering a variety of speakers, genders, ages, and accents, to ensure data diversity as sample clean speech.

[0094] Noisy speech samples refer to speech samples containing noise. This noise may originate from factors such as the environment, equipment, or abnormalities in the speech organs. In this scenario, noisy speech samples are used to train a speech synthesis model, enabling it to adapt to and reduce the impact of noise on speech synthesis.

[0095] Noise data was collected on hoarseness caused by abnormalities of the speech organs, covering different degrees and types of pathological features. Clean speech data was mixed with noise of different types and intensities to generate more noisy speech data, which served as sample noisy speech. Here, the sample clean speech and sample noisy speech belong to the same speaker.

[0096] The sample text corresponding to the noisy speech sample is the text content that corresponds to the noisy speech sample. When training the speech synthesis model, the sample text is used to guide the model on how to convert text into speech.

[0097] By training with speech samples containing different levels of noise (noisy speech samples) and clean speaker speech samples (clean speech samples), the extracted speaker representations are equipped with noise reduction capabilities, which can counteract the hoarse noise caused by abnormalities in the speech organs in pathological speech.

[0098] The method provided in this invention, based on the speaker features of the corrected text and pathological speech, uses a speech synthesis model to generate high-quality restored speech, which can further improve the audibility and comprehensibility of pathological speech.

[0099] Based on any of the above embodiments Figure 3 This is a schematic diagram of the training steps of the speech synthesis model provided by the present invention, as shown below. Figure 3 As shown, the training steps for the speech synthesis model include:

[0100] Step 310: Extract speaker features from the noisy speech samples;

[0101] Step 320: Based on the speaker features and the sample text corresponding to the noisy sample speech, perform speech synthesis to obtain the predicted synthesized speech;

[0102] Step 330: Based on the difference between the predicted synthesized speech and the clean speech sample, the parameters of the pre-trained initial model are iterated to obtain the speech synthesis model.

[0103] Specifically, the pre-trained initial model can be an initial speech synthesis model pre-trained with a large amount of clean speech data, which can achieve zero-shot speech synthesis.

[0104] Preprocessing of the noisy speech samples, including noise reduction and echo removal, improves the accuracy of subsequent feature extraction. Professional audio editing software or algorithms are then used to extract speaker features from the preprocessed noisy speech. Speaker features can include the speaker's age, gender, timbre, and speech rate. For speaker feature extraction, spectrum analysis tools can be used to observe the audio's spectral distribution to more accurately locate the human voice signal. By inverting the phase of the human voice and accompaniment in the left and right channels, the accompaniment is eliminated, thus extracting the pure human voice for further speaker feature analysis. Furthermore, bandpass filters or notch filters can be applied to further purify the human voice signal, removing noise and interference.

[0105] The sample text corresponding to the noisy speech is analyzed, including text content, grammatical structure, and vocabulary selection. Speaker features are mapped to the sample text to generate an input for a speech synthesis model containing these features. Speech synthesis algorithms (such as Hidden Markov Model (HMM) based speech synthesis techniques or deep learning-based speech synthesis techniques) are then used to convert the mapped input into a speech signal, resulting in the predicted synthesized speech.

[0106] Using speech quality evaluation metrics (such as speech similarity, clarity, naturalness, and fluency), the difference between the predicted synthesized speech and the clean sample speech is calculated. Based on the difference calculation results, the parameters of the pre-trained initial model are adjusted to improve the accuracy of speech synthesis. The above process of difference calculation and parameter adjustment is repeated until the difference between the predicted synthesized speech and the clean sample speech reaches a preset threshold or the number of iterations reaches a preset upper limit, at which point the final speech synthesis model is obtained.

[0107] Noisy speech data is used as training data, and the target speech is the corresponding clean speech. This is used to train the noise reduction capability and speaker representation extraction of the model. The speech synthesis model trained based on the method of this embodiment can extract robust speaker features from noisy speech.

[0108] Preferably, a noise reduction mechanism can also be integrated into the speech synthesis model, such as introducing a denoising autoencoder to enhance noise reduction capabilities.

[0109] Based on any of the above embodiments, step 110, identifying the phoneme sequence of pathological speech, includes:

[0110] Step 111: Based on the trained phoneme recognition model, identify the phoneme sequence of pathological speech. The phoneme recognition model is trained using sample speech data and its corresponding labeled phoneme sequence, based on a pre-trained speech model and a phoneme classification layer.

[0111] Specifically, the phoneme sequence for identifying pathological speech can be achieved using a trained phoneme recognition model. This model is built upon a pre-trained speech model and a phoneme classification layer. The pre-trained speech model has been trained on a large amount of speech data and is able to capture the basic features of the speech signal, such as spectrum, pitch, and intensity. These features provide the foundation for subsequent phoneme classification.

[0112] The phoneme classification layer is responsible for mapping the features extracted by the pre-trained speech model to specific phoneme categories. The phoneme classification layer can be implemented using structures such as fully connected networks, convolutional neural networks (CNNs), or recurrent neural networks (RNNs).

[0113] Acquire a large amount of diverse speech data, including recordings of different languages, accents, and pronunciation styles. Perform noise reduction, silent segment removal, and normalization on the collected speech data to improve data quality, thereby obtaining sample speech data.

[0114] Professional annotation tools are used to annotate the preprocessed sample speech data at the phoneme level, generating corresponding phoneme sequences, i.e., annotated phoneme sequences.

[0115] During training, the model learns the mapping relationship between sample speech data and its labeled phoneme sequences. Through multiple parameter iterations, the model's performance on the validation set stabilizes or the preset number of iterations is reached. This allows for the training of an accurate phoneme recognition model that can convert input speech signals into corresponding phoneme sequences, providing a foundation for subsequent phoneme correction and speech synthesis tasks.

[0116] Based on any of the above embodiments Figure 4 This is a schematic diagram of the training steps of the phoneme recognition model provided by the present invention, as shown below. Figure 4 As shown, the training steps for the phoneme recognition model include:

[0117] Step 410: Based on the feature extraction layer of the pre-trained speech model, extract the acoustic features of the sample speech data, and perform phoneme recognition on the acoustic features based on the phoneme classification layer to obtain the predicted phoneme sequence.

[0118] Step 420: Freeze the feature extraction layer of the pre-trained speech model, and perform the first stage training of the phoneme classification layer based on the difference between the predicted phoneme sequence and the labeled phoneme sequence.

[0119] Step 430: Unfreeze the feature extraction layer of the pre-trained speech model. Based on the difference between the predicted phoneme sequence and the labeled phoneme sequence, perform a second-stage joint training on the pre-trained speech model and the phoneme classification layer to determine the pre-trained speech model and the phoneme classification layer as the phoneme recognition model.

[0120] Specifically, a self-supervised pre-trained speech model is selected. These models, trained on large-scale unlabeled speech data, are capable of learning rich speech feature representations. The pre-trained self-supervised model, including its parameters and network structure, is loaded using a deep learning framework. One or more fully connected layers are added on top of the pre-trained speech model as phoneme classification layers. The number of neurons in the last layer equals the total number of phoneme categories, and the activation function can be chosen as Softmax to output the probability distribution of each phoneme.

[0121] The feature extraction layer of a pre-trained speech model is used to process the sample speech data and extract the acoustic features of the speech signal. These features typically include spectral features, temporal features, etc., which can reflect the basic properties and structure of the speech signal.

[0122] The extracted acoustic features are input into the phoneme classification layer for phoneme recognition. The phoneme classification layer outputs a predicted phoneme sequence based on the input features and pre-trained weights. This embodiment employs a segmented training strategy.

[0123] The first training phase involves freezing the feature extraction layer of the pre-trained speech model, i.e., not updating its parameters. This maintains the stability of the feature extraction layer and avoids introducing excessive noise during training. Based on the difference between the predicted and labeled phoneme sequences, the phoneme classification layer is trained using an appropriate loss function (such as cross-entropy loss). The parameters of the phoneme classification layer are updated using the backpropagation algorithm to minimize the loss function. The goal of this first training phase is to enable the phoneme classification layer to gradually adapt to the input acoustic features and initially acquire the ability to recognize phonemes.

[0124] The second training phase involves unfreezing the feature extraction layer of the pre-trained speech model, allowing its parameters to be updated during training. The pre-trained speech model and the phoneme classification layer are still jointly trained based on the difference between the predicted and labeled phoneme sequences. At this point, the entire model (including the feature extraction layer and the phoneme classification layer) will have its parameters updated according to the gradient of the loss function. The purpose of this second training phase is to further optimize the performance of the entire model, enabling the feature extraction layer to better extract acoustic features relevant to phoneme recognition, and the phoneme classification layer to more accurately identify phonemes.

[0125] Evaluate model performance on the validation set, calculating metrics such as accuracy and recall to ensure model effectiveness. Based on the validation results, adjust the model structure and hyperparameters, and repeat the training process to improve performance. Save the trained model as a phoneme recognition model for use in subsequent phoneme recognition tasks.

[0126] The method provided in this invention can train an accurate phoneme recognition model. This model can extract acoustic features from sample speech data using the feature extraction layer of a pre-trained speech model and perform phoneme recognition through a phoneme classification layer. By using a staged training approach, the performance of the model is gradually optimized, giving it stronger generalization ability and accuracy.

[0127] Based on any of the above embodiments Figure 5 This is the second flowchart of the voice restoration method provided by the present invention, as shown below. Figure 5 As shown, the method includes:

[0128] S1, identify the pathological speech to be repaired.

[0129] S2, based on a trained phoneme recognition model, identifies the phoneme sequences of pathological speech. The phoneme recognition model is trained using sample speech data and their corresponding labeled phoneme sequences, based on a pre-trained speech model and a phoneme classification layer. The training steps of the phoneme recognition model include:

[0130] Based on the feature extraction layer of the pre-trained speech model, acoustic features of the sample speech data are extracted. Phoneme recognition is performed on the acoustic features based on the phoneme classification layer to obtain the predicted phoneme sequence. The feature extraction layer of the pre-trained speech model is frozen. Based on the difference between the predicted phoneme sequence and the labeled phoneme sequence, the phoneme classification layer is trained in the first stage. The feature extraction layer of the pre-trained speech model is unfrozen. Based on the difference between the predicted phoneme sequence and the labeled phoneme sequence, the pre-trained speech model and the phoneme classification layer are trained in the second stage. The pre-trained speech model and the phoneme classification layer are then determined as the phoneme recognition model.

[0131] S3, based on a phoneme correction model, corrects phoneme sequences to obtain corrected text. The phoneme correction model is trained based on sample damaged phoneme sequences and their corresponding standard texts. Specifically, it includes:

[0132] Based on the phoneme correction model, the semantic correlation between each phoneme in the phoneme sequence is applied to perform context encoding on the phoneme sequence to obtain phoneme context features; based on the positional relationship between each phoneme in the phoneme sequence, the phoneme sequence is positionally encoded to obtain positional features of each phoneme; based on the phoneme context features and positional features of each phoneme, the corrected text is obtained by decoding.

[0133] S4, based on a speech synthesis model, extracts speaker features from the pathological speech and synthesizes the corrected text based on these features to obtain the restored speech. The speech synthesis model is trained using noisy and clean sample speech from the same speaker, as well as the corresponding sample text for the noisy sample speech. The specific training steps of the speech synthesis model include:

[0134] Extract speaker features from the noisy sample speech; perform speech synthesis based on the speaker features and the corresponding sample text of the noisy sample speech to obtain the predicted synthesized speech; iterate the parameters of the pre-trained initial model based on the difference between the predicted synthesized speech and the clean sample speech to obtain the speech synthesis model.

[0135] The speech restoration device provided by the present invention is described below. The speech restoration device described below and the speech restoration method described above can be referred to in correspondence.

[0136] Based on any of the above embodiments Figure 6 This is a schematic diagram of the voice restoration device provided by the present invention, as shown below. Figure 6 As shown, the voice restoration device includes:

[0137] Phoneme recognition unit 610 is used to determine the pathological speech to be repaired and to recognize the phoneme sequence of the pathological speech;

[0138] The phoneme correction unit 620 is used to correct the phoneme sequence based on the phoneme correction model to obtain the corrected text. The phoneme correction model is trained based on the sample damaged phoneme sequence and its corresponding standard text.

[0139] The speech synthesis unit 630 is used to perform speech synthesis on the corrected text to obtain the restored speech of the pathological speech.

[0140] The speech restoration device provided in this invention identifies the phoneme sequence of pathological speech. Due to the loss of content information and impaired sentence integrity, the phoneme sequence of pathological speech is typically a semantically damaged phoneme sequence. A phoneme correction model can correct the damaged phoneme sequence to obtain corrected text containing complete semantic information. The corrected text is then used for speech synthesis to obtain restored speech of the pathological speech. Through a more comprehensive pathological speech restoration technology, not only is the sound quality improved, but the semantic information in the pathological speech is also further preserved and restored, thereby comprehensively improving the intelligibility and communication efficiency of the restored speech.

[0141] Based on any of the above embodiments, the phoneme correction unit is specifically used for:

[0142] Based on the phoneme correction model, the semantic correlation between each phoneme in the phoneme sequence is applied to perform context encoding on the phoneme sequence to obtain phoneme context features;

[0143] The corrected text is obtained by decoding the phoneme context features.

[0144] Based on any of the above embodiments, the phoneme correction unit is specifically used for:

[0145] Based on the positional relationship between each phoneme in the phoneme sequence, the phoneme sequence is positionally encoded to obtain the positional features of each phoneme;

[0146] The corrected text is obtained by decoding based on the phoneme context features and the position features of each phoneme.

[0147] Based on any of the above embodiments, the speech synthesis unit is specifically used for:

[0148] Based on the speech synthesis model, the speaker features of the pathological speech are extracted, and based on the speaker features, the corrected text is synthesized to obtain the restored speech.

[0149] The speech synthesis model is trained based on noisy and clean speech samples from the same speaker, as well as the sample text corresponding to the noisy speech samples.

[0150] Based on any of the above embodiments, a speech synthesis model training unit is further included, for:

[0151] Extract speaker features from the noisy speech samples;

[0152] Based on the speaker features and the sample text corresponding to the noisy sample speech, speech synthesis is performed to obtain predicted synthesized speech;

[0153] Based on the difference between the predicted synthesized speech and the clean speech sample, the parameters of the pre-trained initial model are iterated to obtain the speech synthesis model.

[0154] Based on any of the above embodiments, the phoneme recognition unit is specifically used for:

[0155] Based on a trained phoneme recognition model, the phoneme sequence of the pathological speech is identified. The phoneme recognition model is trained using sample speech data and its corresponding labeled phoneme sequences, based on a pre-trained speech model and a phoneme classification layer.

[0156] Based on any of the above embodiments, a phoneme recognition model training unit is further included, for:

[0157] Based on the feature extraction layer of the pre-trained speech model, acoustic features of the sample speech data are extracted, and phoneme recognition is performed on the acoustic features based on the phoneme classification layer to obtain a predicted phoneme sequence.

[0158] Freeze the feature extraction layer of the pre-trained speech model, and perform the first stage training of the phoneme classification layer based on the difference between the predicted phoneme sequence and the labeled phoneme sequence;

[0159] Unfreeze the feature extraction layer of the pre-trained speech model, and based on the difference between the predicted phoneme sequence and the labeled phoneme sequence, perform a second stage of joint training on the pre-trained speech model and the phoneme classification layer to determine the pre-trained speech model and the phoneme classification layer as the phoneme recognition model.

[0160] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a speech restoration method, which includes: determining the pathological speech to be restored and identifying the phoneme sequence of the pathological speech; correcting the phoneme sequence based on a phoneme correction model to obtain a corrected text, wherein the phoneme correction model is trained based on a sample damaged phoneme sequence and its corresponding standard text; and performing speech synthesis on the corrected text to obtain the restored speech of the pathological speech.

[0161] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0162] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the speech restoration method provided by the above methods. The method includes: determining the pathological speech to be restored and identifying the phoneme sequence of the pathological speech; correcting the phoneme sequence based on a phoneme correction model to obtain a corrected text, wherein the phoneme correction model is trained based on a sample damaged phoneme sequence and its corresponding standard text; and performing speech synthesis on the corrected text to obtain the restored speech of the pathological speech.

[0163] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the speech restoration method provided by the above methods. The method includes: determining the pathological speech to be restored and identifying the phoneme sequence of the pathological speech; correcting the phoneme sequence based on a phoneme correction model to obtain a corrected text, wherein the phoneme correction model is trained based on a sample damaged phoneme sequence and its corresponding standard text; and performing speech synthesis on the corrected text to obtain the restored speech of the pathological speech.

[0164] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0165] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A voice restoration method, characterized in that, include: Identify the pathological speech to be repaired and recognize the phoneme sequence of the pathological speech; Based on the phoneme correction model, the phoneme sequence is corrected to obtain the corrected text. The phoneme correction model is trained based on the sample damaged phoneme sequence and its corresponding standard text. The corrected text is processed by speech synthesis to obtain the restored speech of the pathological speech; The phoneme correction model corrects the phoneme sequence to obtain corrected text, including: Based on the phoneme correction model, the semantic correlation between each phoneme in the phoneme sequence is applied to perform context encoding on the phoneme sequence to obtain phoneme context features; Based on the positional relationship between each phoneme in the phoneme sequence, the phoneme sequence is positionally encoded to obtain the positional features of each phoneme; The corrected text is obtained by decoding based on the phoneme context features and the position features of each phoneme.

2. The voice restoration method according to claim 1, characterized in that, The step of performing speech synthesis on the corrected text to obtain the restored speech of the pathological speech includes: Based on the speech synthesis model, the speaker features of the pathological speech are extracted, and based on the speaker features, the corrected text is synthesized to obtain the restored speech. The speech synthesis model is trained based on noisy and clean speech samples from the same speaker, as well as the sample text corresponding to the noisy speech samples.

3. The voice restoration method according to claim 2, characterized in that, The training steps of the speech synthesis model include: Extract speaker features from the noisy speech samples; Based on the speaker features and the sample text corresponding to the noisy sample speech, speech synthesis is performed to obtain predicted synthesized speech; Based on the difference between the predicted synthesized speech and the clean speech sample, the parameters of the pre-trained initial model are iterated to obtain the speech synthesis model.

4. The speech restoration method according to any one of claims 1 to 3, characterized in that, The phoneme sequence for recognizing the pathological speech includes: Based on a trained phoneme recognition model, the phoneme sequence of the pathological speech is identified. The phoneme recognition model is trained using sample speech data and its corresponding labeled phoneme sequences, based on a pre-trained speech model and a phoneme classification layer.

5. The voice restoration method according to claim 4, characterized in that, The training steps of the phoneme recognition model include: Based on the feature extraction layer of the pre-trained speech model, acoustic features of the sample speech data are extracted, and phoneme recognition is performed on the acoustic features based on the phoneme classification layer to obtain a predicted phoneme sequence. Freeze the feature extraction layer of the pre-trained speech model, and perform the first stage training of the phoneme classification layer based on the difference between the predicted phoneme sequence and the labeled phoneme sequence; Unfreeze the feature extraction layer of the pre-trained speech model, and based on the difference between the predicted phoneme sequence and the labeled phoneme sequence, perform a second stage of joint training on the pre-trained speech model and the phoneme classification layer to determine the pre-trained speech model and the phoneme classification layer as the phoneme recognition model.

6. A voice restoration device, characterized in that, include: A phoneme recognition unit is used to determine the pathological speech to be repaired and to identify the phoneme sequence of the pathological speech. A phoneme correction unit is used to correct the phoneme sequence based on a phoneme correction model to obtain corrected text. The phoneme correction model is trained based on the sample damaged phoneme sequence and its corresponding standard text. A speech synthesis unit is used to synthesize the corrected text into a restored speech of the pathological speech; The phoneme correction unit is specifically used for: Based on the phoneme correction model, the semantic correlation between each phoneme in the phoneme sequence is applied to perform context encoding on the phoneme sequence to obtain phoneme context features; Based on the positional relationship between each phoneme in the phoneme sequence, the phoneme sequence is positionally encoded to obtain the positional features of each phoneme; The corrected text is obtained by decoding based on the phoneme context features and the position features of each phoneme.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the voice restoration method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the voice restoration method as described in any one of claims 1 to 5.

Images