Intelligibility enhanced speech recovery

Abstract

The present disclosure relates to a computer-implemented method of training a neural network-based language model to recover clean speech from distorted speech waveforms. The method includes obtaining an input time-frequency representation of a distorted speech waveform; determining, using a neural network-based multi-stage speech encoder, one or more audio features (e.g., semantic representations) associated with the distorted speech waveform from the input time-frequency representation; determining, based at least in part on the one or more audio features, conditioning information for the language model; and jointly training the language model and the speech encoder, wherein the language model is trained based at least in part on the conditioning information.

Description

Cross-references to related applications

[0001] This application claims priority to International Patent Application No. PCT / CN2024 / 080815, filed March 8, 2024; U.S. Provisional Application Serial No. 63 / 664,566, filed June 26, 2024; International Patent Application No. PCT / CN2024 / 116230, filed September 2, 2024; Spanish Patent Application No. P202430693, filed September 2, 2024; U.S. Provisional Application Serial No. 63 / 722,192, filed November 19, 2024; and U.S. Provisional Application Serial No. 63 / 758,701, filed February 14, 2025, each of which is incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure relates to speech recovery, and more particularly to methods and systems for speech recovery by training a speech encoder incorporating a machine learning model to provide quality-enhanced recovered speech. Background Technology

[0003] Speech restoration refers to the task of providing high-quality speech from degraded speech, and specifically aims to recover full-band speech with enhanced quality and intelligibility from corrupted signals. Unlike traditional speech enhancement, which typically only refers to speech denoising and dereverberation, speech restoration needs to consider more types of degradation, such as noise, reverberation, low bandwidth, clipping, packet loss, etc. In particular, speech restoration solves a wide range of tasks, including inherently generative tasks such as bandwidth expansion and packet loss concealment. Existing methods for speech restoration include adversarial learning and diffusion-based methods. Recently, language models have also been used for speech enhancement; however, these models only consider speech denoising.

[0004] Generally, training language models on a large scale is much easier than adversarial learning. However, using some existing language models may require a large number of sampling steps during the inference process, which places an extremely high computational burden on speech recovery systems. Furthermore, existing models (or neural networks) may introduce artifacts during training / inference, leading to distortions in the processed speech, including spectral defects, missing high-frequency components, speaker alterations, and partial word truncation. This distortion caused by neural networks has always been problematic for speech enhancement, even though neural networks can successfully remove noise. Traditionally, this problem has been addressed by incorporating a loss function design to penalize excessive suppression of speech during training.

[0005] A further problem is that the improved perceptual quality after processing with existing models may not translate into improved intelligibility (which can typically be measured by the word error rate (WER) of an automatic speech recognition (ASR) system), because distortion removal may unintentionally alter the speech content.

[0006] Therefore, there is a need to develop improved techniques for speech recovery methods and systems that can remove the aforementioned distortions in speech in a computationally efficient and reliable manner. In particular, it is desirable to develop a deep generative method that effectively eliminates the distortions / artifacts introduced by neural network models during the recovery process with reduced computational effort. Further improvements in the intelligibility of the generated speech are also desired. Summary of the Invention

[0007] In view of this need, this disclosure provides computer-implemented methods and corresponding systems for recovering high-quality speech from all types of distortion, as well as computer programs and computer-readable storage media having the features of the respective independent claims.

[0008] Specifically, this disclosure proposes jointly training a speech encoder and a machine learning model (e.g., a neural network-based language model, such as a masked language model or an autoregressive language model) for speech recovery. It should be recognized that a masked language model based on neural network processing can require significantly fewer sampling steps during inference. To achieve efficient speech recovery, this disclosure further proposes employing a fusion network designed to effectively bridge the speech encoder and language model for both the training and inference processes. Specifically, the proposed fusion method may include, for example, concatenating one or more (or all) intermediate features from the speech encoder, or performing a learned weighted sum based on attention. Additionally, this disclosure proposes applying classifier-free guidance (e.g., during the inference phase) to effectively reduce the distortion in the generated clean speech caused by neural network processing (e.g., during the training phase).

[0009] One aspect of this disclosure relates to a computer-implemented method for speech recovery, particularly for training a neural network-based language model in a speech recovery system to generate / recover clean speech from a distorted speech waveform (i.e., during the training phase of the speech recovery system). The method may include obtaining an input time-frequency (also referred to as “spectral-time”) representation of the distorted speech waveform. This input time-frequency representation may, for example, involve local and global spectral and temporal speech patterns of the distorted speech waveform. The method may further include determining one or more audio features associated with the distorted speech waveform (which may involve multiple intermediate speech representations, e.g., semantic representations, of the input time-frequency representation). In particular, one or more audio features may be determined based on the input time-frequency representation using, for example, a neural network-based multi-stage speech encoder. Moreover, the method may include determining conditional information for the language model based at least in part on one or more audio features. For example, one or more audio features (or multiple intermediate speech representations) may be combined (or fused) to provide feature conditions (i.e., conditional information) to the neural network language model. Additionally, the method may also include jointly training the language model and the speech encoder. Notably, the language model may be trained at least in part on the conditional information.

[0010] As configured as described above, by combining audio features obtained from different stages / levels of the speech encoder, optimal speech representations can be provided for conditional determination, which further improves the accuracy of training the speech recovery system. By training the language model and speech encoder simultaneously, different types of distortion can be jointly and efficiently removed from the audio data, thus allowing for the efficient generation of high-quality speech.

[0011] In some embodiments, the method may further consider semantic information within the speech to train the speech recovery system. Specifically, the method may further include obtaining a reference time-frequency representation of the ground truth speech waveform for the distorted speech waveform. The method may additionally include determining one or more semantic representations associated with the distorted speech waveform based on one or more audio features. The method may also include determining one or more reference semantic representations associated with the ground truth speech waveform. More specifically, one or more reference semantic representations may be determined based on the reference time-frequency representation using a pre-trained auxiliary model (e.g., a teacher model). Therefore, an auxiliary loss function can be used to train the speech encoder based on one or more semantic representations and one or more reference semantic representations.

[0012] In some embodiments, determining one or more semantic representations from one or more audio features may include applying a fully connected pooling layer. In some embodiments, one or more semantic representations may include one or more pseudo-speech labels. In particular, one or more reference semantic representations may include one or more reference pseudo-speech labels. In some embodiments, the pre-trained auxiliary model may include a self-supervised model. For example, the pre-trained auxiliary model may be a pre-trained HuBERT (Hidden Unit Bidirectional Encoder Representation from Transformer) model.

[0013] By configuring as described above, and by considering the semantic information of the speech (e.g., one or more semantic representations), both the perceptual quality and intelligibility of the generated / processed speech can be effectively improved, thereby enhancing the performance of the speech recovery system. In particular, the proposed method allows for an optimal trade-off between speech intelligibility and quality for full-band speech recovery.

[0014] In some embodiments, the method may further include (e.g., at the speech encoder) applying a multilayer perceptron to generate a predicted time-frequency representation of a clean speech waveform. In some embodiments, the method may further include obtaining a reference time-frequency representation of the ground truth speech waveform for the distorted speech waveform. In particular, the speech encoder may be trained based on the reference time-frequency representation and the predicted time-frequency representation predicted by the speech encoder.

[0015] In some embodiments, the speech encoder can be trained using a regression loss. For example, training the speech encoder may include minimizing the difference between the predicted time-frequency representation and the reference time-frequency representation by adapting one or more parameters of the multi-stage speech encoder based on a regression loss function.

[0016] In some embodiments, one or more audio features can be determined based on multiple intermediate speech representations of the input time-frequency representation generated within the speech encoder (which may be associated with one or more audio features). In some embodiments, multiple intermediate speech representations can be generated by applying multiple successive transformer layers (which may be provided, for example, at the speech encoder). Specifically, each transformer layer can generate a corresponding intermediate speech representation.

[0017] To determine / generate multiple intermediate speech representations, the method may further include (e.g., by a second multilayer perceptron, which may be provided, for example, at a speech encoder) transforming the frame size of the input time-frequency representation to the transformer layer size to pass the input time-frequency representation to multiple consecutive transformer layers. Furthermore, the method may also include combining additional intermediate speech representations from the second multilayer perceptron and multiple intermediate speech representations from the multiple consecutive transformer layers as conditional information to be provided to the language model.

[0018] In some embodiments, additional intermediate speech representations and multiple intermediate speech representations can be combined by concatenating one or more features from a second multilayer perceptron and the corresponding transformer layer into a feature vector and applying a learnable projection to the feature vector using a linear layer.

[0019] In some embodiments, additional intermediate speech representations and multiple intermediate speech representations can be combined by summing one or more features from a second multilayer perceptron and a corresponding transformer layer using corresponding weight coefficients determined by applying an attention algorithm.

[0020] In some embodiments, the method may further include randomly providing conditional information (e.g., the aforementioned feature conditions associated with the combined intermediate speech representations) to a language model to conditionally train the language model (i.e., to conditionally train a neural network model). The method may also include replacing a portion of the conditional information with one or more learnable embedding parameters as unconditional input to unconditionally train the language model.

[0021] Therefore, training the language model can be further based on unconditional inputs that include one or more learnable embedding parameters. For example, the number of training epochs used for conditionally training the language model can be less than the total number of training epochs used to train the language model. More specifically, the number of training epochs used for unconditionally training the language model can be given by a predetermined ratio. For example, the duration of unconditionally training the language model can be 10% of the total duration of training the language model.

[0022] In some embodiments, the method may include segmenting a reference waveform associated with a ground truth speech signal to convert the reference waveform into a codemap of a predefined size containing discrete tokens for training a language model. In some embodiments, the language model may be trained based on a cross-entropy loss function. In some embodiments, the language model may include an autoregressive language model, or the language model may include a masked language model.

[0023] In some embodiments where the masked language model is implemented as a neural network model, the method may include randomly masking one or more discrete lexical units in a codemap to generate a partially masked codemap, and converting the partially masked codemap into multiple embedding vectors. The method may also include summing the multiple embedding vectors, and providing the sum of the summed embedding vectors and the conditional information as training input to the masked language model.

[0024] More specifically, the language model may contain multiple transformer blocks, followed by a multilayer perceptron. Therefore, to train the language model, the method may further include extracting multiple masking lemmas from the ground truth speech signal. The method may also include estimating multiple predicted indices corresponding to the masking lemmas by the multilayer perceptron. Furthermore, the method may include determining the difference between the predicted and actual indices of the masking lemmas. Moreover, the method may include minimizing the difference between the predicted and actual indices of the masking lemmas based on a cross-entropy loss function.

[0025] As configured as described above, by designing and employing a fusion network (by means of, for example, concatenating all intermediate features of the speech encoder, or performing a learned weighted sum based on attention), the speech encoder and language model can be effectively bridged for both the training and inference processes, thereby achieving efficient performance in speech recovery and enhancement. It should be further recognized that by jointly optimizing the loss function (e.g., cross-entropy loss and / or regression loss) to train the language model and speech encoder, the speech recovery system can be trained in a computationally efficient and reliable manner. The feasibility of the method proposed in this disclosure is further recognized because the proposed method can be cascaded (combined) with existing models to repair artifacts introduced by existing models.

[0026] One aspect of this disclosure relates to a computer-implemented method for speech recovery, particularly for recovering clean speech from distorted speech waveforms using a neural network-based language model (i.e., during the inference phase of a speech recovery system). Similar to the proposed training method described above, this method may include obtaining an input time-frequency (also referred to as “spectral time”) representation of the distorted speech waveform. The method may further include determining one or more audio features associated with the distorted speech waveform (which may involve multiple intermediate speech representations, e.g., semantic representations, of the input time-frequency representation). Specifically, a neural network-based multi-stage speech encoder may be used to determine one or more audio features based on the input time-frequency representation. Moreover, the method may include determining conditional information for a language model based at least in part on one or more audio features. For example, one or more audio features (or multiple intermediate speech representations) may be combined (or fused) to provide feature conditions (i.e., conditional information) to a neural network model, similar to the proposed training method described above.

[0027] Additionally, the method may further include generating multiple discrete lexical units of a codemap corresponding to a clean speech signal, at least partially based on conditional information. Notably, the multiple discrete lexical units of the codemap can be generated from a fully masked codemap containing multiple masking lexical units. For example, the multiple discrete lexical units can be generated from a fully masked codemap that exclusively contains multiple masking lexical units. Furthermore, the method may include generating a recovered speech waveform associated with the clean speech signal from the multiple discrete lexical units of the codemap.

[0028] As configured as described above, by combining audio features obtained from different stages / levels of the speech encoder, an optimal speech representation can be provided for conditional determination, which further improves the accuracy of training the speech recovery system. In this way, different types of distortion can be removed from speech in a computationally efficient and reliable manner. In other words, distortion / artifacts introduced by the neural network model can be effectively eliminated during the recovery process with reduced computational effort.

[0029] In some embodiments, multiple discrete lexical units of the codemap can be generated based on an iterative sampling procedure. More specifically, the method can apply a classifier-free guided (CFG) approach to the iterative sampling procedure for speech recovery.

[0030] In detail, the iterative sampling procedure according to this disclosure may include, for each iteration, calculating a first logit for a first predicted lexical unit generated from the language model using conditional information, and subsequently calculating a second logit for a second predicted lexical unit generated from the language model without conditional information, or vice versa. The iterative sampling procedure may also include using controllable parameters to combine the first and second logits to obtain a guided logit. Specifically, the controllable parameters may indicate the degree to which the generation of the recovered speech waveform associated with the clean speech signal depends on conditional information. Furthermore, the iterative sampling procedure may then further include sampling one or more output lexical units of the language model generated from previous iterations to remask based on the guided logit.

[0031] In some embodiments, the language model may include an autoregressive language model. In some embodiments, the language model may include a masked language model. In the latter case, the method may further include obtaining a fully masked codemap containing multiple masking lemmas (e.g., obtaining a fully masked codemap that exclusively contains multiple masking lemmas). Moreover, the method may include converting the fully masked codemap into multiple embedding vectors. The method may further include summing the multiple embedding vectors and providing the sum of the summed embedding vectors and conditional information to the masked language model as input based on a guided logit. Therefore, the method may include predicting multiple masking lemmas by the language model to predict multiple discrete lemmas of the codemap.

[0032] In some embodiments, the iterative sampling procedure may further include determining prediction accuracy and prediction gain for each predicted masking word. Specifically, predicting multiple discrete words may include retaining the predicted masking word if the corresponding prediction accuracy and / or corresponding prediction gain associated with the predicted masking word is greater than or equal to a predefined value. Alternatively, predicting multiple discrete words may include discarding the predicted masking word if the corresponding prediction accuracy and / or corresponding prediction gain associated with the predicted masking word is less than a predefined value, and may further include re-predicting the discarded masking word by the masking language model.

[0033] In some embodiments, re-predicting dropped masking words may include: updating the fully masked codemap based on the retained predicted masking words and the dropped masking words; converting the updated fully masked codemap into multiple updated embedding vectors; obtaining a guided logit based on the output of the masked language model without conditional information; summing the multiple updated embedding vectors; and providing the summed updated embedding vectors and the sum of conditional information to the masked language model as an updated input for prediction based on the guided logit.

[0034] The proposed method for operating the speech recovery system during the inference phase may also include processing steps similar to those of the proposed training method (for operation during the training phase) described above. In some embodiments, one or more audio features may be determined based on multiple intermediate speech representations of the input time-frequency representation generated within the speech encoder. In some embodiments, the multiple intermediate speech representations may be generated by applying multiple successive transformer layers, each generating a corresponding intermediate speech representation. The method may include, in order to determine the multiple intermediate speech representations, transforming the frame size of the input time-frequency representation to the transformer layer size by a second multilayer perceptron to pass the input time-frequency representation to multiple successive transformer layers.

[0035] In some embodiments, the method may further include combining additional intermediate speech representations from a second multilayer perceptron and multiple intermediate speech representations from multiple consecutive transformer layers as conditional information to be provided to the language model. For example, the additional intermediate speech representations and multiple intermediate speech representations can be combined by concatenating one or more features from the second multilayer perceptron and the corresponding transformer layers into a feature vector and applying a learnable projection to the feature vector using a linear layer. Alternatively, the additional intermediate speech representations and multiple intermediate speech representations can be combined by summing one or more features from the additional multilayer perceptron and the corresponding transformer layer using corresponding weight coefficients determined by applying an attention algorithm.

[0036] As configured as described above, by applying classifier-free guidance during the inference phase to adjust (e.g., via controllable parameters provided by the CFG) the dependence of the generation process on mixing, different types of distortions (such as bandwidth expansion, spectral defect repair, packet loss concealment, etc.) caused by neural network processing on the generated clean speech can be effectively reduced, which enhances the performance of the speech recovery system in a computationally efficient manner.

[0037] One aspect of this disclosure relates to a system for speech recovery, particularly for training (e.g., during the training phase) and using (e.g., during the inference phase) a neural network-based language model in a speech recovery system to recover clean speech from a distorted speech waveform. Specifically, the proposed system can be implemented to perform the proposed training method and / or the proposed inference method as described above.

[0038] The system may include a neural network-based multi-stage speech coding unit configured to obtain an input time-frequency representation of a distorted speech waveform and determine one or more audio features associated with the distorted speech waveform based on the input time-frequency representation. The system may further include a neural network-based language model configured to generate a plurality of discrete lexical units of a codemap corresponding to a clean speech signal. The system may also include a decoding unit configured to generate a recovered speech waveform associated with the clean speech signal from the plurality of discrete lexical units of the codemap. Furthermore, the system may include a fusion element configured to generate conditional information for the language model based on one or more audio features. Specifically, the plurality of discrete lexical units of the codemap may be generated at least partially based on the conditional information.

[0039] In some embodiments, the speech coding unit may include a multilayer perceptron configured to generate a predicted time-frequency representation of a clean speech waveform based on multiple intermediate speech representations of the input time-frequency representation. Furthermore, the speech coding unit may be further configured to determine one or more semantic representations based on one or more audio features. In some embodiments, the speech coding unit may further include multiple consecutive transformer layers, each configured to generate a corresponding intermediate speech representation. In some embodiments, the speech coding unit may further include a second multilayer perceptron configured to transform the frame size of the input time-frequency representation to the transformer layer size, thereby passing the input time-frequency representation to multiple consecutive transformer layers.

[0040] In some embodiments, the fusion element may be configured to combine additional intermediate speech representations from a second multilayer perceptron and multiple intermediate speech representations from multiple successive transformer layers as conditional information to be provided to the language model. In some embodiments, the system may further include a word segmentation unit configured to segment a reference waveform associated with a ground truth speech signal to convert the reference waveform into a reference codemap of a predefined size containing reference discrete lexical units for training the language model.

[0041] In some embodiments, the speech coding unit and the language model may be jointly trained based on a regression loss function and / or a cross-entropy function. In some embodiments, the language model may include an autoregressive language model or a masked language model.

[0042] In some embodiments where the language model includes a masked language model, the system may further include a masking unit configured to randomly mask one or more reference discrete lexical units in a reference codemap to generate a partially masked codemap; an embedding unit configured to convert the partially masked codemap into a plurality of embedding vectors; and a summing unit configured to sum the plurality of embedding vectors and provide the sum of the summed embedding vectors and conditional information to the masked language model as training input.

[0043] In some embodiments, the masked language model may include multiple transformer blocks followed by a multilayer perceptron configured to, in order to train the masked language model, extract multiple masked lexical units from the ground truth speech signal, estimate multiple predicted indices corresponding to the masked lexical units, determine the difference between the predicted and actual indices of the masked lexical units, and minimize the difference between the predicted and actual indices of the masked lexical units based on a cross-entropy loss function.

[0044] In some embodiments, the system may further include a logit estimation unit configured to compute a first logit of a first predicted lexical unit generated from the language model using conditional information, and subsequently compute a second logit of a second predicted lexical unit generated from the language model without conditional information, or vice versa. Furthermore, the logit estimation unit may be further configured to combine the first and second logits using controllable parameters to obtain a guided logit. Specifically, the controllable parameters may indicate the degree to which the generation of the recovered speech waveform associated with the clean speech signal depends on conditional information. Additionally, multiple discrete lexical units of the codemap can be generated using a classifier-free guided (CFG) based on an iterative sampling procedure. Specifically, the iterative sampling procedure may include, for each iteration, sampling one or more output lexical units of the language model generated from previous iterations to remask the guided logit.

[0045] With the configuration described above, the proposed system can perform efficient speech recovery and allows for the generation of quality-enhanced speech in the presence of noise and various types of distortion. It should also be recognized that the proposed system can be used to perform the processing steps of the training and inference methods described above.

[0046] In view of the foregoing, this disclosure proposes improved techniques for methods and systems for speech restoration, which remove speech distortion in a computationally efficient and reliable manner. In particular, this disclosure proposes a deep generative method that eliminates distortion / artifacts introduced by neural network models during the restoration process with reduced computational effort.

[0047] According to another aspect, an apparatus for speech recovery is provided. The apparatus may include a processor and a memory coupled to the processor and storing instructions for the processor. The processor may be configured to perform all steps of the method according to the foregoing aspects and embodiments thereof.

[0048] According to a further aspect, a computer program is described. This computer program may include executable instructions that, when executed by a computing device (e.g., a processor), are used to perform the methods or method steps outlined throughout this disclosure.

[0049] According to another aspect, a computer-readable storage medium is described. This storage medium can store a computer program adapted to execute on a computing device (e.g., a processor) and, when executed on the computing device, to perform the methods or method steps outlined throughout this disclosure.

[0050] It should be noted that the methods and systems outlined in this disclosure (including their preferred embodiments) can be used independently or in combination with other methods and systems disclosed in this document. Furthermore, all aspects of the methods and systems outlined in this disclosure can be combined arbitrarily. In particular, the features of the claims can be combined with each other in any manner.

[0051] It will be recognized that the features of the apparatus and the steps of the method can be interchanged in various ways. In particular, as those skilled in the art will recognize, the details of the disclosed (multiple) methods can be implemented by the corresponding apparatus, and vice versa. Furthermore, any statement above regarding the (multiple) methods (and, for example, their steps) should be understood to apply equally to the corresponding apparatus (and, for example, its blocks, stages, units), and vice versa. Attached Figure Description

[0052] The invention will now be explained by way of example with reference to the accompanying drawings, in which: Figure 1 An example apparatus / system for speech recovery during a training phase according to an embodiment of the present disclosure is illustrated schematically; Figure 2 An embodiment according to the present disclosure is illustrated schematically. Figure 1 Detailed implementation of an example device / system for speech recovery; Figure 3 Another example apparatus / system for speech recovery taking into account semantic information, according to an embodiment of the present disclosure, is illustrated schematically; Figure 4 An example apparatus / system for speech recovery during the inference phase according to an embodiment of the present disclosure is schematically illustrated; Figure 5 An embodiment according to the present disclosure is illustrated schematically. Figure 4 Detailed implementation of an example device / system for speech recovery; Figure 6 An example flowchart of a method 600 for speech recovery during a training phase according to an embodiment of the present disclosure is shown; Figure 7 An example flowchart of a method 700 for speech recovery during an inference phase according to an embodiment of the present disclosure is shown; and Figure 8 An example apparatus for implementing the proposed speech recovery method according to an embodiment of the present disclosure is illustrated schematically. Detailed Implementation

[0053] In the following description, exemplary embodiments of the present disclosure will be described with reference to the accompanying drawings. The same elements in the drawings may be indicated by the same reference numerals, and repeated descriptions thereof may be omitted.

[0054] Figure 1 An example apparatus / system for speech recovery according to an embodiment of the present disclosure is schematically illustrated. In particular, the apparatus / system 100 can operate during a training phase of the speech recovery process, for example, for training a neural network-based language model to recover clean speech from a distorted speech waveform.

[0055] System 100 (as a speech recovery system) may be a computing device (e.g., a computer, mobile phone, etc.), a server (e.g., a computer or computer cluster), or a distributed system between computing devices and servers suitable for performing the speech recovery process, or may be a part thereof. Specifically, speech recovery system 100 includes a speech coding unit 10 (e.g., a neural network-based multistage speech encoder that can be configured to receive or acquire distorted speech waveforms), a machine learning model 11 to be trained for speech recovery (e.g., a neural network-based language model), and a fusion element 12 (as a fusion network) coupled between the multistage speech coding unit 10 and the neural network-based language model 11.

[0056] Specifically, the speech coding unit 10 can be configured to obtain an input time-frequency (time-spectrum) representation of a distorted speech waveform. In some embodiments, the speech coding unit 10 can acquire an artificially created distorted speech waveform and convert it into a spectrogram (e.g., by means of a Short-Time Fourier Transform, STFT). For example, the speech coding unit 10 can acquire an artificially created distorted speech waveform at a sampling rate of 44.1 kHz and convert it into a 1025-dimensional amplitude spectrogram with a frame length of 2048 points and a hop size of 512 points (the size of an STFT frame). It should be noted that, as expressly stated in this disclosure, the foregoing example sampling rates and spectrogram dimensions are non-limiting examples for obtaining an input time-frequency representation of a distorted speech waveform, and other sampling rates and spectrogram dimensions are feasible and within the scope of this disclosure.

[0057] Furthermore, the speech coding unit 10 can also determine one or more audio features 2-1, 2-2, ..., 2-N (e.g., semantic representations or other non-semantic representations) associated with the distorted speech waveform used to train the speech recovery system 100. The one or more audio features 2-1, 2-2, ..., 2-N can be determined based on the input time-frequency representation 1 obtained by the speech coding unit 10. The determined one or more audio features 2-1, 2-2, ..., 2-N can then be provided to the fusion element 12, which can subsequently (at least partially) determine conditional information 4 for the neural network-based language model 11 based on the one or more audio features 2-1, 2-2, ..., 2-N.

[0058] During the training phase of the speech recovery system 100, the speech coding unit 10 and the neural network-based language model 11 can be jointly trained (using, for example, regression loss and / or cross-entropy loss) to recover high-quality denoised speech from distorted speech waveforms, which may include different types of distortion, such as spectral defects, missing high-frequency components, speaker voice alteration, partial word truncation, etc. Specifically, the neural network-based language model 11 can be trained (at least partially) based on the conditional information 4 provided by the fusion element 12. Alternatively, the speech coding unit 10 and the neural network-based language model 11 can also be trained separately (e.g., based on their respective loss functions).

[0059] Specifically, conditional information 4 can be randomly provided to the neural network-based language model 11 for conditional training. Alternatively, a portion of the conditional information 4 can be replaced with one or more learnable embedding parameters (e.g., using an embedding table that can be provided in the language model 11) as unconditional input to unconditionally train the language model 11. Therefore, the neural network-based language model 11 can be further trained based on unconditional input including one or more learnable embedding parameters.

[0060] In other words, the training of language model 11 can include both conditional and unconditional training. As an example of implementation, the number of training cycles used for conditionally training language model 11 is less than the total number of training cycles used for training language model 11. In particular, the ratio of the number of training cycles used for conditionally training language model 11 to the total number of training cycles can correspond to a predetermined ratio, or the number of training cycles used for unconditionally training language model 11 can be given by a predetermined ratio.

[0061] It should be recognized that the neural network-based language model 11 can use discrete lexical units (such as words in natural language processing) as its input / output. Therefore, in order to utilize the language model 11, audio segmentation can be additionally performed by the speech recovery system 100. As a preferred embodiment according to this disclosure, the speech recovery system 100 may further include a segmentation unit 13 configured to segment a reference waveform associated with the true speech signal 3.

[0062] Specifically, the word segmentation unit 13 can obtain / receive a reference waveform associated with the true speech signal 3 and convert the reference waveform into a code of a predefined size containing discrete tokens. Figure 5 , for use in training language model 11. Code Figure 5The size can be represented by K × T, where K is the number of codebooks and T is the number of frames determined based on the segmentation unit 13. For example, such a segmentation unit 13 can be implemented by a pre-trained neural codec that can perform both encoding and decoding functions. However, other types of segmenters and / or other methods for audio segmentation are feasible and within the scope of this disclosure.

[0063] Preferably, the masked language model can be used to implement a neural network-based language model 11, as will be described below. Figure 2 As described in the text. Furthermore, the cross-entropy loss function can be applied to train neural network-based language models11 (e.g., masked language models), as follows: Figure 2 As stated above.

[0064] Figure 2 An example apparatus / system for speech recovery according to embodiments of the present disclosure is schematically illustrated. Similar to... Figure 1 The system architecture shown (i.e., speech recovery system 100a) can be a computing device (e.g., a computer, mobile phone, etc.), a server (e.g., a computer or computer cluster), or a distributed system between a computing device and a server suitable for performing the speech recovery process, or can be part of such a system and can operate during the training phase of the speech recovery process, for example, for training a neural network-based language model (e.g., a masked language model) to recover clean speech from distorted speech waveforms. Furthermore, speech recovery system 100a can include components similar to those in speech recovery system 100, such as a speech coding unit 10, a neural network-based language model 11, a fusion element 12, and a word segmentation unit 13, which perform corresponding functions similar to those in speech recovery system 100. For the sake of brevity, the details of these components in speech recovery system 100a that are similar to those in speech recovery system 100 will not be repeated herein.

[0065] The speech recovery system 100a can be considered as an example implementation of the speech recovery system 100. More specifically, the speech encoding unit 10 of the speech recovery system 100a can be further configured to generate multiple intermediate speech representations (now shown) of the input time-frequency representation 1 (or associated with the input time-frequency representation) to determine one or more audio features 2-1, 2-2, ..., 2-N. As an example implementation of generating intermediate speech representations, the speech encoding unit 10 of the speech recovery system 100a can be configured to apply multiple consecutive transformer layers (or blocks) 101 (e.g., three transformer layers 101-1, 101-2, 101-3), and each transformer layer can generate a corresponding intermediate speech representation. Furthermore, the speech encoding unit 10 of the speech recovery system 100a can also be configured to pass the input time-frequency representation 1 to multiple consecutive transformer layers 101 by transforming the frame size of the input time-frequency representation 1 to the transformer layer size via a multilayer perceptron (MLP) 102-1 to determine additional intermediate speech representations (among the multiple intermediate speech representations).

[0066] Therefore, the intermediate speech representation from the corresponding transformer layer 101 and the additional intermediate speech representation from the multilayer perceptron 102-1 can then be provided to the fusion element 12 to determine the conditional information 4. For example, the conditional information 4 can be determined by combining the additional intermediate speech representation from the multilayer perceptron 102-1 and multiple intermediate speech representations from multiple consecutive transformer layers 101, and this conditional information can then be used to train the language model 11 as described above.

[0067] As an example of the speech coding unit 10 implementing the speech recovery system 100a, a multilayer perceptron (MLP) 102-1 may comprise two affine layers with a GELU (Gaussian Error Linear Unit) nonlinear activation function between them, which transforms each 1025-dimensional STFT frame to D dimensions (D = 512 for example). The D-dimensional speech representation can then be passed through a stack of N multi-head self-attention transformer blocks 101 (N = 6 for example), which models local and global spectral and temporal speech patterns. Those skilled in the art will recognize that other spectrogram dimensions and other numbers of transformer blocks / layers are equally applicable and within the scope of this disclosure.

[0068] Furthermore, the speech encoding unit 10 of the speech recovery system 100a can be configured to further obtain a reference time-frequency representation of the true speech waveform 1' (e.g., the true speech STFT) for the distorted speech waveform. Moreover, the speech encoding unit 10 can be configured to further apply another (second) multilayer perceptron 102-2 to generate a predicted time-frequency representation of the clean speech waveform (e.g., a predicted clean STFT). Therefore, the speech encoder 10 of the speech recovery system 100a can be trained using, for example, a regression loss 103 based on the reference time-frequency representation and the predicted time-frequency representation predicted by the speech encoder 10. For example, the speech encoding unit 10 can be trained by minimizing the difference between the predicted time-frequency representation and the reference time-frequency representation. This difference can be minimized, for example, by adapting one or more parameters of the speech encoding unit 10 based on a regression loss function.

[0069] As an example of the speech coding unit 10 implementing the speech recovery system 100a, another multilayer perceptron (MLP) 102-2 can map the D-dimensional output features from the last transformer block back to 1025 dimensions (e.g., the dimension of the input spectrogram) to generate a predicted amplitude spectrogram (e.g., a predicted clean STFT). During training, given ground truth speech 1', any regression loss function can be used to minimize the distance between the predicted clean speech STFT and the ground truth speech STFT. Such examples can include simple L1 or L2 losses, or other more advanced losses, such as multi-STFT losses. Additionally, the STFT can be transformed back to the waveform domain via inverse STFT to compute losses in the waveform (time) domain (e.g., L1, L2, or log / MSE (mean squared error) losses, SI-SNR (scale-invariant signal-to-noise ratio) losses, etc.). Furthermore, the STFT can be grouped into perceptually excited band domains to compute losses defined on band energy (e.g., the mel band).

[0070] It is further recognized that the speech encoder (e.g., the speech coding unit 10 of the proposed speech recovery systems 100, 100a) can be built on any architecture, not necessarily a transformer block. Moreover, the regression loss used to train the speech encoder can be any loss defined in any domain, and the speech encoder can be trained alone or jointly with the language model.

[0071] It should be noted that the function of the speech encoder (e.g., the speech coding unit 10 of the proposed speech recovery systems 100, 100a) is to preprocess distorted speech, making the intermediate speech representation cleaner than the original input. This can be achieved by performing speech denoising / dereverberation tasks, for example, using a standalone speech encoder. However, training the speech encoder solely with regression loss (in any domain) may not be effective in performing generative speech recovery tasks, such as bandwidth expansion, spectral defect repair, packet loss concealment, etc. Therefore, by training the speech encoder solely as described above, the proposed speech recovery system can preprocess noisy and distorted speech for the recovery process, and in order to deliver the final output (the recovered clean speech), this disclosure develops further methods to enhance the capabilities of the proposed speech recovery system.

[0072] It should be further noted that the fusion element (as a fusion network) 12 can serve as a bridge between the speech encoder 10 and the language model 11. In other words, the fusion element can fuse intermediate speech representations from all transformer blocks 101 and from the initial MLP 102-1, and send the fused features to the language model 11 for further processing. The fusion network can obtain features only from the last transformer block. However, since the speech encoder may cause distortion (e.g., over-suppressing clean speech) when removing noise from noisy input, the last transformer layer may not provide the optimal speech representation. Therefore, two example methods for implementing the fusion element 12 of the speech recovery system 100a are presented below, which use features from one or more (or all) transformer blocks 101, including the output from the initial MLP layer 102-1.

[0073] A. Concatenation: This method concatenates features from all N+1 layers, thereby generating D for each frame. (N+1) dimensional feature vectors. Then, the linear layer will... (N+1) elements are projected onto D dimensions. Therefore, all features from the N+1 layers are combined through learnable projections.

[0074] B. Attention: The attention mechanism is designed to learn the relative importance of each layer in the speech encoder from the data. Furthermore, this relative importance depends on the design time and channels. Specifically, for each frame from time t, it contains tensors... Features from all N+1 layers, and perform the following processing:

[0075]

[0076]

[0077] in, and It is a learnable projection matrix that first projects the D-dimensional feature vector at each frame t from any of the N+1 layers to an M-dimensional vector (for example, D = 512, M = 128), and then projects the resulting feature vector back to D-dimensionality, using a non-linear function in between. (For example, the ReLU (Revised Linear Unit) function). After normalization along the layer dimensions using the softmax function, we can obtain N+1 attention weights for each frame and each channel (a channel refers to an element in a D-dimensional feature vector). For each of the D dimensions, the tensor... These attention weights in the process sum to 1 across N+1 layers. Finally, execution... and Element-wise multiplication (by The resulting tensors are summed along the layer dimensions to obtain the aggregated feature vector for frame t. Therefore, each aggregated feature vector at time t is a weighted sum of all features from the N+1 layers at the same frame t, and the weights are learned through an attention algorithm.

[0078] Therefore, the fusion element 12 of the speech recovery system 100a can be configured to combine additional intermediate speech representations and multiple intermediate speech representations by concatenating one or more features from the multilayer perceptron 102-1 and the corresponding transformer layer 101 into a feature vector and applying a learnable projection to this feature vector using a linear layer. Alternatively, the fusion element 12 of the speech recovery system 100a can be configured to combine additional intermediate speech representations and multiple intermediate speech representations by summing one or more features from the multilayer perceptron 102-1 and the corresponding transformer layer 101 using corresponding weight coefficients determined by applying an attention algorithm.

[0079] It should be noted that, as expressly stated in this disclosure, the aforementioned example speech encoder 10 (speech coding unit) and fusion element (fusion network) 12 of the proposed speech recovery system 100a are non-limiting examples for recovering distorted speech waveforms. Other architectures and methods for implementing the speech encoder and fusion network are feasible and within the scope of this disclosure.

[0080] Figure 2Further illustrations depict detailed example implementations of a neural network-based language model 11 according to embodiments of the present disclosure. Herein, the language model 11 may include a masked language model 114, and a masking module 111 may be provided for generating training inputs for the masked language model 114. More specifically, the masking module 111 may be configured to randomly mask (output by the word segmentation unit 13 as described above) codes. Figure 5 One or more discrete lexical units are used to generate a partially masked codemap. Furthermore, one or more embedding tables 112 can be applied to convert the partially masked codemap into multiple embedding vectors, which can then be summed. A summing unit 113 can also be provided to sum the (summed) embedding vectors with conditional information 4 to provide training input to the masked language model 114.

[0081] Similar to encoding unit 10, masked language model 114 may include multiple transformer blocks followed by a multilayer perceptron (not shown). Therefore, in addition to encoding unit 10, the masked language model can also be trained based on natural language processing. Specifically, to train masked language model 114, multiple masking lexical units can be extracted from the ground truth speech signal 3, and multiple predicted indices corresponding to the masking lexical units can then be estimated (e.g., via the multilayer perceptron within the language model). The difference between the predicted indices and the actual indices 6 of the masking lexical units can then be determined, and this difference can be minimized based on the cross-entropy loss function 14.

[0082] As an example of the language model 11 implementing the speech recovery system 100a, MaskGIT (mask-generated image transformer) can be used as the backbone algorithm of the language model. Specifically, during training, the ground truth clean speech 3 can be converted into a code of size 9 × T by the word segmentation unit 13, such as the Descript audio codec (DAC) encoder / decoder described below. Figure 5 Furthermore, random masking (DAC) codes can be achieved by replacing the tokens to be masked with a special token index of 1024 (because the original token index ranges from 0 to 1023). Figure 5The percentage of (randomly selected) words in the codebook is then used. The partially masked codemap is then transformed into nine tensors, each representing a codebook, using nine learnable embedding tables (each with 1025 entries) 112. Each tensor is T × D in size, where T is the number of frames and D is the dimension of the transformer layer. Furthermore, each D-dimensional feature vector in a row of this tensor can embed words from the codebook at frame t, and the embeddings from all nine tensors can be summed at the same time instance to produce a tensor of size T × D containing information about all unmasked words. Finally, the speech representation from the speech encoder 10, merged by the fusion network 12, can also be summed with the DAC embeddings.

[0083] The sum of the DAC embeddings and the speech encoder embeddings can then be fed into a masked language model 114, which may include L transformer blocks (L = 12 for example), followed by an MLP layer (not shown). The MLP layer can predict the indices of masked lemmas extracted from ground truth clean speech, with a vocabulary size of 9 × 1024. Therefore, the language model 11 and the speech encoder 10 can be trained by jointly optimizing the cross-entropy loss 14 and the regression loss 103.

[0084] As a further example, once further conditioned on the target speaker embedding in both the speech encoder and the language model, the system can perform target speaker extraction, which aims to extract the target speaker's voice from a distortion mixture containing multiple speakers. This conditioned state can be implemented in all transformer blocks via adaptive layer normalization.

[0085] It should be noted that, as expressly stated in this disclosure, the foregoing example masking language model and loss function are non-limiting examples of the neural network-based language model 11 used for implementing and training the speech recovery system 100a. Other types of loss functions and other types of language models (e.g., large language models, such as autoregressive or non-autoregressive language models, etc.) may also be utilized and are within the scope of this disclosure. Furthermore, in some embodiments, the language model 11 and the speech encoder 10 may also be trained separately.

[0086] Figure 3Another example apparatus / system for speech recovery according to an embodiment of the present disclosure is schematically illustrated. Similar to speech recovery systems 100, 100a, speech recovery system 100b may be a computing device (e.g., a computer, mobile phone, etc.), a server (e.g., a computer or computer cluster), or a distributed system between a computing device and a server suitable for performing a speech recovery process, or may be part of such a system, and may operate during the training phase of the speech recovery process, for example, for training a neural network-based language model to recover clean speech from a distorted speech waveform. Furthermore, speech recovery system 100b may include components similar to those of speech recovery system 100, such as a speech encoding unit 10, a neural network-based language model 11, a fusion element 12, and a word segmentation unit 13, which perform corresponding functions similar to those of the speech recovery system 100 described above. In particular, the speech encoding unit 10 may also determine one or more audio features 2-1, 2-2, ..., 2-N associated with the distorted speech waveform used to train speech recovery system 100b.

[0087] The speech recovery system 100b can also be considered as an example implementation of the speech recovery system 100. In the speech recovery system 100b, semantic representations (e.g., truth speech 3) extracted from the target speech can be predicted to introduce semantic knowledge distillation (KD) into the speech encoder 10, thereby improving the speech intelligibility of the speech recovery system 100b.

[0088] More specifically, the speech coding unit 10 of the speech recovery system 100b can be further configured to obtain a reference time-frequency representation of the true speech waveform (e.g., true speech 3) for the distorted speech waveform. Furthermore, in the speech recovery system 100b, one or more semantic representations 304 associated with the distorted speech waveform can be determined based on one or more audio features 2-1, 2-2, ..., 2-N already determined by the speech encoder 10 according to the input time-frequency representation 1 (e.g., by the projection layer 301).

[0089] Additionally, the speech recovery system 100b may further include a pre-trained auxiliary model (e.g., a teacher model) 302 to determine one or more reference semantic representations 305 associated with the ground truth speech waveform 3 based on the reference time-frequency representation. Therefore, the speech encoder 10 of the speech recovery system 100b can be trained using an auxiliary loss function 303 based on one or more semantic representations 304 and one or more reference semantic representations 305.

[0090] As an example of training the speech recovery system 100b, one or more semantic representations 304 can be determined from one or more audio features 2-1, 2-2, ..., 2-N by applying a fully connected pooling layer 301 to project (normalized) audio features to the desired dimension. In a more specific example, the one or more semantic representations 304 may include one or more pseudo-speech labels, and one or more reference semantic representations 305 may include one or more reference pseudo-speech labels. Furthermore, the pre-trained auxiliary model 302 can be implemented by a self-supervised model. For example, the pre-trained auxiliary model 302 could be a pre-trained HuberT model.

[0091] Taking the speech encoder 10 of the speech recovery systems 100 and 100a as an example, the speech encoder 10 of the speech recovery system 100b can further apply fully connected (FC) layers to project normalized features onto d dimensions, and can use a stack of self-attention transformer blocks to learn latent embedding sequences. The output from the last transformer block can be used to tune the language model, as described above. Compared to the speech recovery system 100a, the speech recovery system 100b can employ auxiliary loss to train the speech encoder 10, aiming to learn semantic content without transcribed data. Furthermore, a pre-trained HuBERT base model can be considered to allow self-supervised pre-training to encode semantic information by predicting masked pseudo-speech labels derived solely from audio. For example, given a paired 44.1 kHz target speech signal, a frozen HuBERT target sequence can be extracted by downsampling the target speech signal to 16 kHz. Due to HuBERT's low frame rate, an average pooling layer can be appended after the last transformer block in the speech encoder 10 to align the sequence, and the FC layer can output the predicted sequence. Figure 3 As shown, the pooling layer and the FC layer can form the projection layer 301.

[0092] It should be noted that, as expressly stated in this disclosure, the foregoing example models and architectures are non-limiting examples of training the proposed speech recovery system 100b in consideration of semantic information. Other types of models and architectures for determining semantic information and training the system based on semantic information, as well as other methods, are feasible and within the scope of this disclosure.

[0093] although Figure 2 and Figure 3Two different example implementations of the speech recovery system 100 are illustrated, but those skilled in the art will recognize that these systems 100, 100a, 100b can be combined to incorporate semantic information-based training methods with the speech recovery system 100a. In this way, the speech encoder of the proposed speech recovery system can be further improved to enhance speech intelligibility.

[0094] Figure 4 An example apparatus / system for speech recovery according to an embodiment of the present disclosure is schematically illustrated. In particular, the apparatus / system 200 can operate during the inference phase of the speech recovery process, for example, in which a trained neural network-based language model (e.g., one already used as...) can be employed. Figures 1 to 3 The language models trained by the speech recovery systems 100, 100a, and 100b shown recover clean speech from distorted speech waveforms.

[0095] Similar to systems 100, 100a, and 100b, speech recovery system 200 can be a computing device (e.g., a computer, mobile phone, etc.), a server (e.g., a computer or computer cluster), or a distributed system between computing devices and servers suitable for performing the speech recovery process, or can be a part of such a system. Specifically, speech recovery system 200 includes a speech coding unit 20 (e.g., a neural network-based multistage speech encoder that can be configured to receive or acquire distorted speech waveforms), a machine learning model 21 (which has been trained for speech recovery) (e.g., a neural network-based language model), and a fusion element 22 (as a fusion network) coupled between the multistage speech coding unit 20 and the neural network-based language model 21.

[0096] The speech coding unit 20 and fusion element 22 of the speech recovery system 200 can be implemented using an architecture / model / method similar to that of the speech coding unit 10 and fusion element 12 of the speech recovery systems 100, 100a, and 100b. Correspondingly, the speech coding unit 20 and fusion element 22 of the speech recovery system 200 can also perform functions similar to those of the speech coding unit 10 and fusion element 12 of the speech recovery systems 100, 100a, and 100b, respectively.

[0097] Specifically, speech coding unit 20 can be configured to obtain an input time-frequency (time spectrum) representation of the distorted speech waveform. Furthermore, speech coding unit 20 can determine one or more audio features 2-1, 2-2, ..., 2-N (e.g., semantic representations, or other non-semantic representations) associated with the distorted speech waveform to determine the conditional information 4 to be provided to language model 21. The one or more audio features 2-1, 2-2, ..., 2-N can be determined based on the input time-frequency representation using a method similar to that described above for speech coding unit 10 of system 100.

[0098] Since system 200 can operate during the inference phase, its neural network-based language model 21 can be configured to generate codes corresponding to clean speech signals. Figure 7 Multiple discrete tokens. Specifically, the code... Figure 7 Multiple discrete tokens can be at least partially based on conditional information 4 from a fully masked code that (exclusively) contains, for example, multiple masking tokens. Figure 5 'Generate.' Specifically, code. Figure 7 Multiple discrete lexical units can be generated based on an iterative sampling procedure.

[0099] Furthermore, system 200 may further include a decoding unit 25, which is configured as a slave code. Figure 7 Multiple discrete lexical units generate a recovered speech waveform 8 associated with a clean speech signal. Further, similar to fusion element 12 of system 100, fusion element 22 of speech recovery system 200 can be configured to generate conditional information 4 for language model 21 based on one or more audio features 2-1, 2-2, ..., 2-N to generate a code. Figure 7 Multiple discrete word units.

[0100] It should be recognized that the decoding unit 25 can be integrated with the word segmentation unit 13 described above for training the language model. This can be achieved, for example, by utilizing a pre-trained neural codec (e.g., released by Descript). As an example implementation, the audio (as word segmentation unit 13) associated with the ground truth speech signal 3 can be segmented using a Descript audio codec (DAC) based on a convolutional encoder / decoder architecture, where the encoder converts the input waveform sampled at 44.1 kHz into a latent space, and the decoder converts the latent space back into a waveform to reconstruct the input. For example, a sequence of nine residual vector quantizers between the encoder and decoder can discretize each frame of the latent continuous feature vector into nine discrete tokens, each represented by an integer index ranging from 0 to 1023. Therefore, after word segmentation, the reference waveform 3 can be transformed into a code of size K × T. Figure 5The codemap contains discrete values ​​between 0 and 1023, where K = 9 is the number of codebooks and T is the number of frames, and its hop size can be 512 samples (according to the design of the convolutional layers in the DAC encoder). It should be recognized that the same hop size of 512 can also be used to generate a spectrogram (e.g., when calculating the STFT in speech coding unit 10) to accurately align STFT frames with DAC frames one by one. Therefore, according to this disclosure, this type of word segmenter (e.g., a DAC word segmenter) can also be referred to as a "decoder". Furthermore, the pre-trained DAC can be frozen at any time during the training / inference of the language models 11, 21.

[0101] It should be noted that, as expressly stated in this disclosure, the aforementioned example DAC is a non-limiting example of the word segmentation unit 13 (and decoding unit 25) for implementing the proposed speech recovery system. Other types of decoders and other types of word segmenters and / or other methods for word segmenting audio are also feasible (for replacing the aforementioned DAC word segmenter) and are within the scope of this disclosure.

[0102] Preferably, the masked language model can be used to implement a neural network-based language model 21, which will be as follows: Figure 5 As described in the text. Additionally, classifier-free guidance (CFG) can be applied to mitigate distortion caused by neural network-based speech recovery systems, as described below. Figure 5 As described in [the text].

[0103] The CFG technique was developed to improve the alignment between the generative model and the conditions. According to this disclosure, the conditions used for the language model may include fused speech encoder features (such as those output by fusion elements 12, 22), which may contain representations of various levels of input mixtures. Furthermore, CFG can be used to reduce distortion in the generated clean speech caused by neural network processing using the proposed speech recovery system, because CFG allows for controllable parameters to adjust the degree to which the generation process depends on the mixtures.

[0104] Figure 5 An example apparatus / system for speech recovery according to embodiments of the present disclosure is schematically illustrated. Similar to... Figure 4 The system architecture shown (i.e., speech recovery system 200) indicates that device / system 200a may be a computing device (e.g., computer, mobile phone, etc.), a server (e.g., computer or computer cluster), or a distributed system between computing devices and servers suitable for performing the speech recovery process, or may be part of such a system and may operate during the inference phase of the speech recovery process. For example, during the inference phase, a trained neural network-based language model (e.g., one already used as...) may be employed. Figures 1 to 3The speech recovery systems 100, 100a, and 100b illustrated in the diagram use language models trained to recover clean speech from distorted speech waveforms. Furthermore, speech recovery system 200a may include components similar to those in speech recovery system 200, such as a speech coding unit 20, a neural network-based language model 21, a fusion element 22, and a decoding unit 25, which perform similar functions to those in speech recovery system 200. For the sake of brevity, details of these components similar to those in speech recovery system 200 will not be repeated here.

[0105] The speech recovery system 200a can be considered as an example implementation of the speech recovery system 200. Similar to the speech recovery system 100a, the speech recovery system 200a may also include a masked language model 214, one or more embedding tables 212, and a summing unit 213, which perform corresponding functions similar to those of the masked language model 114, one or more embedding tables 112, and summing unit 113 in the system 100a described above.

[0106] In order to perform an iterative sampling procedure to generate codes Figure 7 Given multiple discrete lexical units, system 200a can consider the logit (e.g., logit score) of the lexical units output from the language model to sample the output lexical units of language model 214. More specifically, system 200a may further include a logit scoring module 215 configured (for each iteration) to calculate a first logit of a first predicted lexical unit generated from language model 214 using conditional information 4 (as determined by fusion element 22), and subsequently calculate a second logit of a second predicted lexical unit generated from language model 214 without conditional information 4, or vice versa. For each iteration, logit scoring module 215 may be further configured to combine the first and second logits using controllable parameters to obtain a guided logit. The controllable parameters may indicate the degree to which the generation of the recovered speech waveform 8 associated with the clean speech signal depends on conditional information 4. Subsequently, one or more output lexical units of language model 214 generated from previous iterations can be sampled for remasking based on the guided logit.

[0107] In addition, the code that is completely masked as described above can be obtained. Figure 5(The codemap may (exclusively) contain, for example, multiple masking terms), and the codemap can then be converted into multiple embedding vectors (via embedding table 212). The multiple embedding vectors can then be summed (via summing unit 213), and the sum of the summed embedding vectors and conditional information can be provided to the masked language model 214 as input based on the guided logit. Therefore, the language model 214 can be configured to predict multiple masking terms to predict / generate codes. Figure 7 Multiple discrete lexical units (e.g., by sampling the output lexical units of language model 214 to remask based on guided logit).

[0108] Furthermore, to predict multiple discrete word units, the iterative sampling procedure performed by the speech recovery system 200a may further include determining prediction accuracy and prediction gain for each predicted masking word unit. Therefore, multiple discrete word units can be generated by comparing the corresponding prediction accuracy and / or corresponding prediction gain associated with the predicted masking word unit with predefined values. For example, if the corresponding prediction accuracy and / or corresponding prediction gain associated with the predicted masking word unit is greater than or equal to the predefined value, multiple discrete word units can be predicted by retaining the predicted masking word unit. Alternatively, if the corresponding prediction accuracy and / or corresponding prediction gain associated with the predicted masking word unit is less than the predefined value, multiple discrete word units can be predicted by discarding the predicted masking word unit. Subsequently, the discarded masking word unit can be re-predicted by the masked language model 214.

[0109] More specifically, in order to re-predict the discarded masking words, the fully masked code can be updated based on the retained predicted masking words and the discarded masking words. Figure 5 Furthermore, the updated fully masked codemap can then be converted into multiple updated embedding vectors (via embedding table 212). Additionally, the guided logit can be obtained (e.g., by the logit scoring module 215) based on the output of the masked language model 214 in the absence of conditional information 4. Subsequently, the multiple updated embedding vectors can be summed, and the sum of the summed updated embedding vectors and conditional information 4 can be used as the updated input to the masked language model 214 for prediction, based on the (obtained) guided logit.

[0110] As an example of implementing CFG in speech recovery system 200a, predicted lexical units generated during the training and inference phases can be considered to determine the guided logit. Specifically, during training, the fused speech encoder features can be randomly discarded 10% of the time, which is achieved by replacing them with learnable embeddings of the unconditional language model with labels (as described above for systems 100, 100a, and 100b). During inference, the logit of the lexical units (the output of the language model) can be computed with and without the fused speech encoder features (e.g., conditional information 4). The corresponding logits of the lexical units can then be combined according to the following formula to obtain the guided logit:

[0111] in, and are the logit values ​​of the predicted lexical units under conditional and unconditional conditions, respectively, and w is the positive leading level. It should be recognized that a larger w may imply a greater dependence on the fused speech encoder representation (e.g., the combined output features from fused elements 12 and 22).

[0112] It should be noted that distortions in the processed speech caused by neural networks can degrade the speech enhancement performance of a speech recovery system, even if the neural network can successfully remove noise. As indicated above, distortions can include spectral defects, missing high-frequency components, altered speaker voice, truncated parts of words, etc. To mitigate the impact of such distortions in speech recovery, this disclosure proposes the use of classifier-free guidance as described above. It should be particularly noted that a guidance level w can be defined to determine the degree to which the generation of the recovered speech waveform is conditionally dependent; for example, a larger guidance w can maintain a more natural speech, at the cost of bringing back a moderate amount of noise. In practice, the guidance level w can be selected / chosen by listening to and measuring objective quality metrics as a function of w.

[0113] During inference processing, as described above, the language model (e.g., masked language model 214) can use the fused speech representation from the fusion network 22 (in conjunction with the speech encoder 20) as condition 4 for the generation process. This language model can employ fully masked codes. Figure 5 The pre-trained DAC decoder 25, as described above, performs an iterative sampling procedure to gradually generate one or more (or all) lexical units of clean speech. For example, the iterative sampling procedure according to this disclosure can be based on MaskGIT. Furthermore, classifier-free guidance can be used to compute the logit of the predicted lexical units as described above. Finally, the pre-trained DAC decoder 25 can convert the generated clean code... Figure 7 Convert to (clean) waveform 8.

[0114] It should be noted that, as expressly stated in this disclosure, the foregoing example masking language model and CFG method are non-limiting examples for inference using the neural network-based language model 21 within the speech recovery system 200a. Other types of language models (e.g., large language models, such as autoregressive or non-autoregressive language models) and other similar algorithms for determining the logit in guided cases may also be utilized, and are within the scope of this disclosure.

[0115] Figure 6 An example flowchart of a method 600 for speech recovery according to an embodiment of the present disclosure is shown. Method 600 may be implemented in software, hardware, or a combination thereof (e.g., at an audio generation device) as a computing device, a server, or a distributed system between a computing device and a server suitable for performing the speech recovery process, or part thereof, and may operate during a training phase of the speech recovery process. Figure 6 As shown, method 600 includes a processing chain formed by steps S610 to S640 for training a neural network-based language model (e.g., language models 11, 21 as described above) to recover clean speech from a distorted speech waveform.

[0116] Method 600 includes a step S610 of obtaining an input time-frequency representation of a distorted speech waveform. The method further includes a step S620 of determining one or more audio features associated with the distorted speech waveform. One or more audio features can be determined based on the input time-frequency representation using a neural network-based multi-stage speech encoder (e.g., encoding units 10, 20 as described above). As indicated above, the one or more audio features may also be associated with, for example, a semantic representation. Additionally, the method includes a step S630 of determining conditional information for a language model based at least in part on the one or more audio features. Furthermore, the method further includes a step S640 of jointly training the language model and the speech encoder. Specifically, the language model may be trained at least in part on the conditional information.

[0117] Further note that steps S610 to S640 can be executed for each of multiple processing cycles of the voice recovery system (e.g., voice recovery systems 100, 100a, 100b), and do not need to be executed sequentially. Figure 6 Execute in the order shown.

[0118] Figure 7 An example flowchart of a method 700 for speech recovery according to an embodiment of the present disclosure is shown. Method 700 may be implemented in software, hardware, or a combination thereof (e.g., at an audio generation device) as a computing device, a server, or a distributed system between a computing device and a server suitable for performing the speech recovery process, or part thereof, and may operate during the inference phase of the speech recovery process. Figure 7As shown, method 700 includes a processing chain formed by steps S710 to S750 for recovering clean speech from distorted speech waveforms using a neural network-based language model (e.g., language models 11, 21 as described above).

[0119] Method 700 includes step S710 of obtaining an input time-frequency representation of a distorted speech waveform. The method further includes step S720 of determining one or more audio features associated with the distorted speech waveform. One or more audio features can be determined based on the input time-frequency representation using a neural network-based multi-stage speech encoder (e.g., coding units 10, 20 as described above). As indicated above, the one or more audio features may also be associated with, for example, a semantic representation. Additionally, the method includes step S730 of determining conditional information for a language model based at least in part on one or more audio features. Furthermore, the method further includes step S740 of generating a plurality of discrete lexical units of a codemap corresponding to a clean speech signal, based at least in part on the conditional information. Specifically, the plurality of discrete lexical units can be generated from a fully masked codemap containing a plurality of masking lexical units. For example, the plurality of discrete lexical units can be generated from a fully masked codemap that exclusively contains a plurality of masking lexical units. Furthermore, the method includes step S750 of generating a recovered speech waveform associated with a clean speech signal from the plurality of discrete lexical units of the codemap.

[0120] Further note that steps S710 to S750 can be executed for each of multiple processing cycles of the voice recovery system (e.g., voice recovery system 200, 200a), and do not need to be executed sequentially. Figure 7 Execute in the order shown.

[0121] It should be understood that this disclosure also relates to apparatus (e.g., computer-implemented apparatus or apparatus with general processing capabilities) for performing or implementing the methods and techniques described herein. For example, such apparatus may relate to an audio decoder. The audio decoder may be, for example, a low-latency audio decoder suitable for general audio.

[0122] exist Figure 8 An example of such a device 800 is schematically illustrated. Specifically, device 800 includes a processor 810 and a memory 820 coupled to the processor 810. The memory 820 may store instructions for execution by the processor 810. Depending on the use case and / or implementation, the processor 810 may, among other things, receive suitable input data 830 (e.g., input frames of time-frequency coefficients, etc.). Depending on the use case and / or implementation, the processor 810 may be adapted to perform or implement the methods / techniques described in this disclosure and generate corresponding output data 840.

[0123] The apparatus 800 can be configured to perform the functions of the aforementioned components (e.g., speech encoder, fusion network / model, language model, word segmenter / decoder, etc.) of the proposed speech recovery system. For example, according to this disclosure, the apparatus 800 can be configured (e.g., via a speech encoder) to determine features of a speech signal including one or more audio distortions and to fuse the determined features. As indicated above, feature fusion can be performed by employing, for example, a fusion model. In specific embodiments, the fusion model can use splicing and / or attention mechanisms to fuse features. Additionally, the apparatus 800 can also be configured to generate discrete lexical units of a codemap corresponding to a clean speech signal based on the fused features (e.g., via a masked language model). Therefore, the apparatus 800 can be further configured (e.g., via a decoder) to generate a clean speech signal from the discrete lexical units. In some embodiments, the decoder can be, for example, an audio word segmenter as described above.

[0124] This disclosure also relates to corresponding computer programs, computer program products, and computer-readable storage media for storing such computer programs or computer program products.

[0125] Aspects of the methods and apparatus / systems described herein can be implemented in a suitable computer-based audio processing network environment (e.g., a server or cloud environment) to process digital or digitized audio files. A portion of the audio system may include one or more networks comprising any desired number of independent machines, including one or more routers (not shown) for buffering and routing data transmitted between computers. Such networks can be built on a variety of different network protocols and can be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0126] One or more components, blocks, processes, or other functional units may be implemented by a computer program executed by a processor-based computing device controlling the system. It should also be noted that any number of combinations of hardware, firmware, and / or data and / or instructions embodied in various machine-readable or computer-readable media may be used to describe the various functions disclosed herein in terms of their behavior, register transfers, logical components, and / or other characteristics. Computer-readable media that may embody such formatted data and / or instructions include, but are not limited to, various forms of physical (non-transitory), non-volatile storage media, such as optical, magnetic, or semiconductor storage media.

[0127] Specifically, it should be understood that embodiments may include hardware, software, and electronic components or modules, which may be illustrated and described for discussion purposes as if most components were implemented solely in hardware. However, those skilled in the art, and based on a reading of this specific embodiment, will recognize that in at least one embodiment, the electronic aspects may be implemented in software (e.g., stored on a non-transitory computer-readable medium) executable by one or more electronic processors (e.g., microprocessors and / or application-specific integrated circuits (“ASICs”)). Therefore, it should be noted that embodiments may be implemented using multiple hardware and software-based devices and multiple different structural components. For example, the aforementioned apparatus (e.g., an encoder) may include one or more electronic processors, one or more computer-readable medium modules, one or more input / output interfaces, and various connections (e.g., system buses) connecting the various components.

[0128] While one or more embodiments have been described by way of example and specific examples, it should be understood that one or more embodiments are not limited to the disclosed embodiments. Rather, they are intended to cover various modifications and similar arrangements that will be obvious to those skilled in the art. Therefore, the scope of the appended claims should be given the broadest interpretation in order to cover all such modifications and similar arrangements.

[0129] Similarly, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be considered restrictive. The use of “including,” “comprising,” or “having,” and variations thereof, is intended to cover the items listed thereafter and their equivalents, as well as additional items. Unless otherwise specified or limited, the terms “installation,” “connection,” “support,” and “coupled,” and variations thereof, are used extensively and cover direct and indirect installation, connection, support, and coupling.

[0130] Various aspects and embodiments of the present invention can also be understood from the following enumerated exemplary embodiments (EEE), which are not claims.

[0131] EEE A1. A computer-implemented method for speech recovery, comprising: The method involves identifying features of a speech signal that includes one or more audio distortions; fusing the features; generating discrete lexical units of a codemap corresponding to a clean speech signal based on the fused features; and generating the clean speech signal from the discrete lexical units.

[0132] EEE A2. The method according to EEE A1, wherein the feature includes intermediate speech representation.

[0133] EEE A3. The method according to EEE A1 or EEE A2, wherein the clean speech signal includes one or more audio distortions reduced compared to the one or more audio distortions.

[0134] EEE A4. The method according to any one of EEE A1 to EEE A3, wherein spectral patterns and / or speech patterns are modeled to determine the characteristics of the speech signal.

[0135] EEE A5. The method according to any one of EEE A1 to EEE A4, wherein fusing the features includes splicing and / or attention mechanisms.

[0136] EEE A6. The method according to any one of EEE A1 to EEE A5, wherein generating the clean speech signal from the discrete lexical units comprises: generating an audio waveform from the discrete lexical units, wherein the clean speech signal comprises the audio waveform.

[0137] EEE A7. The method according to any one of EEE A1 to EEE A6 further includes: performing classifier-free guidance to improve speech quality and mitigate excessive suppression.

[0138] EEE A8. The method according to any one of EEE A1 to EEE A7, wherein the one or more audio distortions include at least one of spectral defects, missing high-frequency components, missing speech, presence of noise and reverberation, clipping, and / or alteration of the speaker's voice.

[0139] EEE A9. A system for speech recovery, comprising: The speech encoder is configured to determine features of a speech signal including one or more audio distortions; a fusion model is configured to fuse the features; a language model is configured to generate discrete lexical units of a codemap corresponding to a clean speech signal based on the fused features; and a decoder is configured to generate the clean speech signal from the discrete lexical units.

[0140] EEE A10. The system according to EEE A9, wherein the speech encoder and the language model are jointly trained.

[0141] EEE A11. A system according to EEE A9 or EEE A10, wherein the feature includes intermediate speech representation.

[0142] EEE A12. The system according to any one of EEE A9 to EEE A11, wherein the clean speech signal includes one or more audio distortions reduced compared to the one or more audio distortions.

[0143] EEE A13. A system according to any one of EEE A9 to EEE A12, wherein the speech encoder comprises: one or more multilayer perceptrons; and one or more transformer blocks, wherein the one or more transformer blocks are configured to determine features of the speech signal.

[0144] EEE A14. The system according to EEE A13, wherein the one or more transformer blocks model the spectral patterns and / or speech patterns of the speech signal to determine the features.

[0145] EEE A15. A system according to any one of EEE A9 to EEE A14, wherein the speech encoder is trained to minimize a regression loss function or a cross-entropy loss function, and the language model is trained to minimize the regression loss function or the cross-entropy loss function.

[0146] EEE A16. The system according to any one of EEE A9 to EEE A15, wherein the fusion model is configured to fuse the features via splicing and / or via an attention mechanism.

[0147] EEE A17. A system according to any one of EEE A9 to EEE A16, wherein the decoder is configured to generate an audio waveform from the discrete lexical units, wherein the clean speech signal includes the audio waveform.

[0148] EEE A18. The system according to any one of EEE A9 to EEE A17 is further configured to improve speech quality and mitigate excessive suppression via classifier-free guidance.

[0149] EEE A19. A system according to any one of EEE A9 to EEE A18, wherein the one or more audio distortions include at least one of spectral loss, missing high-frequency components, missing speech, presence of noise and reverberation, clipping, and / or alteration of speaker voice.

[0150] EEE A20. The system according to any one of EEE A9 to EEE A19, wherein the language model comprises a non-autoregressive language model.

[0151] EEE A21. The system according to any one of EEE A9 to EEE A20, wherein the language model is a masked language model including a MaskGIT model.

[0152] EEE A22. The system according to any one of EEE A9 to EEE A19, wherein the language model comprises an autoregressive language model.

[0153] EEE A23. An apparatus for speech recovery, the apparatus comprising a processor and a memory coupled to the processor and storing instructions for the processor, wherein the processor is configured to perform all steps of the method according to any one of EEE A1 to EEE A8.

[0154] EEE A24. A computer program comprising instructions that, when executed by a processor, cause the processor to perform all the steps of the method according to any one of EEE A1 to EEE A8.

[0155] EEE A25. A computer-readable storage medium storing a computer program according to EEE A24.

[0156] EEE B1. A computer-implemented method for recovering clean speech from distorted speech waveforms using a neural network-based language model, the method comprising: Obtain the input time-frequency representation of the distorted speech waveform; A neural network-based multistage speech encoder is used to determine one or more audio features associated with the distorted speech waveform based on the input time-frequency representation; Conditional information for the language model is determined at least in part based on the one or more audio features; At least in part based on the conditional information, generate multiple discrete lexical units of a codemap corresponding to a clean speech signal from a fully masked codemap containing multiple masked lexical units; and A recovered speech waveform associated with the clean speech signal is generated from the plurality of discrete lexical units of the codemap.

[0157] EEE B2. The method according to EEE B1, wherein the plurality of discrete tokens of the codemap are generated based on an iterative sampling procedure, the iterative sampling procedure comprising, for each iteration: Calculate the first logit of the first predicted lexical unit generated from the language model using the conditional information, and then calculate the second logit of the second predicted lexical unit generated from the language model without the conditional information, or vice versa; Use controllable parameters to combine the first logit and the second logit to obtain the bootstrap logit; and One or more output lexical units from the language model generated from previous iterations are sampled for re-masking based on the guided logit.

[0158] EEE B3. The method according to EEE B2, wherein the controllable parameter indicates the degree to which the generation of the recovered speech waveform associated with the clean speech signal depends on the conditional information.

[0159] EEE B4. The method according to any one of EEE B1 to B3, wherein the language model comprises an autoregressive language model.

[0160] EEE B5. The method according to EEE B3, wherein the language model includes a masked language model, and the method further includes: Obtain the code image of a fully masked word containing multiple masking terms; The fully masked code map is converted into multiple embedding vectors; Summing the plurality of embedding vectors; Based on the guided logit, the sum of the summed embedding vector and conditional information is provided to the masked language model as input; and The language model predicts the plurality of masking lexical units to predict the plurality of discrete lexical units in the code map.

[0161] EEE B6. The method according to EEE B5, wherein the iterative sampling procedure further includes determining prediction accuracy and prediction gain for each predicted masking word, and wherein predicting the plurality of discrete words includes: If the corresponding prediction accuracy and / or corresponding prediction gain associated with the predicted masking term is greater than or equal to a predefined value, then the predicted masking term is retained, or If the corresponding prediction accuracy and / or corresponding prediction gain associated with the predicted masking term is less than a predefined value, then the predicted masking term is discarded; and The discarded masking lexical units are re-predicted by the masking language model.

[0162] EEE B7. According to the method described in EEE B6, wherein re-predicting the discarded masking terminology includes: The fully masked code map is updated based on the retained predicted masking words and the discarded masking words; The updated fully masked codemap is converted into multiple updated embedding vectors; The guided logit is obtained based on the output of the masked language model in the absence of the aforementioned conditional information; Sum the multiple updated embedding vectors; Based on the guided logit, the sum of the summed and updated embedding vectors and the conditional information is provided to the masked language model as an updated input for prediction.

[0163] EEE B8. The method according to any one of EEE B1 to B7, wherein the one or more audio features are determined based on a plurality of intermediate speech representations of the input time-frequency representation generated within the speech encoder.

[0164] EEE B9. The method according to EEE B8, wherein the plurality of intermediate speech representations are generated by applying a plurality of successive transformer layers, each generating a corresponding intermediate speech representation.

[0165] EEE B10. The method according to EEE B9 further includes: in order to determine the plurality of intermediate speech representations, a second multilayer perceptron transforms the frame size of the input time-frequency representation to the transformer layer size to pass the input time-frequency representation to the plurality of consecutive transformer layers.

[0166] EEE B11. The method according to EEE B10 further includes combining additional intermediate speech representations from the second multilayer perceptron and the plurality of intermediate speech representations from the plurality of successive transformer layers as the conditional information to be provided to the language model.

[0167] EEE B12. The method according to EEE B11, wherein the additional intermediate speech representation and the plurality of intermediate speech representations are combined by concatenating one or more features from the second multilayer perceptron and the corresponding transformer layer into a feature vector and applying a learnable projection to the feature vector using a linear layer.

[0168] EEE B13. The method according to EEE B12, wherein the additional intermediate speech representation and the plurality of intermediate speech representations are combined by summing one or more features from the additional multilayer perceptron and the corresponding transformer layer using corresponding weight coefficients determined by applying an attention algorithm.

[0169] EEE B14. A computer-implemented method for training a neural network-based language model to recover clean speech from distorted speech waveforms, the method comprising: Obtain the input time-frequency representation of the distorted speech waveform; A neural network-based multistage speech encoder is used to determine one or more audio features associated with the distorted speech waveform based on the input time-frequency representation; Conditional information for the language model is determined at least in part based on the one or more audio features; and The language model and the speech encoder are trained jointly, wherein the language model is trained at least in part based on the conditional information.

[0170] EEE B15. The method according to EEE B14 further includes: Obtain a reference time-frequency representation of the true speech waveform for the distorted speech waveform; Based on the one or more audio features, determine one or more semantic representations associated with the distorted speech waveform; and A pre-trained auxiliary model is used to determine one or more reference semantic representations associated with the ground truth speech waveform based on the reference time-frequency representation. The speech encoder is trained using an auxiliary loss function based on the one or more semantic representations and the one or more reference semantic representations.

[0171] EEE B16. The method according to EEE B15, wherein determining the one or more semantic representations from the one or more audio features includes applying a fully connected pooling layer.

[0172] EEE B17. The method according to EEE B15 or B16, wherein the one or more semantic representations include one or more pseudo-speech tags, and wherein the one or more reference semantic representations include one or more reference pseudo-speech tags.

[0173] EEE B18. The method according to any one of EEE B15 to B17, wherein the pre-trained auxiliary model is a self-supervised model.

[0174] EEE B19. The method according to any one of EEE B15 to B18, wherein the pre-trained auxiliary model is a pre-trained HuBERT model.

[0175] EEE B20. The method according to any one of EEE B14 to B19 further comprises: Obtain a reference time-frequency representation of the true speech waveform for the distorted speech waveform. The speech encoder is trained using regression loss based on the reference time-frequency representation and the predicted time-frequency representation of the clean speech waveform predicted by the speech encoder.

[0176] EEE B21. The method according to EEE B20 further includes applying a multilayer perceptron to generate the predicted time-frequency representation.

[0177] EEE B22. The method according to EEE B20 or B21, wherein training the speech encoder includes: The difference between the predicted time-frequency representation and the reference time-frequency representation is minimized by adapting one or more parameters of the multi-stage speech encoder based on a regression loss function.

[0178] EEE B23. The method according to any one of EEE B14 or EEE B20 to B22, wherein the one or more audio features are determined based on a plurality of intermediate speech representations of the input time-frequency representation generated within the speech encoder.

[0179] EEE B24. The method according to EEE B23, wherein the plurality of intermediate speech representations are generated by applying a plurality of successive transformer layers, each generating a corresponding intermediate speech representation.

[0180] EEE B25. The method according to EEE B24 further includes: in order to determine the plurality of intermediate speech representations, a second multilayer perceptron transforms the frame size of the input time-frequency representation to the transformer layer size to pass the input time-frequency representation to the plurality of consecutive transformer layers.

[0181] EEE B26. The method according to EEE B25 further includes combining additional intermediate speech representations from the second multilayer perceptron and the plurality of intermediate speech representations from the plurality of successive transformer layers as the conditional information to be provided to the language model.

[0182] EEE B27. The method according to EEE B26, wherein the additional intermediate speech representation and the plurality of intermediate speech representations are combined by concatenating one or more features from the second multilayer perceptron and the corresponding transformer layer into a feature vector and applying a learnable projection to the feature vector using a linear layer.

[0183] EEE B28. The method according to EEE B26, wherein the additional intermediate speech representation and the plurality of intermediate speech representations are combined by summing one or more features from the second multilayer perceptron and the corresponding transformer layer using corresponding weight coefficients determined by applying an attention algorithm.

[0184] EEE B29. The method according to any one of EEE B14 to B28 further comprises: The conditional information is randomly provided to the language model to conditionally train the language model; and A portion of the conditional information is replaced with one or more learnable embedding parameters as unconditional input for training the language model, wherein training the language model is further based on the unconditional input including the one or more learnable embedding parameters.

[0185] EEE B30. The method according to EEE B29, wherein the number of training cycles for conditionally training the language model is less than the total number of training cycles for training the language model, and the number of training cycles for unconditionally training the language model is given by a predetermined ratio.

[0186] EEE B31. The method according to any one of EEE B14 to B30 further includes segmenting a reference waveform associated with a true speech signal to convert the reference waveform into a codemap of a predefined size containing discrete lexical units for training the language model.

[0187] EEE B32. The method according to any one of EEE B14 to B31, wherein the language model is trained based on the cross-entropy loss function.

[0188] EEE B33. The method according to any one of EEE B14 to B32, wherein the language model comprises an autoregressive language model.

[0189] EEE B34. The method according to any one of EEE B14 to B33, wherein the language model includes a masked language model.

[0190] EEE B35. The method according to EEE B34 further includes randomly masking one or more discrete tokens in the codemap to generate a partially masked codemap, converting the partially masked codemap into multiple embedding vectors, summing the multiple embedding vectors, and providing the sum of the summed embedding vectors and the conditional information to the masked language model as training input.

[0191] EEE B36. The method according to EEE B35, wherein the language model comprises multiple transformer blocks, followed by a multilayer perceptron, the method further comprising, for training the language model: Extract multiple masking words from the true speech signal; The multilayer perceptron estimates multiple prediction indices corresponding to the masked lexical units; Determine the difference between the predicted index and the actual index of the masking term; and The difference between the predicted index and the actual index of the masking term is minimized based on the cross-entropy loss function.

[0192] EEE B37. A system for speech recovery, comprising: The neural network-based multi-stage speech coding unit is configured as follows: Obtain the input time-frequency representation of the distorted speech waveform, and One or more audio features associated with the distorted speech waveform are determined based on the input time-frequency representation; A neural network-based language model is configured to generate multiple discrete lexical units of a codemap corresponding to a clean speech signal; A decoding unit configured to generate, from the plurality of discrete lexical units of the codemap, a recovered speech waveform associated with the clean speech signal; and A fusion element configured to generate conditional information for the language model based on one or more audio features, wherein the plurality of discrete lexical units of the codemap are generated at least in part based on the conditional information.

[0193] EEE B38. The system according to EEE B37, wherein the speech coding unit includes a multilayer perceptron configured to generate a predicted time-frequency representation of a clean speech waveform based on a plurality of intermediate speech representations of the input time-frequency representation.

[0194] EEE B39. The system according to EEE B38, wherein the speech coding unit is further configured as follows: One or more semantic representations are determined based on the one or more audio features.

[0195] EEE B40. The system according to EEE B38 or B39, wherein the speech coding unit further includes a plurality of successive transformer layers, each transformer layer being configured to generate a corresponding intermediate speech representation.

[0196] EEE B41. The system according to EEE B40, wherein the speech coding unit further includes a second multilayer perceptron configured to transform the frame size of the input time-frequency representation to a transformer layer size to pass the input time-frequency representation to the plurality of consecutive transformer layers.

[0197] EEE B42. The system according to EEE B41, wherein the fusion element is configured to combine additional intermediate speech representations from the second multilayer perceptron and the plurality of intermediate speech representations from the plurality of successive transformer layers as the conditional information to be provided to the language model.

[0198] EEE B43. The system according to any one of EEE B37 to B42 further includes a word segmentation unit configured to segment a reference waveform associated with a truth speech signal to convert the reference waveform into a reference codemap of a predefined size containing reference discrete lexical units for training the language model.

[0199] EEE B44. The system according to any one of EEE B37 to B43, wherein the speech coding unit and the language model are jointly trained based on a regression loss function and / or a cross-entropy function.

[0200] EEE B45. The system according to any one of EEE B37 to B44, wherein the language model comprises an autoregressive language model or a masked language model.

[0201] EEE B46. A system according to EEE B43, wherein the language model includes a masked language model, the system further includes a masking unit configured to randomly mask one or more reference discrete tokens in the reference codemap to generate a partially masked codemap, an embedding unit configured to convert the partially masked codemap into a plurality of embedding vectors, and a summing unit configured to sum the plurality of embedding vectors and provide the sum of the summed embedding vectors and the conditional information as training input to the masked language model.

[0202] EEE B47. The system according to EEE B46, wherein the masked language model comprises multiple transformer blocks, followed by a multilayer perceptron, the multilayer perceptron being configured to, in order to train the masked language model: Extract multiple masking words from the true speech signal; Estimate multiple prediction indices corresponding to the masking terminology; Determine the difference between the predicted index and the actual index of the masking term; and The difference between the predicted index and the actual index of the masking term is minimized based on the cross-entropy loss function.

[0203] EEE B48. The system according to any one of EEE B37 to B47 further includes a logit estimation unit, said logit estimation unit being configured to: Calculate the first logit of the first predicted lexical unit generated from the language model using the conditional information, and subsequently calculate the second logit of the second predicted lexical unit generated from the language model without the conditional information, or vice versa; and A controllable parameter is used to combine the first logit and the second logit to obtain a guided logit, the controllable parameter indicating the degree to which the generation of the recovered speech waveform associated with the clean speech signal depends on the conditional information, wherein the plurality of discrete lexical units of the codemap are generated by a classifier-free guided (CFG) based on an iterative sampling procedure, the iterative sampling procedure comprising, for each iteration: sampling one or more output lexical units of the language model generated from the previous iteration to remask based on the guided logit.

[0204] EEE B49. An apparatus for speech recovery, the apparatus comprising a processor and a memory coupled to the processor and storing instructions for the processor, wherein the processor is configured to perform all steps of the method according to any one of EEEB1 to B36.

[0205] EEE B50. A computer program comprising instructions that, when executed by a computing device, cause the computing device to perform all the steps of the method according to any one of EEE B1 to B36.

[0206] EEE B51. A computer-readable storage medium storing a computer program according to EEE B50.

Claims

1. A computer-implemented method for recovering clean speech from distorted speech waveforms using a neural network-based language model, the method comprising: Obtain the input time-frequency representation of the distorted speech waveform; A neural network-based multistage speech encoder is used to determine one or more audio features associated with the distorted speech waveform based on the input time-frequency representation; Conditional information for the language model is determined at least in part based on the one or more audio features; At least in part based on the conditional information, multiple discrete lexical units of a codemap corresponding to a clean speech signal are generated from a fully masked codemap containing multiple masked lexical units. as well as A recovered speech waveform associated with the clean speech signal is generated from the plurality of discrete lexical units of the codemap.

2. The method according to claim 1, wherein, The plurality of discrete tokens in the code map are generated based on an iterative sampling procedure, which includes, for each iteration: Calculate the first logit of the first predicted lexical unit generated from the language model using the conditional information, and then calculate the second logit of the second predicted lexical unit generated from the language model without the conditional information, or vice versa; Use controllable parameters to combine the first logit and the second logit to obtain the bootstrap logit; and One or more output lexical units from the language model generated from previous iterations are sampled for re-masking based on the guided logit.

3. The method according to claim 2, wherein, The controllable parameter indicates the degree to which the generation of the recovered speech waveform associated with the clean speech signal depends on the conditional information.

4. The method according to any one of claims 1 to 3, wherein, The language model includes an autoregressive language model.

5. The method according to claim 3, wherein, The language model includes a masked language model, and the method further includes: Obtain the code image of a fully masked word containing multiple masking terms; The fully masked code map is converted into multiple embedding vectors; Summing the plurality of embedding vectors; Based on the guided logit, the sum of the summed embedding vector and conditional information is provided to the masked language model as input; and The language model predicts the plurality of masking lexical units to predict the plurality of discrete lexical units in the code map.

6. The method according to claim 5, wherein, The iterative sampling procedure further includes determining prediction accuracy and prediction gain for each predicted masking word, wherein predicting the plurality of discrete words includes: If the corresponding prediction accuracy and / or corresponding prediction gain associated with the predicted masking term is greater than or equal to a predefined value, then the predicted masking term is retained, or If the corresponding prediction accuracy and / or corresponding prediction gain associated with the predicted masking term is less than a predefined value, then the predicted masking term is discarded; and The discarded masking lexical units are re-predicted by the masking language model.

7. The method according to claim 6, wherein, Re-predicting the discarded masking terms includes: The fully masked code map is updated based on the retained predicted masking words and the discarded masking words; The updated fully masked codemap is converted into multiple updated embedding vectors; The guided logit is obtained based on the output of the masked language model in the absence of the aforementioned conditional information; Sum the multiple updated embedding vectors; Based on the guided logit, the sum of the summed and updated embedding vectors and the conditional information is provided to the masked language model as an updated input for prediction.

8. The method according to any one of claims 1 to 7, wherein, The one or more audio features are determined based on multiple intermediate speech representations of the input time-frequency representation generated within the speech encoder.

9. The method according to claim 8, wherein, The multiple intermediate speech representations are generated by applying multiple consecutive transformer layers, each of which generates its own intermediate speech representation.

10. The method of claim 9, further comprising: To determine the plurality of intermediate speech representations, a second multilayer perceptron transforms the frame size of the input time-frequency representation into the transformer layer size, so as to pass the input time-frequency representation to the plurality of consecutive transformer layers.

11. The method of claim 10, further comprising combining additional intermediate speech representations from the second multilayer perceptron and the plurality of intermediate speech representations from the plurality of successive transformer layers as the conditional information to be provided to the language model.

12. The method according to claim 11, wherein, The additional intermediate speech representations and the plurality of intermediate speech representations are combined by concatenating one or more features from the second multilayer perceptron and the corresponding transformer layer into a feature vector and applying a learnable projection to the feature vector using a linear layer.

13. The method according to claim 12, wherein, The additional intermediate speech representations and the plurality of intermediate speech representations are combined by summing one or more features from the additional multilayer perceptron and the corresponding transformer layer using corresponding weight coefficients determined by applying an attention algorithm.

14. A computer-implemented method for training a neural network-based language model to recover clean speech from distorted speech waveforms, the method comprising: Obtain the input time-frequency representation of the distorted speech waveform; A neural network-based multistage speech encoder is used to determine one or more audio features associated with the distorted speech waveform based on the input time-frequency representation; Conditional information for the language model is determined at least in part based on the one or more audio features; as well as The language model and the speech encoder are trained jointly, wherein the language model is trained at least in part based on the conditional information.

15. The method of claim 14, further comprising: Obtain a reference time-frequency representation of the true speech waveform for the distorted speech waveform; Based on the one or more audio features, determine one or more semantic representations associated with the distorted speech waveform; and A pre-trained auxiliary model is used to determine one or more reference semantic representations associated with the ground truth speech waveform based on the reference time-frequency representation. The speech encoder is trained using an auxiliary loss function based on the one or more semantic representations and the one or more reference semantic representations.

16. The method according to claim 15, wherein, Determining the one or more semantic representations from the one or more audio features includes applying a fully connected pooling layer.

17. The method according to claim 15 or 16, wherein, The one or more semantic representations include one or more pseudo-speech tags, and wherein the one or more reference semantic representations include one or more reference pseudo-speech tags.

18. The method according to any one of claims 15 to 17, wherein, The pre-trained auxiliary model is a self-supervised model.

19. The method according to any one of claims 15 to 18, wherein, The pre-trained auxiliary model is a pre-trained HuBERT model.

20. The method according to any one of claims 14 to 19, further comprising: Obtain a reference time-frequency representation of the true speech waveform for the distorted speech waveform. The speech encoder is trained using regression loss based on the reference time-frequency representation and the predicted time-frequency representation of the clean speech waveform predicted by the speech encoder.

21. The method of claim 20, further comprising applying a multilayer perceptron to generate the predicted time-frequency representation.

22. The method according to claim 20 or 21, wherein, Training the speech encoder includes: The difference between the predicted time-frequency representation and the reference time-frequency representation is minimized by adapting one or more parameters of the multi-stage speech encoder based on a regression loss function.

23. The method according to claim 14 or any one of claims 20 to 22, wherein, The one or more audio features are determined based on multiple intermediate speech representations of the input time-frequency representation generated within the speech encoder.

24. The method according to claim 23, wherein, The multiple intermediate speech representations are generated by applying multiple consecutive transformer layers, each of which generates its own intermediate speech representation.

25. The method of claim 24, further comprising: To determine the plurality of intermediate speech representations, a second multilayer perceptron transforms the frame size of the input time-frequency representation into the transformer layer size, so as to pass the input time-frequency representation to the plurality of consecutive transformer layers.

26. The method of claim 25, further comprising combining additional intermediate speech representations from the second multilayer perceptron and the plurality of intermediate speech representations from the plurality of successive transformer layers as the conditional information to be provided to the language model.

27. The method according to claim 26, wherein, The additional intermediate speech representations and the plurality of intermediate speech representations are combined by concatenating one or more features from the second multilayer perceptron and the corresponding transformer layer into a feature vector and applying a learnable projection to the feature vector using a linear layer.

28. The method according to claim 26, wherein, The additional intermediate speech representation and the plurality of intermediate speech representations are combined by summing one or more features from the second multilayer perceptron and the corresponding transformer layer using corresponding weight coefficients determined by applying an attention algorithm.

29. The method according to any one of claims 14 to 28, further comprising: The conditional information is randomly provided to the language model to conditionally train the language model; as well as A portion of the conditional information is replaced with one or more learnable embedding parameters as unconditional input for training the language model, wherein training the language model is further based on the unconditional input including the one or more learnable embedding parameters.

30. The method according to claim 29, wherein, The number of training cycles used for conditionally training the language model is less than the total number of training cycles used to train the language model, and the number of training cycles used for unconditionally training the language model is given by a predetermined ratio.

31. The method according to any one of claims 14 to 30, further comprising segmenting a reference waveform associated with a truth speech signal to convert the reference waveform into a codemap of a predefined size containing discrete lexical units for training the language model.

32. The method according to any one of claims 14 to 31, wherein, The language model is trained based on the cross-entropy loss function.

33. The method according to any one of claims 14 to 32, wherein, The language model includes an autoregressive language model.

34. The method according to any one of claims 14 to 33, wherein, The language model includes a masked language model.

35. The method of claim 34, further comprising randomly masking one or more discrete tokens in the codemap to generate a partially masked codemap, converting the partially masked codemap into a plurality of embedding vectors, summing the plurality of embedding vectors, and providing the sum of the summed embedding vectors and the conditional information to the masked language model as training input.

36. The method according to claim 35, wherein, The language model comprises multiple transformer blocks, followed by a multilayer perceptron, and the method further includes methods for training the language model: Extract multiple masking words from the true speech signal; The multilayer perceptron estimates multiple prediction indices corresponding to the masked lexical units; Determine the difference between the predicted index and the actual index of the masked word; as well as The difference between the predicted index and the actual index of the masking term is minimized based on the cross-entropy loss function.

37. A system for speech recovery, comprising: The neural network-based multi-stage speech coding unit is configured as follows: Obtain the input time-frequency representation of the distorted speech waveform, and One or more audio features associated with the distorted speech waveform are determined based on the input time-frequency representation; A neural network-based language model is configured to generate multiple discrete lexical units of a codemap corresponding to a clean speech signal; A decoding unit configured to generate a recovered speech waveform associated with the clean speech signal from the plurality of discrete lexical units of the codemap; as well as A fusion element configured to generate conditional information for the language model based on one or more audio features, wherein the plurality of discrete lexical units of the codemap are generated at least in part based on the conditional information.

38. The system according to claim 37, wherein, The speech coding unit includes a multilayer perceptron configured to generate a predicted time-frequency representation of a clean speech waveform based on multiple intermediate speech representations of the input time-frequency representation.

39. The system according to claim 38, wherein, The speech coding unit is further configured as follows: One or more semantic representations are determined based on the one or more audio features.

40. The system according to claim 38 or 39, wherein, The speech coding unit further includes multiple consecutive transformer layers, each of which is configured to generate a corresponding intermediate speech representation.

41. The system according to claim 40, wherein, The speech coding unit further includes a second multilayer perceptron, which is configured to transform the frame size of the input time-frequency representation to the transformer layer size to pass the input time-frequency representation to the plurality of consecutive transformer layers.

42. The system according to claim 41, wherein, The fusion element is configured to combine additional intermediate speech representations from the second multilayer perceptron and the multiple intermediate speech representations from the multiple successive transformer layers as the conditional information to be provided to the language model.

43. The system according to any one of claims 37 to 42, further comprising a word segmentation unit configured to segment a reference waveform associated with a truth speech signal to convert the reference waveform into a reference codemap of a predefined size containing reference discrete lexical units for training the language model.

44. The system according to any one of claims 37 to 43, wherein, The speech coding unit and the language model are jointly trained based on a regression loss function and / or a cross-entropy function.

45. The system according to any one of claims 37 to 44, wherein, The language model includes an autoregressive language model or a masked language model.

46. ​​The system according to claim 43, wherein, The language model includes a masked language model, and the system further includes: a masking unit configured to randomly mask one or more reference discrete lexical units in the reference codemap to generate a partially masked codemap; an embedding unit configured to convert the partially masked codemap into multiple embedding vectors; and a summing unit configured to sum the multiple embedding vectors and provide the sum of the summed embedding vectors and the conditional information to the masked language model as training input.

47. The system according to claim 46, wherein, The masked language model comprises multiple transformer blocks, followed by a multilayer perceptron, which is configured to train the masked language model as follows: Extract multiple masking words from the true speech signal; Estimate multiple prediction indices corresponding to the masking terminology; Determine the difference between the predicted index and the actual index of the masked word; as well as The difference between the predicted index and the actual index of the masking term is minimized based on the cross-entropy loss function.

48. The system according to any one of claims 37 to 47, further comprising a logit estimation unit, the logit estimation unit being configured to: Calculate the first logit of the first predicted lexical unit generated from the language model using the conditional information, and subsequently calculate the second logit of the second predicted lexical unit generated from the language model without the conditional information, or vice versa; and A controllable parameter is used to combine the first logit and the second logit to obtain a guided logit, wherein the controllable parameter indicates the degree to which the generation of the recovered speech waveform associated with the clean speech signal depends on the conditional information, wherein... The plurality of discrete lexical units in the codemap are generated by a classifier-free guided (CFG) iterative sampling procedure, which includes, for each iteration: sampling one or more output lexical units of the language model generated from the previous iteration to remask based on the logit of the guided case.

49. An apparatus for speech recovery, the apparatus comprising a processor and a memory, the memory being coupled to the processor and storing instructions for the processor, wherein, The processor is configured to perform all the steps of the method according to any one of claims 1 to 36.

50. A computer program comprising instructions that, when executed by a computing device, cause the computing device to perform all the steps of the method according to any one of claims 1 to 36.

51. A computer-readable storage medium storing a computer program according to claim 50.

Images