Training method and system for self-supervised training predictor for speech separation

Abstract

The embodiment of the present application provides a training method and system of a self-supervised training predictor for speech separation. The method comprises: extracting self-supervised training features of each single person voice source speech by using a pre-training model; extracting shallow features for speech representation and deep features for context information in the self-supervised training features, and determining the shallow features and the deep features of each single person voice source speech as training labels of the self-supervised training predictor; inputting training mixed speech generated by each single person voice source speech into the self-supervised training predictor to obtain estimated features of each single person voice source speech; and training the self-supervised training predictor based on a loss function determined based on the estimated features and the training labels corresponding to each single person voice source speech. The self-supervised training predictor is trained and applied in a speech separation model in the embodiment of the present application, so that the accuracy of the self-supervised training features is improved, the performance of the speech separation system is improved, and the model parameters and the calculation complexity are reduced.

Description

Technical Field

[0001] This invention relates to the field of intelligent speech, and in particular to a training method and system for a self-supervised predictor for speech separation. Background Technology

[0002] For speech recognition of a single speaker, a high level of accuracy can be achieved. However, for the cocktail party problem, for example, when multiple speakers are speaking together, the speech recognition rate for each speaker will decrease.

[0003] To improve the accuracy of speech recognition when multiple speakers are speaking together, existing technologies typically use unsupervised pre-trained models.

[0004] Large-scale unsupervised pre-trained models are self-supervised using a masking-prediction criterion based on a large amount of unlabeled data. During training, the model attempts to model the contextual information of the speech signal, thereby learning strong deep embedding features. These large-scale pre-unsupervised trained models have achieved good results on various downstream speech tasks.

[0005] In the process of realizing this invention, the inventors discovered at least the following problems in the related technology:

[0006] Most pre-trained models are trained using datasets that primarily consist of single-speaker speech, but speech separation inputs often involve a mixture of speech from multiple speakers. Directly utilizing pre-trained features from single-speaker speech yields poor results in speech separation tasks.

[0007] Most pre-trained models are trained on datasets consisting primarily of single-speaker speech, but speech separation inputs typically involve a mixture of speech from multiple speakers. Directly utilizing pre-trained features from single-speaker speech yields poor results in speech separation tasks. Another issue to consider and optimize when applying pre-trained models to speech separation tasks is model complexity and computational cost. Most pre-trained models are designed for general downstream tasks and are trained on large-scale datasets. For speech separation tasks, their size may be too large, resulting in prohibitively high computational costs. Summary of the Invention

[0008] In order to at least address the problem of high complexity and cost of using pre-trained models for speech separation tasks in existing technologies.

[0009] In a first aspect, embodiments of the present invention provide a training method for a self-supervised predictor for speech separation, comprising:

[0010] Self-supervised training features of speech from each individual voice source were extracted using a pre-trained model.

[0011] The shallow features used for speech representation and the deep features used for context information in the self-supervised training features are extracted, and the shallow features and deep features of each single-person voice source speech are determined as the training labels of the self-supervised training predictor.

[0012] The training mixed speech generated from the individual voice sources is input into the self-supervised training predictor to obtain the estimated features of each individual voice source speech.

[0013] The self-supervised predictor is trained based on the estimated features of each individual voice source speech and the loss function determined by the training labels corresponding to each individual voice source speech, thus obtaining the trained self-supervised predictor.

[0014] Secondly, embodiments of the present invention provide a training system for a self-supervised predictor for speech separation, comprising:

[0015] The feature extraction module is used to extract self-supervised training features of each individual voice source speech using a pre-trained model.

[0016] The training label determination module is used to extract shallow features for speech representation and deep features for context information from the self-supervised training features, and to determine the shallow features and deep features of each single-person voice source speech as the training labels of the self-supervised training predictor.

[0017] The estimated feature determination module is used to input the training mixed speech generated from the speech of each individual voice source into the self-supervised training predictor to obtain the estimated features of each individual voice source speech.

[0018] The training module is used to train the self-supervised predictor based on the estimated features of each individual voice source speech and the loss function determined by the training labels corresponding to each individual voice source speech, so as to obtain the trained self-supervised predictor.

[0019] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of a training method for a self-supervised training predictor for speech separation according to any embodiment of the present invention.

[0020] Fourthly, embodiments of the present invention provide a storage medium storing a computer program thereon, characterized in that, when the program is executed by a processor, it implements the steps of a training method for a self-supervised training predictor for speech separation according to any embodiment of the present invention.

[0021] The beneficial effects of this invention are as follows: a self-supervised training predictor is designed and trained, which can directly extract the self-supervised training features of a single speaker from speech mixtures. Applying the self-supervised training predictor to the speech separation model improves the accuracy of predicting the self-supervised training features of a single speaker, thereby improving the overall performance of the speech separation system. Furthermore, compared to speech separation systems that directly utilize mixed speech for training, this method requires fewer model parameters and has lower computational complexity. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a training method for a self-supervised predictor for speech separation, provided in an embodiment of the present invention.

[0024] Figure 2 This is a schematic diagram of shallow and deep features of Wav2vec 2.0, a training method for a self-supervised predictor for speech separation, provided in an embodiment of the present invention.

[0025] Figure 3 This is a schematic diagram of the structure of a training method for a self-supervised training predictor for speech separation provided in an embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram of a speech separation model with self-supervised training feature embedding, which is a training method for a self-supervised training predictor for speech separation according to an embodiment of the present invention.

[0027] Figure 5 This is a schematic diagram of the speech separation results on the LibriMix dataset using STOI and SDR, which are provided in an embodiment of the present invention for training a self-supervised predictor for speech separation.

[0028] Figure 6 This is a schematic diagram illustrating the prediction accuracy of different embeddings in a training method for a self-supervised predictor for speech separation, provided in an embodiment of the present invention.

[0029] Figure 7 This is a schematic diagram showing the cumulative computation rate and model size of a training method for a self-supervised predictor for speech separation, provided in an embodiment of the present invention.

[0030] Figure 8This is a schematic diagram of STOI and SDR for speech separation on the LibriMix dataset, illustrating a training method for a self-supervised predictor for speech separation provided in an embodiment of the present invention.

[0031] Figure 9 This is a schematic diagram of the structure of a training system for a self-supervised training predictor for speech separation, provided in an embodiment of the present invention.

[0032] Figure 10 This is a schematic diagram of an embodiment of an electronic device for training a self-supervised predictor for speech separation, provided by an embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] like Figure 1 The diagram shows a flowchart of a training method for a self-supervised predictor for speech separation according to an embodiment of the present invention, including the following steps:

[0035] S11: Use a pre-trained model to extract the self-supervised training features of each individual voice source speech.

[0036] S12: Extract the shallow features used for speech representation and the deep features used for context information from the self-supervised training features, and determine the shallow features and deep features of each single-person voice source speech as the training labels of the self-supervised training predictor.

[0037] S13: Input the training mixed speech generated from each individual voice source speech into the self-supervised training predictor to obtain the estimated features of each individual voice source speech;

[0038] S14: The self-supervised training predictor is trained based on the estimated features of each individual voice source speech and the loss function determined by the training labels corresponding to each individual voice source speech, to obtain the trained self-supervised training predictor.

[0039] In this embodiment, to avoid the problem of excessively large pre-trained models in speech separation tasks, a lightweight approach is proposed to utilize knowledge extracted from pre-trained large-scale SSL (self-supervised learning) models for training. This method trains a self-supervised predictor, directly extracting individual speaker self-supervised training estimation features from mixed speech from multiple speakers. These self-supervised training estimation features are then used in the speech separation model, thereby improving speech separation performance while reducing the required computational complexity.

[0040] For step S11, a pre-trained model is needed to provide self-supervised training features of unlabeled raw audio from single-speaker sources for the self-supervised training predictor of this method. As one implementation, an unsupervised pre-trained model, such as wav2vec 2.0, can be used. Wav2vec 2.0 is an SSL (self-supervised learning) model that is pre-trained using a large amount of unlabeled raw audio. This model consists of a feature encoder network and a context network combined with a quantization module. The encoder network maps the raw audio to latent speech representations over a long period using multi-layer temporal CNNs (Convolutional Neural Networks). The context network has the same architecture as the Transformer. It learns high-level contextualized representations from basic features. Before feature output, the encoder network sends it to the context network, which then passes it through a quantization module. This module uses product quantization to discretize the basic speech unit features into a finite set of latent speech units with multiple codebooks. Self-supervised training features for each single-speaker source speech can then be extracted using the pre-trained model.

[0041] For step S12, in the prior art, the contextual representation embedding determined by the context network is used for downstream tasks (e.g., recognition of specific domains such as home appliances, automobiles, consumer electronics, finance, logistics, real estate, government affairs, and healthcare). However, the effectiveness of deep embeddings containing contextual representations in speech separation has not been well studied. In this method, two different embeddings from the Wav2vec2.0 model are considered. Specifically, shallow features for speech representation are extracted from the self-supervised training features based on a temporal convolutional neural network model; deep features for contextual information are extracted from the self-supervised training features based on a Transformer. In this embodiment, as... Figure 2The first example shown is a basic embedding extracted by a temporal CNN and a quantization module. This is called a shallow feature. The other is a high-level contextualized embedding extracted by a Transformer, called a deep feature. Shallow and deep features are technical terms specific to convolutional neural networks. The selection of shallow and deep features in Wav2vec 2.0 is based on the following considerations: On one hand, shallow features are extracted by a temporal CNN without processing by a deep context network; they have a relatively small receptive field and are consistent with the speech input. The extracted shallow features are more acoustically oriented. On the other hand, deep features are processed by a context network. Deep features can be considered more context-sensitive. Each frame in the deep features is focused on the entire input sequence by the self-attention mechanism in the Transformer. In the Wav2vec 2.0 model, shallow features are quantized using a codebook, making them discretized, while deep features are continuous. For a more complete comparison, a quantization module is also used to obtain discrete deep features. The resulting shallow and deep features are then used as training labels for a self-supervised predictor.

[0042] For step S13, since Wav2vec 2.0 (and most other available pre-trained SSL models) is trained using a dataset primarily composed of single-speaker speech, directly extracting SSL features from speech mixtures may not be optimal for speech separation tasks. Therefore, this method designs a self-supervised training predictor to estimate the SSL feature embeddings for each speaker. The goal is to extract knowledge from pre-trained large-scale SSL models using the feature predictor. Furthermore, the SSL features are estimated through speech mixtures rather than clean input. This self-supervised training predictor is trained using oracle SSL features extracted from parallel clean speech. The self-supervised training predictor comprises a temporal convolutional neural network for extracting temporal speech signals and a dual-path recurrent neural network for context modeling, specifically as follows... Figure 3 As shown.

[0043] The self-supervised predictor consists of several parts. The first part is the encoder, a multi-layer convolutional network that encodes the input temporal mixed speech into hidden features. The second part is the separation process for the obtained features. Based on the deep or shallow features to be estimated, the separator designs a TCN (temporal convolutional network). Compared with the existing pre-trained Wav2vec 2.0 model, the self-supervised predictor of this method is smaller and requires less computation. Detailed comparisons can be obtained in the following experiments. The self-supervised predictor also includes a simple linear projection layer. In continuous feature prediction, the linear projection layer produces estimated features with the same dimension as the oracle SSL features. In discrete feature prediction, the softmax algorithm can be used to obtain the probabilities of each class in the codebook, thereby predicting the estimated features of each individual speech source.

[0044] For step S14, after obtaining the estimated features of each individual speech source and the corresponding training labels, cross-entropy loss (CE) is used to perform classification training on the discrete features. For example, the mean squared error (MSE) between the estimated features and the training labels (i.e., the oracle SSL features extracted from the clean speech) is used as the loss function for permutation-invariant training. Training can be stopped when the mean squared error converges.

[0045] As one implementation, after obtaining the trained self-supervised predictor, the method further includes:

[0046] The received mixed speech containing multiple speakers is input into the speech separation model, wherein the speech separation includes: an encoder, a speech separator, and a decoder;

[0047] The encoder encodes the mixed speech to obtain deep features of the mixed speech, and uses the self-supervised training predictor to determine the self-supervised training estimated features of each speaker in the mixed speech. Based on the self-supervised training estimated features of each speaker and the deep features of the mixed speech, the fusion features are determined.

[0048] The speech separator determines the feature code for each speaker in the fused features;

[0049] The decoder decodes the feature encoding to obtain the separate speech of each speaker in the mixed speech.

[0050] In this embodiment, considering the applicability of the self-supervised predictor trained by this method to speech separation tasks, the Conv-TasNet fully convolutional temporal audio separation network is used as the backbone for speech separation. Conv-TasNet uses a linear encoder to generate a speech waveform representation optimized for individual speaker separation. Speaker separation is achieved by applying a set of weighting functions (masks) to the encoder output. Then, a linear decoder inverts the modified encoder representation back into a waveform. This mask is implemented using a temporal convolutional network (TCN) composed of stacked one-dimensional extended convolutional blocks, which allows for modeling the long-term correlation of speech signals while maintaining a small model size. The improved speech separation model equipped with the self-supervised predictor of this method adds a trained self-supervised predictor to Conv-TasNet to further extract deeper self-supervised training features, and also avoids the high complexity and computational cost caused by directly training existing speech separation models using mixed speech. Figure 4 As shown, the speech separation model based on a self-supervised predictor includes an encoder, a speech separator, and a decoder. The encoder is a temporal convolutional layer that maps temporal audio to a feature embedding space. The separator is a sequence modeling network used to process the encoded embeddings and estimate the separated features for each target speaker. The decoder is a transposed convolutional layer used to map the processed features back to the temporal target speech signal. The self-supervised predictor-based speech separation model can be applied to various types of smart devices, such as smart speakers, smart TVs, and smart car systems. In scenarios equipped with smart speakers, smart TVs, or smart car systems, multiple speakers often speak simultaneously. For example, two users are watching TV, and a song playing on the smart TV evokes past memories, leading to conversation. When the song ends, one user wants to listen to another song and, during their conversation, tells the smart TV to "play song X." At this point, the smart TV receives a mixed speech from multiple speakers and inputs this mixed speech into the speech separation model of this method.

[0051] Upon receiving mixed speech, the speech separation model based on a self-supervised predictor resamples the self-supervised training features estimated by the predictor in the time dimension to have the same sampling rate as the TasNet encoder output. Then, the self-supervised training features for multiple speakers are concatenated with the TasNet encoder output to obtain fused features. These fused features are projected through a linear layer and input into a TCN separator to obtain feature codes for each speaker. Finally, a decoder decodes these feature codes to obtain the separated speech for each speaker.

[0052] This implementation demonstrates that by designing and training a self-supervised predictor, the self-supervised training features of a single speaker can be directly extracted from the speech mixture. Applying the self-supervised predictor to the speech separation model improves the accuracy of predicting the self-supervised training features of a single speaker, thereby enhancing the overall performance of the speech separation system. Furthermore, compared to speech separation systems that directly utilize mixed speech for training, this method requires fewer model parameters and has lower computational complexity.

[0053] Experiments are conducted to illustrate this method, analyzing both the trained self-supervised predictor and the speech separation model equipped with the self-supervised predictor. The method can be evaluated using the publicly available LibriMix dataset, a widely used reference dataset in speech separation. Furthermore, this dataset also provides two main datasets: two-speaker mixture and three-speaker mixture. This method selected the two-speaker 16k min Libri2Mix dataset (with noise).

[0054] For the self-supervised predictor, a multi-layer convolutional network was used for feature extraction in the experiments. The same feature extraction network as the Wav2Vec2.0 model was selected, using a 7-layer one-dimensional convolutional neural network with 512 channels, kernel sizes of (10, 3, 3, 3, 3, 3, 2, 2), and strides of (5, 2, 2, 2, 2).

[0055] After feature extraction, a TCN network is used for feature estimation. For shallow features, the convolutional stack repeats once for shallow features and four times for deep features, with each repetition containing eight convolutions. For both predictors, each convolutional block contains 512 units, and the bottleneck dimension is 256. For discrete embeddings, the 256-dimensional features are projected into two 320-dimensional features, and training is performed using cross-entropy as the loss function. For continuous embeddings, the prediction performance is estimated by the mean squared error between the predicted and actual features.

[0056] For the speech separation model, the Conv-TasNet model was chosen, with the following parameter configuration: The encoder uses a 256-channel one-dimensional convolutional network with a kernel length of 40 and a stride of 20 to encode speech features. The separation layer uses a temporal convolutional network with four convolutional blocks, each containing eight convolutional layers. The bottleneck dimension is set to 256, and the hidden layer dimension is also set to 256. The decoder needs to reconstruct the speech, so it uses the same configuration as the encoder.

[0057] Using the existing "Oracle" label as the upper limit for self-supervised training, we analyzed the improvements to the speech separation baseline experiments using three different features extracted by Wav2Vec 2.0. The first is SSL features extracted directly from speech mixtures. The latter two are SSL features extracted from parallel clean speech sources. Experimental results are as follows: Figure 5 As shown, STOI (Short-Time Objective Intelligibility); SDR (Source to Distortion Ratio); and SI-SNR are the temporal loss functions. It can be observed that using Wav2Vec 2.0 to extract embeddings from clean, single-speaker speech improves performance by nearly 3dB compared to the baseline model, with deep features showing the greatest improvement in SDR (2.79dB). Meanwhile, the shallow model improves SDR by 2.63dB, achieving results very close to the deep model. However, when extracting features directly from mixed speech using the wav2vec 2.0 model, there is almost no enhancement, so this pre-trained model cannot be directly applied to separation.

[0058] SSL features extracted from clean speech play a powerful role in speech separation tasks. However, existing techniques cannot directly achieve this. Therefore, this method requires a self-supervised predictor trained on a single speaker to extract SSL features from mixed speech.

[0059] The discrete and continuous predictors described above are used to predict the shallow and deep feature embeddings extracted from the pre-trained model, respectively. The prediction accuracy is as follows: Figure 6 As shown, for discrete shallow features, using two codebooks of size 320 to classify the prediction results, the optimal prediction accuracy was 42.5%. However, for deep continuous features, the prediction performance cannot be judged by prediction accuracy alone; it can only be observed by observing the mean squared error side-by-side. To visualize the prediction accuracy of deep features and reduce overfitting, the deep continuous features extracted from the pre-trained model were also discretized using codebooks of the same size to obtain deep discrete features. A discrete predictor was then trained for prediction, and by adjusting the model parameters, the optimal prediction accuracy was 17.3%.

[0060] The multiplication accumulation operation (mac) and the number of trainable parameters in existing pre-trained models and the self-supervised training predictor of this method were also analyzed. Figure 7As can be seen, this method reduces the number of parameters in the shallow feature embedding of the self-supervised predictor to 3.0% of the original model and the number of MACs to 30.0% of the original model. Although the reduction in the number of parameters in the deep feature embedding is not as significant as that in the shallow feature embedding, the number of parameters is still reduced to 10.0% of the original model and the number of MACs is reduced to 19.0% of the original model. This method makes the self-supervised predictor lighter, with fewer parameters and lower computational cost.

[0061] When training a separable model equipped with a self-supervised predictor, a course learning approach is employed to improve model performance. Experiments were also conducted using shallow discrete features, deep continuous features, and deep discrete features in the predictive embeddings. Oracles and predictive embeddings were integrated into different types of pre-trained embeddings.

[0062] like Figure 8 The metrics analysis for these model types is listed. Here, O represents parallel clean speech features from oracle SSL; P represents predicted SSL features from speech mixtures; SSL-tr represents self-supervised training features during model training; and SSL-tt represents self-supervised training features during testing. Using the existing Conv-TasNet separation model as a benchmark, the separation model equipped with a self-supervised predictor achieves better separation results when fusing only shallow discrete features, improving SDR by 0.55 dB. After adding oracle embeddings to assist training, the shallow discrete feature embedding shows almost no improvement, while the SDR of deep continuous features improves by 0.67 dB. The results indicate that using a separation model equipped with a self-supervised predictor can significantly improve temporal speech separation performance.

[0063] In summary, this method designs and trains a self-supervised predictor for speech separation to extract individual speaker SSL features from mixed speech. These predicted SSL features effectively improve the performance of the speech separation model. Furthermore, the speech separation model based on the self-supervised predictor consumes significantly fewer computational resources than existing primitive models.

[0064] like Figure 9 The diagram shown is a schematic diagram of a training system for a self-supervised training predictor for speech separation according to an embodiment of the present invention. The system can execute the training method for a self-supervised training predictor for speech separation described in any of the above embodiments and is configured in a terminal.

[0065] This embodiment provides a training system 10 for a self-supervised training predictor for speech separation, comprising: a feature extraction module 11, a training label determination module 12, an estimated feature determination module 13, and a training module 14.

[0066] The feature extraction module 11 is used to extract self-supervised training features of each individual voice source speech using a pre-trained model; the training label determination module 12 is used to extract shallow features for speech representation and deep features for context information from the self-supervised training features, and determine the shallow and deep features of each individual voice source speech as training labels for the self-supervised training predictor; the estimation feature determination module 13 is used to input the training mixed speech generated by each individual voice source speech into the self-supervised training predictor to obtain the estimated features of each individual voice source speech; the training module 14 is used to train the self-supervised training predictor based on the estimated features of each individual voice source speech and the loss function determined by the training labels corresponding to each individual voice source speech, to obtain the trained self-supervised training predictor.

[0067] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the training method for a self-supervised training predictor for speech separation in any of the above method embodiments.

[0068] In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows:

[0069] Self-supervised training features of speech from each individual voice source were extracted using a pre-trained model.

[0070] The shallow features used for speech representation and the deep features used for context information in the self-supervised training features are extracted, and the shallow features and deep features of each single-person voice source speech are determined as the training labels of the self-supervised training predictor.

[0071] The training mixed speech generated from the individual voice sources is input into the self-supervised training predictor to obtain the estimated features of each individual voice source speech.

[0072] The self-supervised predictor is trained based on the estimated features of each individual voice source speech and the loss function determined by the training labels corresponding to each individual voice source speech, thus obtaining the trained self-supervised predictor.

[0073] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by a processor, they perform the training method for the self-supervised training predictor for speech separation in any of the above method embodiments.

[0074] Figure 10This is a schematic diagram of the hardware structure of an electronic device for a training method of a self-supervised predictor for speech separation, as provided in another embodiment of this application. Figure 10 As shown, the device includes:

[0075] One or more processors 1010 and memory 1020, Figure 10 Taking a processor 1010 as an example, the device for training a self-supervised predictor for speech separation may further include an input device 1030 and an output device 1040.

[0076] The processor 1010, memory 1020, input device 1030, and output device 1040 can be connected via a bus or other means. Figure 10 Taking the example of a connection between China and Israel via a bus.

[0077] The memory 1020, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the training method for the self-supervised training predictor for speech separation in the embodiments of this application. The processor 1010 executes various server functions and data processing by running the non-volatile software programs, instructions, and modules stored in the memory 1020, thereby implementing the training method for the self-supervised training predictor for speech separation described in the above embodiments.

[0078] The memory 1020 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data, etc. Furthermore, the memory 1020 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1020 may optionally include memory remotely located relative to the processor 1010, and these remote memories can be connected to the mobile device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0079] Input device 1030 can receive input numerical or character information. Output device 1040 may include display devices such as a display screen.

[0080] The one or more modules are stored in the memory 1020, and when executed by the one or more processors 1010, they execute the training method for the self-supervised training predictor for speech separation in any of the above method embodiments.

[0081] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0082] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0083] This invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of a training method for a self-supervised training predictor for speech separation according to any embodiment of this invention.

[0084] The electronic devices described in this application exist in various forms, including but not limited to:

[0085] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0086] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as tablet computers.

[0087] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0088] (4) Other electronic devices with data processing functions.

[0089] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0090] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0091] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a self-supervised predictor for speech separation, comprising: Self-supervised training features of speech from each individual voice source were extracted using a pre-trained model. The shallow features used for speech representation and the deep features used for context information are extracted from the self-supervised training features. The shallow features and deep features of each single-speaker speech source are determined as the training labels of the self-supervised training predictor. The shallow features are extracted based on a temporal convolutional neural network model, and the deep features are extracted based on a Transformer model. The training mixed speech generated from the individual voice sources is input into the self-supervised training predictor to obtain the estimated features of each individual voice source speech. The self-supervised training predictor includes a temporal convolutional neural network for extracting temporal speech signals and a dual-path recurrent neural network for context modeling; The self-supervised training predictor is trained based on the estimated features of each individual voice source speech and the loss function determined by the training labels corresponding to each individual voice source speech, to obtain the trained self-supervised training predictor; wherein, the loss function is the mean squared error loss function, to perform permutation invariance training on the self-supervised training predictor.

2. The method according to claim 1, wherein, The pre-trained models include: the Wav2vec unsupervised pre-trained model.

3. The method according to claim 1, wherein, After obtaining the trained self-supervised predictor, the method further includes: The received mixed speech containing multiple speakers is input into the speech separation model, wherein the speech separation includes: an encoder, a speech separator, and a decoder; The encoder encodes the mixed speech to obtain deep features of the mixed speech, and uses the self-supervised training predictor to determine the self-supervised training estimated features of each speaker in the mixed speech. Based on the self-supervised training estimated features of each speaker and the deep features of the mixed speech, the fusion features are determined. The speech separator determines the feature code for each speaker in the fused features; The decoder decodes the feature encoding to obtain the separate speech of each speaker in the mixed speech.

4. A training system for a self-supervised predictor for speech separation, comprising: The feature extraction module is used to extract self-supervised training features of each individual voice source speech using a pre-trained model. The training label determination module is used to extract shallow features for speech representation and deep features for context information from the self-supervised training features, and to determine the shallow features and deep features of each single-person voice source speech as the training labels of the self-supervised training predictor; wherein the shallow features are extracted based on a temporal convolutional neural network model, and the deep features are extracted based on a Transformer. The estimated feature determination module is used to input the training mixed speech generated from the speech of each individual voice source into the self-supervised training predictor to obtain the estimated features of each individual voice source speech. The self-supervised training predictor includes a temporal convolutional neural network for extracting temporal speech signals and a dual-path recurrent neural network for context modeling; The training module is used to train the self-supervised training predictor based on the estimated features of each individual voice source speech and the loss function determined by the training labels corresponding to each individual voice source speech, to obtain the trained self-supervised training predictor; wherein, the loss function is the mean squared error loss function, so as to perform permutation invariance training on the self-supervised training predictor.

5. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-3.

6. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-3.

Images