Weakly supervised speech pre-training method, electronic device, and storage medium

Abstract

The application discloses a weakly supervised speech pre-training method, an electronic device and a storage medium. The weakly supervised speech pre-training method is used for pre-training a model. An input of the pre-training model comprises registered speech and a main input sentence. The method comprises the following steps: introducing the registered speech of a target speaker as auxiliary speaker identity information in a pre-training process; mixing an interference speaker voice in the main input sentence at random; taking a discrete label of the main input sentence as a training target; and training the pre-training model by using masked speech prediction as a training criterion, wherein the main input sentence and the registered speech belong to the same target speaker. Since the registered speech of the speaker is used instead of the speaker embedding representation, the pre-training model can avoid bias to a specific speaker domain, avoid dependence on a specific external speaker model, and reduce the influence on the generalization and robustness of the pre-training model.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and in particular to a weakly supervised speech pre-training method, electronic device, and storage medium. Background Technology

[0002] In recent years, pre-training based on self-supervised learning (SSL) has greatly advanced the research progress of speech processing, showing great potential in a wide range of downstream speech tasks. Existing SSL models can be broadly classified into three categories based on their pre-training objectives: generative, contrastive, and predictive methods. These SSL models are typically pre-trained on large amounts of unlabeled data in an application-agnostic manner, and then fine-tuned on downstream speech tasks by updating the entire network or only a small number of parameters. The learned representations have been found to be applicable to a range of speech tasks, such as Automatic Speech Recognition (ASR), Text-to-Speech (TTS), Speaker Verification (SV), and Speech Enhancement (SE).

[0003] While existing SSL models are highly effective at extracting rich representations from single-speaker utterances, their ability to eliminate interference from overlapping speech remains limited. However, this ability is particularly important when dealing with the well-known cocktail party problem, where multiple speakers address each other simultaneously in a noisy environment. Therefore, it is natural to ask whether current speech pre-training paradigms can be improved to take this into account. Recently, some research has focused on this direction. Chen et al. proposed enhancing the input speech by overlapping it with another audio source, forcing the SSL model to learn masked speech denoising and prediction simultaneously. Wang et al. proposed explicitly predicting multiple labels corresponding to all corpora in the overlapping input speech, enabling the SSL model to learn masked speech prediction while learning denoising and separation. These efforts still fall within the self-supervised speech pre-training framework, where no information other than the speech itself is utilized. Summary of the Invention

[0004] This invention provides a weakly supervised speech pre-training method, an electronic device, and a storage medium to at least solve one of the aforementioned technical problems.

[0005] In a first aspect, embodiments of the present invention provide a weakly supervised speech pre-training method for pre-training a model, wherein the input of the pre-training model includes a registered speech and a main input statement, and the method includes: introducing the registered speech of the target speaker as auxiliary speaker identity information during the pre-training process; randomly mixing interfering speaker speech into the main input statement; using the discrete label of the main input statement as the training target; and using masked speech prediction as the training criterion to train the pre-training model, wherein the main input statement and the registered speech belong to the same target speaker.

[0006] In a second aspect, embodiments of the present invention provide an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute any of the weakly supervised speech pre-training methods described above.

[0007] Thirdly, embodiments of the present invention provide a storage medium storing one or more programs including execution instructions, which can be read and executed by electronic devices (including but not limited to computers, servers, or network devices) to perform any of the weakly supervised speech pre-training methods described above.

[0008] Fourthly, embodiments of the present invention also provide a computer program product, the computer program product including a computer program stored on a storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to execute any of the above-mentioned weakly supervised speech pre-training methods.

[0009] The solution in this application is more suitable for downstream tasks related to the cocktail party problem because it explicitly considers the use of additional speaker information during pre-training and directly separates and predicts specific speaker signals in multi-speaker aliased speech during training. Furthermore, by using speaker-registered speech instead of speaker embedding representations, it avoids biases in the pre-trained model towards specific speaker domains, avoids dependence on specific external speaker models, and reduces the impact on the generalization and robustness of the pre-trained model. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating a weakly supervised speech pre-training method according to an embodiment of the present invention;

[0012] Figure 2 A schematic diagram of a weakly supervised speech pre-training model based on speech data with speaker annotations provided in an embodiment of the present invention;

[0013] Figure 3 A speaker-perceived speech mixing strategy provided in an embodiment of the present invention:

[0014] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] Please refer to Figure 1 The diagram illustrates a flowchart of a weakly supervised speech pre-training method according to an embodiment of the present invention, used for pre-training a model, wherein the input of the pre-trained model includes registered speech and main input statement.

[0017] like Figure 1 As shown, in step 101, the registered voice of the target speaker is introduced as auxiliary speaker identity information during the pre-training process;

[0018] In step 102, interference speaker speech is randomly mixed into the main input statement. The discrete label of the main input statement is used as the training target, and masked speech prediction is used as the training criterion to train the pre-trained model. The main input statement and the registered speech belong to the same target speaker.

[0019] In this embodiment of the application, by explicitly introducing the registered speech of the target speaker during the pre-training process, the speaker information can be used to better cope with subsequent multi-speaker speech processing tasks. Randomly mixing interfering speaker speech with an overlap rate between 0% and 100% into the main input sentence can force the model to use the additional target speaker registered speech information to eliminate interference.

[0020] The solution in this application is more suitable for downstream tasks related to the cocktail party problem because it explicitly considers the use of additional speaker information during pre-training and directly separates and predicts specific speaker signals in multi-speaker aliased speech during training. Furthermore, by using speaker-registered speech instead of speaker embedding representations, it avoids biases in the pre-trained model towards specific speaker domains, avoids dependence on specific external speaker models, and reduces the impact on the generalization and robustness of the pre-trained model.

[0021] In some optional embodiments, the pre-trained model includes two bottom-level convolutional neural network encoders and a top-level Transformer encoder. The method further includes: encoding the registered speech and the main input sentence using the two convolutional neural network encoders respectively, obtaining corresponding registered speech feature sequences and main sentence feature sequences; performing a masking operation on a portion of the frame vectors of the main sentence feature sequence; concatenating the registered speech feature sequence and the masked main sentence feature sequence in the time dimension to obtain a concatenated sequence; inputting the concatenated sequence into the Transformer encoder for modeling; obtaining the frame-by-frame classification results output by the Transformer encoder; comparing the portion of the frame-by-frame classification results corresponding to the masked frames with the discrete labels of the main input sentence one by one, calculating a cross-entropy-based loss function, and using it for backpropagation to update the model parameters of the pre-trained model. By repeating the above training process on a large amount of speech data with only speaker annotations, a speech pre-trained model with modeling target speaker information and interference resistance can be obtained.

[0022] In some optional embodiments, concatenating the registered speech feature sequence and the masked main sentence feature sequence in the time dimension to obtain the concatenated sequence includes: performing additional relative position encoding on both the registered speech feature sequence and the masked main sentence feature sequence; applying a registered speech bias to the relative position encoding of the registered speech feature sequence and a main sentence bias to the relative position encoding of the masked main sentence feature sequence, wherein the registered speech bias and the main sentence bias are two vectors trained together with the pre-trained model, and are added frame-by-frame with the corresponding feature sequences during use to complete the biasing process. Thus, through additional relative position encoding and the separately applied registered speech bias and main sentence bias, the difference between the two feature sequences can be increased, allowing the Transformer encoder to better distinguish between the two different feature sequences.

[0023] In some alternative embodiments, the two convolutional neural network encoders are identical and share inputs. This reduces the variability introduced during speaker information encoding, making the pre-trained model more stable.

[0024] In some alternative embodiments, the two convolutional neural network encoders are independent convolutional neural network encoders. This allows the model to model the two inputs more freely in different dimensions, resulting in more discriminative target speaker information.

[0025] In some optional embodiments, the trained pre-trained model is used for the target speech recognition task. The method further includes: using a linear projection layer on top of the pre-trained model to map features to the output dimension, and using connection-temporal classification as a loss function during end-to-end speech recognition training. Thus, the pre-trained model can be used for the target speech recognition task by fine-tuning it by adding a linear projection layer, resulting in a target speaker speech recognition model with good performance.

[0026] In some alternative embodiments, the trained pre-trained model is used for a target speech recognition task, and the method further includes: inserting a lightweight speaker adaptation layer into the pre-trained model for joint fine-tuning, wherein the speaker adaptation layer takes pre-extracted speaker embedding vectors as input to guide intermediate representations in the pre-trained model toward the target speaker.

[0027] Among related technologies, similar speech pre-training techniques currently include HuBERT (Hidden-Unit BERT) and WavLM.

[0028] The HuBERT technique proposes using a large model based on a cascaded structure of a CNN encoder and a Transformer encoder for self-supervised pre-training on unlabeled speech data. The training criterion is masked speech prediction, which involves partially masking the speech feature sequence processed by the CNN encoder, and then predicting the discrete labels corresponding to the masked speech features using the Transformer encoder. These discrete labels come from an external acoustic unit discovery system, typically obtained by performing a fixed number of K-Means clusters on speech features (such as Mel-frequency cepstral coefficients (MFCCs) or the output features of a specific layer of the Transformer encoder). Through this self-supervised pre-training on a large amount of speech data, the HuBERT model can learn representations with rich acoustic and semantic information, allowing it to be fine-tuned in downstream tasks with only a small amount of labeled data and achieve excellent performance.

[0029] WavLM is an improvement on Hubert, primarily in three aspects. First, it introduces a gating relative position bias into the attention mechanism of the Transformer encoder in Hubert's model structure, enhancing model performance with only a small increase in parameters. Second, WavLM augments the input speech data during the pre-training phase. For each sample in each training batch, it employs a dynamic sentence mixing strategy, randomly selecting a sentence from other samples in the current batch as interference, superimposing it onto the current sample at a certain signal-to-noise ratio, and ensuring that the aliased portion does not exceed 50% of the current sample length, thus improving the model's robustness against interference. Third, WavLM expands the amount of pre-training data from a maximum of 60,000 hours to 94,000 hours, and observes the performance gain resulting from this increased data volume.

[0030] The aforementioned techniques all belong to self-supervised learning (SSL) methods, which inherently lack the utilization of additional information. However, the pre-training data collected in practice often contains partial data annotations, such as relative speaker identities. Due to the lack of explicit utilization of this meta-information, these techniques struggle to effectively leverage the additional relevant information during fine-tuning in downstream tasks, limiting their performance across different downstream tasks. Furthermore, none of these techniques explicitly model scenarios involving multiple speakers speaking simultaneously during the pre-training phase, such as the cocktail party problem. Although WavLM adds data augmentation with added perturbations to the pre-training data, it is essentially still a noisy version of the original training samples and does not guide the model to separate different speech segments in the input signal. Therefore, when applied to downstream tasks related to multi-speaker speech processing (such as speech separation and target speaker speech recognition), these techniques often offer only minor performance improvements compared to traditional methods that do not employ pre-trained models.

[0031] Since the aforementioned techniques do not explicitly consider multi-speaker speech processing tasks or utilize additional target speaker information during pre-training, they are also difficult to achieve good performance for multi-speaker speech processing tasks in cocktail party scenarios when fine-tuning in downstream tasks.

[0032] To mitigate these shortcomings, other practitioners in this field typically introduce additional information, such as the target speaker, during the fine-tuning stage of downstream tasks. This can be achieved by inserting a new speaker adaptation layer into the pre-trained model to incorporate speaker information, or by fixing the parameters of the pre-trained model and using it as a feature extractor, while focusing the target speaker modeling task on the downstream task model. They generally do not attempt to improve the pre-trained model specifically for the target speaker task because the associated costs could be high, and they typically use existing speaker representations as speaker information input, rather than directly modeling the registered speech.

[0033] The weakly supervised speech pre-training method proposed in this application explicitly introduces the registered speech of the target speaker as auxiliary speaker identity information during the pre-training process, and uses a Transformer encoder to achieve implicit speaker information utilization. Furthermore, it randomly mixes interfering speaker speech with an overlap rate between 0% and 100% into the input main sentence, still using the discrete labels of the main sentence as the training target. Compared to WavLM, this significantly increases the difficulty of pre-training, forcing the model to utilize the additional target speaker registered speech information to eliminate interference. These two modifications explicitly consider the utilization of additional speaker information during pre-training and directly separate and predict the specific speaker signals in multi-speaker aliased speech during training, thus making it more suitable for downstream tasks related to the cocktail party problem.

[0034] Please refer to Figure 2 The diagram illustrates a weakly supervised speech pre-training model based on speech data with speaker annotations, according to an embodiment of this application.

[0035] Figure 2 This diagram illustrates our proposed weakly supervised speech pre-training model based on speaker-annotated speech data, hereinafter referred to as TS-HuBERT. Its input consists of two parts: the main input utterance y and the target speaker registration utterance e. During pre-training, the main input utterance is randomly mixed with interference speech from other speakers, with an overlap ratio between 0% and 100%. The target speaker registration utterance, on the other hand, comes from the same speaker as the main input utterance and is used to provide auxiliary information about the target speaker. Our proposed method adopts a structure similar to the WavLM model, mainly composed of a low-level CNN encoder and a high-level Transformer encoder. We use the discrete labels corresponding to the main input utterance as the pre-training target and masked speech prediction as the training criterion.

[0036] In designing the TS-HuBERT model, we did not use common speaker embedding representations as input speaker information. Instead, we directly modeled the original registered speech. This was to avoid introducing a dependency on external speaker embedding representation models into the pre-trained model, which would affect the model's generalization performance.

[0037] The TS-HuBERT model uses the same CNN encoder to encode features from two input speech sequences, resulting in two feature sequences: the registered speech feature sequence [X1', X2', ..., X...]. L '] and the main statement feature sequence [X1,X2,…,X TThe inventors mask some frame vectors of the main sentence feature sequence by replacing them with zero vectors, thus increasing the difficulty of pre-training and making the model's learned representations more robust and generalizable. To enable the Transformer encoder to distinguish between the two different sequences and focus on the masked speech prediction task of the main sentence, we perform additional relative position encoding (implemented via a convolutional neural network) on each sequence before concatenating and inputting them into the Transformer encoder. We also apply corresponding registered speech biases and main sentence biases to each sequence to further increase the difference between them. The registered speech bias and main sentence bias are two vectors trained along with the model and are added frame-by-frame with the corresponding feature sequences during use to complete the biasing process. Finally, we concatenate the two processed feature sequences in the time dimension to form a longer sequence, which is then input into the Transformer encoder for modeling. The final output of the Transformer encoder is the frame-by-frame classification result of its input sequence. We assume that the first few frames correspond to the registered speech of the target speaker and will not be used, while the remaining prediction results correspond to the main input sentence. We compare the parts corresponding to the masked frames with the discrete labels one by one, calculate the loss function based on cross-entropy, and use it for backpropagation to update the model parameters.

[0038] By repeating this process on a large amount of speech data with only speaker annotations, we can obtain a pre-trained speech model that models target speaker information and is robust to interference. In the downstream target speaker speech recognition task, we directly insert a linear layer on top of the pre-trained TS-HuBERT model for CTC training, using mixed main sentences and target speaker registered speech as inputs, to obtain a target speaker speech recognition model with good performance.

[0039] In implementing this application, the inventors also explored several alternative solutions. One such solution was to replace the input target speaker information from the registered speech with a pre-extracted speaker embedding representation, which is typically just a vector and therefore more compact. During pre-training, this vector can be concatenated to the front of the main input sentence feature sequence in a similar manner as a prompt to guide the model to exclude interfering speaker information in the main input sentence and focus on the corresponding target speaker's speech.

[0040] The advantage of this approach is that it is more compact than the target speaker's registered speech, with a fixed length (only 1). Therefore, the increase in GPU memory / RAM and computational cost when modeling with a Transformer encoder is negligible. The disadvantage is that the speaker embedding representation relies on a pre-trained speaker model on specific speech data for extraction. If used for large-scale pre-training, it is likely to cause the pre-trained model to develop a bias towards a specific speaker domain, i.e., a dependence on a specific external speaker model, thus affecting the generalization and robustness of the pre-trained model.

[0041] In designing the TS-HuBERT model of this application embodiment, the inventors also considered changing the CNN encoder shared by the two inputs in the model structure to two independent CNN encoders, each processing one of the two inputs. The advantage of this approach is that it allows the model to more freely model the two inputs from different dimensions, thereby obtaining more discriminative target speaker information. However, the disadvantage is that it introduces more variability in the encoding of speaker information. Since there is no supervision signal to explicitly constrain the feature information of the corresponding speaker's registered speech during pre-training, this independent CNN encoder design may lead to instability in the model's pre-training process, thus affecting the final performance.

[0042] This application aims to explore a novel direction for speech pre-training: weakly supervised speech pre-training, which allows the use of additional information about the data (weak labels) to facilitate better speech pre-training. While similar ideas have been explored in computer vision and natural language processing, they have been less studied in the field of speech processing. The motivation for this application is that the collected speech data used for pre-training may contain annotations of meta-information, such as relative speaker identity. Although SSL methods typically omit this information during pre-training, its proper utilization during pre-training can be helpful for relevant downstream tasks.

[0043] The TS-HuBERT method proposed in this application not only provides a target speaker speech pre-training model for scenarios where multiple people speak simultaneously at a cocktail party, effectively improving the performance of the pre-trained model in target speaker speech processing tasks, but also provides a new weakly supervised speech pre-training paradigm. Unlike traditional self-supervised speech pre-training, it emphasizes the effective utilization of meta-information such as relative speaker labels in speech data, and has greater potential than traditional pre-training methods, which may inspire more research and applications of speech pre-training based on weakly supervised methods.

[0044] Therefore, in this embodiment, we first attempt to use speaker-registered speech for weakly supervised speech pre-training. The only assumption is that all speech samples can be divided into S groups, each corresponding to a different speaker. The speaker division can be obtained from clustering or data annotation. Based on this assumption, this embodiment proposes a novel speech pre-training model—Target Speaker HuBERT, or TS-HuBERT for short. It adopts the same masked speech prediction target proposed in HuBERT, which is a widely used SSL model. During pre-training, the main input speech is randomly mixed with speech from different speakers, and auxiliary speech samples from the same main speaker are provided to guide the modeling of the main speaker's speech. Using these two waveforms as input, the proposed pre-training model learns to predict discrete targets in masked frames while eliminating interference in overlapping speech. Furthermore, the inventors investigated the effectiveness of the proposed TS-HuBERT model in downstream target speech recognition tasks, which aim to identify the speech of the target speaker in overlapping speech. Our experiments show that the proposed model achieves significantly better ASR performance compared to the state-of-the-art SSL model WavLM with denoising capabilities.

[0045] Speech pre-training under weak supervision: model design

[0046] Figure 2 An overview of the proposed TS-HuBERT model is presented. The entire model is built on a mature architecture, consisting of a convolutional neural network (CNN) encoder and a Transformer encoder. Furthermore, gated relative position bias is used in the self-attention mechanism of the Transformer encoder layer to improve performance. The input to the model proposed in this application includes a main input phrasal for masked speech prediction (y) and an auxiliary input e containing target speaker information. Details of the pre-training data will be described later.

[0047] Now, embodiments of this application consider integrating the target speaker's information with the aforementioned backbone. Two types of data are frequently used to provide this information: the registered audio from the target speaker and the speaker embedding vector. While the latter can directly provide a compact and efficient representation of the target speaker, it entangles the pre-trained model with a specific external speaker embedding model. This inevitably reduces the versatility and flexibility of the speech pre-trained model. Therefore, the inventors choose the original registered audio as auxiliary input and rely on the pre-trained model to implicitly extract the target speaker's information. Inspired by the success of cross-modal modeling based on temporal concatenation, embodiments of this application propose fusing these two input streams through temporal concatenation. More specifically, the same CNN encoder is first used to extract features from the registered speech e and the main input statement y:

[0048]

[0049]

[0050] Here, X and X′ represent the main statement feature and the registered speech feature, respectively. T and L are the number of corresponding feature frames, and D is the feature dimension. Later, we will concatenate these two features along the time dimension and utilize a Transformer encoder to learn the implicit association between them. However, the simple concatenation of the two data streams makes them difficult to distinguish in a self-attention mechanism. To alleviate this problem, embodiments of this application apply two independent convolution-based relative position encoding (rPE) layers to inject temporal sequence information into the two data streams. Furthermore, the difference between the two sequences is further amplified by adding a learnable partial vector to each sequence:

[0051] X in =Conv rel_pos (X)+B main (3)

[0052] X′ in =Conv′ rel_pos (X′)+B enroll (4)

[0053] in, and Let X represent the bias vector of the main corpus and the registration bias vector, respectively. The updated feature X in and X′ in Finally, the data are concatenated and fed into a Transformer encoder to predict the frame-level labels corresponding to the main statement. The output labels corresponding to the registered frame are discarded through slicing.

[0054] Speaker-known masked speech prediction

[0055] This section introduces the training objective of the proposed pre-training method, which follows the design of the HuBERT model. The basic idea of ​​the masked speech prediction objective in HuBERT is to predict the frame-by-frame discrete labels of the input speech based on the partial feature frames generated by the masked CNN encoder. The masking prediction loss (actually the cross-entropy loss) is only calculated for the masked frames, while the labels are generated by one or more iterative K-means clusters on the features of the input speech.

[0056] Since the primary objective of this application is to improve the interference cancellation capability of a speech pre-trained model through weak supervision, the inventors have extended the original objective in HuBERT to accommodate this objective. Given a main input utterance y and a registered utterance e of a target speaker, the inventors only mask the main input utterance feature X obtained in Equation (1), while leaving the registered utterance feature X′ unchanged. During pre-training, only the cross-entropy loss of the main input utterance is calculated, with its label pre-prepared, similar to the HuBERT training process. This is to force the model to focus on the extraction of the target speaker with complete registration. Furthermore, to explicitly consider the cocktail party problem, the inventors mix the main input utterance y with randomly sampled corpora from another speaker to achieve online data augmentation. The detailed process is described later.

[0057] In our implementation, the mask length is fixed at 10 frames, and the number of masks is proportional to the length of the main utterance. The maximum proportion of masked frames is 80%, but this is usually not reached due to potential overlap between different selected masks. Masked feature frames are simply replaced with zeros.

[0058] Preparation of pre-training data

[0059] This application proposes a speaker-aware corpus mixing strategy to simulate online aliasing pre-training data. The core algorithm is described in Algorithm 1. The main differences compared to WavLM are: (1) We extract interfering speech from the entire database instead of the current batch. (2) We set the overlap rate range to [0, 100%] instead of [0, 50%]. (3) An additional registered speech e from the same speaker is added to provide information about the target speaker.

[0060] Application: Target speech recognition

[0061] This section introduces several fine-tuning methods and applies the proposed TS-HuBERT model to a downstream target speech recognition task. Its aim is to identify the target speaker's voice in overlapping speech, a typical task in the cocktail party problem. A straightforward fine-tuning approach is to use a linear projection layer on top of the pre-trained model to map features to the output dimension (vocabulary size). Connectionist Temporal Classification (CTC) loss is used for end-to-end training. During fine-tuning, the parameters of the entire pre-trained model (except for the CNN encoder) are updated to adapt to the downstream task. Since TS-HuBERT can extract target speaker information from registered speech, it is naturally well-suited for this fine-tuning method.

[0062] Figure 3 This invention provides a speaker-perceived speech mixing strategy according to an embodiment of the present invention.

[0063] For SSL models that cannot be directly utilized with registration, an alternative adaptation-based fine-tuning method can be used. This involves inserting a lightweight speaker adaptation layer into the pre-trained model for joint fine-tuning. The newly inserted layer will use the pre-extracted speaker embedding vector e emb As input, the intermediate representations of the pre-trained model are used to guide the approximation of the target speaker. Here, the inventors evaluated the following three adaptation layers using TS-HuBERT:

[0064] 1. Addition: e emb The hidden dimension is matched by directly adding it to the output X of the CNN encoder through linear projection.

[0065] 2.FiLM: e emb Feature-wise Linear Modulation (FiLM) was used to estimate the output X of a CNN.

[0066]

[0067] w(·) and b(·) are two linear projection layers.

[0068] 3.cLN:e emb The FiLM transform used to estimate the layer normalizations (LNs) in the first Transformer encoder layer is converted into conditional LNs (cLNs):

[0069]

[0070] Where γ and β are the learnable proportion and bias parameters in the standard LN, respectively. μ and σ are the mean and standard deviation of the input X, respectively.

[0071] The inventors applied approximately equal initialization to all adaptation layers, which proved crucial in initial experiments.

[0072] The following experiments verify the beneficial effects of the embodiments of this application compared to the prior art.

[0073] Experimental setup

[0074] The experiments in this embodiment were conducted using the fairseq tool. The TS-HuBERT model follows the same structure as the WavLM base model, enhancing the HuberT Transformer encoder by employing gated relative position biases in the self-attention mechanism. Due to space limitations, detailed parameters are provided in the WavLM paper. As mentioned earlier, this embodiment also inserts two convolutional rPE layers and two 768-dimensional learnable bias vectors before the Transformer encoder. Each rPE layer consists of a 16-group convolutional layer with a kernel size of 128 and Gaussian error linear units (GELU) activation. These newly added layers account for 10 million new parameters, resulting in a total of 104.37 million parameters. The TS-HuBERT model was pre-trained on 960 hours of Librispeech data for one iteration (400k steps) using labels generated by 500-class K-means clustering from the output of the 9th Transformer layer of the HuberT base model. The batch size per GPU was a maximum of 312 seconds of audio. Other hyperparameters are the same as those in the base Hubert model. The registered speech e was randomly truncated to 48,000 samples to avoid excessive memory consumption during pre-training.

[0075] For fine-tuning of downstream tasks, the inventors used character-level CTC loss in all experiments. The inventors evaluated the performance of target speaker speech recognition on two commonly used datasets: noisy Libri2Mix and WSJ0-2mix. All speech data were sampled at 16kHz. In noisy Libri2Mix, the training, validation, and evaluation sets contained 13,900, 3,000, and 3,000 samples, respectively. In WSJ0-2mix, the sample counts for these three datasets were 20,000, 5,000, and 3,000, respectively. For the validation and evaluation sets of Libri2Mix and WSJ0-2mix, the inventors used existing sample registration lists provided in previous work. For the training set, the inventors randomly selected another sentence from the same speaker online for each sample as the registration speech to increase diversity. The batch size for fine-tuning was 125 seconds of audio per GPU. The peak learning rate was 2e-5, with 8,000 warm-up steps at the beginning of training. During fine-tuning, the inventors always froze the CNN encoder parameters in the pre-trained model. For adaptive fine-tuning, the inventors used a ResNet-34 model pre-trained on the VoxCeleb corpus to extract 256-dimensional speaker embedding vectors. During evaluation, the inventors employed the Viterbi algorithm for decoding and calculated the word error rate (WER) as a metric. Unless otherwise specified, language models were not used in the experimental results described below. The inventors used eight RTX 2080Ti GPUs in all experiments.

[0076] Evaluation of standard speech recognition

[0077] Before applying the pre-trained model proposed in this application embodiment to target speaker speech recognition, the inventors first evaluated the model's performance on standard ASR tasks. To this end, the inventors fine-tuned the TS-HuBERT model on three different subsets of Librispeech and evaluated it on the test-clean and test-other subsets. The inventors largely adopted the fine-tuning configuration from previous work HuBERT and used a batch size of 200s per GPU. Since there is no registered speech in standard ASR tasks, the inventors removed the processing associated with registered speech and only fed the main input phrasing features into the Transformer encoder (denoted as "speakerless embedding vector"). The inventors compared three speech pre-trained models in Table 1, all of which were pre-trained for 400k steps on 960 hours of Librispeech data. Interestingly, although the model proposed in this application embodiment always has an additional registered input during pre-training, it can still achieve comparable performance to the two SSL models even without registered speech. This reveals the potential of the proposed TS-HuBERT model in this application to adapt to different tasks.

[0078]

[0079] Table 1: WER (%) evaluation of pre-trained speech recognition models for standard speech recognition on the Librispeech test set. Note that none of the models listed used a language model.

[0080] Evaluation of target speech recognition

[0081] Next, the performance of the pre-trained speech model on the target speech recognition task is evaluated. Experiments were first conducted on the WSJ0-2mix dataset. The results are shown in Table 2. First, comparisons were made with two state-of-the-art multi-speaker ASR methods on WSJ0-2mix: a jointly trained speech separation and ASR model (denoted as "DPRNN-ASR") and a single multi-speaker ASR model (denoted as "PIT-ASR"). For the SSL-based method, the inventors compared it with the WavLM base model, as it is a state-of-the-art SSL model with denoising capabilities and has a similar pre-training setup to the embodiments of this application. The adaptive fine-tuning method described above was also applied to the WavLM base model to achieve target speech recognition.

[0082] As shown in Table 2, directly fine-tuning TS-HuBERT without using an adaptation layer easily yields very promising results (WER < 7%). Adaptive fine-tuning methods also work well in both WavLM Base and TS-HuBERT; increasing the number of fine-tuning steps slightly improves performance. The Transformer encoder in the pre-trained model is typically frozen in the first 10k fine-tuning steps. Here, the inventors found through experiments that this technique is detrimental to target speech recognition performance and therefore disabled it in the experiments described below. Notably, the TS-HuBERT model proposed in this application achieves new state-of-the-art performance (WER = 6.1%) on WSJ0-2mix, which significantly outperforms the WavLM Base model.

[0083] Furthermore, the inventors evaluated these models on the noisy Libri2Mix dataset, which is more challenging due to the noise. Due to space limitations, only the best results of the adaptive fine-tuning method are presented. Additionally, the inventors compared the models with the ESPnet multi-speaker ASR method (denoted as "PIT-ASR"), and enhanced it by adding more training data (denoted as "+train-360") and further applying velocity perturbation (SP) and the Transformer Language Model (LM). Although the pre-trained models were trained using only a subset of train-100, they achieved substantial performance improvements over the PIT-ASR method, even though the latter used more training data ("+train-360"). The TS-HuBERT proposed in this application again outperforms the WavLM base model, with a 10% reduction in relative word error rate. However, their performance still lags behind the double-enhanced PIT-ASR method ("+velocity perturbation & language model"), meaning there is room for further improvement in noisy cases.

[0084]

[0085] Table 2: Evaluation of different pre-trained speech recognition models on the WSJ0-2mix test set. "Number of parameters" indicates the number of trainable parameters. "Freeze parameters in the first 10k steps" indicates whether the pre-trained model is frozen in the first 10k steps. "50k" and "100k" indicate the fine-tuning steps.

[0086]

[0087] Table 3: Evaluation of different pre-trained speech recognition models for target speech recognition on the noisy Libri2mix test set. "Number of parameters" indicates the number of trainable parameters. "100k" and "250k" indicate fine-tuning steps.

[0088] In this application, a novel weakly supervised speech pre-training model, TS-HuBERT, is proposed. It utilizes additional registration information to leverage target speaker information. By pre-training on overlapping speech data with a masked speech prediction target, TS-HuBERT can effectively utilize registered speech information to learn to eliminate interfering human voices. Experiments show that TS-HuBERT works well on both standard ASR and target speaker speech recognition tasks.

[0089] In other embodiments, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the weakly supervised speech pre-training method in any of the above method embodiments for pre-training a model, wherein the input of the pre-trained model includes a registered speech and a main input statement;

[0090] In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows:

[0091] The registered voice of the target speaker is introduced as auxiliary speaker identity information during the pre-training process;

[0092] The speech of the interfering speaker is randomly mixed into the main input statement. The discrete label of the main input statement is used as the training target, and the masked speech prediction is used as the training criterion to train the pre-trained model. The main input statement and the registered speech belong to the same target speaker.

[0093] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the voice editing device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely located relative to the processor, and this remote memory may be connected to the voice editing device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0094] This invention also provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer, cause the computer to perform any of the aforementioned weakly supervised speech pre-training methods.

[0095] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present invention, such as... Figure 4 As shown, the device includes one or more processors 410 and a memory 420. Figure 4 Taking a processor 410 as an example, the device for the weakly supervised speech pre-training method may further include an input device 430 and an output device 440. The processor 410, memory 420, input device 430, and output device 440 can be connected via a bus or other means. Figure 4 Taking a bus connection as an example, the memory 420 is the aforementioned non-volatile computer-readable storage medium. The processor 410 executes various server functions and data processing by running non-volatile software programs, instructions, and modules stored in the memory 420, thereby implementing the weakly supervised speech pre-training method described in the above embodiment. The input device 430 can receive input digital or character information and generate key signal inputs related to user settings and function control of the communication compensation device. The output device 440 may include a display screen or other display device.

[0096] The above-described product can execute the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.

[0097] In one implementation, the aforementioned electronic device is used in a weakly supervised speech pre-training device for pre-training a model. The input to the pre-training model includes registered speech and a main input statement. The device includes: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to:

[0098] The registered voice of the target speaker is introduced as auxiliary speaker identity information during the pre-training process;

[0099] The speech of the interfering speaker is randomly mixed into the main input statement. The discrete label of the main input statement is used as the training target, and the masked speech prediction is used as the training criterion to train the pre-trained model. The main input statement and the registered speech belong to the same target speaker.

[0100] The electronic devices described in this application exist in various forms, including but not limited to:

[0101] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones, etc.

[0102] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0103] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes: audio and video players (e.g., iPods), handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0104] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0105] (5) Other electronic devices with data interaction functions.

[0106] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0107] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A weakly supervised speech pre-training method for pre-training models, wherein, The input to the pre-trained model includes the registered speech and the main input statement, and the method includes: The registered voice of the target speaker is introduced as auxiliary speaker identity information during the pre-training process; The speech of the interfering speaker is randomly mixed into the main input statement. The discrete label of the main input statement is used as the training target, and the masked speech prediction is used as the training criterion to train the pre-trained model. The main input statement and the registered speech belong to the same target speaker.

2. The method according to claim 1, wherein, The pre-trained model includes two convolutional neural network encoders at the bottom layer and a Transformer encoder at the top layer, wherein the method further includes: The two convolutional neural network encoders are used to encode the registered speech and the main input sentence respectively, to obtain the corresponding registered speech feature sequence and main sentence feature sequence respectively; Perform a masking operation on a portion of the frame vectors of the main statement feature sequence; The registered speech feature sequence and the main sentence feature sequence after masking are concatenated in the time dimension to obtain the concatenated sequence. The spliced ​​sequence is input into the Transformer encoder for modeling; Obtain the frame-by-frame classification results output by the Transformer encoder; The portion of the frame corresponding to the masked operation in the frame-by-frame classification result is compared one by one with the discrete label of the main input statement. A loss function based on cross-entropy is calculated and used for backpropagation to update the model parameters of the pre-trained model.

3. The method according to claim 2, wherein, The concatenated sequence obtained by concatenating the registered speech feature sequence and the masked main statement feature sequence in the time dimension includes: Additional relative position encoding is performed on the registered speech feature sequence and the main sentence feature sequence after the masking operation, respectively; A registered speech bias is applied to the relative position encoding of the registered speech feature sequence, and a main sentence bias is applied to the relative position encoding of the main sentence feature sequence after the masking operation. The registered speech bias and the main sentence bias are two vectors trained together with the pre-trained model. When used, they are added to the corresponding feature sequences frame by frame to complete the biasing process.

4. The method according to claim 2, wherein, The two convolutional neural network encoders are the same encoders and share the same input.

5. The method according to claim 2, wherein, The two convolutional neural network encoders are the same encoders and the two convolutional neural network encoders are independent of each other.

6. The method according to any one of claims 1-5, wherein, The pre-trained model is used for the target speech recognition task, and the method further includes: A linear projection layer is used on top of the pre-trained model to map features to the output dimension, and connection-temporal classification is used as the loss function during end-to-end speech recognition training.

7. The method according to any one of claims 1-5, wherein, The pre-trained model is used for the target speech recognition task, and the method further includes: A lightweight speaker adaptation layer is inserted into the output of the CNN encoder or the Transformer encoder layer of the pre-trained model and jointly fine-tuned with the pre-trained model. The speaker adaptation layer takes the speaker embedding vector extracted from the registered speech as input to guide the intermediate representation in the pre-trained model toward the target speaker.

8. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images