Front-end adapter training method, electronic device, and storage medium

Abstract

This invention discloses a front-end adapter training method, electronic device, and storage medium. A front-end adapter includes a first stage and a second stage. The first stage includes: inputting a waveform format of speech into an original front-end to obtain a first output of the original front-end; inputting other formats of the speech into the front-end adapter to obtain a second output of the front-end adapter; calculating a first loss between the first output and the second output to train the front-end adapter; and inputting the second output of the front-end adapter into a backbone Transformer model to calculate a second loss, wherein the first loss and the second loss simultaneously optimize a preset epoch. By minimizing the distance between different front-end outputs, features of other speech formats can also be compatible with a self-supervised learning model pre-trained using waveforms.

Description

Technical Field

[0001] This invention belongs to the field of front-end adapter training technology, and particularly relates to front-end adapter training methods, electronic devices, and storage media. Background Technology

[0002] Among related technologies, self-supervised speech models include the wav2vec series, HuBERT, and data2vec. These models are trained using massive amounts of unlabeled data during the pre-training phase and then trained using a small amount of labeled data during the fine-tuning phase.

[0003] Therefore, although the aforementioned efforts in keyword discovery tasks have greatly improved performance under certain specific conditions, some unresolved issues limit the generalizability of these methods.

[0004] In the process of developing this application, the inventors discovered that these models require input data in waveform format during the pre-training, fine-tuning, and decoding stages, which is not suitable for production scenarios. Summary of the Invention

[0005] This invention provides a front-end adapter training method, an electronic device, and a storage medium to at least solve one of the above-mentioned technical problems.

[0006] In a first aspect, embodiments of the present invention provide a front-end adapter training method, wherein the training method includes a first stage and a second stage, the first stage including: inputting the waveform format of speech into an original front-end to obtain a first output of the original front-end; inputting other formats of the speech into the front-end adapter to obtain a second output of the front-end adapter; calculating a first loss of the first output and the second output to train the front-end adapter; and inputting the second output of the front-end adapter into a backbone Transformer model to calculate a second loss, wherein the first loss and the second loss simultaneously optimize a preset epoch.

[0007] In a second aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the front-end adapter training method of any embodiment of the present invention.

[0008] Thirdly, embodiments of the present invention also provide a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the steps of the front-end adapter training method of any embodiment of the present invention.

[0009] The method in this application embodiment, by training front-end adapters using other formats of the same speech, enables front-end adapters using other formats to achieve essentially the same effect as those using waveform formats. By minimizing the distance between different front-end outputs, the features of other speech formats can also be compatible with self-supervised learning models pre-trained using waveforms. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating a front-end adapter training method according to an embodiment of the present invention;

[0012] Figure 2 This is the first stage of a front-end adapter provided in an embodiment of the present invention;

[0013] Figure 3 The second stage of a front-end adapter provided in an embodiment of the present invention;

[0014] Figure 4 This section explains three different types of self-supervised learning models for related technologies.

[0015] Figure 5 The waveform-based fine-tuning bit error rate result provided in an embodiment of the present invention;

[0016] Figure 6 This invention provides a variation of the Euclidean distance between the front-end output based on waveform and Fbank, as provided in one embodiment of the invention.

[0017] Figure 7 The bit error rate results of three self-supervised learning models provided in an embodiment of the present invention on the Librispeech and Gigaspeech test sets;

[0018] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please refer to Figure 1 The diagram illustrates a flowchart of an embodiment of the front-end adapter training method of this application. The above method can be used for self-supervised learning speech models, wherein the front-end adapter training method includes a first stage and a second stage. The training steps of the first stage are as follows:

[0021] like Figure 1 As shown, in step 101, the waveform format of the speech is input to the original front end, and the first output of the original front end is obtained;

[0022] In step 102, other formats of the speech are input to the front-end adapter, the second output of the front-end adapter is obtained, and the first loss of the first output and the second output is calculated to train the front-end adapter;

[0023] In step 103, the second output of the front-end adapter is input to the backbone Transformer model to calculate the second loss, wherein the first loss and the second loss are simultaneously optimized for a preset epoch.

[0024] In this embodiment, by training front-end adapters using other formats of the same speech, it is possible to achieve essentially the same effect as a front-end using the waveform format. By minimizing the distance between different front-end outputs, features of other speech formats can also be compatible with a self-supervised learning model pre-trained using waveforms.

[0025] In some optional embodiments, the second stage includes: inputting other formats of the speech into the front-end adapter, then inputting the obtained third output of the front-end adapter into the backbone Transformer model, and calculating only the second loss.

[0026] In some optional embodiments, the first loss is L2 loss, the second loss is CTC loss, and the preset epoch is epoch 200.

[0027] In some alternative embodiments, the other formats include the Fbank format and the MFCC format.

[0028] In some embodiments, this application also provides a self-supervised speech model, including a front-end adapter and a backbone Transformer model trained according to the aforementioned method. Thus, the self-supervised speech model including the front-end adapter can accept speech input in any format.

[0029] In some optional embodiments, the self-supervised speech model is a speech recognition model.

[0030] It should be noted that the above method steps are not intended to limit the execution order of each step. In fact, some steps may be executed simultaneously or in the reverse order of the steps, and this application does not impose any restrictions on this.

[0031] The following description addresses some problems encountered by the inventors in realizing this invention and provides a specific embodiment of the final solution, so that those skilled in the art can better understand the solution of this application.

[0032] The inventors found that the above defects were mainly caused by the following reasons: the front end of the model is fixed and the same at different stages.

[0033] To address the aforementioned shortcomings in related technologies, those skilled in the art typically choose waveform format data for training. The reasons why this is not readily apparent to those skilled in the art are: the local loss function for the front-end adapter is not easily conceived, and the need to use different loss functions for different training stages is not readily apparent.

[0034] In this embodiment of the application, given a pre-trained model with waveform as the front-end input, our model trains a front-end adapter during the fine-tuning stage, so that input in FBank or MFCC format can be used during decoding.

[0035] Please refer to Figure 2 and Figure 3 The diagram illustrates a block diagram of a front-end adapter training system provided in an embodiment of this application. Figure 2 The first stage of a front-end adapter is shown as a concrete example. Figure 3 The second phase of a front-end adapter is shown as a specific example.

[0036] Phase 1: Input the waveform and FBank formats of the same speech into the original front end and the adapter front end respectively, and calculate the L2 loss at the output position of the front end;

[0037] Input the output of the adapter front end into the backbone Transformer model and calculate the CTC loss;

[0038] Both losses are optimized simultaneously for 200 epochs.

[0039] The second stage: After the FBank format input is adapted to the front end, it enters the main Transformer model, and finally only the CTC loss is calculated.

[0040] The solution proposed in this application allows self-supervised speech models to be used without being limited to waveform-formatted input, but rather with any given speech features, thus enhancing the flexibility of self-supervised speech models. Furthermore, its application extends beyond speech recognition; other fields can also utilize the front-end adapter of this application on their datasets, demonstrating broad application prospects.

[0041] SSL: Self-Supervised Learning, refers to learning a model where the learning objective is not given in advance, but rather the model learns by obtaining supervision signals from the data itself.

[0042] The following specific experiments and experimental data verify the beneficial effects of the embodiments of this application compared to the prior art.

[0043] Self-supervised learning (SSL) has attracted increasing research interest in deep learning across multiple domains, including text, image, and speech processing. Its basic idea is to recover or predict itself in an unsupervised manner based on its contextual information. Compared to text and image processing, speech signals, due to their continuous nature and strong short-term correlations, present a less straightforward task for reconstruction or classification. In recent years, significant research has been conducted on speech-oriented SSL, yielding promising progress. Several representative SSL models have been developed, such as Wav2vec, HuBERT, WavLM, and Data2vec. These SSL techniques have largely been examined in Automatic Speech Recognition (ASR) tasks. Reports indicate that by using a small amount of supervised data and pre-training the SSL model on a large amount of unlabeled audio, performance has been significantly improved. Recently, a series of works have demonstrated that these SSL models can extract superior general speech representations for various downstream speech-related tasks, such as speaker verification and emotion recognition.

[0044] Due to the continuous nature of speech signals, different types of front-ends can be used as input to SSL models, such as waveform-based and Mel-spectrum features. In practice, the optimal front-end and stride size depend on the task, prioritizing accuracy and efficiency. However, SSL models published by Fairseq from Meta typically use waveforms as front-end input, CNN blocks as feature extractors, a fixed stride of 20ms, and then feed them into a Transformer-based backbone model during pre-training. To fully utilize the information learned from SSL training, using a consistent front-end is crucial during pre-training, fine-tuning, and inference phases. However, waveform-based front-ends may not be optimal for all tasks, such as speech recognition. Therefore, this may limit the potential applications of publicly available speech-based SSL models. It would be highly valuable to remove this constraint on front-end consistency and allow the use of alternative front-ends to leverage the power of SSL models in downstream tasks.

[0045] This paper aims to alleviate this front-end discrepancy problem in speech-based SSL models. More specifically, we explore the feasibility of using Fbank features in a waveform-trained SSL model for speech recognition tasks. In addition to standard ASR losses (such as CTC loss), we introduce an additional loss to minimize the output distance between the waveform-based and Fbank-based front-ends during the fine-tuning phase. Therefore, the representation of the Fbank front-end can be adjusted to match the waveform front-end representation to work well with other components of the SSL model. During pre-training and fine-tuning, the stride of the input to the Transformer backbone model may also be mismatched. For example, in practice, speech recognition systems may tend to use larger strides (e.g., 40ms or 80ms) instead of the default stride used in SSL models, typically 20ms. This paper also investigates this stride size mismatch. Experimental results show that our proposed adapter can quickly adapt the SSL model from the original waveform-based front-end to the Fbank-based front-end, achieving comparable performance across different front-end configurations. To our knowledge, this is the first work to explore a front-end adapter for speech-based SSL models.

[0046] Overview of Speech-Based Self-Supervised Learning

[0047] Due to the continuous nature and strong short-term correlation of speech signals, self-supervised learning of speech is not as straightforward as learning from text and images. A series of algorithms have been proposed to generate discrete or continuous representations from speech, thereby facilitating speech-based self-supervised learning. In this section, we will briefly introduce three popular SSL models: Wave2vec2.0, HuBERT, and Data2vec. Figure 4 The diagrams for these three SSL models are provided.

[0048] 2.1.Wav2vec 2.0

[0049] A series of ongoing research efforts aim to transform raw waveforms into continuous high-dimensional vectors, then use contrastive loss to distinguish between the current input representation and competing representations. Wav2vec 2.0 has proven to be quite effective, providing an end-to-end solution for self-supervised learning. A general pipeline for Wav2vec 2.0 can be used... Figure 4 The green and blue blocks in the image illustrate this. In the Wav2vec 2.0 model, the output of the front-end processing block is quantized using either product vector quantization or KMeans. Similar to masked LM in BERT, the audio input is randomly masked sequentially and then fed into the Transformer block. The output of Wav2vec 2.0 aims to recover the masked input by applying a contrastive loss, where the quantized vector of the masked input is used as the target, and other quantized vectors from the encoding library are randomly sampled as negative samples.

[0050] 2.2.HuBERT

[0051] In Wav2vec 2.0, HuBERT does not use a learnable codebook and contrastive loss to minimize reconstruction error, but instead assigns a pseudo-label to each speech frame. This pseudo-label can be obtained from the K-means algorithm, on top of standard MFCC features, or from a refined speech representation obtained from a trained HuBERT model. The cluster ID obtained from K-means is used as the target for each frame, and the corresponding input frame is masked to better utilize contextual information and avoid information leakage. HuBERT has become increasingly popular, and several variants for SSL have been developed based on HuBERT [5]. Since each frame is associated with a pseudo-label, cross-entropy can be simply used as the loss function, and HuBERT training has been found to be more stable than the contrastive loss of Wav2vec 2.0. The HuBERT data flow can be described as Figure 4 The green and yellow blocks in the middle.

[0052] 2.3.Data2vec

[0053] Data2vec is proposed to unify the training algorithm for SSL models across different modes, including audio, image, and text. Two parallel models serve as the teacher and student models, respectively. In each training step, the teacher model is allowed to see the full input, while the student model consumes the masked input. The objective function is to minimize the distance between the outputs of the teacher and student models in the masked regions, where a smooth L1 loss is applied. The gradients computed from the loss function are backpropagated only to the student model; the teacher model is a delayed version of the student model, achieved by applying the EMA technique. At the end of training, the teacher model is discarded, and the student model is retained as the resulting SSL model. Figure 4 The green and gray blocks in the image show the overall framework of the Data2vec model.

[0054] After pre-training the SSL model on unlabeled audio data, Figure 4 The front-end processing block and converter layer corresponding to the green block in the middle are retained as acoustic encoders, and then fine-tuned together with other task-related model components in downstream tasks, such as speech.

[0055] Figure 4 The diagram illustrates three different types of SSL models: green blocks represent the shared waveform front-end and Transformer backbone models; blue blocks represent the Wav2vec 2.0 model; yellow blocks correspond to the Hubert model; and gray blocks represent Data2vec.

[0056] Speaker recognition and verification are crucial for speech processing. There are several options for front-end input, such as waveforms and Melspectrum features. The three SSL models mentioned above used waveforms as the front-end input in their original papers. Code for pre-training and fine-tuning the SSL models has been released in Meta's Fairseq toolkit. Furthermore, Meta provides a series of well-trained SSL models available to the public. These models are widely used in the community for various downstream tasks. In all models released by Fairseq, waveforms were chosen as the front-end input. The waveform-based front-end consists of several CNN layers, producing acoustic feature representations spanning 20ms, such as... Figure 4 As shown, the speech front-end input should be consistent across the pre-training, fine-tuning, and inference phases. This means that if we want to fully leverage the power of existing SSL models, we need to use the same speech front-end. However, in many existing solutions or production systems, waveforms may not be the optimal choice considering accuracy and efficiency. For example, in ASR tasks, it is better to use Fbank features as input with a larger step size, such as 40ms, to reduce computation time during testing. Therefore, the requirement for the front-end input should be consistent between pre-training and fine-tuning.

[0057] This could limit the potential applications of these published SSL models. In this application, we aim to mitigate this mismatch and investigate the use of Fbank features in the fine-tuning process of SSL models initially pre-trained with waveforms as front-end input, which will be detailed in the next section.

[0058] SSL model front-end adapter

[0059] As mentioned in the previous section, pre-training an SSL model is computationally expensive. A common practice is to download a publicly available SSL model and then fine-tune it for downstream tasks using a small amount of labeled data. In this paper, we primarily focus on speech recognition as a downstream task for our experiments. Consistent front-end inputs are necessary during both pre-training and fine-tuning phases, with waveforms serving as the default front-end input for various SSL models, from Fairseq. However, given the existence of different front-end inputs in speech signals, people might prefer to use different front-end inputs rather than waveforms, such as Fbank features, in real-world scenarios. Therefore, this discrepancy poses a potential challenge to fully utilizing existing and well-trained SSL models. In this section, we propose a simple and effective framework for adapting the front-end, allowing the use of different front-ends during pre-training and fine-tuning.

[0060] like Figure 4 As shown, the green blocks (such as the CNN layer front-end and Transformer layers) are retained after pre-training and then fine-tuned with labeled data. More specifically, in speech recognition, the CTC output layer can be added to the output of the Transformer layer. Based on our earlier attempts, simply replacing the waveform front-end with a randomly initialized Fbank front-end did not converge well. This indicates that the mismatch between different front-ends is detrimental to a well-trained SSL model. To address this potential problem, we propose a novel front-end adapter to mitigate the gap between waveform-based and Fbank-based front-ends.

[0061] Figure 2 and Figure 3 This illustrates the two stages of the front-end adapter in the SSL model. Among them, Figure 2 The first stage of the front-end adapter is shown: both CTC loss and L2 loss are applied, L2 loss does not backpropagate to the waveform front-end block, and CTC loss does not backpropagate to the Fbank front-end in the first stage. Figure 3 The second phase of the front-end adapter is shown: all modules are jointly optimized with CTC loss.

[0062] Figure 3The Fbank-based front-end shown was used to replace the original waveform front-end. However, due to differences in various front-end processing blocks, directly replacing the front-end input does not work well. To address this mismatch, the output of the original waveform front-end was used to normalize the output of the Fbank front-end, such as... Figure 2 The bottom block is shown. The original waveform frontend is frozen during training and used to normalize the output of the Fbank frontend. An L2 loss is applied to minimize the distance between the outputs of the waveform-based and Fbank-based frontends. Notably, this additional L2 loss is applied only in the early stages of training for a specific number of steps, known as the adapter warm-up phase. In this paper, we set the adapter warm-up phase to the first 3 durations as an empirical value. Furthermore, we find that during the adapter warm-up phase, when both the CTC loss and L2 loss are used to update the Fbank frontend block, the gradient of the CTC loss backpropagation may outweigh the gradient of the L2 loss. Therefore, to avoid this effect, during the adapter warm-up phase, the gradient of the CTC loss is only used to update the transformer and CTC layer, not the Fbank frontend block. The L2 loss between the waveform and the Fbank frontend is used to update the Fbank frontend. After the adapter warm-up phase, the Fbank frontend, Transformer, and CTC layer are jointly optimized with the CTC loss as standard CTC-based ASR training. In summary, the loss function can be written as follows:

[0063]

[0064] Another important factor to consider is the ability to use variable span sizes for the outputs of different front-end blocks. For the SSL model published in Fairseq, the waveform span size is typically downsampled to 20ms. In contrast, for efficiency reasons, larger spans, such as 40ms, are preferred in practice. This will result in a length mismatch between the waveform output and the Fbank front end. To address this, we simply downsample the output of the waveform front end to match the Fbank front end, for example, to 40ms, and then calculate the L2 distance.

[0065] In general, fine-tuning the SSL model using the proposed front-end adapter can be done in two phases.

[0066] 1. Front-end adapter warm-up: When the update step is less than the warm-up step (e.g., the first 3 durations mentioned above), both CTC loss and L2 loss are applied. Note that the L2 loss of the gradient is not used for backpropagation to the waveform front-end block, and the gradient of the CTC loss is not backpropagated to the Fbank front-end at this stage.

[0067] 2. Fine-tuning: After the adapter warm-up phase, all modules must be fine-tuned using standard CTC losses.

[0068] It's worth noting that the front-end adapter framework can be easily extended to the three SSL models introduced in Section 2: Wav2vec 2.0, Hubert, and Data2vec, because only the front-end adapter and subsequent fine-tuning phases require... Figure 4 The waveform front-end and transformer layer in the middle.

[0069] experiment

[0070] This section examines three popular SSL models from the Fairseq repository: Wav2vec2.0, HuBERT, and Data2vec. All three SSL models were pre-trained on 960 hours of Librispeech data with a "basic" model configuration, resulting in approximately 97 million parameters, making them comparable. A subset of the two public English datasets, Librispeech and Gigaspeech, was used for downstream speech recognition tasks to validate the effectiveness of the front-end adapter proposed in Section 3. For Gigaspeech, 100 hours of speech were randomly selected as fine-tuning data, and performance was evaluated using standard dev and test data. For Librispeech, 100 hours of training-cleaned speech were selected as fine-tuning data, and evaluation was performed using dev-other and test-other data. Letters were used as model units, serving as the output sequences for speech recognition. The CTC criterion was used as the loss function for ASR training. SpecAug was applied to the Fbank feature front-end to improve speech recognition performance, while the waveform front-end used Fairseq's default fine-tuning configuration. For simplicity, the WER results of a greedy search without any LM are reported for performance evaluation.

[0071] The first experiment investigated the impact of the proposed front-end adapter on a 100-hour subset of Librispeech. The HuBERT model was selected as the SSL model for the experimental analysis. Figure 5The WER trends during fine-tuning with different front-ends and strides are plotted. It can be seen that the waveform-based front-end (blue curve) converges quickly, which is expected since the same front-end is used during pre-training and fine-tuning. Compared to Hubert, the adaptive Fbank front-end with a sample span size (i.e., 20ms, colored orange) achieves similar performance on the waveform. The adaptive Fbank front-end with a stride of 40ms (grey curve) also converges quickly, although its performance is slightly worse than that with a stride of 20ms. This degradation can be explained by the limited amount of fine-tuning data. The yellow curve depicts the bit error rate trend during fine-tuning when simply replacing the waveform front-end with the Fbank front-end without using the adaptive front-end. It can be seen that without a front-end adapter, the mismatch between the Fbank front-end and the Transformer encoder significantly reduces the bit error rate during fine-tuning. Therefore, using a front-end adapter during fine-tuning is crucial when using different front-ends.

[0072] Figure 5 The waveform-based fine-tuned bit error rate (BER) results are shown, with the BER based on the Fbank front-end at different step sizes in Hubert. The SSL model was fine-tuned on 100 hours of Librispeech data, and the BER is reported on the dev-other set.

[0073] Figure 6 The variation of the Euclidean distance between the waveform and the front-end output of the Fbank was also plotted. It can be seen that in the first stage of the front-end adapter (the first 200 updates), the Euclidean distance continuously decreases due to the additional L2 loss. In the second stage of the front-end adapter (after the first 200 updates), no L2 loss is applied to the objective function, and the Euclidean distance remains stable. After the first stage, the Fbank front-end can be tuned to work well with the Transformer encoder.

[0074] Figure 6 The Euclidean distance between the waveform and the Fbank front end during training is shown. The first 200 updates were used for the first phase of the front-end adapter.

[0075] Nevertheless, the Fbank front-end model adapted from Data2vec still outperforms all models fine-tuned on Wav2vec 2.0 and HuberT.

[0076] Figure 7 The results show the bit error rate of three SSL models on the Librispeech and Gigaspeech test sets. Gigaspeech uses the standard dev and test sets, while the evaluation uses the dev-other and test-other sets.

[0077] Figure 7 The following table shows the bit error rate results after fine-tuning three types of SSL models, such as... Figure 7 As shown, these models have different front-ends (Fbank vs waveform) and stride sizes (20ms vs 40ms). It can be seen that the front-end adapter in this embodiment can effectively adapt a waveform-based front-end to an Fbank-based front-end. For the Wav2vec 2.0 and HuberT models, the 20ms stride Fbank front-end outperforms the waveform front-end, while the 40ms stride Fbank produces performance comparable to the waveform front-end. In Data2vec, the waveform-based front-end gives the best WER among the three configurations.

[0078] Conclusions and Discussion

[0079] In recent years, speech-based self-supervised learning (SSL) has shown promising performance improvements on a range of downstream tasks, including speech recognition. Currently, most popular and publicly released SSL models use waveforms as their front-end input. To fully leverage the power of SSL models, it is necessary to use consistent front-end inputs during both pre-training and fine-tuning phases. This can limit the potential applications of SSL models, as waveforms are not the optimal choice for many tasks in real-world scenarios. In this paper, we propose a simple yet effective front-end adapter framework to alleviate the mismatch between different front-end inputs for speech-based SSL models. Our proposed front-end adapter allows Fbank features to be used with SSL models initially trained on waveforms while preserving comparable performance in speech recognition tasks.

[0080] In other embodiments, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the front-end adapter training method in any of the above method embodiments. A front-end adapter training method includes a first stage and a second stage, wherein the first stage includes:

[0081] In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows:

[0082] Input the waveform format of the speech into the original front end, and obtain the first output of the original front end;

[0083] Other formats of the speech are input to the front-end adapter, the second output of the front-end adapter is obtained, and the first loss of the first output and the second output is calculated to train the front-end adapter.

[0084] The second output of the front-end adapter is input to the backbone Transformer model to calculate the second loss, wherein the first loss and the second loss simultaneously optimize the preset epoch.

[0085] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the front-end adapter training system, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely configured relative to the processor, which can be connected to the front-end adapter training system via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0086] This invention also provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer, cause the computer to perform any of the aforementioned front-end adapter training methods.

[0087] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present invention, such as... Figure 8 As shown, the device includes one or more processors 810 and a memory 820. Figure 8 Taking a processor 810 as an example, the device for the front-end adapter training method and system may further include an input device 830 and an output device 840. The processor 810, memory 820, input device 830, and output device 840 can be connected via a bus or other means. Figure 8 Taking a bus connection as an example, the memory 820 is the aforementioned non-volatile computer-readable storage medium. The processor 810 executes various server functions and data processing by running non-volatile software programs, instructions, and modules stored in the memory 820, thereby implementing the front-end adapter training method and system described in the above embodiment. The input device 830 can receive input digital or character information and generate key signal inputs related to user settings and function control of the communication compensation device. The output device 840 may include a display screen or other display device.

[0088] The above-described product can execute the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.

[0089] In one implementation, the above-described electronic device is applied to a front-end adapter training system. The training method includes a first stage and a second stage. The first stage includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions, when executed by the at least one processor, enable the at least one processor to:

[0090] Input the waveform format of the speech into the original front end, and obtain the first output of the original front end;

[0091] Other formats of the speech are input to the front-end adapter, the second output of the front-end adapter is obtained, and the first loss of the first output and the second output is calculated to train the front-end adapter.

[0092] The second output of the front-end adapter is input to the backbone Transformer model to calculate the second loss, wherein the first loss and the second loss simultaneously optimize the preset epoch.

[0093] The electronic devices described in this application exist in various forms, including but not limited to:

[0094] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones, multimedia phones, feature phones, and low-end phones, etc.

[0095] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, etc.

[0096] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes: audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0097] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0098] (5) Other electronic devices with data interaction functions.

[0099] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0100] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A front-end adapter training method, wherein, The training method includes a first stage and a second stage, wherein the first stage includes: Input the waveform format of the speech into the original front end and obtain the first output of the original front end; Other formats of the speech are input to the front-end adapter, the second output of the front-end adapter is obtained, and the first loss of the first output and the second output is calculated to train the front-end adapter. The second output of the front-end adapter is input to the backbone Transformer model to calculate the second loss, wherein the first loss and the second loss are optimized simultaneously in a preset epoch, the first loss is the L2 loss, and the second loss is the CTC loss; The second stage includes: inputting other formats of the speech into the front-end adapter, then inputting the obtained third output of the front-end adapter into the backbone Transformer model, and calculating only the second loss; In the first stage: both the CTC loss and the L2 loss are applied, the L2 loss is not backpropagated to the waveform front-end block, and the CTC loss is not backpropagated to the Fbank front-end in the first stage; in the second stage: all modules are optimized together with the CTC loss, the original waveform front-end is frozen during training and used to normalize the output of the Fbank front-end, and the L2 loss is applied to minimize the distance between the outputs of the waveform-based and Fbank-based front-ends.

2. The method according to claim 1, wherein, The preset era is 200.

3. The method according to claim 1, wherein, The other formats include Fbank format and MFCC format.

4. A self-supervised speech model, comprising a front-end adapter and a backbone Transformer model trained according to any one of claims 1-3.

5. The model according to claim 4, wherein, The self-supervised speech model is a speech recognition model.

6. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 3.

7. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

Images