Self-supervised speech representation training method and system, electronic device and storage medium
By constructing a self-supervised speech representation learning backbone network and combining offline and online target extractors, the problems of poor performance of HuBERT and slow convergence of data2vec are solved, achieving more accurate speech representation learning and faster convergence speed, while reducing training costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-04
- Publication Date
- 2026-04-07
AI Technical Summary
Among existing technologies, HuBERT has poor speech representation learning performance, data2vec has a slow convergence speed during training, and UniSpeech requires labeled speech data, which is costly.
A self-supervised speech representation learning backbone network is constructed, which is combined with an encoding network and a content network. The masked speech representation is input into the offline and online target extractors, and the online extraction loss and offline extraction loss are calculated respectively. Gradient backpropagation training is performed until the model converges.
It improves the accuracy of speech representation and the convergence speed of the model, reduces training costs, and maintains model performance while saving training costs.
Smart Images
Figure CN115860052B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent speech technology, and in particular to a self-supervised speech representation training method, system, electronic device, and storage medium. Background Technology
[0002] Self-supervised learning (SSL) has achieved significant success in the field of representation learning and has been applied to computer vision, natural language processing, and speech processing. For speech representation learning, SSL methods are typically used in the pre-training phase to obtain supervisory signals from large amounts of unlabeled audio data.
[0003] Current techniques for learning speech representations typically use the input audio directly as the training target, either by comparing positive and negative samples or by reconstructing the waveform and acoustic features of the original audio. Common methods include:
[0004] HuBERT obtains the training target by clustering features and adopts an iterative training method. The first iteration performs clustering on MFCC features, and the second iteration performs clustering on the intermediate layer features of the HuBERT model obtained in the first iteration to obtain speech representation.
[0005] data2vec is a full-modal self-supervised learning method that can use the same structure and training ideas to train different modalities of tasks to obtain speech representations.
[0006] UniSpeech introduces a multi-task learning framework for labeled data, using a multi-task learning framework to enhance the model's representational capabilities.
[0007] In the process of realizing this invention, the inventors discovered at least the following problems in the related technology:
[0008] HuBERT performs poorly in learning fixed speech representations during training;
[0009] The data2vec target is not fixed during training, resulting in a slow convergence speed.
[0010] UniSpeech requires labeled speech data, which is expensive to obtain. Summary of the Invention
[0011] To at least address the problems of the prior art, in a first aspect, embodiments of the present invention provide a self-supervised speech representation training method, comprising:
[0012] A self-supervised speech representation learning backbone network is constructed, wherein the self-supervised speech representation learning backbone network includes: an encoding network and a content network;
[0013] Speech representation is obtained by sampling the original speech based on the coding network;
[0014] The speech representation is masked and then used as input to the content network, outputting an aggregated speech representation with masked representation.
[0015] The masking representation in the aggregated speech representation is input into the online target extractor and the self-supervised pre-trained offline target extractor, respectively. The online extraction loss and the offline extraction loss are determined based on the masking representation and the original speech.
[0016] The self-supervised speech representation learning backbone network is trained by gradient backpropagation based on the online extraction loss and the offline extraction loss until the self-supervised speech representation learning backbone network converges.
[0017] Secondly, embodiments of the present invention provide a self-supervised speech representation training system, comprising:
[0018] A network construction module is used to construct a self-supervised speech representation learning backbone network, wherein the self-supervised speech representation learning backbone network includes: an encoding network and a content network;
[0019] A speech representation determination module is used to sample the original speech based on the coding network to obtain a speech representation;
[0020] The occlusion representation determination module is used to process the occlusion of the speech representation and use it as the input of the content network, and output an aggregated speech representation with occlusion representation.
[0021] The loss extraction module is used to input the occlusion representation in the aggregated speech representation to the online target extractor and the self-supervised pre-trained offline target extractor, respectively, and determine the online extraction loss and the offline extraction loss based on the occlusion representation and the original speech.
[0022] The speech representation training module is used to perform gradient backpropagation training on the self-supervised speech representation learning backbone network based on the online extraction loss and the offline extraction loss until the self-supervised speech representation learning backbone network converges.
[0023] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the self-supervised speech representation training method of any embodiment of the present invention.
[0024] Fourthly, embodiments of the present invention provide a storage medium storing a computer program thereon, characterized in that, when the program is executed by a processor, it implements the steps of the self-supervised speech representation training method of any embodiment of the present invention.
[0025] The beneficial effects of this invention are as follows: It integrates a multi-objective framework for improving self-supervised speech representation learning, using offline and online object extractors respectively, and integrates multiple objectives through multi-task learning, which can learn more accurate speech representations. At the same time, the convergence speed of the model is also improved, thereby improving the convergence speed of the model without reducing the effect, thus saving training costs. Attached Figure Description
[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0027] Figure 1 This is a flowchart of a self-supervised speech representation training method provided in an embodiment of the present invention;
[0028] Figure 2 This is an overall framework diagram of a self-supervised speech representation training method provided in an embodiment of the present invention;
[0029] Figure 3 This is a schematic diagram of the training result of a self-supervised speech representation training method provided in an embodiment of the present invention;
[0030] Figure 4 This is a schematic diagram of the training results of a self-supervised speech representation training method provided in another embodiment of the present invention;
[0031] Figure 5 This is a schematic diagram showing the word error rate and pre-training cycle between different SSL models of a self-supervised speech representation training method provided in an embodiment of the present invention;
[0032] Figure 6 This is a schematic diagram of the structure of a self-supervised speech representation training system provided in an embodiment of the present invention;
[0033] Figure 7 This is a schematic diagram of an embodiment of an electronic device for self-supervised speech representation training provided by an embodiment of the present invention. Detailed Implementation
[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0035] like Figure 1 The diagram shown is a flowchart of a self-supervised speech representation training method according to an embodiment of the present invention, which includes the following steps:
[0036] S11: Construct a self-supervised speech representation learning backbone network, wherein the self-supervised speech representation learning backbone network includes: an encoding network and a content network;
[0037] S12: Sampling the original speech based on the coding network to obtain a speech representation;
[0038] S13: After masking the speech representation, use it as the input to the content network and output an aggregated speech representation with masked representation.
[0039] S14: Input the masking representation in the aggregated speech representation into the online target extractor and the self-supervised pre-trained offline target extractor respectively, and determine the online extraction loss and the offline extraction loss based on the masking representation and the original speech;
[0040] S15: Based on the online extraction loss and the offline extraction loss, perform gradient backpropagation training on the self-supervised speech representation learning backbone network until the self-supervised speech representation learning backbone network converges.
[0041] In this implementation, the method finds that all Target Extractors (TEs) used in the SSL model can be divided into two categories: Off-TEs and On-TEs. Off-TEs can be pre-trained extractors or other types, and are not updated during the self-supervised pre-training phase. On-TEs can be pre-trained or randomly initialized. During pre-training, the TEs are continuously updated. Offline targets from Off-TEs have a coarser granularity, making the Off-TE model easier to train and converge faster. On-TEs, like course learning, provide increasingly finer online targets during training, resulting in better performance.
[0042] For step S11, in order to use the two types of target extractors mentioned above, such as Figure 2 For the overall architecture of this method, a baseline model can be used for the backbone network of self-supervised speech representation learning, which includes an encoding network and a content network.
[0043] For step S12, the encoding network can be set as a 7-layer 1D convolutional neural network with a kernel size of (5, 2, 2, 2, 2, 2, 2) and a stride of (10, 3, 3, 3, 3, 2, 2). The original audio X for speech representation learning is input into the encoding network, and the sampling rate is set to 16000Hz, denoted as . The encoding network downsamples the original audio. The output represents... The frequency is 50Hz and the dimension is 512. Then, a linear projection is used to transform the dimension from 512 to 768 to obtain the speech representation.
[0044] For step S13, the speech representation is masked. Specifically, a masking matrix can be used to mask a portion of the speech representation. The masked speech representation is then used as the input to the content network, denoted as...
[0045] Specifically, the content network can be a 12-layer Transformer model with learnable convolutional position encoding. Each Transformer is set to a model size of 768, an inner size of 3072, and 12 attention heads corresponding to the 12 layers. The content network can use: The final output is an aggregated, masked speech representation. Used for classification and regression processing of TE.
[0046] For step S14, the masked speech representation from step S13 is compared with the discrete target extracted by the offline target extractor and the online target extracted by the online target extractor to calculate the loss.
[0047] For offline extraction loss, specifically, the self-supervised pre-training offline target is determined based on the occlusion representation;
[0048] The original speech is input into an offline target extractor, and a predicted offline representation is obtained through mean clustering.
[0049] The offline extraction loss is determined based on the predicted occlusion representation and the self-supervised pre-trained offline target.
[0050] In this implementation, the model parameters of the Off-TE (Offline Target Extractor) are not updated during the pre-training phase. The K-means algorithm can be used as the Off-TE. Before pre-training, the K-means algorithm can be used to train the offline target extractor model. This model converts speech features into C clusters. Therefore, the index of the cluster center is used to represent each speech tag. Assuming X is the original audio utterance sampled from X, the offline target can be obtained as follows:
[0051] Y f =TE f (X)
[0052] Among them TE f (·) represents the extraction operation. It is a self-training offline target. Suppose Z = [z1, ..., z2] T ] T It is a masked version obtained from the same original audio X via the backbone network. Represents the mask index, where z is the mask index if t∈M. t Replaced with a mask marker. Dimensional transformation using a projection layer can be written as:
[0053] Z f =W f Z
[0054] in, and The offline extraction loss is defined as:
[0055]
[0056] Where CE(·) calculates Y f and Z f Cross-entropy loss between them.
[0057] For online extraction loss, specifically, the online target is determined based on the occlusion representation;
[0058] The original speech is sampled to obtain a speech representation, which is then input into an online target extractor. A gradient-free teacher network is used to determine the predicted online representation.
[0059] The online extraction loss is determined based on the online objective and the predicted online representation.
[0060] In this implementation, the model parameters of the On-TE (Online Target Extractor) are continuously updated during the pre-training phase. A teacher network without gradients can be used as the On-TE to obtain the online target. This process can be viewed as a special type of knowledge distillation or noisy student training. Assume H is derived from... The hidden representation of the mid-sampled object is obtained from X. For convolutional sub-sampling, the online target can be obtained in the following way:
[0061] Y n =TE n (H)
[0062] Among them TE n (·) represents the extraction operation. This is the self-training objective. Similar to offline speech tagging, a projection layer can be used for dimensionality transformation, written as:
[0063] Z n =W n Z
[0064] Among them, Z n Having the same characteristics as Y n Same size. Online extraction loss is defined as:
[0065]
[0066] Where MSE(·) calculates Z n and Y n The mean squared error between them. The parameters Δ of the teacher network are initialized using the parameters θ of the backbone network. The parameters of the teacher network are updated using an exponential moving average within each mini-batch, and assigned the following values:
[0067] Δ=τΔ+(1-τ)θ
[0068] Where τ is a parameter that increases linearly during pre-training.
[0069] For step S15, after determining the online extraction loss and offline extraction loss in step S14, these losses are integrated to enhance the representational ability of self-supervised learning, resulting in:
[0070]
[0071] Here, α is an adjustable parameter. It is important to note that only the loss of the masked portion of the speech is calculated. Using the determined integration loss, the self-supervised speech representation learning backbone network is trained through gradient backpropagation until it converges, enabling the trained backbone network to learn more accurate speech representations.
[0072] As can be seen from this implementation, the framework for integrating multi-objectives to enhance self-supervised speech representation learning, by using offline and online object extractors respectively and integrating multiple objectives through multi-task learning, can learn more accurate speech representations. At the same time, the convergence speed of the model is also improved, thereby increasing the convergence speed of the model without reducing the effect and thus saving training costs.
[0073] The method is illustrated through experiments. For the unsupervised pre-training dataset, the LibriSpeech corpus (train-clean-360) with 360 hours of unlabeled data is used. For supervised fine-tuning, 1-hour and 10-hour segments are extracted from the Libri-light corpus, and 100-hour segments are extracted from the LibriSpeech corpus. The model is evaluated based on clean / other (or noisy) dev sets and clean / other (or noisy) test sets from the mainstream test sets in the LibriSpeech corpus.
[0074] The MT4SSL (multi-task self-supervised learning) model proposed in this method can be considered a simplification and fusion of an online and an offline object extractor. Specifically, the offline object extractor can be the HuBERT model, and the online object extractor can be the data2vec model. Therefore, the hyperparameter inheritance between the two is maximized to demonstrate the model's effectiveness. Given limited computational resources, only some empirical configurations need to be selected to train the MT4SSL model without extensive hyperparameter search.
[0075] During the pre-training phase, the model was trained using 360 hours of unlabeled LibriSpeech data. Training was performed on an NVIDIA GeForce RTX 3090 GPU, simulating 16 GPUs by using k GPUs and setting the update frequency to 16 / k. In the methodology, k was set to 4. For the masking policy, each time step had a probability of p = 0.065 as the starting index, and subsequent time steps of l = 10 were masked. This resulted in the mask embedding covering an average of 49% of all tokens. For the optimization policy, the Adam policy was used with a learning rate of 0.0005 and a weight decay of 0.01. MT4SSL was trained for 800 epochs with warm-up, hold, and linear decay ratios of [3%, 90%, 7%]. The hyperparameter α controlling the loss weights was set to 1, meaning that both losses had the same weights.
[0076] For the offline objective, features are extracted from the Hubert model, and Off-TE is trained before pre-training. Specifically, the objective is achieved by running k-means clustering with 500 clusters on the output of the 6th layer transformer of the Hubert model in the first iteration.
[0077] For online targets, the average of the first 8 blocks of the transformer layer output from the teacher network is used as the data2vec model design. For parameter updates of the teacher model, for the first 7.5% of training steps, the values of τ from τ...s =0.99 to τ e =0.999, employing a linearly increasing strategy. The parameter τ remains constant throughout the remaining training steps.
[0078] During the fine-tuning phase, CTC (Connectionist Temporal Classification) loss is used to maintain consistency with the baseline model. The hyperparameters during the fine-tuning phase remain consistent with mainstream models.
[0079] like Figure 3 As shown, MT4SSL compares with other state-of-the-art models on the LibriSpeech benchmark. The model was pre-trained on the LibriSpeech 360-hour dataset (train-clean-360) and fine-tuned on subsets of Libri-light (1 hour, 10 hours, and 100 hours). Performance on dev-clean / other and test / other datasets with and without a language model was compared. A 4-gram language model trained on the official LibriSpeech language modeling data was used. Given 10 hours of labeled data, MT4SSL achieves relative WER reductions of 13.0% (dev-clean), 13.9% (dev-other), 13.1% (test-clean), and 15.5% (test-other) without a language model, and 18.4% (dev-clean), 10.2% (dev-other), 15.1% (test-clean), and 11.2% (test-other) with a 4-gram language model. For fine-tuning with 1 hour and 100 hours of labeled data, MT4SSL's improvements are consistent with other models.
[0080] like Figure 4 As shown, the results of MT4SSL trained with 360 hours of audio data are compared with the results of a state-of-the-art model trained with 960 hours of data. The results of MT4SSL are compared with those in the wav2vec 2.0 and WavLM papers, as well as those in the publicly available HuberT release page. Despite using less data, our model is comparable to, or even better than, state-of-the-art models.
[0081] The MT4SSL of this method is analyzed quantitatively and qualitatively from the perspective of model convergence. All experiments were conducted with the following configuration: all models were pre-trained on 360 hours of unlabeled LibriSpeech data, fine-tuned on 10 hours of labeled LibriSpeech data, and evaluated on the dev-other subset of the LibriSpeech corpus.
[0082] MT4SSL not only achieved better results in benchmark tests, but also demonstrated good convergence. Figure 5 As shown, the WER trend relative to the number of training times is plotted. By comparing MT4SSL, data2vec, and Hu-BERT, it can be seen that HuBERT, which utilizes an offline objective, has better convergence than data2vec, which utilizes an online objective. However, at full training time, data2vec performs better than HuBERT. MT4SSL combines the advantages of all three and converges quickly to a relatively low WER. Fixed offline objectives are easier to learn than constantly changing online objectives. Therefore, models using offline objectives converge faster. However, online objectives have finer granularity than offline objectives, so models using online objectives have better representational power. The learning of these two objectives is unlikely to conflict but rather cooperates, resulting in superior efficiency and effectiveness for MT4SSL.
[0083] like Figure 6 The diagram shown is a structural schematic of a self-supervised speech representation training system provided in an embodiment of the present invention. The system can execute the self-supervised speech representation training method described in any of the above embodiments and is configured in a terminal.
[0084] This embodiment provides a self-supervised speech representation training system 10, which includes: a network construction module 11, a speech representation determination module 12, an occlusion representation determination module 13, a loss extraction module 14, and a speech representation training module 15.
[0085] The network construction module 11 is used to construct a self-supervised speech representation learning backbone network, which includes an encoding network and a content network. The speech representation determination module 12 is used to sample the original speech based on the encoding network to obtain speech representations. The masking representation determination module 13 is used to mask the speech representations and use them as input to the content network, outputting an aggregated speech representation with masking representations. The loss extraction module 14 is used to input the masking representations in the aggregated speech representations to an online target extractor and a self-supervised pre-trained offline target extractor, respectively, and determine the online extraction loss and offline extraction loss based on the masking representations and the original speech. The speech representation training module 15 is used to perform gradient backpropagation training on the self-supervised speech representation learning backbone network based on the online extraction loss and the offline extraction loss until the self-supervised speech representation learning backbone network converges.
[0086] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the self-supervised speech representation training method in any of the above method embodiments.
[0087] In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows:
[0088] A self-supervised speech representation learning backbone network is constructed, wherein the self-supervised speech representation learning backbone network includes: an encoding network and a content network;
[0089] Speech representation is obtained by sampling the original speech based on the coding network;
[0090] The speech representation is masked and then used as input to the content network, outputting an aggregated speech representation with masked representation.
[0091] The masking representation in the aggregated speech representation is input into the online target extractor and the self-supervised pre-trained offline target extractor, respectively. The online extraction loss and the offline extraction loss are determined based on the masking representation and the original speech.
[0092] The self-supervised speech representation learning backbone network is trained by gradient backpropagation based on the online extraction loss and the offline extraction loss until the self-supervised speech representation learning backbone network converges.
[0093] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by a processor, the self-supervised speech representation training method in any of the above method embodiments is executed.
[0094] Figure 7 This is a schematic diagram of the hardware structure of an electronic device for a self-supervised speech representation training method according to another embodiment of this application, as shown below. Figure 7 As shown, the device includes:
[0095] One or more processors 710 and memory 720, Figure 7 Taking a processor 710 as an example, the device for the self-supervised speech representation training method may also include an input device 730 and an output device 740.
[0096] The processor 710, memory 720, input device 730, and output device 740 can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.
[0097] The memory 720, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the self-supervised speech representation training method in the embodiments of this application. The processor 710 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 720, thereby implementing the self-supervised speech representation training method in the above-described method embodiments.
[0098] The memory 720 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data, etc. Furthermore, the memory 720 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 720 may optionally include memory remotely located relative to the processor 710, and these remote memories can be connected to the mobile device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0099] Input device 730 can receive input numerical or character information. Output device 740 may include display devices such as a display screen.
[0100] The one or more modules are stored in the memory 720, and when executed by the one or more processors 710, they execute the self-supervised speech representation training method in any of the above method embodiments.
[0101] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.
[0102] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0103] This invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the self-supervised speech representation training method of any embodiment of this invention.
[0104] The electronic devices described in this application exist in various forms, including but not limited to:
[0105] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.
[0106] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as tablet computers.
[0107] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.
[0108] (4) Other electronic devices with data processing functions.
[0109] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.
[0110] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0111] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A self-supervised speech representation training method, comprising: A self-supervised speech representation learning backbone network is constructed, wherein the self-supervised speech representation learning backbone network includes: an encoding network and a content network; Speech representation is obtained by sampling the original speech based on the coding network; The speech representation is masked and then used as input to the content network, outputting an aggregated speech representation with masked representation. The masking representation in the aggregated speech representation is input into the online target extractor and the self-supervised pre-trained offline target extractor, respectively. The online extraction loss and the offline extraction loss are determined based on the masking representation and the original speech. The self-supervised speech representation learning backbone network is trained by gradient backpropagation based on the online extraction loss and the offline extraction loss until the self-supervised speech representation learning backbone network converges.
2. The method according to claim 1, wherein, The step of inputting the masking representation in the aggregated speech representation into the online target extractor and the self-supervised pre-trained offline target extractor respectively, and determining the online extraction loss and offline extraction loss based on the masking representation and the original speech, includes: Determine the self-supervised pre-training offline target based on the occlusion representation; The original speech is input into an offline target extractor, and the predicted masking representation is obtained through mean clustering. The offline extraction loss is determined based on the predicted occlusion representation and the self-supervised pre-trained offline target.
3. The method according to claim 1, wherein, The step of inputting the masking representation in the aggregated speech representation into the online target extractor and the self-supervised pre-trained offline target extractor respectively, and determining the online extraction loss and offline extraction loss based on the masking representation and the original speech, includes: Online targets are determined based on the aforementioned occlusion characterization; The original speech is sampled to obtain a speech representation, which is then input into an online target extractor. A gradient-free teacher network is used to determine the predicted online representation. The online extraction loss is determined based on the online objective and the predicted online representation.
4. The method according to claim 3, wherein, After determining the online extraction loss, the method further includes: The model parameters of the online target extractor are iteratively updated using the online extraction loss.
5. A self-supervised speech representation training system, comprising: A network construction module is used to construct a self-supervised speech representation learning backbone network, wherein the self-supervised speech representation learning backbone network includes: an encoding network and a content network; A speech representation determination module is used to sample the original speech based on the coding network to obtain a speech representation; The occlusion representation determination module is used to process the occlusion of the speech representation and use it as the input of the content network, and output an aggregated speech representation with occlusion representation. The loss extraction module is used to input the occlusion representation in the aggregated speech representation to the online target extractor and the self-supervised pre-trained offline target extractor, respectively, and determine the online extraction loss and the offline extraction loss based on the occlusion representation and the original speech. The speech representation training module is used to perform gradient backpropagation training on the self-supervised speech representation learning backbone network based on the online extraction loss and the offline extraction loss until the self-supervised speech representation learning backbone network converges.
6. The system according to claim 5, wherein, The loss extraction module is used for: Determine the self-supervised pre-training offline target based on the occlusion representation; The original speech is input into an offline target extractor, and the predicted masking representation is obtained through mean clustering. The offline extraction loss is determined based on the predicted occlusion representation and the self-supervised pre-trained offline target.
7. The system according to claim 5, wherein, The loss extraction module is also used for: Online targets are determined based on the aforementioned occlusion characterization; The original speech is sampled to obtain a speech representation, which is then input into an online target extractor. A gradient-free teacher network is used to determine the predicted online representation. The online extraction loss is determined based on the online objective and the predicted online representation.
8. The system according to claim 7, wherein, The loss extraction module is also used for: The model parameters of the online target extractor are iteratively updated using the online extraction loss.
9. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-4.
10. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-4.