Low-resource speech keyword detection method based on unsupervised learning and transfer learning

Abstract

The present invention discloses a low-resource speech keyword detection method based on unsupervised learning and transfer learning: using unlabeled speech data for training to obtain an unsupervised network model; freezing the model parameters as a feature extraction network, and constructing a speech keyword detection model together with a precoding layer and a decision network; performing supervised pre-training under rich-resource conditions to obtain a rich-resource speech keyword detection model; under low-resource conditions, migrating the parameters of the trained rich-resource speech keyword detection model to the current model, and then fine-tuning to obtain a low-resource speech keyword detection model; finally, decoding is performed based on the model output to determine whether the sample contains a predetermined keyword. The present invention combines unsupervised learning and transfer learning, fully utilizes unlabeled data and labeled data, improves speech keyword detection performance under low-resource conditions, and alleviates the data shortage in the field of speech keyword detection.

Description

Technical Field

[0001] The present invention relates to the field of speech signal processing, and in particular to a low-resource speech keyword detection method based on unsupervised learning and transfer learning. Background Art

[0002] As one of the current research hotspots in speech technology, voice keyword spotting (KWS) technology refers to detecting pre-defined keywords from continuous speech streams. It is widely used in smart home, voice information retrieval, and device control.

[0003] In recent years, as deep learning has achieved success in many fields such as autonomous driving, human-computer dialogue and chatbots, speech keyword detection technology based on deep learning has also achieved rapid development.

[0004] However, in practical applications, deep learning-based speech keyword detection technology still faces many challenges. Because deep learning is inherently computationally and data-driven, training a high-performance, generalizable speech keyword detection model often requires extensive computing resources and labeled speech data. Without large-scale data support, training can easily lead to overfitting, significantly degrading model performance.

[0005] There are approximately 6,900 languages ​​in the world, and only a few (such as English, Chinese, and Spanish) have sufficient data resources. Most languages ​​are low-resource and lack linguistic expert knowledge and sufficient annotated speech data, making it difficult to train a high-performance speech keyword detection model.

[0006] For example, training a commercial speech keyword detection system requires thousands of hours of labeled data. While speech data can be recorded, the labeled text can be created by human experts, semi-automatically, or fully automatically.

[0007] However, when faced with huge data demands, such work often requires a lot of manpower and material resources. As a result, many companies that provide voice data solutions have emerged, such as Datatang and Hill Beike.

[0008] Although users can pay companies to help them collect and annotate data, building a large corpus requires significant capital costs. While large companies have the ability to collect massive amounts of data, researchers and small businesses generally face limited data resources.

[0009] Currently, many speech keyword detection systems are developed for resource-rich languages ​​such as English and Chinese. For low-resource scenarios such as local dialects and minority languages, it is difficult to complete the development of low-resource speech keyword detection systems due to the lack of sufficient annotated data.

[0010] An effective approach is to leverage resource-rich languages ​​to improve the performance of low-resource languages, thereby improving the performance of the target task. The standard paradigm for unsupervised learning is to use unlabeled data for unsupervised training, then use the trained model as a feature extraction network. Deep features replace traditional handcrafted features, and are then applied to downstream tasks for further supervised fine-tuning, thereby improving model performance with limited data.

[0011] The standard transfer learning paradigm is to pre-train a model using rich-resource data and then transfer the network parameters to a model with low-resource data for fine-tuning. This provides a good starting point for model training and improves model performance under low-resource conditions. Both training paradigms can, to a certain extent, alleviate the problem of low data resources. Summary of the Invention

[0012] The present invention provides a low-resource speech keyword detection method based on unsupervised learning and transfer learning.

[0013] The present invention establishes a training paradigm and voice keyword detection model that efficiently utilizes rich resource data, provides a high-performance voice keyword detection method under low-resource conditions, and effectively solves the problem of poor voice keyword detection performance when the data volume is insufficient in the existing technology.

[0014] The present invention is achieved through the following technical solutions:

[0015] The present invention is based on a low-resource speech keyword detection method for unsupervised learning and transfer learning, which is implemented by the following steps:

[0016] S1. Collect and establish an unlabeled speech dataset Among them, X n is a speech sample, and N1 is the total number of unlabeled speech samples. The original waveform of the unlabeled speech sample is directly input into the unsupervised network model M1 for unsupervised training. The network parameters are iteratively updated until convergence to obtain the model M1.

[0017] S2, reuse the network topology and parameters of the unsupervised network model M1 as the feature extraction network, use the precoding layer to connect the feature extraction network with the decision network, and build a speech keyword detection model;

[0018] S3. Collect and establish a rich resource speech dataset with annotations Among them, Xn is the speech sample, Y n is the corresponding annotated text, N2 is the total number of rich-resource speech samples; the rich-resource speech dataset M2 is used to perform supervised pre-training iterations on the parameters of the model’s precoding layer and decision network until convergence, and the rich-resource speech keyword detection model M2 is obtained;

[0019] S4. Combined with transfer learning, the parameters of the rich-resource speech keyword detection model M2 are transferred to the low-resource speech keyword detection model M3; collect and establish annotated low-resource speech dataset Among them, N3 is the total number of low-resource speech samples. The low-resource speech dataset D3 is used for supervised fine-tuning training until the model M3 converges;

[0020] S5. Input the low-resource speech sample into the trained low-resource speech keyword detection model M3 to obtain the posterior probability output matrix, use the prefix beam search algorithm to decode the output matrix, and determine whether the decoding result contains the predefined keywords.

[0021] The unlabeled speech dataset in the above step S1 is not limited to a language and can be single language or multilingual.

[0022] The unsupervised network model in the above step S1 consists of an encoder and a context representation module, wherein the encoder is composed of multiple one-dimensional convolutional neural network layers, which is used to encode the input speech sequence into a potential pronunciation representation; the context representation module is composed of multiple Transformer layers containing a multi-head attention mechanism, which is used to construct a context representation from the potential pronunciation representation, that is, the high-dimensional deep features of the speech.

[0023] The feature extraction network in the speech keyword detection model in step S2 above is used to convert the original speech into discriminative high-dimensional deep features by using the pronunciation context representation information learned from the unlabeled speech data. The parameters of this network are no longer updated during the subsequent supervised training process.

[0024] The precoding layer in the speech keyword detection model in step S2 above specifically includes a fully connected layer and an activation function for reducing the dimensionality of high-dimensional deep features:

[0025] O=f(W T I+b)

[0026] Where I represents the high-dimensional feature matrix obtained by the feature extraction network, O represents the low-dimensional feature matrix after dimensionality reduction, W is the learnable weight parameter matrix of the fully connected layer, b is the corresponding bias, and the linear rectification function f(x) = max(0, x) is used as the activation function for nonlinear transformation. Using learnable nonlinear transformations to reduce the dimensionality of high-dimensional deep features is used to reduce redundant information in high-dimensional deep features and reduce the subsequent computational complexity of the access decision network.

[0027] The decision network in the speech keyword detection model in step S2 is a convolutional recurrent neural network, specifically including a convolutional neural network module, a recurrent neural network module, and a fully connected network module. The convolutional neural network module is used to capture local information, the recurrent neural network module is used to learn temporal information, and the fully connected network module is used for further learning and output dimension matching. The output dimension of the last fully connected layer is consistent with the number of target label categories of the corresponding speech keyword. The role of the decision network is to model the deep features after dimensionality reduction and the target label sequence, and play a final decision-making role.

[0028] The supervised pre-training of the rich-resource speech keyword detection model M2 in step S3 above is as follows:

[0029] S3.1. Input the rich-resource speech dataset D2 with annotations directly into model M2, passing it through the feature extraction network, precoding layer, and decision network in sequence to obtain the posterior probability output matrix;

[0030] S3.2. Compute the Connectionist Temporal Classification (CTC) loss using the posterior probability output matrix and the target label sequence:

[0031]

[0032] Where x represents the input sequence corresponding to the speech sample X, that is, the original digital sequence quantized by sampling, l represents the target label sequence corresponding to the annotated text Y, and B represents the many-to-one mapping function between the target label sequence l and all feasible paths π. The loss is specifically calculated using a forward-backward algorithm based on dynamic programming.

[0033] S3.3. Update the parameters of the precoding layer and decision network. If the model M2 converges, end the training; otherwise, repeat S3.1 to S3.3.

[0034] The supervised fine-tuning training of the low-resource speech keyword detection model M3 in step S4 is as follows:

[0035] S4.1. Based on unsupervised learning, we further combine transfer learning and migrate the parameters of the rich-resource speech keyword detection model M2 trained in step S3 to model M3 for initialization. This serves as the starting point for supervised fine-tuning training. The migrated parameters include those of the precoding layer and the decision network:

[0036]

[0037]

[0038] in, and Represent the parameters of the precoding layer and decision network of model M3 that are randomly initialized, and Represents the trained parameters of the precoding layer and decision network of model M2;

[0039] S4.2. Replace the last fully connected layer of model M3 with a fully connected layer that matches the target output dimension and randomly initialize the parameters.

[0040] S4.3: The feature extraction network of model M3 remains frozen. The labeled low-resource speech dataset D3 is directly fed into model M3, and the parameters are fine-tuned using the training method in step S3.

[0041] During low-resource speech keyword detection in step S5, the model outputs a posterior probability matrix. This matrix is ​​further decoded using the prefix beam search algorithm to obtain an output label sequence. Finally, a determination is made as to whether the output label sequence contains the predefined keyword. The prefix beam search algorithm, an improvement on the beam search algorithm trained using the CTC method, aligns and combines the probabilities of repeated characters and blank characters in the prefix during the search, resulting in superior decoding performance compared to the beam search algorithm.

[0042] Compared with the prior art, the present invention has the following advantages and effects:

[0043] The present invention utilizes unsupervised learning, performs unsupervised training on unlabeled data, and replaces traditional manual features with more discriminative deep features provided by the unsupervised model. By combining the unsupervised feature extraction network, a speech keyword detection model is constructed, which efficiently utilizes unlabeled data, improves the performance of low-resource speech keyword detection systems, and reduces the cost of data annotation.

[0044] The present invention proposes to combine transfer learning on the basis of unsupervised learning with supervised pre-training under rich-resource conditions to perform parameter migration, providing a better training starting point under low-resource conditions; further combining the use of labeled data on the basis of utilizing unlabeled data, effectively alleviating the problem of poor performance of the speech keyword detection model under low-resource conditions, and improving the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is an overall flow chart of the low-resource speech keyword detection method based on unsupervised learning and transfer learning of the present invention;

[0046] Figure 2 Schematic diagram of the training of the low-resource speech keyword detection model based on unsupervised learning and transfer learning of the present invention;

[0047] Figure 3 This is a network architecture diagram of the speech keyword detection model of the present invention. DETAILED DESCRIPTION

[0048] The present invention is described in further detail below with reference to specific embodiments.

[0049] like Figure 1 and Figure 2 As shown, this embodiment provides a low-resource speech keyword detection method based on unsupervised learning and transfer learning, comprising the following steps:

[0050] S1. Collect and establish an unlabeled speech dataset Among them, X n is a speech sample, and N1 is the total number of unlabeled speech samples. The original waveform of the unlabeled speech sample is directly input into the unsupervised network model M1 for unsupervised training. The network parameters are iteratively updated until convergence to obtain the model M1.

[0051] In this embodiment, the unlabeled speech dataset D1, which is not limited to language types, uses about 10,000 hours of speech data, and N1 is 9,439,892, which mainly consists of Mandarin speech samples and a small number of English speech samples; the unsupervised network model M1 consists of an encoder and a context representation module, where the encoder

[0052] It consists of 7 one-dimensional convolutional neural network layers, each one-dimensional convolutional layer uses layer normalization and GELU activation function, the convolution kernel sizes are [10, 3, 3, 3, 3, 2, 2], the strides are [5, 2, 2, 2, 2, 2, 2], and the number of channels is 512; the encoder is used to encode the input speech sequence into a potential pronunciation representation; the context representation module is composed of 24 Transformer layers with a multi-head attention mechanism, each Transformer layer uses an output dimension of 1024, an internal dimension of the feedforward layer of 4096, and 16 attention heads; the context representation module is used to construct a context representation from the potential pronunciation representation, that is, the high-dimensional deep feature of the speech, and the dimension of the high-dimensional deep feature is 1024; the loss function for training the unsupervised network model M1 uses contrastive loss, and training is based on the mask prediction method.

[0053] S2. Reuse the network topology and parameters of the unsupervised network model M1 and solidify it as a feature extraction network. Use the precoding layer to connect the feature extraction network with the decision network to build a speech keyword detection model.

[0054] The feature extraction network is directly reused from the unsupervised model trained in S1. It has the same network structure and freezes the network parameters as the feature extraction network. Its function is to convert the original speech into discriminative high-dimensional deep features by using the pronunciation information learned from the unlabeled speech data. The parameters of this network are no longer updated in the subsequent supervised training process.

[0055] The precoding layer specifically includes a fully connected layer and an activation function. The input dimension of the fully connected layer is 1024 and the output dimension is 80, which is used to reduce the dimensionality of high-dimensional deep features:

[0056] O=f(W T I+b)

[0057] Where I represents the high-dimensional feature matrix obtained by the feature extraction network, O represents the low-dimensional feature matrix after dimensionality reduction, W is the learnable weight parameter matrix of the fully connected layer, b is the corresponding bias, and the linear rectification function f(x) = max(0, x) is used as the activation function for nonlinear transformation. Using learnable nonlinear transformations to reduce the dimensionality of high-dimensional deep features is used to reduce redundant information in high-dimensional deep features and reduce the subsequent computational complexity of the access decision network.

[0058] The decision network is a convolutional recurrent neural network (CNN), consisting of three CNN layers, two RNN layers, and two fully connected layers. The CNN layers all have a 3×3 kernel size, a channel number of [16, 32, 32], and a stride of 1. Each CNN layer uses batch normalization, linear rectification activation, and max pooling to capture local information. Both RNN layers use bidirectional gated units of dimension 256 to learn temporal information. Two fully connected layers are used for further learning and output dimension matching. The output dimension of the first fully connected layer is 128, and the output dimension of the last fully connected layer matches the number of target label categories for the corresponding speech keyword. The decision network models the reduced deep features and the target label sequence, making the final decision.

[0059] S3. Collect and establish a rich resource speech dataset with annotations Among them, X n is the speech sample, Y n is the corresponding annotated text, N2 is the total number of rich-resource speech samples, the rich-resource speech dataset D2 is directly input into the network, and the parameters of the precoding layer and decision network are supervised pre-trained iteratively until convergence, and the rich-resource speech keyword detection model M2 is obtained;

[0060] In this embodiment, the rich-resource speech sample dataset D2 uses 819.2 hours of annotated Mandarin speech data, N2 is 822468, of which the number of positive samples is 205615 and the number of negative samples is 6165853. Positive samples refer to samples containing keywords, and negative samples refer to samples not containing keywords. 70 keywords are used as training targets, the keyword modeling unit is a toned syllable, and the number of output label categories is 113. During training, the optimizer uses Adam, the initial learning rate is 0.001, the batch size is 64, and the model convergence standard is 30 epochs. The specific steps of the supervised pre-training of the rich-resource speech keyword detection model M2 are as follows:

[0061] S3.1. Input the rich-resource speech dataset D2 with annotations directly into model M2, passing it through the feature extraction network, precoding layer, and decision network in sequence to obtain the posterior probability output matrix;

[0062] S3.2. Calculate CTC loss using the posterior probability output matrix and target label sequence:

[0063]

[0064] Where x represents the input sequence corresponding to the speech sample X, that is, the original digital sequence quantized by sampling, l represents the target label sequence corresponding to the annotated text Y, and B represents the many-to-one mapping function between all feasible paths π of the target label sequence l. The loss is calculated using a forward-backward algorithm based on dynamic programming.

[0065] S3.3. Update the parameters of the precoding layer and decision network. If the model M2 converges, end the training; otherwise, repeat S3.1 to S3.3.

[0066] S4. Combined with transfer learning, the parameters of the rich-resource speech keyword detection model M2 are transferred to the low-resource speech keyword detection model M3, and a low-resource speech dataset with annotations is collected and established. Among them, N3 is the total number of low-resource speech samples. The low-resource speech dataset D3 is used for supervised fine-tuning training until the model M3 converges;

[0067] In this embodiment, the low-resource speech sample dataset D3 uses a 5.09-hour annotated Cantonese speech dataset, N3 is 3922, of which the number of positive samples is 161 and the duration is 0.22 hours; five keywords are used as training targets, namely: "self", "IKEA", "tonight", "why", and "like"; the modeling unit also uses toned syllables, and the number of output label categories is 12; the optimizer uses Adam during training, the initial learning rate is set to 0.0001, the batch size is 64, and the model convergence criterion is 15 epochs; the specific steps of supervised fine-tuning training of the low-resource speech keyword detection model M3 are as follows:

[0068] S4.1. Based on unsupervised learning, we further combine transfer learning and transfer the parameters of the rich-resource speech keyword detection model M2 trained in step S3 to model M3 for initialization. This serves as the starting point for supervised fine-tuning training, including the precoding layer and decision network parameters:

[0069]

[0070]

[0071] in, and Represent the parameters of the precoding layer and decision network of model M3 that are randomly initialized, and Represents the trained parameters of the precoding layer and decision network of model M2;

[0072] S4.2. Replace the last fully connected layer of model M3 with a fully connected layer that matches the target output dimension and randomly initialize the parameters.

[0073] S4.3: The feature extraction network of model M3 remains frozen. The labeled low-resource speech dataset D3 is directly fed into model M3, and the parameters are fine-tuned using the training method in step S3.

[0074] S5. Input the low-resource speech sample into the trained low-resource speech keyword detection model M3 to obtain the posterior probability output matrix, use the prefix beam search algorithm to decode the output matrix, and determine whether the decoding result contains the predefined keywords.

[0075] In this embodiment, the low-resource speech keyword model M3 trained in step S4 is used, and the test speech sample is directly input into the model M3, and passes through the feature extraction network for deep feature extraction and precoding layer dimensionality reduction in sequence, and finally output through the decision network; the output matrix is ​​decoded using the prefix beam search algorithm, and the beam width is set to 20, and the predefined keywords contained in the decoding result are judged as the final detection result.

[0076] The present invention combines unsupervised learning and transfer learning, fully utilizes unlabeled data and labeled data, improves the performance of speech keyword detection under low-resource conditions, and alleviates the data shortage in the field of speech keyword detection.

[0077] The implementation methods of the present invention are not limited to the above-mentioned embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principles of the present invention shall be considered as equivalent replacement methods and shall be included in the scope of protection of the present invention.

Claims

1. A low-resource speech keyword detection method based on unsupervised learning and transfer learning, characterized by The steps include: S1. Using unlabeled speech dataset Chinese Voice Sample X n The original waveform is directly input into the unsupervised network model M1 for training; Among them, N1 is the total number of unlabeled speech samples; S2, reuse the network model M1 as the feature extraction network, use the precoding layer to connect it with the decision network, and build the speech keyword detection model M2; S3. Use a rich resource speech dataset with annotations Train the precoding layer and decision network in M2 to obtain the speech keyword detection model M2; Among them, Y n For speech sample X n The corresponding annotated text, N2 is the total number of rich resource speech samples; S4. Migrate the rich-resource speech keyword detection model M2 to the low-resource speech keyword detection model M3, and then use the annotated low-resource speech dataset Fine-tune the training model M3; Among them, N3 is the total number of low-resource speech samples; S5. Input the low-resource speech sample into the speech keyword detection model M3 to obtain the posterior probability output matrix, use the prefix beam search algorithm to decode it, and determine whether the decoding result contains the predefined keywords; The unsupervised network model in step S1 consists of an encoder and a context representation module; The encoder consists of multiple one-dimensional convolutional neural network layers to encode the input speech sequence into a latent pronunciation representation; The contextual representation module consists of multiple Transformer layers with a multi-head attention mechanism, which is used to construct contextual representations from potential pronunciation representations, i.e., high-dimensional deep features of speech. The precoding layer in the speech keyword detection model in step S2 specifically includes a fully connected layer and an activation function, which is used to reduce the dimensionality of high-dimensional deep features: O=f(W T I+b); Where I represents the high-dimensional feature matrix obtained by the feature extraction network, O represents the low-dimensional feature matrix after dimensionality reduction, W is the learnable weight parameter matrix of the fully connected layer, b is the corresponding bias, and the linear rectification function f(x) = max(0, x) is used as the activation function for nonlinear transformation; The decision network in the speech keyword detection model in step S2 is a convolutional recurrent neural network, which specifically includes a convolutional neural network module, a recurrent neural network module, and a fully connected network module. The output dimension of the last fully connected layer is consistent with the number of target label categories of the corresponding speech keyword; The supervised fine-tuning training of the low-resource speech keyword detection model M3 in step S4 is as follows: S4.1: Migrate the parameters of the precoding layer and decision network of model M2 to model M3; S4.2: Replace the last fully connected layer of model M3 with a fully connected layer that matches the target output dimension and randomly initialize the parameters. S4.3: The feature extraction network of model M3 remains frozen, and the annotated low-resource speech dataset D3 is directly input into model M3. The training method in step S3 is used for parameter fine-tuning.

2. The low-resource speech keyword detection method based on unsupervised learning and transfer learning according to claim 1, characterized in that The feature extraction network in the speech keyword detection model in step S2 will no longer update parameters in the subsequent supervised training process.

3. The low-resource speech keyword detection method based on unsupervised learning and transfer learning according to claim 1, characterized in that The supervised pre-training of the rich resource speech keyword detection model M2 in step S3 is as follows: S3.1: The rich, annotated speech dataset D2 is directly fed into the model M2, passing through the feature extraction network, precoding layer, and decision network in sequence to obtain the posterior probability output matrix. S3.2: Compute the connectionist temporal classification loss using the posterior probability output matrix and the target label sequence: Where x represents the input sequence corresponding to the speech sample X, that is, the original digital sequence quantized by sampling, l represents the target label sequence corresponding to the annotated text Y, and B represents the many-to-one mapping function between the target label sequence l and all feasible paths π; S3.3: Update the parameters of the precoding layer and decision network. If the model M2 converges, end the training; otherwise, repeat S3.1 to S3.

3.

4. The low-resource speech keyword detection method based on unsupervised learning and transfer learning according to claim 1, characterized in that When performing low-resource speech keyword detection in step S5, the output of the model is a posterior probability matrix, which is further decoded using a prefix beam search algorithm to obtain a final output label sequence, and finally a determination is made as to whether the output label sequence contains predefined keywords; The prefix beam search algorithm is an improvement on the beam search algorithm under the CTC training method. During the search, the probabilities of repeated characters and blank characters in the prefix are aligned and merged.

5. The low-resource speech keyword detection method based on unsupervised learning and transfer learning according to any one of claims 1 to 4, characterized in that: The unlabeled speech dataset in step S1 is single language or multi-language.

Images