A method, device and storage medium for compressing a speech recognition model

Abstract

Embodiments of the present application disclose a method and device for compressing a speech recognition model and a storage medium. The method comprises: using an unlabelled speech data set as first training data, training an original speech recognition model using a knowledge distillation algorithm to obtain a first speech recognition model; using the unlabelled speech data set and a labelled speech data set as second training data, fine-tuning the first speech recognition model for a speech recognition task to obtain a second speech recognition model; using the labelled speech data set as third training data, training the second speech recognition model using a neural network structure search algorithm to obtain a third speech recognition model comprising a plurality of sub-speech recognition models; extracting a sub-speech recognition model from the plurality of sub-speech recognition models to obtain a compressed speech recognition model. Thus, the speech recognition model can be effectively compressed to obtain a speech recognition model that meets the computing resources and has high recognition accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of speech recognition, and more particularly, to a method and device for compressing a speech recognition model and a storage medium. BACKGROUND

[0002] Speech recognition, also known as automatic speech recognition (ASR), is a key component of human-computer interaction and communication, and its main purpose is to enable machines to "understand" human speech and convert speech signals into text signals.

[0003] Currently, in order to pursue higher speech recognition performance, the network structure of the speech recognition model inevitably becomes more complex, and the network computation and network parameters increase exponentially, which results in high storage space and computing resource consumption.

[0004] However, in some resource-constrained scenarios such as smart phones and vehicle-mounted scenarios, the huge computation and parameter amount makes it difficult to deploy the speech recognition model to intelligent terminal devices, and the speech recognition use requirements of the resource-constrained scenarios cannot be met. SUMMARY

[0005] Therefore, the embodiments of the present application disclose a method and device for compressing a speech recognition model and a storage medium, which effectively compress the speech recognition model and obtain a speech recognition model that meets the computing resources and has high recognition accuracy.

[0006] The technical solutions provided by the embodiments of the present application are as follows:

[0007] In a first aspect, the embodiments of the present application provide a method for compressing a speech recognition model, comprising:

[0008] using a knowledge distillation algorithm to train an original speech recognition model using an unlabeled speech data set as first training data, to obtain a first speech recognition model;

[0009] using the unlabeled speech data set and a labeled speech data set as second training data, fine-tuning the first speech recognition model for a speech recognition task, to obtain a second speech recognition model;

[0010] using a neural network structure search algorithm to train the second speech recognition model using the labeled speech data set as third training data, to obtain a third speech recognition model; wherein the third speech recognition model comprises a plurality of sub-speech recognition models;

[0011] Extract a sub-voice recognition model from the plurality of sub-voice recognition models to obtain a compressed voice recognition model; the compressed voice recognition model is used for recognizing voice data to be recognized.

[0012] In a possible implementation, the training of the original voice recognition model by using the knowledge distillation algorithm with the unlabeled voice data set as the first training data to obtain the first voice recognition model comprises:

[0013] Adding a linear layer to the original voice recognition model to obtain a fourth voice recognition model; the linear layer is used to map the output dimension of the fourth voice recognition model to be consistent with the output dimension of the original voice recognition model;

[0014] The original voice recognition model is used as a teacher model of the knowledge distillation algorithm, and the fourth voice recognition model is used as a student model of the knowledge distillation algorithm, and the teacher model is used to guide the training of the student model to obtain the first voice recognition model; wherein the original voice recognition model does not mask the unlabeled voice data set, and the fourth voice recognition model masks the unlabeled voice data set.

[0015] In a possible implementation, the training of the original voice recognition model by using the knowledge distillation algorithm with the unlabeled voice data set as the first training data to obtain the first voice recognition model comprises:

[0016] The unlabeled voice data set is input into the original voice recognition model for training to obtain the output result of each Transformer layer in the Transformer module of the original voice recognition model;

[0017] The output results of the first K Transformer layers are averaged to obtain a first output result;

[0018] The unlabeled voice data set is input into the fourth voice recognition model for training to obtain a second output result; the second output result is the output result of the Transformer module of the fourth voice recognition model;

[0019] The first loss value is calculated by using the first output result and the second output result;

[0020] The fourth voice recognition model is iteratively trained with the first training target of minimizing the first loss value until the fourth voice recognition model converges to obtain the first voice recognition model.

[0021] In a possible implementation, the fine-tuning of the first speech recognition model on the speech recognition task by using the unlabeled speech data set and the labeled speech data set as second training data to obtain a second speech recognition model comprises the following steps.

[0022] adding a linear mapping module to the first speech recognition model to obtain a fifth speech recognition model;

[0023] inputting the labeled speech data set into the fifth speech recognition model to fine-tune the speech recognition task, and obtaining a third output result of the linear mapping module;

[0024] inputting the unlabeled speech data set into the fifth speech recognition model to fine-tune the speech recognition task, and obtaining a fourth output result of the linear mapping module;

[0025] calculating a second loss value by using the third output result and the fourth output result;

[0026] taking the minimization of the second loss value as a second training target, iteratively training the fifth speech recognition model until the fifth speech recognition model converges, and obtaining the second speech recognition model.

[0027] In a possible implementation, the calculating of the second loss value by using the third output result and the fourth output result comprises the following steps.

[0028] calculating a first sub-loss value by using the third output result and a label corresponding to the labeled speech data set;

[0029] calculating a second sub-loss value by using the fourth output result and a pseudo label corresponding to the unlabeled speech data set, wherein the pseudo label is obtained by decoding the unlabeled speech data set;

[0030] calculating the second loss value by using the first sub-loss value and the second sub-loss value.

[0031] In a possible implementation, the training of the second speech recognition model by using a neural network structure search algorithm with the labeled speech data set as third training data to obtain a third speech recognition model comprises the following steps.

[0032] constructing a search space of the second speech recognition model by using variable parameters of a Transformer module;

[0033] inputting the labeled speech data set into the second speech recognition model, randomly sampling a plurality of sub-speech recognition models from the search space to perform forward propagation calculation, and obtaining a fifth output result;

[0034] The third loss value is calculated by using the fifth output result and a label corresponding to the labeled speech data set;

[0035] The second speech recognition model is iteratively trained with a third training target of minimizing the third loss value until the second speech recognition model converges, and the third speech recognition model is obtained.

[0036] In a possible implementation, the variable parameters of the Transformer module include at least one of the following: an input-output dimension of a Transformer layer, a number of attention heads in a multi-head attention mechanism of the Transformer layer, a dimension of a feedforward neural network in the Transformer layer, a ratio of the input-output dimension of the Transformer layer, and a number of Transformer layers in the Transformer module.

[0037] In a second aspect, an apparatus for compressing a speech recognition model is provided, and the apparatus includes:

[0038] A first training unit configured to train an original speech recognition model by using a knowledge distillation algorithm with a no-labeled speech data set as first training data, and obtain a first speech recognition model;

[0039] A second training unit configured to fine-tune the first speech recognition model for a speech recognition task by using the no-labeled speech data set and a labeled speech data set as second training data, and obtain a second speech recognition model;

[0040] A third training unit configured to train the second speech recognition model by using a neural network structure search algorithm with the labeled speech data set as third training data, and obtain a third speech recognition model; the third speech recognition model includes a plurality of sub-speech recognition models.

[0041] An extraction unit configured to extract one sub-speech recognition model from the plurality of sub-speech recognition models, and obtain a compressed speech recognition model; the compressed speech recognition model is used for recognizing speech data to be recognized.

[0042] In a third aspect, an apparatus for compressing a speech recognition model is provided, and the apparatus includes a processor, a memory, and a system bus;

[0043] The processor and the memory are connected through the system bus;

[0044] The memory is configured to store one or more programs including instructions that, when executed by the processor, cause the processor to perform the method for compressing a speech recognition model according to any one of the first aspect.

[0045] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, and the computer-readable storage medium stores instructions. When the instructions run on a terminal device, the terminal device performs the method for compressing a speech recognition model according to any one of the first aspect.

[0046] In a fifth aspect, an embodiment of the present application provides a computer program product, and the computer program product runs on a terminal device. When the computer program product runs on the terminal device, the terminal device performs the method for compressing a speech recognition model according to any one of the first aspect.

[0047] Based on the above technical solutions, the present application has the following beneficial effects:

[0048] The embodiment of the present application discloses a method, device and storage medium for compressing a speech recognition model. The method comprises: using an untagged speech data set as first training data, training an original speech recognition model by using a knowledge distillation algorithm to obtain a first speech recognition model; using the untagged speech data set and a tagged speech data set as second training data, fine-tuning the first speech recognition model for a speech recognition task to obtain a second speech recognition model; using the tagged speech data set as third training data, training the second speech recognition model by using a neural network structure search algorithm to obtain a third speech recognition model; wherein the third speech recognition model comprises a plurality of sub-speech recognition models; extracting a sub-speech recognition model from the plurality of sub-speech recognition models to obtain a compressed speech recognition model; and the compressed speech recognition model is used to recognize speech data to be recognized. As can be seen, in the embodiment of the present application, the knowledge distillation algorithm and the fine-tuning of the speech recognition task are used to improve the recognition accuracy of the speech recognition model, thereby improving the recognition accuracy of the subsequent compressed speech recognition model; the neural network structure search algorithm is used to compress the speech recognition model, so that a sub-speech recognition model meeting the computing resources can be obtained, and each sub-speech recognition model is a subset of the third speech recognition model, sharing the knowledge learned by the third speech recognition model, so that the loss of recognition accuracy of the sub-speech recognition model can be effectively reduced; that is, the knowledge distillation algorithm, the fine-tuning of the speech recognition task and the neural network structure search algorithm are combined in the embodiment of the present application to effectively compress the speech recognition model, so that a speech recognition model meeting the computing resources and having high accuracy can be obtained, so that the compressed speech recognition model can be deployed in a resource-limited scene, and the speech recognition use demand of the resource-limited scene can be met. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced. Obviously, the accompanying drawings in the following description only represent some of the embodiments of the present application, and other drawings can be obtained by those of ordinary skill in the art without any creative effort based on the disclosed drawings.

[0050] Figure 1 A flow chart of a method for compressing a speech recognition model according to an embodiment of the present application;

[0051] Figure 2 A flow chart of another method for compressing a speech recognition model according to an embodiment of the present application;

[0052] Figure 3 A structural schematic diagram of a device for compressing a speech recognition model according to an embodiment of the present application. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments only represent some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without any creative effort fall within the scope of protection of the present application.

[0054] In the present specification, the terms “include”, “contain”, “have” and their variants all mean “include but not limited to”, unless otherwise specifically emphasized. It should be noted that in the description of the embodiments of the present application, the terms “first”, “second”, etc. are only used for distinguishing the description purpose, and cannot be understood as indicating or implying relative importance, nor indicating or implying sequence.

[0055] The existing model compression schemes mostly use pruning and quantization methods. Pruning can be divided into two categories: unstructured pruning and structured pruning. Unstructured pruning is to prune the single or entire row, entire column weight values in the weight matrix that are below the threshold according to the preset threshold. The new weight matrix after pruning will become a sparse matrix, that is, the pruned values will be set to 0, and then the precision is gradually trimmed and restored through iteration, so as to achieve the purpose of compressing the model without reducing the accuracy. Structured pruning usually prunes the filter or entire network layer as the basic unit. Structured pruning mainly includes channel-wise, filter-wise and layer-wise, etc. Channel-wise and filter-wise are pruning in the channel dimension, the difference is that channel-wise is pruning the input layer channel, filter-wise is pruning the output layer channel, and layer-wise is pruning the entire layer. Quantization is the process of converting the trained deep neural network weights, activation values, etc. from high precision to low precision, such as converting 32-bit floating-point numbers to 8-bit integers int8. Model quantization can reduce memory and storage occupation, reduce power consumption and improve computing speed.

[0056] However, the inventors have found through research that although the implementation of unstructured pruning is very simple, it can be achieved by simply setting the matrix weight to zero, but most deep learning frameworks and hardware cannot accelerate the calculation of sparse matrices at present, which means that the consumption of storage space and computing resources for actual application cannot bring substantial performance improvement. Structured pruning can conveniently change the structural characteristics of the model to achieve the effect of compressing the model, but due to the roughness of this pruning method, to achieve a high compression rate means that the accuracy loss is very serious, and the training is not easy to converge, the training time is longer, and the model sparsity is low. Model quantization inevitably introduces accuracy loss, the lower the bit rate, the greater the accuracy loss; for a relatively difficult task such as speech recognition, the accuracy loss caused by model quantization will also be greater; at the same time, the implementation of quantization algorithms for different chip backends has different characteristics, and different hardware needs to research different quantization schemes for hardware, which will lead to a decline in the migration ability of the quantized model on the hardware platform. It can be seen that through the above several compression schemes to compress the end-to-end speech recognition model, to achieve a high compression rate to meet the deployment conditions of mobile phones, vehicle-mounted and other intelligent terminal devices will bring great accuracy loss.

[0057] To this end, the embodiment of the present application discloses a method, device and storage medium for compressing a speech recognition model. The method comprises: using an unlabeled speech data set as first training data, training an original speech recognition model using a knowledge distillation algorithm to obtain a first speech recognition model; using the unlabeled speech data set and a labeled speech data set as second training data, fine-tuning the first speech recognition model for a speech recognition task to obtain a second speech recognition model; using the labeled speech data set as third training data, training the second speech recognition model using a neural network structure search algorithm to obtain a third speech recognition model; wherein the third speech recognition model comprises a plurality of sub-speech recognition models; extracting a sub-speech recognition model from the plurality of sub-speech recognition models to obtain a compressed speech recognition model; and the compressed speech recognition model is used to recognize speech data to be recognized. As can be seen, in the embodiment of the present application, the knowledge distillation algorithm and the fine-tuning of the speech recognition task are used to improve the recognition accuracy of the speech recognition model, thereby improving the recognition accuracy of the subsequent compressed speech recognition model; the neural network structure search algorithm is used to compress the speech recognition model, so that a sub-speech recognition model meeting the computing resources can be obtained, and each sub-speech recognition model is a subset of the third speech recognition model, sharing the knowledge learned by the third speech recognition model, so that the recognition accuracy loss of the sub-speech recognition model can be effectively reduced; that is, the combination of the knowledge distillation algorithm, the fine-tuning of the speech recognition task and the neural network structure search algorithm in the embodiment of the present application can effectively compress the speech recognition model, obtain a speech recognition model meeting the computing resources and having high accuracy, so that the compressed speech recognition model can be deployed in a resource-limited scene to meet the speech recognition use requirements of the resource-limited scene.

[0058] Referring to Figure 1 The embodiment of the present application discloses a flowchart of a method for compressing a speech recognition model, which comprises:

[0059] S101, using an unlabeled speech data set as first training data, training an original speech recognition model using a knowledge distillation algorithm to obtain a first speech recognition model;

[0060] It should be noted that in the embodiment of the present application, the unlabeled speech data set in the current scene is used for pre-training, so that the first speech recognition model can learn the speech characteristics in the current scene, thereby better adapting to the downstream speech recognition task.

[0061] The core idea of the knowledge distillation (KD) algorithm is to use the knowledge learned by the teacher model to guide the training of the student model to improve the performance and accuracy of the student model.

[0062] The original speech recognition model in the embodiments of the present application can be a Wav2vec2 model, and the specific implementation is not limited and can be selected according to actual needs.

[0063] S102, fine-tuning the first speech recognition model for a speech recognition task by taking the unlabeled speech data set and the labeled speech data set as second training data, to obtain a second speech recognition model;

[0064] It should be noted that the fine-tuning for the speech recognition task in the embodiments of the present application enables the second speech recognition model to learn information related to the speech recognition task, thereby improving the recognition accuracy of the second speech recognition model.

[0065] S103, training the second speech recognition model by taking the labeled speech data set as third training data and using a neural network structure search algorithm, to obtain a third speech recognition model; wherein the third speech recognition model includes a plurality of sub-speech recognition models.

[0066] It should be noted that each sub-speech recognition model in the embodiments of the present application is a subset of the third speech recognition model and shares the knowledge learned by the third speech recognition model.

[0067] Neural Architecture Search (NAS) is a technology for automatically designing neural networks. It can automatically design a high-performance network structure from a sample set through an algorithm. In each iteration of the search process, a "sample" is generated from the search space, which is a neural network structure, called "subnetwork".

[0068] S104, extracting a sub-speech recognition model from the plurality of sub-speech recognition models to obtain a compressed speech recognition model; the compressed speech recognition model is used for recognizing speech data to be recognized.

[0069] As can be seen, the embodiments of the present application improve the recognition accuracy of the speech recognition model through the knowledge distillation algorithm and the fine-tuning for the speech recognition task, thereby improving the recognition accuracy of the subsequent compressed speech recognition model. The neural network structure search algorithm is used to compress the speech recognition model, so that a sub-speech recognition model that meets the computing resources can be obtained. Each sub-speech recognition model is a subset of the third speech recognition model and shares the knowledge learned by the third speech recognition model, which can effectively reduce the loss of recognition accuracy of the sub-speech recognition model. That is, the embodiments of the present application combine the knowledge distillation algorithm, the fine-tuning for the speech recognition task, and the neural network structure search algorithm to effectively compress the speech recognition model, so as to obtain a speech recognition model that meets the computing resources and has high accuracy. The compressed speech recognition model can be deployed in a resource-limited scenario to meet the speech recognition use requirements of the resource-limited scenario.

[0070] Referring to Figure 2 Another flowchart of a method for compressing a speech recognition model disclosed in an embodiment of the present application. Figure 2 Stage1 in corresponds to S101 described above, Stage2 corresponds to S102 described above, and Stage3 corresponds to S103 described above. Figure 2 Waveform Unlabel in corresponds to an unlabeled speech dataset, Waveform Label indicates a labeled speech dataset, Original indicates that no mask is applied to the speech dataset, and Masked indicates that a mask is applied to the speech dataset. Figure 2 Teacher Model in corresponds to a teacher model, and Student Model corresponds to a student model; Feature Extractor corresponds to a feature extraction module, Transformer Layers corresponds to a Transformer module, Pred-Head corresponds to a linear layer, and LinearProj corresponds to a linear mapping module. Figure 2 Top-K Ayg in corresponds to an average value of output results of the top K Transformer layers in the Transformer module of the teacher model, DistillLoss corresponds to a distillation loss, i.e., a first loss value described below; Pseudo Label corresponds to a pseudo label corresponding to the unlabeled speech dataset, Ground Truth corresponds to a label corresponding to the labeled speech dataset; CTC-unlabel corresponds to a Connectionist Temporal Classification (CTC) loss corresponding to the unlabeled speech data, i.e., a first sub-loss value described below; CTC-label corresponds to a CTC loss corresponding to the labeled speech data, i.e., a second sub-loss value described below; and CTC Loss corresponds to a CTC loss corresponding to the sub-speech recognition model, i.e., a third loss value described below.

[0071] In a possible implementation, S101 in the method for compressing a speech recognition model provided in an embodiment of the present application specifically includes:

[0072] S1011, a linear layer is added to the original speech recognition model to obtain a fourth speech recognition model; the linear layer is used to map an output dimension of the fourth speech recognition model to be consistent with an output dimension of the original speech recognition model.

[0073] S1012, use the original speech recognition model as a teacher model of a knowledge distillation algorithm, use the fourth speech recognition model as a student model of the knowledge distillation algorithm, train the student model by using the teacher model to guide the student model, and obtain the first speech recognition model; wherein the original speech recognition model does not mask the unlabeled speech data set, and the fourth speech recognition model masks the unlabeled speech data set.

[0074] It should be noted that the structure of the student model and the structure of the teacher model in the embodiments of the present application are similar, and only one layer of linear layer is added to align the output dimensions of the teacher model and the student model.

[0075] The mask is a means in training, which is used to mask the original signal. For example, the content of a speech is "turn on the TV", and after the mask, it may be "turn on the machine", and "TV" is masked into silence. It can be understood that the above is only an example and should not be construed as a limitation of the present application.

[0076] S1012 in the embodiments of the present application can specifically include:

[0077] S10121, input the unlabeled speech data set into the original speech recognition model for training to obtain the output result of each Transformer layer in the Transformer module of the original speech recognition model;

[0078] It should be noted that each Transformer layer in the Transformer module is mainly composed of two sub-layers of a multi-head-attention layer and a feedforward neural network layer. The Transformer layer learns the relationship between words through the multi-head-attention mechanism, encodes the context information, and then outputs the vector representation of each word that integrates the context features, i.e. the context representation, through a feedforward neural network after nonlinear transformation.

[0079] In the embodiments of the present application, the unlabeled speech data set x is extracted by the feature extraction module of the teacher model during training, and then the context representation is constructed by the Transformer module of the teacher model The output result of the i-th Transformer layer in the Transformer module is.

[0080] S10122, average the output results of the first K Transformer layers to obtain a first output result;

[0081] S10123, input the unlabeled speech data set into the fourth speech recognition model for training to obtain a second output result; the second output result is an output result of a Transformer module of the fourth speech recognition model;

[0082] In the embodiment of the application, the Masked unlabeled speech data set x' is extracted by the feature extraction module of the student model during training, and then the Transformer module of the student model is used to predict the representation f of the complete data s (x').

[0083] S10124, a first loss value is calculated using the first output result and the second output result;

[0084] In the embodiment of the application, the calculation formula of the first loss value is as follows:

[0085]

[0086] wherein, denotes the first loss value, denotes the time step corresponding to the mask part of the unlabeled speech data input into the student model, denotes the mean value of the output results of the first K Transformer layers in the Transformer module of the teacher model, i.e., the first output result f s (x') denotes the output result of the Transformer module of the student model, i.e., the second output result.

[0087] S10125, the fourth speech recognition model is iteratively trained with the minimization of the first loss value as the first training target until the fourth speech recognition model converges, and the first speech recognition model is obtained.

[0088] As can be seen, in the embodiment of the application, the knowledge distillation algorithm is used to enable the first speech recognition model to learn the speech characteristics in the current scenario, so that the downstream speech recognition task can be better adapted, and the recognition accuracy of the subsequent compressed speech recognition model can be further improved.

[0089] In a possible implementation manner, the method for compressing the speech recognition model provided in the embodiment of the application specifically includes the following steps:

[0090] S1021, a linear mapping module is added to the first speech recognition model to obtain a fifth speech recognition model;

[0091] It should be noted that the probability distribution of the modeling unit, such as the probability distribution of 26 English letters, is obtained by adding a linear mapping module in the embodiment of the present application. It can be understood that the above is only an exemplary description and should not be understood as a limitation of the present application.

[0092] S1022, inputting the labeled speech data set into the fifth speech recognition model to perform fine tuning of the speech recognition task, to obtain a third output result of the linear mapping module;

[0093] S1023, inputting the unlabeled speech data set into the fifth speech recognition model to perform fine tuning of the speech recognition task, to obtain a fourth output result of the linear mapping module;

[0094] It should be noted that the labeled speech data set will be masked in S1022, and the unlabeled speech data set will also be masked in S1023. The fifth speech recognition model is trained in combination with the unlabeled speech data set and the labeled speech data set, so that the fifth speech recognition model can learn task-specific task-related information, and further improve the recognition accuracy of the model.

[0095] S1024, calculating a second loss value by using the third output result and the fourth output result;

[0096] It should be noted that the model is optimized by using CTC Loss in the embodiment of the present application. CTC mainly solves the problem of misalignment between the label of the neural network and the output result. The advantage is that the label can be variable length without forced alignment, and only the input sequence and the supervised label sequence are required for training. In the field of text recognition and language recognition, the loss can be calculated more flexibly, and gradient descent can be performed.

[0097] In the embodiment of the present application, S1024 can specifically include:

[0098] S10241, calculating a first sub-loss value by using the third output result and the label corresponding to the labeled speech data set;

[0099] In the embodiment of the present application, the calculation formula of the first sub-loss value is as follows:

[0100]

[0101] wherein, represents the first sub-loss value, y represents the label corresponding to the labeled speech data set, and x' represents the output result corresponding to the labeled speech data set, i.e. the third output result.

[0102] S10242, a second sub-loss value is calculated by using the fourth output result and the pseudo label corresponding to the unlabeled speech data set; wherein the pseudo label is obtained by decoding the unlabeled speech data set;

[0103] In the embodiment of the application, the calculation formula of the second sub-loss value is as follows:

[0104]

[0105]

[0106] wherein, denotes the second sub-loss value, denotes the pseudo label corresponding to the unlabeled speech data set, and x' denotes the output result corresponding to the unlabeled speech data set, i.e. the fourth output result. is the result obtained by greedy decoding of the unlabeled speech data set, and the greedy decoding is to take the probability maximum y as the decoding result.

[0107] S10243, a second loss value is calculated by using the first sub-loss value and the second sub-loss value.

[0108] In the embodiment of the application, the calculation formula of the second loss value is as follows:

[0109]

[0110] wherein, denotes the second loss value, denotes the first sub-loss value, denotes the second sub-loss value, and a denotes a hyperparameter used to control the proportion of the loss function.

[0111] S1025, the fifth speech recognition model is iteratively trained with the second loss value as the second training target until the fifth speech recognition model converges, and the second speech recognition model is obtained.

[0112] It can be seen that in the embodiment of the application, the fine tuning of the speech recognition task is performed by using the speech labeled data and the unlabeled speech data, and the optimization is performed by using the CTC Loss, so that the recognition model can learn the information related to the speech recognition task, and the recognition accuracy of the model is further improved.

[0113] In a possible implementation manner, the method for compressing the speech recognition model provided in the embodiment of the application specifically includes the following steps:

[0114] S1031, a search space of the second speech recognition model is constructed by using the variable parameters of the Transformer module;

[0115] The variable parameters of the Transformer module include at least one of the following: an input-output dimension of a Transformer layer (Embedding-dim), a number of attention heads in a multi-head attention mechanism of the Transformer layer (Attention-head), a ratio of a dimension of a feed-forward neural network in the Transformer layer to the input-output dimension of the Transformer layer (FFN-ratio), and a number of Transformer layers in the Transformer module (Layer-num).

[0116] It should be noted that the embodiments of the present application construct a search space for the Transformer module, and define the search space as a super network Supernet, each subnet Subnet in the space being a subset of the Supernet, and all the Subnets sharing the weights of the common part. Referring to Figure 2 , Figure 2 The black part of Stage3 in the above table represents the module with variable dimensions and shared parameters in the model.

[0117] For example, the model structure is {1024, [16, 4.0] x 24}, which indicates that the model has 24 Transformer layers, the Embedding-dim is 1024, the Attention-heads of each Transformer layer is 16, and the FFN-ratio is 4.0. Assuming that the search space is defined as Embed_dim e {512, 768, 1024}, Atten_head e {8, 12, 16}, FFN_ratio e {3.0, 4.0}, Layer_num e {24}, and the subnet structure randomly searched at present is {512, [16, 4.0] x 2 + [12, 3.0] + [8, 4.0] + [12, 4.0] x 20}, which indicates that the subnet model has 24 Transformer layers, the Embedding-dim is 512, the Attention-heads of the first two Transformer layers is 16, the FFN-ratio is 4.0, the Attention-heads of the third Transformer layer is 12, the FFN-ratio is 3.0, the Attention-heads of the fourth Transformer layer is 8, the FFN-ratio is 4.0, and the Attention-heads of the last 20 Transformer layers is 12, the FFN-ratio is 4.0. The above search space contains 1.4 x 10 19The range of the model parameter quantity is [70M-317M], so that different computing resource requirements of small intelligent terminals such as mobile phones and vehicles can be met. It can be understood that the above is only an example and should not be understood as a limitation of the present application.

[0118] S1032, input the labeled speech data set into the second speech recognition model, randomly sample a plurality of sub-speech recognition models from the search space for forward propagation calculation to obtain a fifth output result;

[0119] It should be noted that the labeled speech data set is masked in S1032.

[0120] S1033, calculating a third loss value by using the fifth output result and the label corresponding to the labeled speech data set;

[0121] It should be noted that the calculation formula of the third loss value is similar to that of the first sub-loss value, so it will not be described again.

[0122] S1034, taking minimizing the third loss value as a third training target, iteratively training the second speech recognition model until the second speech recognition model converges to obtain the third speech recognition model.

[0123] The search strategy in the embodiment of the present application is random sampling, and the evaluation criterion is to evaluate the pros and cons of the model according to the loss value. The loss value of the present application includes the first loss value, the second loss value and the third loss value. The lower the loss value, the better the model effect.

[0124] As can be seen, in the embodiment of the present application, a search space containing a large number of sub-speech recognition models can be constructed by setting the variable quantity of the Transformer module of different dimensions. When training, a sub-speech recognition model is randomly sampled from the search space for forward calculation, so that a small model meeting different computing resources can be obtained. In this way, only the search space, the search strategy and the evaluation criterion need to be determined to automatically search for the optimal model architecture without human intervention. Moreover, all small models in the search space are subsets of the large model, sharing the knowledge learned by the large model, so that the precision loss of the small model can be effectively reduced.

[0125] Referring to Figure 3 , the embodiment of the present application discloses a structure diagram of a device for compressing a speech recognition model. The device comprises:

[0126] The first training unit 301 is configured to use a knowledge distillation algorithm to train an original speech recognition model by taking the unlabeled speech data set as the first training data to obtain a first speech recognition model.

[0127] The second training unit 302 fine-tunes the first speech recognition model on a speech recognition task by taking the unlabeled speech data set and the labeled speech data set as second training data, to obtain a second speech recognition model.

[0128] The third training unit 303 is configured to train the second speech recognition model by taking the labeled speech data set as third training data and using a neural network structure search algorithm, to obtain a third speech recognition model; wherein the third speech recognition model includes a plurality of sub speech recognition models.

[0129] The extraction unit 304 is configured to extract one sub speech recognition model from the plurality of sub speech recognition models, to obtain a compressed speech recognition model; the compressed speech recognition model is used for recognizing speech data to be recognized.

[0130] The embodiments of the present application disclose a method and device for compressing a speech recognition model and a storage medium. The original speech recognition model is trained by taking an unlabeled speech data set as first training data and using a knowledge distillation algorithm, to obtain a first speech recognition model. The first speech recognition model is fine-tuned on a speech recognition task by taking the unlabeled speech data set and a labeled speech data set as second training data, to obtain a second speech recognition model. The second speech recognition model is trained by taking the labeled speech data set as third training data and using a neural network structure search algorithm, to obtain a third speech recognition model. The third speech recognition model includes a plurality of sub speech recognition models. One sub speech recognition model is extracted from the plurality of sub speech recognition models, to obtain a compressed speech recognition model. The compressed speech recognition model is used for recognizing speech data to be recognized. It can be seen that, in the embodiments of the present application, the knowledge distillation algorithm and the fine-tuning of the target task are used to improve the recognition accuracy of the speech recognition model, so as to improve the recognition accuracy of the compressed speech recognition model. The neural network structure search algorithm is used to compress the speech recognition model, so that a sub speech recognition model meeting the computing resource requirement can be obtained. Each sub speech recognition model is a subset of the third speech recognition model, shares the knowledge learned by the third speech recognition model, and can effectively reduce the loss of recognition accuracy of the sub speech recognition model. That is, the knowledge distillation algorithm, the fine-tuning of the target task and the neural network structure search algorithm are combined in the embodiments of the present application to effectively compress the speech recognition model, so that a speech recognition model meeting the computing resource requirement and having high accuracy can be obtained. The compressed speech recognition model can be deployed in a resource-limited scene, to meet the speech recognition use requirement of the resource-limited scene.

[0131] In a possible implementation, the first training unit 301 in the device for compressing a speech recognition model provided in the embodiments of the present application specifically includes:

[0132] The first increasing unit is configured to add a linear layer to the original speech recognition model to obtain a fourth speech recognition model; the linear layer is configured to map an output dimension of the fourth speech recognition model to be consistent with an output dimension of the original speech recognition model.

[0133] The first training subunit is configured to use the original speech recognition model as a teacher model of a knowledge distillation algorithm, use the fourth speech recognition model as a student model of the knowledge distillation algorithm, and train the student model by using the teacher model to obtain the first speech recognition model; wherein the original speech recognition model does not mask the unlabeled speech data set, and the fourth speech recognition model masks the unlabeled speech data set.

[0134] In a possible implementation, the first training subunit in the device for compressing a speech recognition model provided in the embodiments of the present application specifically includes:

[0135] The first result unit is configured to input the unlabeled speech data set into the original speech recognition model to train the original speech recognition model, and obtain an output result of each Transformer layer in a Transformer module of the original speech recognition model.

[0136] The first calculation unit is configured to perform mean calculation on the output results of the first K Transformer layers to obtain a first output result.

[0137] The second result unit is configured to input the unlabeled speech data set into the fourth speech recognition model to train the fourth speech recognition model, and obtain a second output result; the second output result is an output result of a Transformer module of the fourth speech recognition model.

[0138] The second calculation unit is configured to calculate a first loss value by using the first output result and the second output result.

[0139] The first iterative training unit is configured to perform iterative training on the fourth speech recognition model with the minimization of the first loss value as a first training target, until the fourth speech recognition model converges, to obtain the first speech recognition model.

[0140] In a possible implementation, the second training unit 302 in the device for compressing a speech recognition model provided in the embodiments of the present application specifically includes:

[0141] The second increasing unit is configured to add a linear mapping module to the first speech recognition model to obtain a fifth speech recognition model.

[0142] a third result unit, configured to input the labeled speech data set into the fifth speech recognition model to perform fine-tuning of a speech recognition task, to obtain a third output result of the linear mapping module;

[0143] a fourth result unit, configured to input the unlabeled speech data set into the fifth speech recognition model to perform fine-tuning of a speech recognition task, to obtain a fourth output result of the linear mapping module;

[0144] a third calculation unit, configured to calculate a second loss value by using the third output result and the fourth output result;

[0145] a second iterative training unit, configured to perform iterative training on the fifth speech recognition model with minimization of the second loss value as a second training target, until the fifth speech recognition model converges, to obtain the second speech recognition model.

[0146] In a possible implementation, the third calculation unit in the device for compressing a speech recognition model provided in the embodiments of the present application specifically includes:

[0147] a first calculation sub-unit, configured to calculate a first sub-loss value by using the third output result and a label corresponding to the labeled speech data set;

[0148] a second calculation sub-unit, configured to calculate a second sub-loss value by using the fourth output result and a pseudo label corresponding to the unlabeled speech data set, wherein the pseudo label is obtained by decoding the unlabeled speech data set;

[0149] a third calculation sub-unit, configured to calculate the second loss value by using the first sub-loss value and the second sub-loss value.

[0150] In a possible implementation, the third training unit 303 in the device for compressing a speech recognition model provided in the embodiments of the present application specifically includes:

[0151] a construction unit, configured to construct a search space of the second speech recognition model by using variable parameters of a Transformer module;

[0152] a fifth result unit, configured to input the labeled speech data set into the second speech recognition model, to perform forward propagation calculation on a plurality of sub-speech recognition models randomly sampled from the search space, to obtain a fifth output result;

[0153] a fourth calculation unit, configured to calculate a third loss value by using the fifth output result and a label corresponding to the labeled speech data set;

[0154] A third iterative training unit is configured to iteratively train the second speech recognition model with the third loss value as a third training target until the second speech recognition model converges, and obtain the third speech recognition model.

[0155] In a possible implementation, in the device for compressing a speech recognition model, the variable parameters of the Transformer module include at least one of the following: an input / output dimension of a Transformer layer, a number of attention heads in a multi-head attention mechanism of the Transformer layer, a dimension of a feedforward neural network in the Transformer layer, a ratio of the input / output dimension of the Transformer layer, and a number of Transformer layers in the Transformer module.

[0156] Further, the embodiment of the present application further provides a device for compressing a speech recognition model, comprising: a processor, a memory, a system bus;

[0157] The processor and the memory are connected through the system bus;

[0158] The memory is configured to store one or more programs, the one or more programs comprising instructions that, when executed by the processor, cause the processor to perform any of the above-mentioned implementation methods for compressing a speech recognition model.

[0159] Further, the embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium storing instructions, when the instructions run on a terminal device, causing the terminal device to perform any of the above-mentioned implementation methods for compressing a speech recognition model.

[0160] Further, the embodiment of the present application further provides a computer program product, when the computer program product runs on a terminal device, causing the terminal device to perform any of the above-mentioned implementation methods for compressing a speech recognition model.

[0161] From the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment methods can be implemented by means of software plus necessary universal hardware platforms. Based on such an understanding, the technical solutions of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) execute the methods described in the various embodiments or some parts of the embodiments of the present application.

[0162] It should be noted that the various embodiments described herein are described with progression of some embodiments as to others, each having different advantages. Certain embodiments that are not specifically described herein are also within the scope and spirit of the claims. Such embodiments should be considered as exemplary and should not be considered as limiting the scope of the claims in any way.

[0163] Finally, it should be noted that the terms "comprises", "comprising", or other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0164] The various embodiments described herein are described with progression of some embodiments as to others, each having different advantages. Certain embodiments that are not specifically described herein are also within the scope and spirit of the claims. Such embodiments should be considered as exemplary and should not be considered limiting the scope of the claims in any way.

[0165] The above description is one non-limiting example of the disclosed embodiments. Many modifications can be made to these embodiments without departing from the spirit of the application. For example, different types of devices can be used in place of the devices described herein. The scope of the present application is not to be construed as limited to the embodiments set forth herein as these embodiments are presented by way of example only. The scope of the present application is set out in the appended claims and encompasses any embodiments within their scope.

Claims

1. A method of compressing a speech recognition model, the method comprising: The method includes: Using an unlabeled speech dataset as the first training data, the original speech recognition model was trained using a knowledge distillation algorithm to obtain the first speech recognition model. Using the unlabeled speech dataset and the labeled speech dataset as the second training data, the first speech recognition model is fine-tuned for the speech recognition task to obtain the second speech recognition model. Using the labeled speech dataset as the third training data, the second speech recognition model is trained using a neural network architecture search algorithm to obtain a third speech recognition model; wherein, the third speech recognition model includes multiple sub-speech recognition models; One sub-speech recognition model is extracted from the plurality of sub-speech recognition models to obtain a compressed speech recognition model; the compressed speech recognition model is used to recognize the speech data to be recognized.

2. The method of claim 1, wherein, The process of using an unlabeled speech dataset as the first training data and training the original speech recognition model using a knowledge distillation algorithm to obtain the first speech recognition model includes: A linear layer is added to the original speech recognition model to obtain a fourth speech recognition model; the linear layer is used to map the output dimension of the fourth speech recognition model to be consistent with the output dimension of the original speech recognition model. The original speech recognition model is used as the teacher model for the knowledge distillation algorithm, and the fourth speech recognition model is used as the student model for the knowledge distillation algorithm. The teacher model guides the student model to be trained to obtain the first speech recognition model. The original speech recognition model does not mask the unlabeled speech dataset, while the fourth speech recognition model masks the unlabeled speech dataset.

3. The method according to claim 2, characterized in that, The process of using the original speech recognition model as the teacher model for the knowledge distillation algorithm and the fourth speech recognition model as the student model for the knowledge distillation algorithm, with the teacher model guiding the student model in training to obtain the first speech recognition model, includes: The unlabeled speech dataset is input into the original speech recognition model for training, and the output results of each Transformer layer in the Transformer module of the original speech recognition model are obtained. The first output result is obtained by averaging the output results of the first K Transformer layers. The unlabeled speech dataset is input into the fourth speech recognition model for training to obtain a second output result; the second output result is the output result of the Transformer module of the fourth speech recognition model; The first loss value is calculated using the first output result and the second output result; Minimizing the first loss value is used as the first training objective. The fourth speech recognition model is iteratively trained until the fourth speech recognition model converges, thus obtaining the first speech recognition model.

4. The method according to claim 1, characterized in that, The step of using the unlabeled speech dataset and the labeled speech dataset as second training data to fine-tune the first speech recognition model for the speech recognition task, thereby obtaining a second speech recognition model, includes: By adding a linear mapping module to the first speech recognition model, a fifth speech recognition model is obtained; The labeled speech dataset is input into the fifth speech recognition model for fine-tuning of the speech recognition task, resulting in the third output of the linear mapping module; The unlabeled speech dataset is input into the fifth speech recognition model for fine-tuning of the speech recognition task, resulting in the fourth output of the linear mapping module; The second loss value is calculated using the third and fourth output results; Minimizing the second loss value is used as the second training objective. The fifth speech recognition model is iteratively trained until the fifth speech recognition model converges, thus obtaining the second speech recognition model.

5. The method according to claim 4, characterized in that, The calculation of the second loss value using the third and fourth output results includes: The first sub-loss value is calculated using the third output result and the labels corresponding to the labeled speech dataset. The second sub-loss value is calculated using the fourth output result and the pseudo-labels corresponding to the unlabeled speech dataset; wherein the pseudo-labels are obtained by decoding the unlabeled speech dataset. The second loss value is calculated using the first sub-loss value and the second sub-loss value.

6. The method according to claim 1, characterized in that, The step of using the labeled speech dataset as the third training data and training the second speech recognition model using a neural network architecture search algorithm to obtain the third speech recognition model includes: The search space of the second speech recognition model is constructed using the variable parameters of the Transformer module; The labeled speech dataset is input into the second speech recognition model, and multiple sub-speech recognition models are randomly sampled from the search space for forward propagation calculation to obtain the fifth output result; The third loss value is calculated using the fifth output result and the labels corresponding to the labeled speech dataset. Minimizing the third loss value is used as the third training objective. The second speech recognition model is iteratively trained until the second speech recognition model converges, thus obtaining the third speech recognition model.

7. The method according to claim 6, characterized in that, The variable parameters of the Transformer module include at least one of the following: the input and output dimensions of the Transformer layer, the number of attention heads in the multi-head attention mechanism of the Transformer layer, the ratio of the dimension of the feedforward neural network in the Transformer layer to the input and output dimensions of the Transformer layer, and the number of Transformer layers in the Transformer module.

8. An apparatus for compressing a speech recognition model, characterized in that, The device includes: The first training unit is used to train the original speech recognition model using an unlabeled speech dataset as the first training data and a knowledge distillation algorithm to obtain the first speech recognition model. The second training unit uses the unlabeled speech dataset and the labeled speech dataset as the second training data to fine-tune the first speech recognition model for the speech recognition task, thereby obtaining the second speech recognition model. The third training unit is used to train the second speech recognition model using the labeled speech dataset as the third training data and a neural network structure search algorithm to obtain a third speech recognition model; wherein the third speech recognition model includes multiple sub-speech recognition models. An extraction unit is used to extract one sub-speech recognition model from the plurality of sub-speech recognition models to obtain a compressed speech recognition model; the compressed speech recognition model is used to recognize the speech data to be recognized.

9. An apparatus for compressing a speech recognition model, characterized in that, The device includes: a processor, a memory, and a system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, the one or more programs including instructions that, when executed by the processor, cause the processor to perform the method for compressing a speech recognition model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to perform the method for compressing a speech recognition model as described in any one of claims 1 to 7.

Images