Model training method and device, storage medium and electronic device

Abstract

The application provides a model training method and device, a storage medium and an electronic device. The method comprises: obtaining training data and a first speech recognition model; performing feature masking on the training data to obtain feature-masked training data; calling the first speech recognition model to perform category prediction on the training data to obtain reference category labels of each speech feature in the training data; calling the first speech recognition model to perform category prediction on the feature-masked training data to obtain predicted category labels of each masked feature in the feature-masked training data; calculating a model loss value by using the difference between the predicted category labels of each masked feature and the reference category labels of the corresponding speech feature of the masked feature; and optimizing the model parameters in the first speech recognition model in the direction of reducing the model loss value to determine a second speech recognition model. The embodiments of the application can reduce the cost of model training and improve the accuracy of the speech recognition model.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a model training method, apparatus, storage medium, and electronic device. Background Technology

[0002] Currently, speech recognition technology is widely used in devices such as smartphones, smart speakers, and smart TVs. Speech recognition technology converts speech into text. Common speech recognition systems are typically end-to-end speech recognition models based on deep learning, such as CTC (Connectionist Temporal Classification), LAS (Listen Attention and Spell), and RNN-T (Recurrent Neural Network-Transducer). These end-to-end speech recognition models require tens or even hundreds of thousands of speech data points (including audio data and corresponding text data, i.e., labeled text) during training. In practical scenarios, audio data is relatively easy and inexpensive to acquire, but the corresponding text data requires manual annotation by professional annotators, making text data acquisition costly and consequently increasing model training costs. Therefore, reducing model training costs and improving the accuracy of speech recognition models has become a research hotspot. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a model training method, apparatus, storage medium, and electronic device to solve the problem that the annotation cost of audio data corresponding to text data in model training is high, resulting in high model training cost; that is, embodiments of the present invention can reduce the cost of model training and improve the accuracy of speech recognition models, i.e., embodiments of the present invention can reduce the cost of model training while ensuring the accuracy of speech recognition models.

[0004] According to one aspect of the present invention, a model training method is provided, the method comprising:

[0005] Acquire training data and acquire a first speech recognition model, wherein the training data includes multiple speech features;

[0006] The training data is feature-masked to obtain feature-masked training data. The feature-masked training data includes at least one mask feature, and a mask feature is the result of the corresponding speech feature being masked.

[0007] The first speech recognition model is invoked to perform category prediction on the training data to obtain the reference category label of each speech feature among the multiple speech features;

[0008] The first speech recognition model is invoked to perform category prediction on the training data after the feature masking, so as to obtain the predicted category label of each mask feature in the at least one mask feature;

[0009] The model loss value of the first speech recognition model is calculated by using the difference between the predicted category label of each mask feature and the reference category label of the corresponding speech feature. The model parameters in the first speech recognition model are optimized in the direction of reducing the model loss value. Based on the optimized first speech recognition model, a second speech recognition model is determined. The second speech recognition model is used for speech recognition.

[0010] According to another aspect of the present invention, a model training apparatus is provided, the apparatus comprising:

[0011] An acquisition unit is used to acquire training data and a first speech recognition model, wherein the training data includes multiple speech features;

[0012] The processing unit is configured to perform feature masking on the training data to obtain feature-masked training data, wherein the feature-masked training data includes at least one masking feature, and a masking feature is the result of a corresponding speech feature being masked.

[0013] The processing unit is further configured to invoke the first speech recognition model to perform category prediction on the training data and obtain reference category labels for each speech feature among the plurality of speech features;

[0014] The processing unit is further configured to call the first speech recognition model to perform category prediction on the training data after the feature masking, and obtain the predicted category label of each mask feature in the at least one mask feature.

[0015] The processing unit is further configured to calculate the model loss value of the first speech recognition model by using the difference between the predicted category label of each mask feature and the reference category label of the corresponding speech feature, and optimize the model parameters in the first speech recognition model in the direction of reducing the model loss value, so as to determine the second speech recognition model based on the optimized first speech recognition model, and the second speech recognition model is used for speech recognition.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device including a processor and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to perform the methods mentioned above.

[0017] According to another aspect of the present invention, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the methods mentioned above.

[0018] In this embodiment of the invention, after obtaining training data and a first speech recognition model, feature masking is performed on the training data to obtain feature-masked training data. The training data includes multiple speech features, and the feature-masked training data includes at least one mask feature. A mask feature is the result of masking a corresponding speech feature, facilitating subsequent model training of the first speech recognition model using the training data and the feature-masked training data. Then, the first speech recognition model can be invoked to predict the category of the training data, obtaining reference category labels for each of the multiple speech features. The first speech recognition model can also be invoked to predict the category of the feature-masked training data, obtaining predicted category labels for each of the at least one mask feature. Based on this, the difference between the predicted category labels of each mask feature and the reference category labels of the corresponding speech features can be used to calculate the model loss value of the first speech recognition model. The model parameters in the first speech recognition model are then optimized in the direction of reducing the model loss value. Based on the optimized first speech recognition model, a second speech recognition model is determined, which is then used for speech recognition. As can be seen, the embodiments of the present invention can use the reference category labels predicted by the first speech recognition model as the text data corresponding to the training data (i.e., the text data corresponding to the audio data used to obtain the training data) to train the model based on each reference category label. This eliminates the need for manual annotation by annotators, allowing for pre-training of the first speech recognition model using a large amount of unlabeled, low-cost data. This effectively reduces the cost of model training and improves the accuracy of the speech recognition model. In other words, it reduces the cost of model training while maintaining the accuracy of the speech recognition model. Furthermore, since the speech recognition model is a neural network model, the embodiments of the present invention can generate reference category labels for unsupervised data through neural network self-clustering, making the predicted reference category labels more accurate, thereby improving the accuracy of the speech recognition model and enabling the speech recognition model to converge faster and better. This gives the embodiments of the present invention good robustness and practical application capabilities. Attached Figure Description

[0019] Further details, features, and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0020] Figure 1 A schematic flowchart of a model training method according to an exemplary embodiment of the present invention is shown;

[0021] Figure 2 A flowchart illustrating another model training method according to an exemplary embodiment of the present invention is shown;

[0022] Figure 3 A flowchart illustrating yet another model training method according to an exemplary embodiment of the present invention is shown;

[0023] Figure 4 A flowchart illustrating another model training method according to an exemplary embodiment of the present invention is shown;

[0024] Figure 5 A schematic block diagram of a model training apparatus according to an exemplary embodiment of the present invention is shown;

[0025] Figure 6 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present invention is shown. Detailed Implementation

[0026] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the invention. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the invention.

[0027] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0028] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0029] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0030] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0031] It should be noted that the execution subject of the model training method provided in this embodiment of the invention can be one or more electronic devices, and this invention does not limit this; wherein, the electronic device can be a terminal (i.e., a client) or a server. Therefore, when the execution subject includes multiple electronic devices, and among the multiple electronic devices includes at least one terminal and at least one server, the model training method provided in this embodiment of the invention can be jointly executed by the terminal and the server. Accordingly, the terminal mentioned herein may include, but is not limited to: smartphones, tablets, laptops, desktop computers, smartwatches, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. The server mentioned herein can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, etc.

[0032] Based on the above description, this embodiment of the invention proposes a model training method, which can be executed by the aforementioned electronic device (terminal or server); or, the model training method can be executed jointly by the terminal and the server. For ease of explanation, the following description will use the execution of the model training method by an electronic device as an example; such as Figure 1As shown, the model training method may include the following steps S101-S105:

[0033] S101, acquire training data and acquire the first speech recognition model. The training data includes multiple speech features.

[0034] The speech features (i.e., speech vectors) can be F-bank (Filter Bank, a type of frequency domain feature) features, MFCC (Mel-frequency cepstral coefficients) features, etc.; this invention does not limit this. It should be noted that the speech recognition model (such as the first speech recognition model and the second speech recognition model, etc.) can be a neural network model; optionally, the speech recognition model can be a Transformer-based (Transformer model, a deep learning model based on a self-attention mechanism) speech recognition model, or a Conformer-based (Convolution-augmented transformer, a transformer model based on a convolutional neural network) speech recognition model, etc.; this invention does not limit this. The Transformer model or Conformer model can be used to output high-level representation features of the speech features; optionally, the speech recognition model can also include, but is not limited to, convolutional neural networks or recurrent neural networks, etc.; this invention does not limit this.

[0035] In this embodiment of the invention, the methods for obtaining training data may include, but are not limited to, the following:

[0036] The first method of acquisition: Electronic devices can obtain the download link of the training data and download the data based on the download link to obtain the training dataset, so that any training data in the training dataset can be used as the aforementioned training data.

[0037] The second method of acquisition: The electronic device can store a training dataset, which includes at least one training data point. The electronic device can then select a training data point from the training dataset and use the selected training data point as the aforementioned training data.

[0038] The third acquisition method: If the speech recognition model includes a speech feature extraction module, the electronic device can acquire speech data (i.e., audio data) and call the speech feature extraction module in the first speech recognition model to extract features from the speech data. This enables speech signal processing of the speech data, thereby extracting continuous speech data into training data containing multiple speech features, thus obtaining the training data. Specifically, the electronic device can extract the speech data into T N-dimensional feature vectors (i.e., T frames of speech features), where T and N are both positive integers. This invention does not limit the specific values ​​of T and N. For example, the electronic device can typically convert 1 second of speech into a 100*80 matrix, where T is 100 and N is 80. Optionally, if the speech recognition model includes a speech feature extraction module, then during the iteration process, the electronic device can update the training data through the speech feature extraction module, that is, update the vector representation of each speech feature, thereby continuously improving the accuracy of the vector representation of the speech features.

[0039] S102, perform feature masking on the training data to obtain feature-masked training data. The feature-masked training data includes at least one masking feature, and a masking feature is the result of the corresponding speech feature being masked.

[0040] In this embodiment of the invention, the electronic device can perform feature masking on the training data based on a specified mask character to obtain the feature-masked training data. In this case, a mask feature may include at least one specified mask character, that is, the electronic device can replace elements in the corresponding speech feature with the specified mask character. The specified mask character can be 0, 1, etc., and the invention does not limit it.

[0041] Optionally, the electronic device can randomly replace at least one speech feature from multiple speech features with a specified mask character in the time dimension. That is, the electronic device can randomly determine at least one speech feature from multiple speech features and perform feature masking on each of the determined at least one speech feature to achieve feature masking of the training data, thereby obtaining the mask feature corresponding to each of the at least one speech feature, i.e., obtaining at least one mask feature. Alternatively, the electronic device can determine a preset mask probability and a preset time length. Then, the electronic device can determine at least one speech feature within the preset time length from multiple audio features and randomly generate a probability. If the generated probability is greater than the preset mask probability, feature masking can be performed on at least one speech feature within the preset time length. If the generated probability is less than the preset mask probability, at least one speech feature within the preset time length can be re-determined from multiple speech features until feature masking of the training data is achieved. In this case, the electronic device uses a certain probability to mask (i.e., cover) all speech features of a fixed time length, and so on. Optionally, the preset mask probability and preset time length can be set based on experience or actual needs; this invention does not limit this.

[0042] It should be understood that each speech feature in the multiple speech features corresponds one-to-one with each feature in the training data after feature masking. That is, one feature in the training data after feature masking corresponds to one speech feature in the training data. In addition, the training data after feature masking also includes at least one unmasked feature. An unmasked feature refers to a speech feature that has not been masked. The sum of the number of features in at least one masked feature and the number of features in at least one unmasked feature is the same as the number of features in the training data.

[0043] For example, suppose the training data includes speech feature 1, speech feature 2, speech feature 3, speech feature 4, and speech feature 5. When performing feature masking on the training data, the electronic device performs feature masking on speech feature 2 and speech feature 3, thereby obtaining the masked features corresponding to speech feature 2 and speech feature 3. In this case, the training data after feature masking may include speech feature 1, the masked features corresponding to speech feature 2, the masked features corresponding to speech feature 3, speech feature 4, and speech feature 5. At this time, at least one masked feature includes the masked features corresponding to speech feature 2 and the masked features corresponding to speech feature 3, and at least one non-masked feature includes speech feature 1, speech feature 4, and speech feature 5.

[0044] S103, invoke the first speech recognition model to perform category prediction on the training data, and obtain the reference category label of each speech feature among multiple speech features.

[0045] It should be noted that the speech recognition model may also include a category prediction module, which may refer to a Softmax (normalized exponential function) classifier. Based on this, when calling the first speech recognition model to predict the category of the training data and obtain reference category labels for each speech feature among multiple speech features, the electronic device can first call the first speech recognition model to extract features from each speech feature among multiple speech features to obtain the high-level representation features of each speech feature. Then, the electronic device can call the category prediction module in the first speech recognition model to perform category prediction on the training data based on the high-level representation features of each speech feature (i.e., perform category prediction on each speech feature), obtaining the reference category labels for each speech feature (the category labels can also be called feature labels), such as... Figure 2 As shown. Among them, Figure 2 The neural network model shown may refer to the high-level representation feature extraction module in a speech recognition model, which is used to extract speech features to output high-level representation features, such as the Transformer model or the Conformer model.

[0046] S104, invoke the first speech recognition model to perform category prediction on the training data after feature masking, and obtain the predicted category label of each mask feature in at least one mask feature.

[0047] Optionally, the electronic device may invoke a first speech recognition model to perform category prediction on the training data after feature masking, and obtain the predicted category label of each feature in the training data after feature masking; then, the electronic device may determine the predicted category label of each mask feature in at least one mask feature from the predicted category labels of each feature in the training data after feature masking.

[0048] S105, using the difference between the predicted category label of each mask feature and the reference category label of the corresponding speech feature, calculate the model loss value of the first speech recognition model, and optimize the model parameters in the first speech recognition model in the direction of reducing the model loss value, so as to determine the second speech recognition model based on the optimized first speech recognition model, and the second speech recognition model is used for speech recognition.

[0049] It should be understood that electronic devices can acquire multiple training data sets, and then use each of these training data sets to train the first speech recognition model. In this case, the optimized first speech recognition model can be obtained by training the first speech recognition model based on multiple training data sets. That is, multiple training data sets can be used for model training in each round. In other words, in a round of model training, all training data sets must be used for model training before a round of model training is completed. For ease of explanation, the following explanation will use a single training data set for model training as an example.

[0050] In this embodiment of the invention, after obtaining training data and a first speech recognition model, feature masking is performed on the training data to obtain feature-masked training data. The training data includes multiple speech features, and the feature-masked training data includes at least one mask feature. A mask feature is the result of masking a corresponding speech feature, facilitating subsequent model training of the first speech recognition model using the training data and the feature-masked training data. Then, the first speech recognition model can be invoked to predict the category of the training data, obtaining reference category labels for each of the multiple speech features. The first speech recognition model can also be invoked to predict the category of the feature-masked training data, obtaining predicted category labels for each of the at least one mask feature. Based on this, the difference between the predicted category labels of each mask feature and the reference category labels of the corresponding speech features can be used to calculate the model loss value of the first speech recognition model. The model parameters in the first speech recognition model are then optimized in the direction of reducing the model loss value. Based on the optimized first speech recognition model, a second speech recognition model is determined, which is then used for speech recognition. As can be seen, the embodiments of the present invention can use the reference category labels predicted by the first speech recognition model as the text data corresponding to the training data (i.e., the text data corresponding to the audio data used to obtain the training data) to train the model based on each reference category label. This eliminates the need for manual annotation by annotators, allowing for pre-training of the first speech recognition model using a large amount of unlabeled, low-cost data. This effectively reduces the cost of model training and improves the accuracy of the speech recognition model. In other words, it reduces the cost of model training while maintaining the accuracy of the speech recognition model. Furthermore, since the speech recognition model is a neural network model, the embodiments of the present invention can generate reference category labels for unsupervised data through neural network self-clustering, making the predicted reference category labels more accurate, thereby improving the accuracy of the speech recognition model and enabling the speech recognition model to converge faster and better. This gives the embodiments of the present invention good robustness and practical application capabilities.

[0051] Based on the above description, this embodiment of the invention also proposes a more specific model training method. Accordingly, this model training method can be executed by the aforementioned electronic device (terminal or server); or, the model training method can be executed jointly by the terminal and the server. For ease of explanation, the following description will use the execution of this model training method by an electronic device as an example; please refer to [link to previous text]. Figure 3 The model training method may include the following steps S301-S307:

[0052] S301, acquire training data and acquire the first speech recognition model. The training data includes multiple speech features.

[0053] Specifically, when acquiring the first speech recognition model, the electronic device can acquire an initial speech recognition model, an initial category prediction model, and initial training data. The initial training data includes H speech features, where H is a positive integer. Each model parameter in at least one of the model parameters included in the initial speech recognition model is randomly initialized. Optionally, the at least one model parameter may include, but is not limited to: all model parameters in the high-level representation feature extraction module of the initial speech recognition model, all model parameters in the speech feature extraction module of the initial speech recognition model, and all model parameters in the category prediction module of the initial speech recognition model, etc.; this invention does not limit this. It should be understood that the model parameters in the initial speech recognition model, other than the at least one model parameter, can be obtained through pre-training or set based on experience; this invention does not limit this. It should be noted that the initial training data can be the same as or different from the above-mentioned training data; this invention does not limit this. Optionally, the initial category prediction model can be a BEST-RQ (BERT-based Speech pre-Training with Random-projectionQuantizer) model, a K-means-based deep clustering model, a spectral clustering-based deep clustering model, etc.; this invention does not limit this. Here, BERT is an abbreviation for Bidirectional Encoder Representation from Transformers, a bidirectional encoder representation based on Transformers.

[0054] Then, the electronic device can perform feature masking on the initial training data to obtain feature-masked initial training data. The feature-masked initial training data includes M masked features, where M is a positive integer and M is less than H. Based on this, the electronic device can call the initial category prediction model to predict the category of the initial training data, obtaining a reference category label for each of the H speech features. It can also call the initial speech recognition model to predict the category of the feature-masked initial training data, obtaining a predicted category label for each of the M masked features. Further, the electronic device can use the difference between the predicted category label of each of the M masked features and the reference category label of the corresponding speech feature to calculate the initial model loss value of the initial speech recognition model. Then, it can optimize the model parameters in the initial speech recognition model in the direction of reducing the initial model loss value, so as to obtain a first speech recognition model based on the optimized initial speech recognition model.

[0055] Similarly, there can be multiple initial training data sets, so electronic devices can use multiple initial training data sets to train the initial speech recognition model; for ease of explanation, the following will use one initial training data set as an example.

[0056] Furthermore, the electronic device can optimize the model parameters in the initial category prediction model in the direction of reducing the initial model loss value, thereby obtaining an optimized initial category prediction model. This improves the model performance of the category prediction model and increases the accuracy of the reference category label predicted by the category prediction model, i.e., improves the accuracy of the category prediction model. Based on this, when obtaining the first speech recognition model based on the optimized initial speech recognition model, the electronic device can obtain the first speech recognition model based on the optimized initial speech recognition model and the optimized initial category prediction model. Specifically, the electronic device can continue to train the optimized initial speech recognition model based on the optimized initial category prediction model and the initial training data until the first speech convergence condition is met, and the speech recognition model that meets the first speech convergence condition is taken as the first speech recognition model. Optionally, the first speech convergence condition may refer to the number of training iterations (i.e., the number of training rounds) reaching a first speech training iteration threshold, or it may refer to the difference between the loss values ​​of two consecutive iterations being less than a first speech loss difference, etc. This invention does not limit this. Optionally, the first speech training iteration threshold and the first speech loss difference may be set empirically or according to actual needs, and this invention does not limit this. The case where the difference between the loss values ​​of two consecutive iterations is less than one speech loss difference can also be referred to as the case where the model stops converging.

[0057] In summary, the embodiments of the present invention can predict reference category labels for speech features through a category prediction model, thereby enabling model training of the initial speech recognition model based on the reference category labels. This can effectively prevent the speech recognition model from getting stuck in local optima, accelerate the convergence speed, and improve the accuracy of the speech recognition model, that is, improve the accuracy of the category labels predicted by the speech recognition model.

[0058] S302, perform feature masking on the training data to obtain feature-masked training data. The feature-masked training data includes at least one masking feature, and a masking feature is the result of the corresponding speech feature being masked.

[0059] S303, invoke the first speech recognition model to perform category prediction on the training data, and obtain the category probability of each speech feature among multiple speech features. The category probability of a speech feature includes the probability value of the corresponding speech feature belonging to each of the multiple categories.

[0060] S304, smooth the category probabilities of each speech feature to obtain the smoothed category probabilities of each speech feature.

[0061] In this embodiment of the invention, for any one of multiple speech features, the electronic device can determine Q neighboring speech features of that speech feature from the multiple speech features. The distance between any one of the Q neighboring speech features and any other speech feature is less than the distance between any other speech feature (excluding the Q neighboring speech features) and any other speech feature, where Q is a positive integer. Then, the electronic device can perform a weighted summation of the class probabilities of each of the Q neighboring speech features to achieve a smoothing of the class probability of any speech feature, obtaining a smoothed class probability of any speech feature. The distance between two speech features can refer to the distance between the two speech features in the time dimension. For example, assuming that the multiple speech features sequentially include speech feature 1, speech feature 2, speech feature 3, speech feature 4, and speech feature 5 in the time dimension, then the distance between speech feature 1 and speech feature 2 can be 1, the distance between speech feature 1 and speech feature 3 can be 2, and the distance between speech feature 1 and speech feature 5 can be 4, and so on. It should be noted that when performing a weighted summation of the class probabilities of each of the Q adjacent speech features, the electronic device can determine the weight corresponding to each adjacent speech feature and perform a weighted summation of the class probabilities of each adjacent speech feature according to the weight corresponding to each adjacent speech feature. Optionally, the weights corresponding to each adjacent speech feature can be the same (i.e., the mean operation) or different (e.g., the weights corresponding to each adjacent speech feature are randomly generated, or the adjacent speech feature with a smaller distance from any speech feature has a larger weight, etc.). This invention does not limit this. For example, taking the case where the weights corresponding to each adjacent speech feature are the same, assuming that the class probabilities of each adjacent speech feature of any speech feature are (0.6, 0.2, 0.2), (0.7, 0.2, 0.1), and (0.7, 0.1, 0.2), then the smoothed class probability of any speech feature can be (0.67, 0.17, 0.17), and so on.

[0062] In one specific implementation, the electronic device can determine a preset distance threshold and, based on the preset distance threshold, determine Q neighboring speech features of any speech feature from multiple speech features. In this case, the electronic device can use speech features among the multiple speech features whose distance to any speech feature is less than or equal to the preset distance threshold as neighboring speech features of any speech feature, thereby determining Q neighboring speech features of any speech feature. That is, the distance between any neighboring speech feature of any speech feature and any speech feature is less than or equal to the preset distance threshold. Optionally, the preset distance threshold can be 1 or 2, etc., and the present invention does not limit this.

[0063] For example, suppose multiple speech features sequentially include speech feature 1, speech feature 2, speech feature 3, speech feature 4 and speech feature 5 in the time dimension, where any one speech feature is speech feature 3, and the preset distance threshold is 1. In this case, the distances between speech features 1-5 and any one speech feature are 2, 1, 0, 1 and 2, respectively. That is, the distances between speech features 2, speech features 3 and speech features 4 and any one speech feature are less than or equal to the preset distance threshold. Then, the electronic device can use speech features 2, speech features 3 and speech features 4 as adjacent speech features of any one speech feature. That is, the Q adjacent speech features of any one speech feature can include speech features 2, speech features 3 and speech features 4.

[0064] In another embodiment, the electronic device can determine a preset quantity threshold (i.e., the value of Q). Based on this preset quantity threshold, the electronic device can determine Q neighboring speech features of any given speech feature from multiple speech features. In this case, the electronic device can use the first preset quantity threshold (i.e., the first Q) speech features closest to any given speech feature as its neighboring speech features, thereby determining the Q neighboring speech features of any given speech feature. Optionally, the preset quantity threshold can be set empirically or according to actual needs; this invention does not limit this.

[0065] For example, assuming a preset threshold of 3, multiple speech features sequentially include speech feature 1, speech feature 2, speech feature 3, speech feature 4, and speech feature 5 in the time dimension; in this case, when any speech feature is speech feature 1, the three speech features closest to any speech feature are speech feature 1, speech feature 2, and speech feature 3, then the electronic device can use speech feature 1, speech feature 2, and speech feature 3 as adjacent speech features of any speech feature; when any speech feature is speech feature 4, the three speech features closest to any speech feature are speech feature 3, speech feature 4, and speech feature 5, then the electronic device can use speech feature 3, speech feature 4, and speech feature 5 as adjacent speech features of any speech feature.

[0066] S305, based on the smoothed class probability of each speech feature, determine the reference class label for each speech feature.

[0067] Specifically, for any one of multiple speech features, the electronic device can use the category label indicated by the maximum probability value among the smoothed category probabilities of any speech feature as the reference category label of any speech feature.

[0068] S306, invoke the first speech recognition model to perform category prediction on the training data after feature masking, and obtain the predicted category label of each mask feature in at least one mask feature.

[0069] S307, using the difference between the predicted category label of each mask feature and the reference category label of the corresponding speech feature, calculate the model loss value of the first speech recognition model, and optimize the model parameters in the first speech recognition model in the direction of reducing the model loss value, so as to determine the second speech recognition model based on the optimized first speech recognition model, and the second speech recognition model is used for speech recognition.

[0070] Specifically, determining the specific implementation method of the second speech recognition model based on the optimized first speech recognition model may include steps s11-s14:

[0071] s11, determine the mask training data corresponding to the training data. The mask training data includes P mask features, and the features in the mask training data correspond one-to-one with the features in the training data. P is a positive integer.

[0072] The mask training data can be the same as or different from the training data after feature masking. That is, the electronic device can use the training data after feature masking as the mask training data, or it can perform feature masking on the training data again to obtain the mask training data, etc. The present invention does not limit this.

[0073] s12 calls the first speech recognition model after model optimization, performs category prediction on the mask training data, and obtains the predicted category label of each mask feature in P mask features.

[0074] s13, from the reference category labels of each speech feature, determine the reference category label of each of the P mask features corresponding to the speech feature.

[0075] s14. Using the difference between the predicted category label of each mask feature in P mask features and the reference category label of the corresponding speech feature, calculate the first model loss value of the optimized first speech recognition model. Then, optimize the model parameters in the optimized first speech recognition model in the direction of reducing the first model loss value to obtain the third speech recognition model. Based on the third speech recognition model, determine the second speech recognition model.

[0076] It should be noted that the electronic device can iteratively execute steps s11-s14 to continue training the third speech recognition model. That is, the electronic device can call the third speech recognition model to predict the category of the current mask training data, and train the third speech recognition model based on the predicted category labels of the mask features in the current mask training data until the second speech recognition convergence condition is met. The speech recognition model that meets the second speech recognition condition is then used as the fourth speech recognition model, and the second speech recognition model can be determined based on the fourth speech recognition model. Therefore, in the process of obtaining the fourth speech recognition model based on the first speech recognition model, the reference category labels of each speech feature can remain unchanged, thereby reducing the computational load, improving the convergence speed, and reducing the model training time.

[0077] Optionally, the second speech convergence condition may refer to the number of training iterations reaching a second speech training iteration threshold, or it may refer to the difference between the loss values ​​of two consecutive iterations being less than the second speech loss difference, etc., and this invention does not limit it in this way; Optionally, the second speech training iteration threshold and the second speech loss difference may be set according to experience or according to actual needs, and this invention does not limit it in this way.

[0078] Furthermore, when determining the second speech recognition model based on the fourth speech recognition model, the electronic device can call the fourth speech recognition model to predict the category of the training data, thereby updating the reference category labels of each speech feature. That is, the electronic device can regenerate the reference category labels of each speech feature through the fourth speech recognition model, and then continue training the fourth speech recognition model based on the updated reference category labels of each speech feature until the third speech convergence condition is met. The speech recognition model that meets the third speech convergence condition is then used as the pre-trained speech recognition model, and the second speech recognition model is determined based on the pre-trained speech recognition model. It should be understood that during the process of obtaining the pre-trained speech recognition model, each time the second speech convergence condition is met, the electronic device can call the speech recognition model at the current system time to update the reference category labels of each speech feature, so as to continue model training until the third speech convergence condition is met. For example, as shown... Figure 4As shown, whenever step 2 is executed to achieve the second speech convergence condition (e.g., after a threshold number of iterations of the second speech training or when the difference between the loss values ​​of two consecutive iterations is less than the second speech loss difference), the electronic device can use the model trained in step 2 as the model for label generation in step 1 to regenerate the reference category labels for each speech feature. Then, it continues to train the speech recognition model (i.e., continues to execute step 2). In other words, the electronic device can repeat steps 1 and 2 until the third speech convergence condition is met. In this embodiment of the invention, the intermediately trained speech recognition model (i.e., the speech recognition model that meets the second speech convergence condition) can be used to update the reference category labels, which can effectively improve the clustering effect, that is, improve the accuracy of the reference category labels, thereby accelerating the convergence speed of the speech recognition model.

[0079] Optionally, the third speech convergence condition may refer to the number of training iterations reaching a third speech training iteration threshold, or it may refer to the difference between the loss values ​​of two consecutive iterations being less than the third speech loss difference, etc., and this invention does not limit this; optionally, the third speech training iteration threshold and the third speech loss difference may be set according to experience or according to actual needs, and this invention does not limit this.

[0080] Accordingly, when determining the second speech recognition model based on the optimized first speech recognition model, the electronic device can acquire target training data, which includes audio data and corresponding text data. Based on the optimized first speech recognition model, it determines a pre-trained speech recognition model (a speech recognition model that has undergone unsupervised training). The device then calls the pre-trained speech recognition model to perform speech recognition on the audio data, obtaining the speech recognition result. Next, it calculates the second model loss value of the pre-trained speech recognition model using the difference between the speech recognition result and the text data. Following the direction of reducing the second model loss value, it optimizes the model parameters in the pre-trained speech recognition model to determine the second speech recognition model based on the optimized pre-trained speech recognition model. Therefore, the electronic device can first determine the pre-trained speech recognition model based on the optimized first speech recognition model, and then continue training the pre-trained speech recognition model to obtain the optimized pre-trained speech recognition model. Finally, it determines the second speech recognition model based on the optimized pre-trained speech recognition model. Figure 4 As shown, the recognition model training in step 4 refers to the supervised training of the pre-trained speech recognition model to obtain the second speech recognition model. It should be noted that the target training data is supervised training data, and the process of training the pre-trained speech recognition model using the target training data is a supervised training process.

[0081] In this embodiment of the invention, after acquiring training data and a first speech recognition model, feature masking is performed on the training data to obtain feature-masked training data. The training data includes multiple speech features, and the feature-masked training data includes at least one mask feature. A mask feature is the result of a corresponding speech feature being masked. Then, the first speech recognition model can be invoked to predict the category of the training data, obtaining the category probability of each speech feature among the multiple speech features. The category probability of a speech feature includes the probability value of the corresponding speech feature belonging to each of the multiple categories. The category probabilities of each speech feature are then smoothed to obtain the smoothed category probabilities of each speech feature. Based on the smoothed category probabilities of each speech feature, the reference category label of each speech feature is determined. Furthermore, a first speech recognition model can be invoked to predict the category of the training data after feature masking, obtaining predicted category labels for each mask feature in at least one mask feature. Then, the difference between the predicted category labels of each mask feature and the reference category labels of the corresponding speech features is used to calculate the model loss value of the first speech recognition model. The model parameters in the first speech recognition model are optimized in the direction of reducing the model loss value. Based on the optimized first speech recognition model, a second speech recognition model is determined, which is then used for speech recognition. It can be seen that this embodiment of the invention can generate reference category labels for the unsupervised training process through self-clustering of the speech recognition model, and has better clustering effects, making the reference category labels more accurate. This allows the unsupervised training speech recognition model to converge faster and better, enabling better supervised training and exhibiting good robustness and practical application capabilities. In addition, during the generation of reference category labels, a time-dimensional smoothing process (i.e., smoothing operation) is performed, making the speech feature categories that are temporally adjacent similar, which is more consistent with the characteristics of speech.

[0082] Based on the description of the relevant embodiments of the above model training method, this invention also proposes a model training device, which can be a computer program (including program code) running in an electronic device; such as Figure 5 As shown, the model training apparatus may include an acquisition unit 501 and a processing unit 502. The model training apparatus can perform... Figure 1 or Figure 3 The model training method shown, i.e., the model training device can run the above-mentioned unit:

[0083] The acquisition unit 501 is used to acquire training data and acquire a first speech recognition model, wherein the training data includes multiple speech features;

[0084] The processing unit 502 is used to perform feature masking on the training data to obtain feature-masked training data. The feature-masked training data includes at least one masking feature, and a masking feature is the result of a corresponding speech feature being masked.

[0085] The processing unit 502 is further configured to call the first speech recognition model to perform category prediction on the training data and obtain reference category labels for each speech feature among the plurality of speech features;

[0086] The processing unit 502 is further configured to call the first speech recognition model to perform category prediction on the training data after the feature mask, and obtain the predicted category label of each mask feature in the at least one mask feature.

[0087] The processing unit 502 is further configured to calculate the model loss value of the first speech recognition model by using the difference between the predicted category label of each mask feature and the reference category label of the corresponding speech feature, and optimize the model parameters in the first speech recognition model in the direction of reducing the model loss value, so as to determine the second speech recognition model based on the optimized first speech recognition model, and the second speech recognition model is used for speech recognition.

[0088] In one embodiment, when acquiring the first speech recognition model, the acquisition unit 501 may specifically be used for:

[0089] Obtain an initial speech recognition model, an initial category prediction model, and initial training data, wherein the initial training data includes H speech features, where H is a positive integer;

[0090] The initial training data is feature-masked to obtain the feature-masked initial training data, which includes M mask features, where M is a positive integer.

[0091] The initial category prediction model is invoked to predict the category of the initial training data, thereby obtaining the reference category label for each of the H speech features;

[0092] The initial speech recognition model is invoked to perform category prediction on the initial training data after feature masking, thereby obtaining the predicted category label of each of the M masked features;

[0093] The initial model loss value of the initial speech recognition model is calculated by using the difference between the predicted category label of each mask feature in the M mask features and the reference category label of the corresponding speech feature. The model parameters in the initial speech recognition model are optimized in the direction of reducing the initial model loss value, so as to obtain the first speech recognition model based on the optimized initial speech recognition model.

[0094] In another embodiment, the processing unit 502 can also be used for:

[0095] Optimize the model parameters in the initial category prediction model in the direction of reducing the loss value of the initial model to obtain the optimized initial category prediction model;

[0096] When acquiring the first speech recognition model based on the optimized initial speech recognition model, the acquisition unit 501 can be specifically used for:

[0097] Based on the optimized initial speech recognition model and the optimized initial category prediction model, a first speech recognition model is obtained.

[0098] In another embodiment, when the processing unit 502 invokes the first speech recognition model to perform category prediction on the training data and obtain reference category labels for each speech feature among the plurality of speech features, it may specifically be used for:

[0099] The first speech recognition model is invoked to perform category prediction on the training data to obtain the category probability of each speech feature among the multiple speech features. The category probability of a speech feature includes the probability value of the corresponding speech feature belonging to each of the multiple categories.

[0100] The category probabilities of each speech feature are smoothed to obtain the smoothed category probabilities of each speech feature.

[0101] Based on the smoothed class probability of each speech feature, a reference class label is determined for each speech feature.

[0102] In another embodiment, when the processing unit 502 smooths the category probabilities of each speech feature to obtain the smoothed category probabilities of each speech feature, it may specifically be used to:

[0103] For any one of the plurality of speech features, Q neighboring speech features are determined from the plurality of speech features. The distance between any one of the Q neighboring speech features and the any one speech feature is less than the distance between any other speech feature (excluding the Q neighboring speech features) and the any one speech feature. Q is a positive integer.

[0104] The class probabilities of each of the Q adjacent speech features are weighted and summed to achieve a smoothed class probability of any speech feature, thus obtaining a smoothed class probability of any speech feature.

[0105] In another implementation, when the processing unit 502 determines the second speech recognition model based on the optimized first speech recognition model, it may specifically be used to:

[0106] Determine the mask training data corresponding to the training data, wherein the mask training data includes P mask features, and the features in the mask training data correspond one-to-one with the features in the training data, where P is a positive integer;

[0107] The optimized first speech recognition model is invoked to perform category prediction on the mask training data, thereby obtaining the predicted category label of each of the P mask features.

[0108] From the reference category labels of each of the P mask features, determine the reference category label of the speech feature corresponding to each mask feature;

[0109] The difference between the predicted category label of each of the P mask features and the reference category label of the corresponding speech feature is used to calculate the first model loss value of the optimized first speech recognition model. The model parameters in the optimized first speech recognition model are optimized in the direction of reducing the first model loss value to obtain the third speech recognition model. Based on the third speech recognition model, the second speech recognition model is determined.

[0110] In another implementation, when the processing unit 502 determines the second speech recognition model based on the optimized first speech recognition model, it may specifically be used to:

[0111] Acquire target training data, which includes audio data and text data corresponding to the audio data;

[0112] Based on the first speech recognition model after model optimization, a pre-trained speech recognition model is determined, wherein the pre-trained speech recognition model is a speech recognition model that has completed unsupervised training.

[0113] The pre-trained speech recognition model is invoked to perform speech recognition on the audio data, and the speech recognition result is obtained.

[0114] The difference between the speech recognition result and the text data is used to calculate the second model loss value of the pre-trained speech recognition model. The model parameters in the pre-trained speech recognition model are optimized in the direction of reducing the second model loss value. Based on the optimized pre-trained speech recognition model, the second speech recognition model is determined.

[0115] According to one embodiment of the present invention, Figure 1 or Figure 3 Each step involved in the method shown can be derived from... Figure 5 The training is performed by individual units in the model training device shown. For example, Figure 1 Step S101 shown can be performed by Figure 5 The acquisition unit 501 shown is executed, and steps S102-S105 can all be performed by... Figure 5 The processing unit 502 shown executes this. For example, Figure 3 Step S301 shown can be performed by Figure 5 The acquisition unit 501 shown is executed, and steps S302-S307 can all be performed by... Figure 5 The processing unit 502 shown executes, etc.

[0116] According to another embodiment of the present invention, Figure 5 Each unit in the model training device shown can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effect of the embodiments of the present invention. The above units are based on logical function division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of the present invention, any model training device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0117] According to another embodiment of the present invention, the following can be performed by running on a general-purpose electronic device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). Figure 1 or Figure 3 The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 5 The diagram illustrates a model training apparatus and a model training method for implementing embodiments of the present invention. The computer program may be stored on, for example, a computer storage medium, loaded onto the aforementioned electronic device via the computer storage medium, and run therein.

[0118] In this embodiment of the invention, after obtaining training data and a first speech recognition model, feature masking is performed on the training data to obtain feature-masked training data. The training data includes multiple speech features, and the feature-masked training data includes at least one mask feature. A mask feature is the result of masking a corresponding speech feature, facilitating subsequent model training of the first speech recognition model using the training data and the feature-masked training data. Then, the first speech recognition model can be invoked to predict the category of the training data, obtaining reference category labels for each of the multiple speech features. The first speech recognition model can also be invoked to predict the category of the feature-masked training data, obtaining predicted category labels for each of the at least one mask feature. Based on this, the difference between the predicted category labels of each mask feature and the reference category labels of the corresponding speech features can be used to calculate the model loss value of the first speech recognition model. The model parameters in the first speech recognition model are then optimized in the direction of reducing the model loss value. Based on the optimized first speech recognition model, a second speech recognition model is determined, which is then used for speech recognition. As can be seen, the embodiments of the present invention can use the reference category labels predicted by the first speech recognition model as the text data corresponding to the training data (i.e., the text data corresponding to the audio data used to obtain the training data) to train the model based on each reference category label. This eliminates the need for manual annotation by annotators, allowing for pre-training of the first speech recognition model using a large amount of unlabeled, low-cost data. This effectively reduces the cost of model training and improves the accuracy of the speech recognition model. In other words, it reduces the cost of model training while maintaining the accuracy of the speech recognition model. Furthermore, since the speech recognition model is a neural network model, the embodiments of the present invention can generate reference category labels for unsupervised data through neural network self-clustering, making the predicted reference category labels more accurate, thereby improving the accuracy of the speech recognition model and enabling the speech recognition model to converge faster and better. This gives the embodiments of the present invention good robustness and practical application capabilities.

[0119] Based on the description of the method and apparatus embodiments above, an exemplary embodiment of the present invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, causes the electronic device to perform a method according to an embodiment of the present invention.

[0120] An exemplary embodiment of the present invention also provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0121] An exemplary embodiment of the present invention also provides a computer program product, including a computer program, wherein, when executed by a computer's processor, the computer program is used to cause the computer to perform a method according to an embodiment of the present invention.

[0122] refer to Figure 6 The present invention will now be described in the form of a structural block diagram of an electronic device 600 that can serve as a server or client of the present invention, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0123] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0124] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, output unit 607, storage unit 608, and communication unit 609. Input unit 606 can be any type of device capable of inputting information to electronic device 600. Input unit 606 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 607 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 608 may include, but is not limited to, disks and optical discs. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0125] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above. For example, in some embodiments, the model training method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 600 via ROM 602 and / or communication unit 609. In some embodiments, the computing unit 601 can be configured to perform the model training method by any other suitable means (e.g., by means of firmware).

[0126] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to the processor or controller of a general-purpose computer, special-purpose computer, or other programmable model training device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0127] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0128] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0129] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0130] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0131] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0132] Furthermore, it should be understood that the above-disclosed embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, any equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A model training method, characterized in that, include: Acquire training data and acquire a first speech recognition model, wherein the training data includes multiple speech features; The training data is feature-masked to obtain feature-masked training data. The feature-masked training data includes at least one mask feature, and a mask feature is the result of the corresponding speech feature being masked. The first speech recognition model is invoked to perform category prediction on the training data to obtain the reference category label of each speech feature among the multiple speech features; The first speech recognition model is invoked to perform category prediction on the training data after the feature masking, so as to obtain the predicted category label of each mask feature in the at least one mask feature; The model loss value of the first speech recognition model is calculated by using the difference between the predicted category label of each mask feature and the reference category label of the corresponding speech feature. The model parameters in the first speech recognition model are optimized in the direction of reducing the model loss value. Based on the optimized first speech recognition model, a second speech recognition model is determined. The second speech recognition model is used for speech recognition.

2. The method according to claim 1, characterized in that, The process of obtaining the first speech recognition model includes: Obtain an initial speech recognition model, an initial category prediction model, and initial training data, wherein the initial training data includes H speech features, where H is a positive integer; The initial training data is feature-masked to obtain the feature-masked initial training data, which includes M mask features, where M is a positive integer. The initial category prediction model is invoked to predict the category of the initial training data, thereby obtaining the reference category label for each of the H speech features; The initial speech recognition model is invoked to perform category prediction on the initial training data after feature masking, thereby obtaining the predicted category label of each of the M masked features; The initial model loss value of the initial speech recognition model is calculated by using the difference between the predicted category label of each mask feature in the M mask features and the reference category label of the corresponding speech feature. The model parameters in the initial speech recognition model are optimized in the direction of reducing the initial model loss value, so as to obtain the first speech recognition model based on the optimized initial speech recognition model.

3. The method according to claim 2, characterized in that, The method further includes: Optimize the model parameters in the initial category prediction model in the direction of reducing the loss value of the initial model to obtain the optimized initial category prediction model; The process of obtaining the first speech recognition model based on the optimized initial speech recognition model includes: Based on the optimized initial speech recognition model and the optimized initial category prediction model, a first speech recognition model is obtained.

4. The method according to any one of claims 1-3, characterized in that, The step of calling the first speech recognition model to perform category prediction on the training data and obtain the reference category label for each of the multiple speech features includes: The first speech recognition model is invoked to perform category prediction on the training data to obtain the category probability of each speech feature among the multiple speech features. The category probability of a speech feature includes the probability value of the corresponding speech feature belonging to each of the multiple categories. The category probabilities of each speech feature are smoothed to obtain the smoothed category probabilities of each speech feature. Based on the smoothed class probability of each speech feature, a reference class label is determined for each speech feature.

5. The method according to claim 4, characterized in that, The step of smoothing the category probabilities of each speech feature to obtain the smoothed category probabilities of each speech feature includes: For any one of the plurality of speech features, Q neighboring speech features are determined from the plurality of speech features. The distance between any one of the Q neighboring speech features and the any one speech feature is less than the distance between any other speech feature (excluding the Q neighboring speech features) and the any one speech feature. Q is a positive integer. The class probabilities of each of the Q adjacent speech features are weighted and summed to achieve a smoothed class probability of any speech feature, thus obtaining a smoothed class probability of any speech feature.

6. The method according to any one of claims 1-3, characterized in that, The process of determining the second speech recognition model based on the optimized first speech recognition model includes: Determine the mask training data corresponding to the training data, wherein the mask training data includes P mask features, and the features in the mask training data correspond one-to-one with the features in the training data, where P is a positive integer; The optimized first speech recognition model is invoked to perform category prediction on the mask training data, thereby obtaining the predicted category label of each of the P mask features. From the reference category labels of each of the P mask features, determine the reference category label of the speech feature corresponding to each mask feature; The difference between the predicted category label of each of the P mask features and the reference category label of the corresponding speech feature is used to calculate the first model loss value of the optimized first speech recognition model. The model parameters in the optimized first speech recognition model are optimized in the direction of reducing the first model loss value to obtain the third speech recognition model. Based on the third speech recognition model, the second speech recognition model is determined.

7. The method according to any one of claims 1-3, characterized in that, The process of determining the second speech recognition model based on the optimized first speech recognition model includes: Acquire target training data, which includes audio data and text data corresponding to the audio data; Based on the first speech recognition model after model optimization, a pre-trained speech recognition model is determined, wherein the pre-trained speech recognition model is a speech recognition model that has completed unsupervised training. The pre-trained speech recognition model is invoked to perform speech recognition on the audio data, and the speech recognition result is obtained. The difference between the speech recognition result and the text data is used to calculate the second model loss value of the pre-trained speech recognition model. The model parameters in the pre-trained speech recognition model are optimized in the direction of reducing the second model loss value. Based on the optimized pre-trained speech recognition model, the second speech recognition model is determined.

8. A model training device, characterized in that, The device includes: An acquisition unit is used to acquire training data and a first speech recognition model, wherein the training data includes multiple speech features; The processing unit is configured to perform feature masking on the training data to obtain feature-masked training data, wherein the feature-masked training data includes at least one masking feature, and a masking feature is the result of a corresponding speech feature being masked. The processing unit is further configured to invoke the first speech recognition model to perform category prediction on the training data and obtain reference category labels for each speech feature among the plurality of speech features; The processing unit is further configured to call the first speech recognition model to perform category prediction on the training data after the feature masking, and obtain the predicted category label of each mask feature in the at least one mask feature. The processing unit is further configured to calculate the model loss value of the first speech recognition model by using the difference between the predicted category label of each mask feature and the reference category label of the corresponding speech feature, and optimize the model parameters in the first speech recognition model in the direction of reducing the model loss value, so as to determine the second speech recognition model based on the optimized first speech recognition model, and the second speech recognition model is used for speech recognition.

9. An electronic device, characterized in that, include: processor; as well as Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

Images