Speech recognition model training method, device, storage medium and electronic device

Abstract

The present disclosure relates to the field of speech recognition, and specifically to a training method, device, storage medium and electronic device for a speech recognition model. The training method of the speech recognition model includes constructing an initial speech recognition model; fixing a second initial parameter, calculating a contrastive learning loss function based on an unlabeled data set to adjust the first initial parameter to a first intermediate parameter; fixing the first intermediate parameter, calculating a first joint loss function based on a labeled data set to adjust the second initial parameter to a second intermediate parameter; calculating a second joint loss function based on a labeled data set, and training the first network and the second network according to the second joint loss function to adjust the first intermediate parameter and the second intermediate parameter to obtain a target speech recognition model. The training method of the speech recognition model provided by the present disclosure can solve the problem of low recognition performance of the speech recognition model when the labeled data is insufficient.

Description

Technical Field

[0001] The present disclosure relates to the field of speech recognition, and in particular to a speech recognition model training method, a speech recognition model training device, a storage medium and an electronic device. Background Art

[0002] In recent years, with the rapid development of deep learning technology, automatic speech recognition (ASR) based on end-to-end deep neural networks has gradually developed into the mainstream technology in the current speech recognition field.

[0003] Since the number of parameters in the end-to-end ASR model is large, the performance of the model often depends on a large amount of annotated data. In addition, the self-supervised ASR method is usually performed under the CTC (Connectionist temporal classification) framework. The CTC framework assumes that the speech feature representation is independent between frames, which is inconsistent with the actual situation and has limited performance. Therefore, it is necessary to further improve the recognition performance of the speech recognition model under the condition of insufficient annotated data.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide a training method for a speech recognition model, a training device for a speech recognition model, a storage medium and an electronic device, aiming to solve the problem of low recognition performance of the speech recognition model when there is insufficient labeled data.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of an embodiment of the present disclosure, a method for training a speech recognition model is provided, comprising: constructing an initial speech recognition model; wherein the initial speech recognition model comprises a first network having a first initial parameter and a second network having a second initial parameter; fixing the second initial parameter, calculating a contrastive learning loss function based on an unlabeled data set, and performing self-supervised training on the first network according to the contrastive learning loss function to adjust the first initial parameter to a first intermediate parameter; fixing the first intermediate parameter, calculating a first joint loss function based on a labeled data set, and training the second network according to the first joint loss function to adjust the second initial parameter to a second intermediate parameter; calculating a second joint loss function based on the labeled data set, and training the first network and the second network according to the second joint loss function to adjust the first intermediate parameter and the second intermediate parameter to obtain a target speech recognition model.

[0008] According to some embodiments of the present disclosure, based on the aforementioned scheme, the first network includes a convolutional neural network module and a convolutional enhancement module.

[0009] According to some embodiments of the present disclosure, based on the aforementioned scheme, the contrastive learning loss function is calculated based on the unlabeled dataset, including: calculating a shallow representation result of an audio sample data in the unlabeled dataset based on the convolutional neural network module; performing masking on the shallow representation result to obtain a masked representation result, and calculating a deep representation result of the masked representation result based on the convolutional enhancement module; and performing a linear transformation on the shallow representation result to obtain a target representation result; and calculating the contrastive learning loss function based on the deep representation result and the target representation result.

[0010] According to some embodiments of the present disclosure, based on the aforementioned scheme, the masking processing of the shallow representation result to obtain the masked representation result includes: randomly selecting a seed sample frame from the shallow representation result based on a random mask probability; replacing the feature vectors of consecutive K frames after the seed sample frame in the shallow representation result with a learnable vector to obtain the masked representation result, where K is a positive integer.

[0011] According to some embodiments of the present disclosure, based on the aforementioned scheme, the contrastive learning loss function is calculated based on the deep representation result and the target representation result, including: selecting M frame anchor samples from the mask part in the deep representation result as the first sample, where M is a positive integer; and selecting M frame anchor samples that correspond one-to-one to the M frame anchor samples in the first sample from the target representation result as the second sample, and selecting S frame negative samples as the third sample, where S is a positive integer; calculating the contrastive learning loss function based on the similarity between the first sample and the second sample and the similarity between the first sample and the third sample.

[0012] According to some embodiments of the present disclosure, based on the aforementioned solution, the second network includes a feature deformation module.

[0013] According to some embodiments of the present disclosure, based on the aforementioned scheme, the method also includes: acquiring audio sample data based on a preset audio sampling rate, and dividing the audio sample data into a first audio sample and a second audio sample; calculating an audio feature matrix of the first audio sample to obtain the unlabeled data set; and obtaining the labeled data set based on the calculated audio feature matrix of the second audio sample and the acquired text labeling result of the second audio sample.

[0014] According to a second aspect of an embodiment of the present disclosure, a training device for a speech recognition model is provided, comprising: a model construction module, used to construct an initial speech recognition model; wherein the initial speech recognition model comprises a first network with a first initial parameter and a second network with a second initial parameter; a first training module, used to fix the second initial parameter, calculate a contrastive learning loss function based on an unlabeled data set, and perform self-supervised training on the first network according to the contrastive learning loss function to adjust the first initial parameter to a first intermediate parameter; a second training module, used to fix the first intermediate parameter, calculate a first joint loss function based on a labeled data set, and train the second network according to the first joint loss function to adjust the second initial parameter to a second intermediate parameter; a model adjustment module, used to calculate a second joint loss function based on the labeled data set, and train the first network and the second network according to the second joint loss function to adjust the first intermediate parameter and the second intermediate parameter to obtain a target speech recognition model.

[0015] According to a third aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the training method of the speech recognition model as in the above embodiment is implemented.

[0016] According to the fourth aspect of the embodiments of the present disclosure, an electronic device is provided, characterized in that it includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the training method of the speech recognition model as in the above-mentioned embodiments.

[0017] The exemplary embodiments of the present disclosure may have some or all of the following beneficial effects:

[0018] In the technical solutions provided in some embodiments of the present disclosure, first, based on the initial speech recognition model, an unlabeled data set is used to design a contrastive learning loss function to pre-train the first network of the model; then, the parameters of the first network are fixed, and the labeled data set is used to calculate the joint loss function to train the second network of the model; finally, the labeled data is used to calculate the joint loss function to train the speech recognition model to fine-tune the parameters of the first network and the second network, and the model is trained until convergence to obtain the final speech recognition model. The training method of the speech recognition model disclosed in the present application, on the one hand, does not rely on a large amount of labeled data during the training process, thereby reducing the cost of labeled data for automatic speech recognition ASR and improving the research and development and optimization progress of the speech recognition model; on the other hand, the model training process is not limited by the CTC framework of the time series data classification algorithm, avoiding the independence of speech feature representation frames, which is more in line with the actual situation, thereby making the recognition accuracy of the speech recognition model higher.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0021] Figure 1 A flowchart schematically illustrates a method for training a speech recognition model in an exemplary embodiment of the present disclosure;

[0022] Figure 2 A schematic diagram schematically illustrates a flow chart of a method for preparing a training data set in an exemplary embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram schematically illustrates a flow chart of a method for calculating a contrastive learning loss function in an exemplary embodiment of the present disclosure;

[0024] Figure 4 A schematic diagram schematically shows a flow chart of a mask processing method in an exemplary embodiment of the present disclosure;

[0025] Figure 5 Schematically illustrates a flow chart of another method for calculating a contrastive learning loss function in an exemplary embodiment of the present disclosure;

[0026] Figure 6 A schematic diagram schematically illustrates the composition of a training device for a speech recognition model in an exemplary embodiment of the present disclosure;

[0027] Figure 7 A schematic diagram schematically illustrates a computer-readable storage medium in an exemplary embodiment of the present disclosure;

[0028] Figure 8 The structure diagram of a computer system of an electronic device in an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0029] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more comprehensive and complete and will fully convey the concept of the example embodiments to those skilled in the art.

[0030] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0031] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0032] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0033] The implementation details of the technical solution of the embodiment of the present disclosure are elaborated in detail below.

[0034] Figure 1 The flowchart of a method for training a speech recognition model in an exemplary embodiment of the present disclosure is schematically shown. Figure 1 As shown, the training method of the speech recognition model includes steps S101 to S104:

[0035] Step S101, constructing an initial speech recognition model; wherein the initial speech recognition model includes a first network having a first initial parameter and a second network having a second initial parameter;

[0036] Step S102, fixing the second initial parameter, calculating a contrastive learning loss function based on an unlabeled data set, and performing self-supervised training on the first network according to the contrastive learning loss function to adjust the first initial parameter to a first intermediate parameter;

[0037] Step S103, fixing the first intermediate parameter, calculating a first joint loss function based on the labeled data set, and training the second network according to the first joint loss function to adjust the second initial parameter to a second intermediate parameter;

[0038] Step S104, calculating a second joint loss function based on the labeled data set, and training the first network and the second network according to the second joint loss function to adjust the first intermediate parameters and the second intermediate parameters to obtain a target speech recognition model.

[0039] In the technical solutions provided in some embodiments of the present disclosure, first, based on the initial speech recognition model, an unlabeled data set is used to design a contrastive learning loss function to pre-train the first network of the model; then, the parameters of the first network are fixed, and the labeled data set is used to calculate the joint loss function to train the second network of the model; finally, the labeled data is used to calculate the joint loss function to train the speech recognition model to fine-tune the parameters of the first network and the second network, and the model is trained until convergence to obtain the final speech recognition model. The training method of the speech recognition model disclosed in the present application, on the one hand, does not rely on a large amount of labeled data during the training process, thereby reducing the cost of labeled data for automatic speech recognition ASR and improving the research and development and optimization progress of the speech recognition model; on the other hand, the model training process is not limited by the CTC framework of the time series data classification algorithm, avoiding the independence of speech feature representation frames, which is more in line with the actual situation, thereby making the recognition accuracy of the speech recognition model higher.

[0040] Below, each step of the method for training the speech recognition model in this example implementation will be described in more detail with reference to the accompanying drawings and embodiments.

[0041] Step S101, constructing an initial speech recognition model; wherein the initial speech recognition model includes a first network having first initial parameters and a second network having second initial parameters.

[0042] In one embodiment of the present disclosure, a randomly initialized speech recognition model is first constructed. The network structure of the speech recognition model may include an embedding layer (i.e., an Embedding layer), a conversion layer (i.e., a Transformer layer), and an output layer. The Transformer layer is composed of a first network and a second network, the first network is an encoder network, and the second network is a decoder network.

[0043] For the initial speech recognition model after random initialization, the first network and the second network both have their own initial parameters, and the network model parameters are adjusted in subsequent model training to obtain the trained speech recognition model.

[0044] In one embodiment of the present disclosure, before performing the training from step S102 to step S104, a data set for training needs to be prepared. Figure 2 The following is a schematic diagram showing a flow chart of a method for preparing a training data set in an exemplary embodiment of the present disclosure. Figure 2 As shown, the training data set preparation method includes:

[0045] Step S201, acquiring audio sample data based on a preset audio sampling rate, and dividing the audio sample data into a first audio sample and a second audio sample;

[0046] Step S202, calculating the audio feature matrix of the first audio sample to obtain the unlabeled data set; and

[0047] Step S203: obtaining the annotated data set according to the calculated audio feature matrix of the second audio sample and the obtained text annotation result of the second audio sample.

[0048] In step S201, audio sampling is performed at a preset audio sampling rate to obtain audio sample data. The sampled audio may be Chinese speech audio or other language audio. For example, an audio sample of a certain duration is obtained by sampling at an audio sampling rate of 16 kHz.

[0049] Afterwards, in order to configure the unlabeled data set and the labeled data set, the sampled audio sample data can be divided into two parts, one part is used to generate the unlabeled data set, which is i in total, and the other part is used to generate the labeled data set, which is j in total.

[0050] It should be noted that, in the process of division, some audio samples can be used as both the first audio samples and the second audio samples, that is, the contents may overlap.

[0051] In step S202, an unlabeled data set is generated. The unlabeled data set does not require speech annotation, so the audio feature matrix of the first audio sample is directly calculated to obtain the unlabeled data set, which is denoted as U = {x i |i∈[1,N u ]}, where x i is the audio feature matrix of the i-th first audio sample, N u is the number of unlabeled first audio samples in the unlabeled dataset.

[0052] In step S203, a labeled data set is generated. In the labeled data set, each audio sample has its corresponding text annotation result. Therefore, the labeled data set can be obtained by calculating the audio feature matrix of the second audio sample and annotating the second audio sample to obtain the text annotation result, which is recorded as L = {x j ,y j |j∈[1,N l ]}, where x j is the audio feature matrix of the jth second audio sample, y j is the audio feature matrix x j The corresponding text annotation result, N l is the number of unlabeled second audio samples in the unlabeled dataset.

[0053] It should be noted that the present disclosure has no limit on the size between the number of unlabeled data sets Nu and the number of labeled data sets Nl. However, in actual operation, considering the cost of speech annotation, the number of unlabeled data sets can be much larger than the number of labeled data sets, that is, Nu>>Nl, for example, the number of unlabeled data sets and labeled data sets are 10,000 hours and 100 hours respectively.

[0054] In steps S202 and S203, when the audio feature matrix of the audio sample is calculated, the audio feature matrix may be an 80-dimensional Mel-spectrogram feature, wherein the duration of each frame of the spectrogram is 25 ms and the step length is 10 ms.

[0055] In step S102, the second initial parameters are fixed, a contrastive learning loss function is calculated based on an unlabeled data set, and self-supervised training is performed on the first network according to the contrastive learning loss function to adjust the first initial parameters to first intermediate parameters.

[0056] In one embodiment of the present disclosure, step S102 is to perform self-supervised training on the first network, where the first network includes a convolutional neural network module and a convolutional enhancement module.

[0057] The first network may be an encoder network, including a convolutional neural network module, namely a CNN (Convolutional Neural Network) module, and a convolution enhancement module, namely a Conformer module. For example, the encoder network is formed by sequentially connecting 5 layers of CNN modules and 12 Conformer modules.

[0058] Figure 3 The following schematically shows a flow chart of a method for calculating a contrastive learning loss function in an exemplary embodiment of the present disclosure. Figure 3 As shown, the method for calculating the contrastive learning loss function includes steps S301 to S304:

[0059] Step S301, calculating a shallow representation result of an audio sample data in the unlabeled data set based on the convolutional neural network module;

[0060] Step S302, performing mask processing on the shallow representation result to obtain a mask representation result, and calculating a deep representation result of the mask representation result based on the convolution enhancement module; and

[0061] Step S303, performing a linear transformation on the shallow representation result to obtain a target representation result;

[0062] Step S304: Calculate the contrastive learning loss function based on the deep representation result and the target representation result.

[0063] Next, steps S301 to S304 are described in detail:

[0064] In step S301, a shallow representation result of an audio sample data in the unlabeled data set is calculated based on the convolutional neural network module.

[0065] Specifically, given an audio sample data x in an unlabeled dataset i ∈U,x i After multi-layer CNN calculations, the shallow representation result is obtained, denoted as e.

[0066] Then, the shallow representation result e is processed in two ways, namely, the two processings of step S302 and step S301, and then the processing results of these two ways are compared.

[0067] In step S302, mask processing is performed on the shallow representation result to obtain a mask representation result, and a deep representation result of the mask representation result is calculated based on the convolution enhancement module.

[0068] Specifically, Figure 4 The flowchart of a mask processing method in an exemplary embodiment of the present disclosure is schematically shown. Figure 4 As shown, the mask processing method includes:

[0069] Step S401, randomly selecting a seed sample frame from the shallow representation result based on a random mask probability;

[0070] Step S402, replacing the feature vectors of the continuous K frames after the seed sample frame in the shallow representation result with the learnable vector to obtain the mask representation result, where K is a positive integer.

[0071] Specifically, we randomly select p percent of sample frames from the shallow representation result e as seed sample frames, and mask the subsequent K consecutive frames of the seed sample frame in e, that is, we use a learnable vector to replace the feature vector of the mask position in the shallow representation e to obtain the mask representation result

[0072] Wherein, p is the random mask probability, which is a preset value, for example, p = 6.5, and K is the continuous frame mask parameter, which is also a preset value, a positive integer, for example, K = 10. Of course, the embodiments of the present disclosure are only exemplary, and the values ​​of the random mask probability and the continuous frame mask parameter can be adaptively adjusted according to actual needs.

[0073] In the obtained mask representation result After that, after calculations through multiple Conformer modules, the deep representation result is obtained, denoted as h.

[0074] In step S303, the shallow representation result is linearly transformed to obtain the target representation result.

[0075] Specifically, a linear transformation is a linear map, which is a mapping from one vector space V to another vector space W while maintaining addition and scalar multiplication. The shallow representation result e is subjected to a linear transformation to obtain the target representation result, denoted as q.

[0076] In step S304, the contrastive learning loss function is calculated based on the deep representation result and the target representation result.

[0077] Figure 5 The flowchart of another method for calculating the contrastive learning loss function in the exemplary embodiment of the present disclosure is schematically shown. Figure 5As shown, the method for calculating the contrastive learning loss function includes:

[0078] Step S501, selecting M frame anchor samples from the mask part in the deep representation result as the first sample, where M is a positive integer;

[0079] Step S502, selecting M frame anchor samples corresponding one-to-one to the M frame anchor samples in the first sample from the target representation result as second samples, and selecting S frame negative samples as third samples, where S is a positive integer;

[0080] Step S503: Calculate the contrastive learning loss function based on the similarity between the first sample and the second sample and the similarity between the first sample and the third sample.

[0081] Specifically, M frame anchor samples are selected from the mask part of the deep representation result h, and each frame sample, that is, the first sample is recorded as h m , M is the number of frames of the anchor sample, which is a preset value and a positive integer, for example, the number of frames of the anchor sample is M=10.

[0082] And select M frames of anchor samples corresponding to the anchor samples in the first sample from the target representation result q, and each frame sample, that is, the second sample, is recorded as q m At the same time, S frames of negative samples are selected from the target representation result q, and each frame sample, that is, the third sample is recorded as S is the number of frames of negative samples, which is a preset value and a positive integer. For example, the number of frames of negative samples is S=100.

[0083] Then calculate x i Contrastive learning loss function for audio samples i , as shown in formula (1):

[0084]

[0085] Among them, sim(h m ,q m ) represents the first sample h m and the second sample q m The similarity between Denotes the first sample h m And the third sample The similarity between them, T is the scale factor, which is a preset value, for example, T=10.

[0086] Specifically, sim() is a similarity function, and the calculation formula is shown in formula (2):

[0087]

[0088] Among them, a and b are two entities whose similarity needs to be calculated. For example, to calculate sim(h m ,q m ), a is the first sample h m , b is the second sample q m , Same reason.

[0089] For every x i Audio samples can calculate the contrastive learning loss function loss i , then for the total contrastive learning loss function loss of all unlabeled datasets U, it is also necessary to integrate the loss functions of each audio sample, such as finding the average.

[0090] Based on the above method, a comparative learning task is designed, and the first network encoder network in the speech recognition model is self-supervised trained through the unlabeled data set U. After the training is completed, the first initial parameter of the encoder network is adjusted to the first intermediate parameter. Since it does not rely on a large amount of labeled data, it can reduce the cost of labeled data for automatic speech recognition ASR and improve the research and development and optimization progress of the speech recognition model.

[0091] In step S103, the first intermediate parameter is fixed, a first joint loss function is calculated based on the labeled data set, and the second network is trained according to the first joint loss function to adjust the second initial parameter to a second intermediate parameter.

[0092] In one embodiment of the present disclosure, step S103 is to train the second network, where the second network includes a feature deformation module.

[0093] The second network may be a decoder network, including one or more feature deformation modules, namely transformer modules. For example, the decoder network is composed of 6 transformer modules.

[0094] After step S102, the encoder network has been trained, but the decoder is still in a randomly initialized state. In order to avoid an imbalance in the training states of the decoder and encoder, in this step, the decoder network is trained using a joint loss function to achieve the purpose of preliminary training of the decoder network.

[0095] In one embodiment of the present disclosure, the decoder network is trained by a joint loss function, which is a CTC-attention joint loss function.

[0096] Specifically, the loss functions currently used in the end-to-end ASR model training process mainly include (1) Connectionist Temporal Classification (CTC) loss function; (2) encoder-decoder loss function based on the attention mechanism; and (3) CTC-attention joint loss function. Among them, the CTC-attention joint loss function combines the advantages of both CTC and attention mechanisms, so the present disclosure uses the CTC-attention joint loss function for model training.

[0097] During model training, the labeled data set L is used to fix the encoder network, that is, the first intermediate parameters are fixed, and the CTC-attention joint loss function is used to complete the model training of the decoder network until the decoder network converges, and then the decoder network is adjusted from the second initial parameters to the second intermediate parameters.

[0098] In step S104, a second joint loss function is calculated based on the labeled data set, and the first network and the second network are trained according to the second joint loss function to adjust the first intermediate parameters and the second intermediate parameters to obtain a target speech recognition model.

[0099] In one embodiment of the present disclosure, step S104 is to fine-tune the parameters of the two networks in the speech recognition model. The loss function is still the CTC-attention joint loss function.

[0100] Specifically, a labeled dataset L is used, the encoder network and the decoder network are opened, and the encoder network and the decoder network are fine-tuned and trained until the model converges by optimizing the CTC-attention joint loss function, so as to adjust the first intermediate parameters and the second intermediate parameters to obtain the final speech recognition model.

[0101] Based on the training method of the speech recognition model provided in the present invention, the model training process is not restricted by the CTC framework of the time series data classification algorithm, which avoids the independence of speech feature representation frames, is more in line with the actual situation, and thus makes the speech recognition model more accurate.

[0102] Figure 6 The following is a schematic diagram showing the composition of a training device for a speech recognition model in an exemplary embodiment of the present disclosure. Figure 6As shown, the speech recognition model training device 600 may include a model building module 601, a first training module 602, a second training module 603 and a model adjustment module 604. Among them:

[0103] A model building module 601 is used to build an initial speech recognition model; wherein the initial speech recognition model includes a first network with a first initial parameter and a second network with a second initial parameter;

[0104] A first training module 602 is used to fix the second initial parameter, calculate a contrastive learning loss function based on an unlabeled data set, and perform self-supervised training on the first network according to the contrastive learning loss function to adjust the first initial parameter to a first intermediate parameter;

[0105] A second training module 603 is used to fix the first intermediate parameter, calculate a first joint loss function based on the labeled data set, and train the second network according to the first joint loss function to adjust the second initial parameter to a second intermediate parameter;

[0106] The model adjustment module 607 is used to calculate a second joint loss function based on the labeled data set, and train the first network and the second network according to the second joint loss function to adjust the first intermediate parameters and the second intermediate parameters to obtain a target speech recognition model.

[0107] According to an exemplary embodiment of the present disclosure, the first network includes a convolutional neural network module and a convolutional enhancement module.

[0108] According to an exemplary embodiment of the present disclosure, the first training module 602 includes a shallow unit, a mask unit, a target unit and a comparison unit, the shallow unit is used to calculate the shallow representation result of an audio sample data in the unlabeled data set based on the convolutional neural network module; the mask unit represents masking processing of the shallow representation result to obtain a masked representation result, and calculates the deep representation result of the masked representation result based on the convolutional enhancement module; and the target unit is used to linearly transform the shallow representation result to obtain a target representation result; the comparison unit is used to calculate the contrast learning loss function based on the deep representation result and the target representation result.

[0109] According to an exemplary embodiment of the present disclosure, the mask unit is also used to randomly select a seed sample frame from the shallow representation result based on a random mask probability; and replace the feature vectors of K consecutive frames after the seed sample frame in the shallow representation result with a learnable vector to obtain the mask representation result, where K is a positive integer.

[0110] According to an exemplary embodiment of the present disclosure, the comparison unit is also used to select M frame anchor samples from the mask part in the deep representation result as the first sample, where M is a positive integer; and select M frame anchor samples from the target representation result that correspond one-to-one to the M frame anchor samples in the first sample as the second sample, and select S frame negative samples as the third sample, where S is a positive integer; calculate the comparative learning loss function based on the similarity between the first sample and the second sample and the similarity between the first sample and the third sample.

[0111] According to an exemplary embodiment of the present disclosure, the second network includes a feature deformation module.

[0112] According to an exemplary embodiment of the present disclosure, the training device 600 of the speech recognition model also includes a data preparation module, which is used to obtain audio sample data based on a preset audio sampling rate, and divide the audio sample data into a first audio sample and a second audio sample; calculate the audio feature matrix of the first audio sample to obtain the unlabeled data set; and obtain the labeled data set based on the calculated audio feature matrix of the second audio sample and the obtained text annotation result of the second audio sample.

[0113] The specific details of each module in the above-mentioned speech recognition model training device 600 have been described in detail in the corresponding speech recognition model training method, so they will not be repeated here.

[0114] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0115] In an exemplary embodiment of the present disclosure, a storage medium capable of implementing the above method is also provided. Figure 7 A schematic diagram schematically illustrates a computer-readable storage medium in an exemplary embodiment of the present disclosure, such as Figure 7 As shown, a program product 700 for implementing the above method according to an embodiment of the present disclosure is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a mobile phone. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0116] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided. Figure 8 The structure diagram of a computer system of an electronic device in an exemplary embodiment of the present disclosure is schematically shown.

[0117] It should be noted that Figure 8 The computer system 800 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0118] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 802 or the program loaded from the storage part 808 to the random access memory (RAM) 803. In the RAM 803, various programs and data required for system operation are also stored. The CPU 801, ROM 802 and RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0119] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed so that a computer program read therefrom is installed into the storage section 808 as needed.

[0120] In particular, according to an embodiment of the present disclosure, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 809, and / or installed from a removable medium 811. When the computer program is executed by a central processing unit (CPU) 801, various functions defined in the system of the present disclosure are executed.

[0121] It should be noted that the computer-readable medium shown in the embodiment of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0122] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0123] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0124] As another aspect, the present disclosure further provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

[0125] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0126] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0127] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0128] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for training a speech recognition model, characterized in that: include: Constructing an initial speech recognition model; wherein the initial speech recognition model includes a first network having a first initial parameter and a second network having a second initial parameter; Fixing the second initial parameter, calculating a contrastive learning loss function based on an unlabeled data set, and performing self-supervised training on the first network according to the contrastive learning loss function to adjust the first initial parameter to a first intermediate parameter; Fixing the first intermediate parameter, calculating a first joint loss function based on the labeled data set, and training the second network according to the first joint loss function to adjust the second initial parameter to a second intermediate parameter; A second joint loss function is calculated based on the labeled data set, and the first network and the second network are trained according to the second joint loss function to adjust the first intermediate parameter and the second intermediate parameter to obtain a target speech recognition model.

2. The method for training a speech recognition model according to claim 1, characterized in that: The first network includes a convolutional neural network module and a convolutional enhancement module.

3. The method for training a speech recognition model according to claim 2, characterized in that: The calculating of the contrastive learning loss function based on the unlabeled data set includes: Calculating a shallow representation result of an audio sample data in the unlabeled data set based on the convolutional neural network module; Performing mask processing on the shallow representation result to obtain a mask representation result, and calculating a deep representation result of the mask representation result based on the convolution enhancement module; and Performing a linear transformation on the shallow representation result to obtain a target representation result; The contrastive learning loss function is calculated based on the deep representation result and the target representation result.

4. The method for training a speech recognition model according to claim 3, characterized in that: The performing masking processing on the shallow representation result to obtain the masked representation result includes: Randomly selecting a seed sample frame from the shallow representation result based on a random mask probability; The mask representation result is obtained by replacing the feature vectors of K consecutive frames after the seed sample frame in the shallow representation result with learnable vectors, where K is a positive integer.

5. The method for training a speech recognition model according to claim 3, characterized in that: The calculating the contrastive learning loss function based on the deep representation result and the target representation result comprises: Selecting M frame anchor samples from the mask part of the deep representation result as the first sample, where M is a positive integer; and Selecting M frame anchor samples corresponding to the M frame anchor samples in the first sample from the target representation result as the second sample, and selecting S frame negative samples as the third sample, where S is a positive integer; The contrastive learning loss function is calculated based on the similarity between the first sample and the second sample and the similarity between the first sample and the third sample.

6. The method for training a speech recognition model according to claim 1, characterized in that: The second network includes a feature deformation module.

7. The method for training a speech recognition model according to claim 1, characterized in that: The method further comprises: Acquire audio sample data based on a preset audio sampling rate, and divide the audio sample data into a first audio sample and a second audio sample; Calculating an audio feature matrix of the first audio sample to obtain the unlabeled data set; and The annotated data set is obtained according to the calculated audio feature matrix of the second audio sample and the obtained text annotation result of the second audio sample.

8. A training device for a speech recognition model, characterized in that: include: A model building module is used to build an initial speech recognition model; wherein the initial speech recognition model includes a first network with a first initial parameter and a second network with a second initial parameter; A first training module, used for fixing the second initial parameter, calculating a contrastive learning loss function based on an unlabeled data set, and performing self-supervised training on the first network according to the contrastive learning loss function to adjust the first initial parameter to a first intermediate parameter; a second training module, configured to fix the first intermediate parameter, calculate a first joint loss function based on a labeled data set, and train the second network according to the first joint loss function to adjust the second initial parameter to a second intermediate parameter; A model adjustment module is used to calculate a second joint loss function based on the labeled data set, and train the first network and the second network according to the second joint loss function to adjust the first intermediate parameter and the second intermediate parameter to obtain a target speech recognition model.

9. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the training method of the speech recognition model as described in any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the training method of the speech recognition model as described in any one of claims 1 to 7.

Images