Speech recognition model training method and device, speech recognition method

Abstract

The method comprises the following steps: calculating a target field voice data set according to a source field voice data set, and inputting the source field voice data set and the target field voice data set into a voice recognition model to be trained respectively to obtain a first text recognition result and a second text recognition result; constructing a first loss function according to the first text recognition result and voice annotation results included in the source field voice data set; constructing a second loss function according to a first covariance and a second covariance, and constructing a target loss function according to the first loss function and the second loss function; and updating parameters included in the voice recognition model to be trained according to the target loss function to obtain a trained voice recognition model. The method improves the accuracy of the voice recognition model.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of machine learning, and in particular, to a speech recognition model training method and device, a speech recognition method, a computer readable storage medium, and an electronic device. BACKGROUND

[0002] In recent years, with the rapid development of deep learning technology, automatic speech recognition (ASR) based on end-to-end deep neural networks has gradually become the mainstream technology in the current speech recognition field, and has been widely used in intelligent customer service, mobile assistants, human-computer interaction and other business scenarios.

[0003] In the existing method for training an ASR model, an unsupervised domain adaptation (UDA) method is usually used to achieve this; specifically, this method aims to train a model using data and labels from the source domain and unlabeled data from the target domain, so that the model can have better generalization ability for the target domain; at the same time, in order to improve the generalization ability of the model, feature alignment can be achieved by aligning the source domain data and the target domain data, for example, by designing a classifier and adding a gradient reversal layer (GRL) to perform domain adversarial training, so that the features obtained by the model for the source domain data and the target domain data cannot be effectively distinguished by this classification network.

[0004] However, the above method has the following defects: since the speech data of ASR is sequence data, the audio lengths of different data are different, and in the adversarial training process, the training of the classifier is prone to overfitting, resulting in poor feature alignment effect, and thus the accuracy of the speech recognition model is low.

[0005] Therefore, it is necessary to provide a new speech recognition model training method and device.

[0006] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0007] The purpose of the present disclosure is to provide a speech recognition model training method, a speech recognition model training device, a speech recognition method, a computer readable storage medium, and an electronic device, thereby at least partially overcoming the problem of low accuracy of the speech recognition model due to the limitations and defects of the related art.

[0008] According to one aspect of the present disclosure, a training method of a speech recognition model is provided, comprising:

[0009] According to the source field speech data set and the preset far-field reverberation impulse response set, the target field speech data set is calculated, and the source field speech data set and the target field speech data set are respectively input into the speech recognition model to be trained to obtain a first text recognition result and a second text recognition result;

[0010] According to the first text recognition result and the speech annotation result included in the source field speech data set, a first loss function is constructed, and a first covariance and a second covariance of the first text recognition result and the second text recognition result are calculated;

[0011] A second loss function is constructed according to the first covariance and the second covariance, and a target loss function is constructed according to the first loss function and the second loss function;

[0012] According to the target loss function, the parameters included in the speech recognition model to be trained are updated to obtain a trained speech recognition model.

[0013] In an exemplary embodiment of the present disclosure, according to the source field speech data set and the preset far-field reverberation impulse response set, the target field speech data set is calculated, comprising:

[0014] The original audio data in the source field speech data set and the impulse response samples in the preset far-field mixed impulse response set are extracted; wherein the original audio data is Chinese speech data;

[0015] The original audio data and the impulse response samples are subjected to convolution calculation to obtain target audio data; wherein the audio feature matrix of the original audio data and the target audio data is 81-dimensional mel spectrum feature;

[0016] The target field speech data set is generated according to a plurality of target audio data.

[0017] In an exemplary embodiment of the present disclosure, according to the first text recognition result and the speech annotation result included in the source field speech data set, a first loss function is constructed, comprising:

[0018] The path set of all paths constituting the speech annotation result included in the source field speech data set in the output matrix included in the first text recognition result is calculated;

[0019] The product of the probability of each element included in each path in the path set belonging to the element at the corresponding position in the speech annotation result is calculated;

[0020] The product of the probabilities of all paths included in the path set is summed, and the first loss function is constructed with the maximum value of the sum result as the target.

[0021] In an example embodiment of the present disclosure, the first covariance and the second covariance of the first text recognition result and the second text recognition result are calculated, including:

[0022] According to the first audio duration of the original audio data included in the source domain voice data set and the second audio duration of the target audio data included in the target domain voice data set, the first feature time dimension length and the second feature time dimension length are calculated;

[0023] According to the first text recognition result, the first feature time dimension length, and the preset first vector, the first covariance is calculated;

[0024] According to the second text recognition result, the second feature time dimension length, and the preset second vector, the second covariance is calculated.

[0025] In an example embodiment of the present disclosure, the second loss function is constructed according to the first covariance and the second covariance, including:

[0026] The feature dimension length of the self-attention module included in the to-be-trained speech recognition model is obtained;

[0027] The matrix norm of the first covariance and the second covariance is calculated, and the second loss function is constructed according to the matrix norm and the feature dimension length of the self-attention module.

[0028] In an example embodiment of the present disclosure, the target loss function is constructed according to the first loss function and the second loss function, including:

[0029] The second loss function and the preset balance parameter are multiplied, and the first loss function and the multiplication result are used to construct the target loss function;

[0030] The preset balance parameter is used to balance the first loss function and the second loss function, the first loss function is used to train the to-be-trained speech recognition model, and the second loss function is used to align the source domain voice data set and the target domain voice data set.

[0031] In an example embodiment of the present disclosure, the source domain voice data set and the target domain voice data set are respectively input into the to-be-trained speech recognition model to obtain the first text recognition result and the second text recognition result, including:

[0032] extracting first local features and second local features of original audio data in the source domain speech data set and target audio data in the target domain speech data set respectively by using a convolutional neural network included in the speech recognition model to be identified;

[0033] calculating first global features of the original audio data and second global features of the target audio data according to the first local features and the second local features respectively by using a self-attention module included in the speech recognition model to be identified;

[0034] classifying the first global features and the second global features respectively by using a fully connected layer included in the speech recognition model to be identified, to obtain first text recognition results of the original audio data and second text recognition results of the target audio data.

[0035] According to one aspect of the present disclosure, a speech recognition method is provided, comprising:

[0036] collecting speech data to be identified; wherein the speech data to be identified includes at least one of near-field speech data and far-field speech data; the far-field speech data includes background noise and / or far-field reverberation;

[0037] inputting the speech data to be identified into a trained speech recognition model to obtain a speech recognition result;

[0038] The trained speech recognition model is obtained by training a speech recognition model to be trained by using the speech recognition model training method according to any one of the preceding aspects.

[0039] According to one aspect of the present disclosure, a speech recognition model training device is provided, comprising:

[0040] a first calculation module configured to calculate a target domain speech data set according to a source domain speech data set and a preset far-field reverberation impulse response set, and input the source domain speech data set and the target domain speech data set into a speech recognition model to be trained respectively to obtain first text recognition results and second text recognition results;

[0041] a second calculation module configured to construct a first loss function according to the first text recognition results and speech annotation results included in the source domain speech data set, and calculate first covariance and second covariance of the first text recognition results and the second text recognition results;

[0042] a target loss function construction module configured to construct a second loss function according to the first covariance and the second covariance, and construct a target loss function according to the first loss function and the second loss function;

[0043] The speech recognition model training module is configured to update parameters included in the speech recognition model to be trained according to the target loss function, and obtain a trained speech recognition model.

[0044] According to an aspect of the present disclosure, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the speech recognition model training method according to any one of the preceding aspects, and the speech recognition method according to the preceding aspects.

[0045] According to an aspect of the present disclosure, an electronic device is provided, which comprises:

[0046] a processor; and

[0047] a memory configured to store executable instructions of the processor;

[0048] The processor is configured to execute the executable instructions to implement the speech recognition model training method according to any one of the preceding aspects, and the speech recognition method according to the preceding aspects.

[0049] The speech recognition model training method provided by the embodiments of the present disclosure has the following advantages. On the one hand, in the process of constructing the loss function, the loss function between the predicted result (the first text recognition result) and the speech annotation result is considered, and the loss function (the first covariance and the second covariance) of the alignment between the source domain speech data set and the target domain speech data set is also considered. In the alignment process, the alignment is performed by the covariance, and the feature alignment is not realized by the classifier, thereby avoiding the problem that the final feature alignment effect is poor due to the overfitting of the training of the classifier, and thus the accuracy of the speech recognition model is low. On the other hand, the target domain speech data set is obtained by adding the far-field reverberation impulse response on the basis of the source domain speech data set, and in the process of constructing the loss function, the source domain speech data set and the target domain speech data set are considered at the same time, thereby avoiding the problem that the accuracy of the recognition result is low when recognizing the speech data to be recognized with the far-field reverberation due to the training of the speech recognition model only by the source domain speech data set in the prior art.

[0050] It should be understood that the general description above and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0051] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, further serve to explain the principles of the present disclosure. It is to be understood that the drawings are designed solely for purposes of illustration to be used in conjunction with the description insofar as specifically described herein. Other drawings can be derived from these drawings by a person of ordinary skill in the art without paying creative effort.

[0052] Figure 1 A flowchart schematically showing a method of training a speech recognition model according to an example embodiment of the present disclosure.

[0053] Figure 2 A structural example diagram of a speech recognition model according to an example embodiment of the present disclosure is schematically shown.

[0054] Figure 3 An example diagram of an 81-dimensional Mel-spectrogram feature matrix according to an example embodiment of the present disclosure is schematically shown.

[0055] Figure 4 A flowchart schematically showing a method of constructing a first loss function according to a first text recognition result and a speech annotation result included in the source domain speech data set according to an example embodiment of the present disclosure.

[0056] Figure 5 A flowchart schematically showing a method of calculating a first covariance and a second covariance of the first text recognition result and the second text recognition result according to an example embodiment of the present disclosure.

[0057] Figure 6 A flowchart schematically showing a method of constructing a second loss function according to the first covariance and the second covariance according to an example embodiment of the present disclosure.

[0058] Figure 7 A flowchart schematically showing another method of training a speech recognition model according to an example embodiment of the present disclosure.

[0059] Figure 8 A block diagram of a training device of a speech recognition model according to an example embodiment of the present disclosure is schematically shown.

[0060] Figure 9 An electronic device for implementing the above-mentioned method of training a speech recognition model according to an example embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0061] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations can be implemented in any

[0062] Moreover, the drawings represent a simplified diagram of the disclosure and are not necessarily to scale. Like reference numerals in the drawings indicate like or similar elements such that their description need not be repeated. Some of the block diagrams shown in the drawings are functional entities that do not necessarily have to correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0063] In the training or optimization process of an ASR (Automatic Speech Recognition) model, the usual implicit assumption is that the data distribution of the training set and the test set of speech data is the same. However, in many practical cases, there can be a large difference between the distribution of the test set and the training set due to the existence of accents, background noise, far-field reverberation, and other special locations in the test set of speech data, which reduces the effect of the ASR model obtained from the training set in the actual application scenario. For example, an ASR model trained by using near-field speech data (for example, the distance between the speaker and the microphone is less than 20 centimeters) usually has a serious decline in accuracy when identifying the test set of far-field speech data (for example, the distance between the speaker and the microphone is greater than 1 meter).

[0064] In order to improve the speech recognition accuracy of a target domain (TD), one of the most direct methods is to collect and label speech data corresponding to the scene for ASR model training. Generally, due to the large number of network parameters of an end-to-end ASR model, training an accurate ASR model often requires large-scale speech data and corresponding manual labeling. However, since the labeling of speech data requires at least one manual listening, the labeling cost is often high; especially when the speech data set contains speech data with poor speech intelligibility or recognition due to accent, background noise, far-field reverberation, etc., the cost and period of manual labeling will be higher.

[0065] In order to improve the speech recognition performance of the TD while avoiding large-scale manual data labeling, the prior art usually adopts a transfer learning (TL) method. The idea of this method is to use a small amount or no labeling of target domain speech data to improve the performance of the model in the target domain on the basis of the existing speech data and labeling of the source domain (SD).

[0066] In the TL-based technology, the most typical method is fine-tuning, which uses the speech data and labeling of the SD to train a pre-trained model, and then fine-tunes the parameters of the model using a small amount of target domain speech data and labeling. However, the fine-tuning ASR model method still requires manual labeling; and due to the small amount of training data, the model may face the problem of overfitting.

[0067] The most ideal ASR transfer learning technology is to completely avoid speech data labeling to improve the generalization ability of the ASR model, that is, the unsupervised ASR transfer learning technology. For example, if there is the word "hello" in the training data and labeling of the SD model, we hope that the model can effectively recognize "hello" in different scenarios (accent, background noise, far-field reverberation, etc.), without relying on using various complex scenarios containing the word "hello" as the labeling data sample of the TD to optimize the recognition effect of the model on the word "hello" in these scenarios.

[0068] To solve the above problems, an unsupervised domain adaptation (UDA) technology is usually used to solve the problem. This method belongs to a kind of transfer learning, which aims to train a model through the data and labels of the source domain and the unlabeled data of the target domain, so that the model can have better generalization ability for the target domain. In order to achieve this, the prior art is to align the features of the source domain data and the target domain data, for example, by designing a classifier and adding a gradient reversal layer (GRL) to perform domain adversarial training, so that the features obtained by the model for SD data and TD data cannot be effectively distinguished by the classification network.

[0069] However, this technology mainly has two shortcomings: first, since the ASR speech data is sequence data, the audio lengths of different data are different, and in the process of adversarial training, the training of the classifier is prone to overfitting, resulting in poor feature alignment effect; second, in many actual scenarios, it is rare to encounter an absolute source domain data and target domain data two-classification task, and most speech data classification is often ambiguous and not suitable for unsupervised domain adaptation using a classifier problem.

[0070] Based on this, the training method of the speech recognition model provided in the example embodiment can be run on a server, a server cluster, or a cloud server, etc. Of course, those skilled in the art can also run the method of the present disclosure on other platforms according to the needs, and the example embodiment does not specially limit this. Referring to Figure 1 As shown in the figure, the training method of the speech recognition model can include the following steps:

[0071] Step S110. According to the source domain speech data set and the preset far-field reverberation impulse response set, the target domain speech data set is calculated, and the source domain speech data set and the target domain speech data set are input into the speech recognition model to be trained respectively, to obtain the first text recognition result and the second text recognition result;

[0072] Step S120. According to the first text recognition result and the speech annotation result included in the source domain speech data set, a first loss function is constructed, and the first covariance and the second covariance of the first text recognition result and the second text recognition result are calculated;

[0073] Step S130. A second loss function is constructed according to the first covariance and the second covariance, and a target loss function is constructed according to the first loss function and the second loss function;

[0074] Step S140. Updating the parameters included in the speech recognition model to be trained according to the target loss function, to obtain a trained speech recognition model.

[0075] In the speech recognition model training method, on the one hand, in the process of constructing the loss function, the loss function between the prediction result (the first text recognition result) and the speech annotation result is considered, and the loss function (the first covariance and the second covariance) of the alignment between the source field speech data set and the target field speech data set is also considered. In the alignment process, the alignment is performed through the covariance, and the feature alignment is not realized through the classifier, thereby avoiding the problem that the final feature alignment effect is poor due to the overfitting of the training of the classifier, and thereby the accuracy of the speech recognition model is low. On the other hand, since the target field speech data set is obtained by adding the far-field reverberation impulse response on the basis of the source field speech data set, and in the process of constructing the loss function, the source field speech data set and the target field speech data set are considered, thereby avoiding the problem that in the prior art, the speech recognition model is trained only through the source field speech data set, and when the speech data to be recognized with the far-field reverberation is recognized, the recognition result accuracy is low.

[0076] In the following, the speech recognition model training method of the example embodiments of the present disclosure will be explained and described in detail with reference to the accompanying drawings.

[0077] First, the invention purpose of the example embodiments of the present disclosure is explained and described. Specifically, in order to efficiently and low-costly improve the recognition performance of the ASR model on the TD, the present disclosure proposes a CORAL-based unsupervised domain adaptation speech recognition method. Based on the SD audio data and the annotation, the TD unannotated data is used to improve the performance of the model on the TD. In the training process of the model, an improved CORAL (Correlation Alignment, CORAL) technology is used to align the features of the SD speech data and the TD speech data, overcome the shortcomings of the GRL-based method in the feature alignment of the variable-length data, improve the feature alignment effect of the variable-length speech data, and finally improve the generalization ability of the model on the TD. Moreover, the feature alignment method based on the improved CORAL under the variable-length input provided by the present disclosure significantly improves the accuracy of the end-to-end ASR model on the TD.

[0078] Secondly, the speech recognition model involved in the example embodiments of the present disclosure is explained and described. Referring to Figure 2As shown, the speech recognition model can include an input layer 201, a convolutional neural network (CNN) 202, a self-attention block (SAB) 203, a fully connected layer (FC) 204, and an output layer 205. The input layer, the convolutional neural network, the self-attention block, the fully connected layer, and the output layer are connected in sequence.

[0079] Specifically, the convolutional neural network can be used to extract local features of the input data, the self-attention block can be used to extract global features based on the local features, and the fully connected layer can be used to classify the global features, and then output through the output layer. In the speech recognition model involved in the present example embodiment, the convolutional neural network includes 3 layers, the self-attention block includes 10 layers, and the fully connected layer includes 2 layers.

[0080] The following will be explained in detail in combination with Figure 2 The steps included in the training method of the speech recognition model will be explained and described in detail.

[0081] In the training method of the speech recognition model provided in the present example embodiment of the present disclosure:

[0082] In step S110, the target domain speech data set is calculated according to the source domain speech data set and the preset far-field reverberation impulse response set, and the source domain speech data set and the target domain speech data set are input into the speech recognition model to be trained respectively to obtain the first text recognition result and the second text recognition result.

[0083] In the present example embodiment, first, historical speech data needs to be obtained, which can be Chinese speech data. Then each historical speech data can be sampled by a traditional short-time Fourier transform (STFT), thereby obtaining a plurality of speech data, and generating an original audio data set based on each speech data. In the process of sampling each speech data, the window size of the short-time Fourier transform can be 20ms, the window sliding step can be 10ms, the sampling rate of the speech data can be 16kHz, the length of each speech data can be between 0.5s and 15s, and the total length of the original audio data set can be 150 hours. It should be noted that the sizes of the window size and the window sliding step can be adjusted as needed, and the present example does not specially limit this.

[0084] Secondly, when the original audio dataset is obtained, the target domain speech dataset can be calculated according to the source domain speech dataset and the preset far-field reverberation impulse response set. Specifically, the original audio data in the source domain speech dataset and the impulse response sample in the preset far-field mixed impulse response set can be extracted; the original audio data and the impulse response sample are convolved to obtain target audio data; wherein the audio feature matrix of the original audio data and the target audio data is 81-dimensional mel spectrum feature; the target domain speech dataset is generated according to a plurality of target audio data.

[0085] For example, the source domain speech dataset involved in the present disclosure can be denoted as D S , and has:

[0086]

[0087] The target domain speech dataset can be denoted as D T , and has:

[0088]

[0089] wherein, and are the audio feature matrices of the source domain speech dataset and the target domain speech dataset respectively, is the text annotation result corresponding to the audio feature (the source domain dataset contains source domain speech data and its corresponding annotated text, and the target domain dataset only contains target domain speech data, without annotation information), N S and N T are the training sample quantities of the source domain speech dataset and the target domain speech dataset respectively.

[0090] Further, the target domain speech dataset is the speech data generated by randomly adding far-field reverberation impulse responses on the basis of the source domain speech dataset. For example, an audio data sample x S is randomly selected from the SD dataset, and an impulse response h∈H is randomly selected from the far-field reverberation impulse response set H, then a target domain data sample can be calculated and obtained, wherein, represents convolution. At the same time, the audio feature matrix adopted by the present disclosure is 81-dimensional mel spectrum feature, which can be specifically as shown in Figure 3 .

[0091] Further, after obtaining the source domain speech data set and the target domain speech data set, the source domain speech data set and the target domain speech data set can be input into the speech recognition model to be trained respectively to obtain a first text recognition result and a second text recognition result. Specifically, it can include: first, using the convolutional neural network included in the speech recognition model to be recognized to extract the first local features and the second local features of the original audio data in the source domain speech data set and the target audio data in the target domain speech data set respectively; second, using the self-attention module included in the speech recognition model to be recognized to calculate the first global features and the second global features of the original audio data and the target audio data according to the first local features and the second local features respectively; further, using the full connection layer included in the speech recognition model to be recognized to classify the first global features and the second global features respectively to obtain the first text recognition result of the original audio data and the second text recognition result of the target audio data.

[0092] In step S120, a first loss function is constructed according to the first text recognition result and the speech annotation result included in the source domain speech data set, and a first covariance and a second covariance of the first text recognition result and the second text recognition result are calculated.

[0093] In the example embodiment, first, a first loss function is constructed according to the first text recognition result and the speech annotation result included in the source domain speech data set. Specifically, referring to FIG. 4, it can include the following steps: Figure 4

[0094] Step S410, calculating a path set of all paths constituting the speech annotation result included in the source domain speech data set in an output matrix included in the first text recognition result;

[0095] Step S420, calculating the product of the probability of each element included in each path in the path set belonging to the element at the corresponding position in the speech annotation result;

[0096] Step S430, summing the product of the probability of all paths included in the path set, and constructing the first loss function with the maximum value of the sum result as the target.

[0097] In the following, step S410-step S430 will be explained and described. Specifically, in the example embodiment, the existing Connectionist Temporal Classification (CTC) loss function is used as the end-to-end ASR loss function (first loss function). The first loss function can be L A ​, which can be shown as the following formula (1):

[0098]

[0099] wherein, represents the model output matrix P(x S )∈R M×K , the c k row k column element of P(x S ), M is the row number of P(x S ), represents the number of categories of output characters plus an empty character “_”, K is the column number of P(x S ), represents the length of the output time frame. C(y S , K) represents the set of all paths of the audio text label y S with a length of K. For example, the audio text label y S is “Hello”, K is 3, and the set C(y k , K) includes “You _ Hello”, “_ You Hello”, “You Hello _”, “You You Hello”, and the like. c S represents the kth element in a path (c1,...c k ) in C(y S , K). It should be noted that the ASR model can be trained using the speech data and labels of D S .

[0100] Secondly, the first covariance and the second covariance of the first text recognition result and the second text recognition result are calculated. Specifically, as shown in Figure 5 , the following steps can be included:

[0101] Step S510, according to the first audio duration of the original audio data included in the source domain speech data set and the second audio duration of the target audio data included in the target domain speech data set, the first feature time dimension length and the second feature time dimension length are calculated;

[0102] Step S520, according to the first text recognition result, the first feature time dimension length and the preset first vector, the first covariance is calculated;

[0103] Step S530, according to the second text recognition result, the second feature time dimension length and the preset second vector, the second covariance is calculated.

[0104] In the following, steps S510-S530 will be explained and described. Specifically, in this example embodiment, the output of the last layer in the 10-layer self-attention module is selected as the deep semantic feature; and let the sample serial number p∈[1, N S] and q ∈ [1, N T ], corresponding sample and Note and The output of the last layer of the model attention module is respectively: (first text recognition result) and (second text recognition result), wherein T p and T q are the first feature time dimension length and the second feature time dimension length of and , and the first feature time dimension length and the second feature time dimension length are proportional to the duration of the corresponding sample audio. It needs to be supplemented here that the first audio duration and the second audio duration can be the same or different, and this example does not specially limit this.

[0105] Further, the first covariance and the second covariance of the source domain voice data and the target domain voice data are calculated respectively, which are C S ∈R d×d (first covariance) and C T ∈R d×d (second covariance), and the calculation method can be shown in the following formula (3) and formula (4):

[0106]

[0107]

[0108] wherein, (predefined first vector), (predefined second vector) are vectors with all elements being 1, and * represents the transpose of a matrix or a vector.

[0109] In step S130, a second loss function is constructed according to the first covariance and the second covariance, and a target loss function is constructed according to the first loss function and the second loss function.

[0110] In the example embodiment, first, a second loss function is constructed according to the first covariance and the second covariance. Specifically, as shown in Figure 6 may include the following steps:

[0111] Step S610, the feature dimension length of the self-attention module included in the voice recognition model to be trained is obtained;

[0112] Step S620, the matrix norm of the first covariance and the second covariance is calculated, and the second loss function is constructed according to the matrix norm and the feature dimension length of the self-attention module.

[0113] The following will explain and illustrate steps S610 and S620. Specifically, the second loss function L... C The specific calculation method can be shown in the following formula (5):

[0114]

[0115] Among them, ||·|| F d represents the Frobenius norm, or F-norm for short, also known as the matrix norm, which can be defined as the sum of the squares of the absolute values ​​of the elements of a matrix; d is the feature dimension length of the self-attention module, which in this example is d = 256.

[0116] Furthermore, after obtaining the second loss function, a target loss function can be constructed based on the first and second loss functions. Specifically, this may include: performing a product operation on the second loss function and a preset balancing parameter, and constructing the target loss function using the first loss function and the product operation result; wherein the preset balancing parameter is used to balance the first and second loss functions, the first loss function is used to train the speech recognition model to be trained, and the second loss function is used to align the source domain speech dataset and the target domain speech dataset.

[0117] Specifically, using the ASR loss function L A (First Loss Function) and CORAL Loss Function L C (Second loss function), calculate the final target loss function L, which can be shown in the following formula (6):

[0118] L = L A +λL C ; Formula (6)

[0119] The parameter λ can be used to balance ASR training and CORAL feature alignment. When λ = 0, the method degenerates into model training based on SD data, and the final model may perform poorly on SD data. When λ → ∞, the method ignores ASR learning and only focuses on SD and TD feature alignment, and the final converged model may output the same result for all audio data. In this disclosure, λ is set to 1.5 × 10⁻⁶. 4 .

[0120] Based on this, it can be known that in the target loss function, since the second loss function is a feature alignment loss function, the second loss function can make the source domain speech data set and the target domain speech data set in the model keep consistent in the deep semantic features in the learning process of the speech recognition model, so that in the case that there is no text annotation result of the source domain speech data, when the trained speech recognition model is used to recognize the to-be-recognized speech data including the far-field reverberation impulse response, the knowledge of the source domain speech data set can be inherited, and the accuracy of the recognition result can be further improved.

[0121] In step S140, the parameters included in the to-be-trained speech recognition model are updated according to the target loss function, and a trained speech recognition model is obtained.

[0122] In the example embodiment, the parameters included in the to-be-trained speech recognition model (neural network model) can be updated by using a back propagation algorithm, and specifically, an existing ADAM optimization algorithm can be used. The final model performance is shown in Table 1.

[0123] Table 1

[0124]

[0125] Based on Table 1, it can be concluded that the ASR model trained only by using the SD speech data and the annotation has an accuracy of 78.58% on the TD test set. In comparison, the unsupervised domain adaptation speech recognition method based on CORAL proposed in the present disclosure has an accuracy of 82.99% on the TD test set, which is an increase of 4.4%. In the case that there is no TD annotation data, the final test set has a very significant improvement effect, which verifies the effectiveness of the method of the present disclosure.

[0126] Hereinafter, the training method of the speech recognition model of the example embodiment of the present disclosure will be explained and described in combination with Figure 7 The training method of the speech recognition model of the example embodiment of the present disclosure will be explained and described in combination with Figure 7 As shown in FIG. 7, the training method of the speech recognition model can include the following steps:

[0127] In step S701, SD speech data and text annotation and TD unannotated speech data are prepared; wherein the SD data set is Chinese speech data, a total of 150 hours, the audio sampling rate is 16 kHz, and the TD data set is speech data randomly added with far-field reverberation impulse response on the basis of the SD data set

[0128] Step S702, designing an end-to-end deep neural network ASR model structure; wherein the end-to-end ASR model structure comprises 3 layers of convolutional neural network (CNN), 10 layers of self-attention module (SAB), and 2 layers of fully connected layer (FC); the ASR model can directly convert a segment of audio feature matrix at the input end into a text recognition result at the output end, and is easy for model training, inference and optimization;

[0129] Step S703, designing a CORAL-based UDA loss function; wherein the loss function comprises an ASR loss function and a CORAL loss function;

[0130] Step S704, using a back propagation algorithm to train the neural network model.

[0131] Therefore, it can be concluded without doubt that the training method of the speech recognition model provided by the example embodiments of the present disclosure can not only avoid the problem in the prior art that in many actual scenarios, absolute SD data and TD data binary classification tasks are rarely encountered, and most speech data classification is often ambiguous and not suitable for UDA using a classifier problem, but also solve the problem that the method of fine-tuning the ASR model still needs manual annotation, and due to the small amount of training data, the model may face the problem of overfitting during training; at the same time, it is also unnecessary to annotate the target field speech data set, avoiding the problem of wasting a large amount of human cost due to poor speech intelligibility or recognition caused by accents, background noise, far-field reverberation, etc. in the speech data; further, since the second loss function is a feature alignment loss function, the second loss function can make the source field speech data set and the target field speech data set in the model have consistent deep semantic features during the learning process, so that when the trained speech recognition model is used to recognize the to-be-recognized speech data including the far-field reverberation impulse response without the text annotation result of the source field speech data, the knowledge of the source field speech data set can be inherited, and the accuracy of the recognition result can be further improved.

[0132] It needs to be further supplemented that the training method of the speech recognition model provided by the present disclosure uses TD unannotated data to improve the performance of the model on TD on the basis of SD audio data and annotation; and also overcomes the shortcomings of the GRL-based method in variable-length audio data feature alignment, and finally significantly improves the accuracy of the end-to-end ASR model on TD.

[0133] Finally, when the trained speech recognition model is obtained, the speech data to be recognized can be recognized according to the trained speech recognition model. Specifically, it can include: first, collecting speech data to be recognized; wherein the speech data to be recognized includes at least one of near-field speech data and far-field speech data; the far-field speech data includes background noise and / or far-field reverberation; second, inputting the speech data to be recognized into the trained speech recognition model to obtain a speech recognition result; wherein the trained speech recognition model is obtained by training the speech recognition model to be trained according to the speech recognition model training method.

[0134] It should be noted here that the speech data to be recognized can be near-field speech data (for example, the initiator of the speech data is less than 20 centimeters away from the microphone), or far-field speech data including background noise and / or far-field reverberation (for example, the initiator of the speech data is more than 1 meter away from the microphone, or there are specific scenarios such as speech data with accent, background noise and far-field reverberation), which is not limited in this example.

[0135] The present disclosure also provides a speech recognition model training device. Referring to Figure 8 The speech recognition model training device can include a first calculation module 810, a second calculation module 820, a target loss function construction module 830, and a speech recognition model training module 840. Wherein:

[0136] The first calculation module 810 can be used to calculate the target field speech data set according to the source field speech data set and the preset far-field reverberation impulse response set, and input the source field speech data set and the target field speech data set into the speech recognition model to be trained respectively, to obtain the first text recognition result and the second text recognition result;

[0137] The second calculation module 820 can be used to construct a first loss function according to the first text recognition result and the speech annotation result included in the source field speech data set, and calculate the first covariance and the second covariance of the first text recognition result and the second text recognition result;

[0138] The target loss function construction module 830 can be used to construct a second loss function according to the first covariance and the second covariance, and construct a target loss function according to the first loss function and the second loss function;

[0139] The speech recognition model training module 840 can be used to update the parameters included in the speech recognition model to be trained according to the target loss function, to obtain a trained speech recognition model.

[0140] In an example embodiment of the present disclosure, a target domain speech dataset is calculated according to a source domain speech dataset and a preset set of far-field reverberation impulse responses, including:

[0141] Extracting original audio data in the source domain speech dataset and impulse response samples in the preset set of far-field reverberation impulse responses; wherein the original audio data is Chinese speech data;

[0142] Performing convolution calculation on the original audio data and the impulse response samples to obtain target audio data; wherein the audio feature matrices of the original audio data and the target audio data are both 81-dimensional mel-spectrum features;

[0143] Generating the target domain speech dataset according to a plurality of target audio data.

[0144] In an example embodiment of the present disclosure, a first loss function is constructed according to a first text recognition result and speech annotation results included in the source domain speech dataset, including:

[0145] Calculating a path set of all paths in an output matrix included in the first text recognition result, which constitute the speech annotation results included in the source domain speech dataset;

[0146] Calculating the product of the probability of each element included in each path in the path set belonging to the element at the corresponding position in the speech annotation result;

[0147] Summing the products of the probabilities of all paths included in the path set, and constructing the first loss function with the maximum sum as the target.

[0148] In an example embodiment of the present disclosure, a first covariance and a second covariance of the first text recognition result and a second text recognition result are calculated, including:

[0149] According to a first audio duration of original audio data included in the source domain speech dataset and a second audio duration of target audio data included in the target domain speech dataset, calculating a first feature time dimension length and a second feature time dimension length;

[0150] According to the first text recognition result, the first feature time dimension length, and a preset first vector, calculating the first covariance;

[0151] According to the second text recognition result, the second feature time dimension length, and a preset second vector, calculating the second covariance.

[0152] In an example embodiment of the present disclosure, the second loss function is constructed according to the first covariance and the second covariance, including:

[0153] The feature dimension length of the self-attention module included in the speech recognition model to be trained is obtained.

[0154] The matrix norm of the first covariance and the second covariance is calculated, and the second loss function is constructed according to the matrix norm and the feature dimension length of the self-attention module.

[0155] In an example embodiment of the present disclosure, the target loss function is constructed according to the first loss function and the second loss function, including:

[0156] The second loss function and the preset balance parameter are multiplied, and the first loss function and the multiplication result are used to construct the target loss function.

[0157] The preset balance parameter is used to balance the first loss function and the second loss function, the first loss function is used to train the speech recognition model to be trained, and the second loss function is used to align the source domain speech data set and the target domain speech data set.

[0158] In an example embodiment of the present disclosure, the source domain speech data set and the target domain speech data set are respectively input into the speech recognition model to be trained to obtain the first text recognition result and the second text recognition result, including:

[0159] The first local feature and the second local feature of the original audio data in the source domain speech data set and the target audio data in the target domain speech data set are extracted by using the convolutional neural network included in the speech recognition model to be recognized.

[0160] The first global feature of the original audio data and the second global feature of the target audio data are calculated according to the first local feature and the second local feature by using the self-attention module included in the speech recognition model to be recognized.

[0161] The first text recognition result of the original audio data and the second text recognition result of the target audio data are obtained by using the full connection layer included in the speech recognition model to be recognized to classify the first global feature and the second global feature, respectively.

[0162] The present disclosure also provides a speech recognition device. The speech recognition device can include a data acquisition module and a speech recognition module. Wherein:

[0163] The data collection module can be configured to collect to-be-recognized voice data; wherein the to-be-recognized voice data comprises at least one of near-field voice data and far-field voice data; and the far-field voice data comprises background noise and / or far-field reverberation.

[0164] The voice recognition module can be configured to input the to-be-recognized voice data into the trained voice recognition model to obtain a voice recognition result; wherein the trained voice recognition model is obtained by training the to-be-trained voice recognition model according to the voice recognition model training method.

[0165] The specific details of the modules in the voice recognition model training apparatus and the voice recognition apparatus have been described in the voice recognition model training method and the voice recognition method, and thus will not be described here again.

[0166] It should be noted that although several modules or units of the device for action execution are mentioned in the foregoing detailed description, such a division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units.

[0167] In addition, although the steps of the method in the present disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired result. In addition or alternatively, some steps can be omitted, multiple steps can be combined into one step, one step can be divided into multiple steps, etc.

[0168] In the exemplary embodiments of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0169] Those skilled in the art can understand that each aspect of the present disclosure can be implemented as a system, a method or a program product. Therefore, each aspect of the present disclosure can be embodied as a whole hardware embodiment, a whole software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "system" herein.

[0170] The electronic device 900 according to this embodiment of the present disclosure will be described below with reference to Figure 9 Figure 9 The displayed electronic device 900 is only an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0171] ​As shown in Figure 9 The components of electronic device 900 can include, but are not limited to, at least one processing unit 910, at least one storage unit 920, a bus 930 connecting the different system components, including the storage unit 920 and the processing unit 910, and a display unit 940.

[0172] The storage unit stores program codes which can be executed by the processing unit 910, so that the processing unit 910 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of the present specification. For example, the processing unit 910 can perform the steps shown in Figure 1 S110: calculating a target field voice data set according to a source field voice data set and a preset far-field reverberation impulse response set, and inputting the source field voice data set and the target field voice data set into a speech recognition model to be trained respectively to obtain a first text recognition result and a second text recognition result; S120: constructing a first loss function according to the first text recognition result and a speech annotation result included in the source field voice data set, and calculating a first covariance and a second covariance of the first text recognition result and the second text recognition result; S130: constructing a second loss function according to the first covariance and the second covariance, and constructing a target loss function according to the first loss function and the second loss function; S140: updating parameters included in the speech recognition model to be trained according to the target loss function to obtain a trained speech recognition model.

[0173] The storage unit 920 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 9201 and / or a cache memory 9202, and can further include a read-only memory (ROM) 9203.

[0174] The storage unit 920 can further include a program / utility 9204 having a set of program modules 9205, including but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or a combination thereof can include implementation of a network environment.

[0175] The bus 930 can represent one or more of several types of bus structures, including a storage unit bus or storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit or a local bus using any of a variety of bus architectures.

[0176] The electronic device 900 can also communicate with one or more external devices 1000 such as a keyboard or pointing device, a Bluetooth device, or a database, and can communicate with one or more devices enabling user interaction with the electronic device 900 (for example, a display, speakers, a haptic output device, or the like) and / or one or more devices enabling communication of the electronic device 900 with other computing devices (for example, a modem, communications port, or the like). Such communication can occur via Input / Output (I / O) interface 950. Still yet, the electronic device 900 can communicate with one or more networks, such as a local area network (LAN), a wide area network (WAN), and / or the Internet, through a network adapter 960. As depicted, the network adapter 960 communicates with the other components of the electronic device 900 via bus 930. It should be appreciated that the network adapter 960 and / or the other hardware and / or software components depicted in FIG. 9 can be utilized in conjunction with the electronic device 900, but have been omitted for the sake of brevity and clarity.

[0177] Through the above description of the embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, or the like) or a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0178] In the example embodiments of the present disclosure, a computer-readable storage medium having stored thereon a program product capable of implementing the above-mentioned method of the present disclosure is also provided. In some possible embodiments, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program codes for causing an end device to perform the steps described in the above-mentioned “example method” section according to various example embodiments of the present disclosure when the program product is run on the end device.

[0179] The program product for implementing the above-mentioned method according to the embodiments of the present disclosure can take the form of a portable compact disc read-only memory (CD-ROM) and include program codes, and can be run on an end device, such as a personal computer. However, the program product of the present disclosure is not limited to this, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in conjunction with an instruction execution system, device, or apparatus.

[0180] The program product can employ any combination of one or more computer-readable media. The computer-readable media can be a computer-readable storage medium or a computer-readable signal medium. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0181] The computer-readable signal medium can include a computer-readable storage medium that is configured to store and deliver a computer-readable program code. The computer-readable program code can be propagated as a computer-readable signal medium.

[0182] The program code embodied on the computer-readable media can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0183] The program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, C++, etc., and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0184] In addition, the above-described flowcharts are merely illustrative of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to limit the purpose. It is easily understood that the processes shown in the above-described flowcharts do not indicate or limit the time sequence of the processes. In addition, it is easily understood that the processes can be executed synchronously or asynchronously, for example, in a plurality of modules.

[0185] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features of the disclosure as set forth above. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

Claims

1. A method for training a speech recognition model, the method comprising: The application relates to a method for training a speech recognition model. According to a source field voice data set and a preset far-field reverberation impulse response set, a target field voice data set is calculated, and the source field voice data set and the target field voice data set are respectively input into a speech recognition model to be trained to obtain a first text recognition result and a second text recognition result. According to the first text recognition result and voice annotation results included in the source field voice data set, a first loss function is constructed, and first covariance and second covariance of the first text recognition result and the second text recognition result are calculated. According to the first covariance and the second covariance, a second loss function is constructed, and the second loss function and a preset balance parameter are multiplied, and the first loss function and the multiplication result are used to construct a target loss function; wherein the first loss function is used for training the speech recognition model to be trained, and the second loss function is used for aligning the source field voice data set and the target field voice data set. According to the target loss function, parameters included in the speech recognition model to be trained are updated to obtain a trained speech recognition model. 2.The method of Claim 1, wherein, According to a source field voice data set and a preset far-field reverberation impulse response set, a target field voice data set is calculated, comprising: extracting original audio data in the source field voice data set and impulse response samples in the preset far-field mixed impulse response set; wherein the original audio data is Chinese voice data; convolution calculation is performed on the original audio data and the impulse response samples to obtain target audio data; wherein the audio feature matrix of the original audio data and the target audio data is 81-dimensional mel spectrum features; According to a plurality of target audio data, the target field voice data set is generated. 3.The method of Claim 1, wherein, According to the first text recognition result and the voice annotation results included in the source field voice data set, a first loss function is constructed, comprising: calculating a path set of all paths constituting the voice annotation results included in the source field voice data set in an output matrix included in the first text recognition result; calculating the product of the probability of each element included in each path in the path set belonging to the corresponding position element in the voice annotation results; summing the products of the probabilities of all paths included in the path set, and taking the maximum value of the summation result as the target to construct the first loss function. 4.The method of Claim 1, wherein, Calculating first covariance and second covariance of the first text recognition result and the second text recognition result, comprising: According to a first audio duration of original audio data included in the source field voice data set and a second audio duration of target audio data included in the target field voice data set, a first feature time dimension length and a second feature time dimension length are calculated. According to the first text recognition result, the first feature time dimension length and a preset first vector, the first covariance is calculated. According to the second text recognition result, the second feature time dimension length and a preset second vector, the second covariance is calculated. 5.The method of Claim 1, wherein, constructing a second loss function according to the first covariance and the second covariance, comprising: obtaining a feature dimension length of a self-attention module included in the to-be-trained speech recognition model; calculating a matrix norm of the first covariance and the second covariance, and constructing the second loss function according to the matrix norm and the feature dimension length of the self-attention module. 6.The method of Claim 1, wherein, The preset balance parameter is used to balance the first loss function and the second loss function. 7.The method of Claim 1, wherein, inputting the source domain speech data set and the target domain speech data set into the to-be-trained speech recognition model respectively to obtain a first text recognition result and a second text recognition result, comprising: extracting first local features and second local features of original audio data in the source domain speech data set and target audio data in the target domain speech data set respectively by using a convolutional neural network included in the to-be-trained speech recognition model; calculating first global features of the original audio data and second global features of the target audio data according to the first local features and the second local features respectively by using a self-attention module included in the to-be-trained speech recognition model; classifying the first global features and the second global features respectively by using a full connection layer included in the to-be-trained speech recognition model to obtain the first text recognition result of the original audio data and the second text recognition result of the target audio data.

8. A voice recognition method characterized by, comprising: collecting to-be-recognized speech data; wherein the to-be-recognized speech data includes near-field speech data and / or far-field speech data; the far-field speech data includes background noise and / or far-field reverberation; inputting the to-be-recognized speech data into a trained speech recognition model to obtain a speech recognition result; wherein the trained speech recognition model is obtained by training a to-be-trained speech recognition model by the speech recognition model training method of any one of claims 1-7. 9.A device for training a speech recognition model, comprising: comprising: a first calculation module, configured to calculate a target domain speech data set according to a source domain speech data set and a preset far-field reverberation impulse response set, and input the source domain speech data set and the target domain speech data set into a to-be-trained speech recognition model respectively to obtain a first text recognition result and a second text recognition result; a second calculation module, configured to construct a first loss function according to a first text recognition result and a speech annotation result included in the source domain speech data set, and calculate a first covariance and a second covariance of the first text recognition result and the second text recognition result; a target loss function construction module, configured to construct a second loss function according to the first covariance and the second covariance, and perform a product operation on the second loss function and a preset balance parameter, and construct the target loss function according to the first loss function and the product operation result; wherein the first loss function is used to train the to-be-trained speech recognition model, and the second loss function is used to align the source domain speech data set and the target domain speech data set. The voice recognition model training module is configured to update parameters included in the voice recognition model to be trained according to the target loss function, and obtain a trained voice recognition model.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to implement the voice recognition model training method in any one of claims 1-7, and the voice recognition method in claim 8.

11. An electronic device, comprising: comprises: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to implement the voice recognition model training method in any one of claims 1-7, and the voice recognition method in claim 8 by executing the executable instructions.

Images