Speech recognition method, device and related equipment

Abstract

The present disclosure provides a speech recognition method, apparatus and related equipment, the method comprising: extracting the frequency spectrum feature matrix corresponding to the audio to be tested in different time intervals; inputting the frequency spectrum feature matrix into a pre-trained speech recognition model to obtain text as the recognition result of the audio to be tested; wherein the speech recognition model is obtained by training a plurality of sample frequency spectrum feature matrices and their corresponding sample speech recognition results and sample phoneme alignment results; and the sample speech recognition result is the recognition result obtained by the sample frequency spectrum feature matrix through the speech recognition model, and the sample phoneme alignment result is obtained by inputting the sample frequency spectrum feature matrix into a pre-trained acoustic model. The speech recognition model trained by the present method has a better convergence effect, and the present method uses the model to effectively improve the accuracy of speech recognition.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a speech recognition method, apparatus and related equipment. Background Art

[0002] In recent years, with the rapid development of deep learning technology, speech recognition models based on end-to-end deep neural networks have gradually developed into the mainstream technology in the current speech recognition field. Speech recognition models based on end-to-end deep neural networks can be trained through self-supervision based on contrastive learning. In model training, contrastive learning requires randomly selecting a certain proportion of fragments of a specific length for each sample for masking. The length parameter of the mask has a great influence on the final performance of the model.

[0003] However, in the prior art, the length parameter of the mask is only obtained through experiments or experience, and how to obtain an accurate length parameter of the mask needs to be solved urgently. Summary of the invention

[0004] In view of the problems existing in the prior art, the present invention provides a speech recognition method, apparatus and related equipment.

[0005] In a first aspect, the present disclosure provides a speech recognition method, comprising: extracting a spectral feature matrix corresponding to the audio to be tested in different time intervals; inputting the spectral feature matrix into a pre-trained speech recognition model to obtain a text as the recognition result of the audio to be tested; wherein the speech recognition model is obtained by training a plurality of sample spectral feature matrices and their corresponding sample speech recognition results and sample phoneme alignment results; and the sample speech recognition result is a recognition result obtained by the sample spectral feature matrix through the speech recognition model, and the sample phoneme alignment result is obtained by inputting the sample spectral feature matrix into a pre-trained acoustic model.

[0006] According to a speech recognition method provided by the present disclosure, the spectral feature matrix is ​​input into a pre-trained speech recognition model, and obtaining the text as the audio recognition result to be tested includes: obtaining a shallow feature sequence based on the spectral feature matrix; obtaining a deep feature sequence based on the shallow feature sequence; obtaining a sentence feature vector matrix based on the deep feature sequence; and obtaining the speech recognition result to be tested based on the sentence feature vector matrix.

[0007] According to a speech recognition method provided by the present disclosure, the speech recognition model is trained in the following manner: based on a sample audio feature matrix, a sample audio shallow feature sequence is obtained; sample frames in the sample audio shallow feature sequence are randomly extracted for masking to obtain a masked sample audio shallow feature sequence; based on the masked sample audio shallow feature sequence, a masked sample audio deep feature sequence is obtained; based on the masked sample audio deep feature sequence, a sample speech recognition result is obtained; differences are calculated based on a loss function; and based on the differences calculated by the loss function and a back propagation algorithm, the speech recognition network model parameters are adjusted.

[0008] In a second aspect, the present disclosure provides a training method for a speech recognition model, comprising: obtaining a shallow feature sequence of sample audio based on a sample audio feature matrix; randomly extracting sample frames from the shallow feature sequence of sample audio for masking to obtain a shallow feature sequence of masked sample audio; obtaining a deep feature sequence of masked sample audio based on the shallow feature sequence of masked sample audio; obtaining a sample speech recognition result based on the deep feature sequence of masked sample audio; calculating the difference based on a loss function; adjusting the parameters of the speech recognition network model based on the difference calculated by the loss function and a back propagation algorithm; wherein the difference is the sum of the contrast learning difference and the speech recognition difference.

[0009] According to a training method for a speech recognition model provided by the present disclosure, after the randomly extracted sample frames in the shallow feature sequence of the sample audio are masked, it also includes: according to the sample phoneme alignment result corresponding to the sample audio feature matrix, masking the sample frames in the adjacent frames corresponding to the sample frame of the anchor sample that are aligned with the anchor sample phoneme result; wherein the anchor sample is the masked sample frame feature.

[0010] According to a training method for a speech recognition model provided by the present disclosure, calculating the difference based on the loss function further includes: calculating the speech recognition difference between the sample speech recognition result and the sample text label based on the connection time classification loss function; calculating the contrastive learning difference between the anchor sample and the contrast sample based on the contrastive learning loss function; summing the speech recognition difference and the contrastive learning difference to obtain the difference; wherein the sample text label is a real text obtained in advance by manually listening to the transcription of the sample audio feature matrix; the contrast sample is a feature in the deep feature sequence of the contrast sample audio obtained by linearly transforming the shallow feature sequence of the sample audio.

[0011] According to a training method for a speech recognition model provided in the present disclosure, the comparison samples include: positive samples and negative samples; wherein the positive samples are features in the comparison sample audio deep feature sequence that are at the same position as the anchor sample; and the negative samples are features in the comparison sample audio deep feature sequence that are different from the anchor sample position and phoneme alignment result.

[0012] According to a training method for a speech recognition model provided by the present disclosure, the formula of the contrastive learning loss function is:

[0013]

[0014] Among them, M is the number of anchor samples, is the mth anchor sample, is the mth anchor sample positive sample, m∈[1,M], is the sth negative sample in the mth anchor sample, S is the number of negative samples of each anchor sample, s∈[1,S].

[0015] In a third aspect, the present disclosure also provides a speech recognition device, comprising an extraction module and an acquisition module; wherein the extraction module is used to extract the spectral feature matrix corresponding to the audio to be tested in different time intervals; the acquisition module is used to input the spectral feature matrix into a pre-trained speech recognition model to obtain text as the recognition result of the audio to be tested; wherein the speech recognition model is obtained by training multiple sample spectral feature matrices and their corresponding sample speech recognition results and sample phoneme alignment results; and the sample speech recognition result is the recognition result obtained by the sample spectral feature matrix through the speech recognition model, and the sample phoneme alignment result is obtained by inputting the sample spectral feature matrix into a pre-trained acoustic model.

[0016] In a fourth aspect, the present disclosure also provides a training device for a speech recognition model, comprising: an acquisition sequence module, a mask module, a self-attention module, an acquisition result module, a calculation module and an adjustment module; wherein the acquisition sequence module is used to acquire a shallow feature sequence of sample audio based on a sample audio feature matrix; the mask module is used to randomly extract sample frames from the shallow feature sequence of sample audio for masking, and acquire a shallow feature sequence of masked sample audio; the self-attention module is used to acquire a deep feature sequence of masked sample audio based on the shallow feature sequence of masked sample audio; the acquisition result module is used to acquire a sample speech recognition result based on the deep feature sequence of masked sample audio; the calculation module is used to calculate the difference based on the loss function; the adjustment module is used to adjust the parameters of the speech recognition network model based on the difference calculated by the loss function and the back propagation algorithm; wherein the difference is the sum of the contrast learning difference and the speech recognition difference.

[0017] In a fifth aspect, the present disclosure further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-described speech recognition methods are implemented.

[0018] In a sixth aspect, the present disclosure further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described speech recognition methods.

[0019] In a seventh aspect, the present disclosure further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any of the above-described speech recognition methods.

[0020] In an eighth aspect, the present disclosure further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the training method of the speech recognition model as described in any one of the above are implemented.

[0021] In a ninth aspect, the present disclosure further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for training a speech recognition model as described in any one of the above.

[0022] In a tenth aspect, the present disclosure further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the training method of the speech recognition model as described in any one of the above.

[0023] The speech recognition method, apparatus and related equipment provided by the present disclosure obtain a masked sample audio shallow feature sequence by randomly extracting sample frames from a sample audio shallow feature sequence obtained based on a sample audio feature matrix; obtain a masked sample audio deep feature sequence based on the masked sample audio shallow feature sequence; obtain a sample speech recognition result based on the masked sample audio deep feature sequence; calculate the difference based on a loss function; wherein the difference is the sum of the contrastive learning difference and the speech recognition difference; adjust the speech recognition network model parameters based on the difference calculated by the loss function and the back propagation algorithm, and train the speech recognition model.

[0024] The sample audio feature matrix used in this method is supervised data. The difference of the sample audio feature matrix is ​​calculated based on the loss function. The difference is the sum of the contrastive learning difference and the speech recognition difference. That is, the contrastive learning task and the speech recognition task are jointly trained, so that the training steps are combined into one. Only one experimental design is needed to train a speech recognition model that meets the requirements. Taking the selection of hyperparameters such as learning rate and number of iterations as an example, the model pre-training and model fine-tuning of the prior art need to design hyperparameters according to the needs, while the training method of the speech recognition model provided by the present disclosure only needs to design hyperparameters once before the start of training, which simplifies the experimental difficulty. At the same time, the accuracy of the model of the prior art mainly depends on the model pre-training to train a large amount of unlabeled data. The sample audio feature matrix used in this method is supervised data, which can improve the accuracy of the trained model.

[0025] The speech recognition method, apparatus and related equipment provided by the present disclosure extract the spectral feature matrix corresponding to the audio to be tested in different time intervals; input the spectral feature matrix into a pre-trained speech recognition model to obtain text as the recognition result of the audio to be tested. The speech recognition model is obtained by training a plurality of sample spectral feature matrices and their corresponding sample speech recognition results and sample phoneme alignment results; and the sample speech recognition result is the recognition result obtained by the sample spectral feature matrix through the speech recognition model, and the sample phoneme alignment result is obtained by inputting the sample spectral feature matrix into a pre-trained acoustic model. The sample audio feature matrix used for training the pre-trained speech recognition model adopted in the method is supervised data, and the supervised data is trained throughout the entire training process, which can improve the accuracy of the output results of the trained model. Therefore, the results output by this method are more accurate than the models trained by the prior art, which improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the present disclosure or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 It is a flowchart of a method for training a speech recognition model provided by the present disclosure;

[0028] Figure 2 is a structural schematic diagram of a training device for a speech recognition model provided by the present disclosure;

[0029] Figure 3 is a schematic diagram of the structure of an electronic device provided by the present disclosure;

[0030] Figure 4 is a flow chart of the speech recognition method provided by the present disclosure;

[0031] Figure 5 is a structural schematic diagram of a speech recognition device provided by the present disclosure;

[0032] Figure 6 It is a schematic diagram of the logical structure of the neural network model based on the training method of the speech recognition model provided by the present disclosure;

[0033] Figure 7 It is a schematic diagram of the structure of an electronic device provided by the present disclosure. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present disclosure.

[0035] In recent years, with the rapid development of deep learning technology, automatic speech recognition technology (Automatic Speech Recognition: ASR) based on end-to-end deep neural networks has gradually developed into the mainstream technology in the current field of speech recognition, and has been widely used in many practical scenarios such as e-commerce, logistics, and finance. The end-to-end ASR model can directly convert a piece of audio data at the input end into a text recognition result at the output end, which is easy to train, reason and optimize the model. Compared with the traditional acoustic model based on GMM-HMM (Gaussian Mixed Model-Hidden Markov Model), the end-to-end deep neural network model has a stronger representation ability for the acoustic features of speech data, which is the key to improving the performance of the speech recognition model.

[0036] Speech recognition technology based on contrastive learning (CL) uses a self-supervised training model to directly use speech data as supervisory information to learn the feature representation of speech data, which is used to improve the performance of downstream speech recognition tasks. In recent years, representative works using contrastive learning for speech recognition model training include wav2vec2.0, etc. This type of model training mainly includes two steps:

[0037] The first step is model pre-training: using the contrastive learning loss function, the model is trained on the basis of unlabeled audio data to obtain a pre-trained model for audio feature representation. In this step, the specific implementation method is to first randomly select a certain proportion of fragments of a specific length for each sample in the shallow representation of the neural network, record them as anchor samples, and mask them; secondly, in the deep representation, the features at the masked moment are recorded as positive samples of the anchor samples, and the features at other moments are recorded as negative samples of the anchor samples, and contrastive learning training is performed. By updating the model parameters, the distance between the anchor samples and the positive samples is reduced, and the distance between the anchor samples and the negative samples is increased, so as to obtain a speech data feature representation model.

[0038] The second step is model fine-tuning training: based on the pre-trained model represented by the audio features obtained in the first step, the loss function of the speech recognition task is used, and the manually annotated text audio data is used as supervision information to fine-tune the pre-trained model parameters, and finally a speech recognition model is obtained.

[0039] Combine the following Figure 1 The training method of the speech recognition model of the embodiment of the present disclosure includes:

[0040] Step S101, obtaining a shallow feature sequence of the sample audio based on the sample audio feature matrix;

[0041] Step S103, randomly extracting a sample frame from the sample audio shallow feature sequence for masking, and obtaining a masked sample audio shallow feature sequence;

[0042] Step S105, obtaining a masked sample audio deep feature sequence based on the masked sample audio shallow feature sequence;

[0043] Step S107, obtaining a sample speech recognition result based on the masked sample audio deep feature sequence;

[0044] Step S109, calculating the difference based on the loss function; wherein the difference is the sum of the contrastive learning difference and the speech recognition difference;

[0045] Step S111: Adjust the speech recognition network model parameters based on the difference calculated by the loss function and the back propagation algorithm.

[0046] The training method of the speech recognition model provided by the present disclosure is to obtain the masked sample audio shallow feature sequence by randomly extracting the sample frames in the sample audio shallow feature sequence obtained based on the sample audio feature matrix; obtain the masked sample audio deep feature sequence based on the masked sample audio shallow feature sequence; obtain the sample speech recognition result based on the masked sample audio deep feature sequence; calculate the difference based on the loss function; wherein the difference is the sum of the contrastive learning difference and the speech recognition difference; adjust the speech recognition network model parameters based on the difference calculated by the loss function and the back propagation algorithm. The sample audio feature matrix used in the method is supervised data, and the difference of the sample audio feature matrix is ​​calculated based on the loss function, and the difference is the sum of the contrastive learning difference and the speech recognition difference, that is, the contrastive learning task and the speech recognition task are jointly trained, so that the training steps are combined into one, and only one experimental design is needed to train a speech recognition model that meets the requirements. Taking the selection of hyperparameters such as learning rate and iteration steps as an example, the model pre-training and model fine-tuning of the prior art need to design hyperparameters according to the needs, while the training method of the speech recognition model provided by the present disclosure only needs to design hyperparameters once before the start of training, which simplifies the experimental difficulty. At the same time, the accuracy of the model in the existing technology mainly relies on model pre-training on a large amount of unlabeled data. The sample audio feature matrix used in this method is supervised data, which can improve the accuracy of the trained model.

[0047] Combine the following Figure 1 Describe each step in detail.

[0048] Step S101, obtaining a shallow feature sequence of the sample audio based on the sample audio feature matrix;

[0049] Specifically, the sample audio feature matrix is ​​an 80-dimensional Mel-spectrogram feature, and the sample audio feature matrix is ​​converted by frame by frame through the sample audio. Among them, the duration of each frame of the sample audio is 20ms, and the step length is 10ms; the sample audio is sampled at an audio sampling rate of 16kHz. Optionally, the sample audio feature matrix is ​​input into the convolutional neural network layer to obtain the shallow feature sequence of the sample audio. In a specific application, the sample audio feature matrix is ​​pre-stored in a training data set, and the training data set is denoted as D={x i ,y i |i∈[1,N]}, where x i is the sample audio feature matrix of the i-th training data sample, y i is the sample audio feature matrix x i The corresponding text annotation results, N is the number of samples in the training data set. By using the traditional GMM-HMM to build an acoustic model, we can obtain the feature matrix x of each sample. i The phoneme alignment result a i , that is, xi The phoneme results corresponding to each frame in the image are helpful for the subsequent phoneme-based masking strategy, which improves the model's ability to represent phonemes, that is, the speech recognition performance. At the same time, the phoneme alignment results can be used to improve the accuracy of positive and negative sample selection of anchor samples, thereby improving the convergence effect of the model.

[0050] Step S103, randomly extracting a sample frame from the sample audio shallow feature sequence for masking, and obtaining a masked sample audio shallow feature sequence;

[0051] Specifically, randomly extract the sample frames in the sample audio shallow feature sequence and mask them into the sample audio shallow feature sequence e of a given sample i , from e i In the example above, p percent of sample frames are randomly selected as anchor samples. In the specific implementation, the random masking probability is p = 6.5. In order to further improve the model effect, the features of 2 or 3 adjacent phonemes can be randomly masked on this basis.

[0052] Step S105, obtaining a masked sample audio deep feature sequence based on the masked sample audio shallow feature sequence;

[0053] Specifically, based on the shallow feature sequence of the masked sample audio, obtaining the deep feature sequence of the masked sample audio can be inputting the shallow feature sequence of the masked sample audio into the self-attention layer for calculation to obtain the deep feature sequence of the masked sample audio.

[0054] Step S107, obtaining a sample speech recognition result based on the masked sample audio deep feature sequence;

[0055] Specifically, the sample speech recognition results are used to compare with the text corresponding to the sample audio feature matrix that was previously transcribed by manual listening.

[0056] Step S109, calculating the difference based on the loss function; wherein the difference is the sum of the contrastive learning difference and the speech recognition difference;

[0057] Specifically, calculating the difference based on the loss function specifically includes calculating the contrastive learning difference and the speech recognition difference.

[0058] Step S111: Adjust the speech recognition network model parameters based on the difference calculated by the loss function and the back propagation algorithm.

[0059] Specifically, the back propagation algorithm can use the existing Adam optimization algorithm. When the difference calculated by the loss function meets the threshold or converges, the parameters of the speech recognition network model are adjusted, that is, the speech recognition network model training is completed.

[0060] In an optional embodiment, after randomly extracting sample frames from the sample audio shallow feature sequence for masking, step S103 further includes:

[0061] According to the sample phoneme alignment result corresponding to the sample audio feature matrix, the sample frames in the adjacent frames corresponding to the sample frame of the anchor sample and the sample frames aligned with the anchor sample phoneme result are masked.

[0062] Specifically, the anchor sample is the masked sample frame feature. The sample phoneme alignment result is obtained by inputting the sample spectrum feature matrix into the pre-trained acoustic model. The acoustic model here can be a different hidden Markov model generated by fitting the distribution of the speech signal using a mixed Gaussian model or a mixed Laplace model and combining it with vector clustering. There is no restriction. Here we use the Gaussian mixture model-hidden Markov model.

[0063] The training method of the speech recognition model provided by the embodiment of the present disclosure masks the sample frames of the anchor sample phoneme alignment results in the adjacent frames of the sample frame corresponding to the anchor sample according to the phoneme alignment results corresponding to the sample audio feature matrix, that is, the masking is performed in units of phonemes. The masking strategy is based on the phonemes, and the parameter selection takes into account the length of the speech recognition acoustic modeling unit to obtain a suitable parameter length. The model training will not be difficult due to the loss of more shallow features due to the mask being too long, nor will the difficulty of model comparison learning be reduced due to the mask being too short. The model trained in this way has better performance.

[0064] In an optional embodiment, step S109 further includes:

[0065] Step S1091, calculating the speech recognition difference between the sample speech recognition result and the sample text label based on the connection time classification loss function;

[0066] Step S1093, calculating the contrastive learning difference between the anchor sample and the contrast sample based on the contrastive learning loss function;

[0067] Step S1095: sum the speech recognition difference and the contrastive learning difference to obtain the difference.

[0068] Specifically, the sample text label is the real text obtained by manually transcribing the sample audio feature matrix; the comparison sample is the feature in the deep feature sequence of the comparison sample audio obtained by linearly transforming the shallow feature sequence of the sample audio.

[0069] In one embodiment, the comparison samples include positive samples and negative samples; wherein the positive samples are features in the comparison sample audio deep feature sequence that are at the same position as the anchor sample; and the negative samples are features in the comparison sample audio deep feature sequence that are different from the anchor sample position and phoneme alignment result.

[0070] In the disclosed embodiment, the negative sample selection has features that are different from the anchor sample position and phoneme alignment results, which improves the accuracy of the positive sample and negative sample selection of the anchor sample, thereby improving the convergence effect of the model. In this way, when performing comparative learning, the distance between the anchor sample and the positive sample is reduced, and the distance between the anchor sample and the negative sample is increased. There will be no situation where the anchor sample and the negative sample are the same phoneme in different positions but the distance between the anchor sample and the negative sample needs to be increased, thereby improving the convergence effect of the model.

[0071] In an optional embodiment, the formula of the contrastive learning loss function is:

[0072]

[0073] Among them, M is the number of anchor samples, is the mth anchor sample, is the mth anchor sample positive sample, m∈[1,M], is the sth negative sample in the mth anchor sample, S is the number of negative samples for each anchor sample, s∈[1,S]. In the specific implementation, the number of anchor samples is M=10, the corresponding number of negative samples is set to S=100, and the scale factor T=10.

[0074] The formula of the sim function is:

[0075] sim(a,b)=a T b / (||a||||b||)

[0076] In a specific application, the training method of the speech recognition model provided in the embodiment of the present disclosure is implemented based on a neural network model. Figure 6 The neural network model mainly includes two parts: the acoustic network and the contrastive learning network. The acoustic model includes 3 convolutional neural network layers, 10 self-attention layers, and 2 fully connected layers in sequence; the contrastive learning network includes a mask layer and a linear transformation layer.

[0077] Table 1 compares the accuracy of speech recognition obtained by the speech recognition model training method provided by the embodiment of the present disclosure, the common ASR training method (baseline method), and the model obtained by using a fixed mask length (set to 7 for best results).

[0078] Table 1. Comparison of the accuracy of the speech recognition model training method proposed in the embodiment of the present disclosure with other methods

[0079]

[0080] The training device for the speech recognition model provided in the embodiment of the present disclosure is described below. The training device for the speech recognition model described below and the training method for the speech recognition model described above can refer to each other.

[0081] Combine the following Figure 2 The training device of the speech recognition model of the embodiment of the present disclosure includes: an acquisition sequence module 20, a mask module 22, a self-attention module 24, an acquisition result module 26, a calculation module 28 and an adjustment module 29; wherein the acquisition sequence module 20 is used to acquire a shallow feature sequence of sample audio based on a sample audio feature matrix; the mask module 22 is used to randomly extract sample frames in the shallow feature sequence of sample audio for masking, and acquire a shallow feature sequence of masked sample audio; the self-attention module 24 is used to acquire a deep feature sequence of masked sample audio based on the shallow feature sequence of masked sample audio; the acquisition result module 26 is used to acquire a sample speech recognition result based on the deep feature sequence of masked sample audio; the calculation module 28 is used to calculate the difference based on the loss function; wherein the difference is the sum of the contrastive learning difference and the speech recognition difference; the adjustment module 29 is used to adjust the speech recognition network model parameters based on the difference calculated by the loss function and the back propagation algorithm.

[0082] The training device of the speech recognition model provided by the present disclosure randomly extracts sample frames from the shallow feature sequence of the sample audio obtained by the sequence acquisition module 20 based on the sample audio feature matrix for masking through the mask module 22, so as to obtain the shallow feature sequence of the masked sample audio; the self-attention module 24 obtains the deep feature sequence of the masked sample audio based on the shallow feature sequence of the masked sample audio; the result acquisition module 26 obtains the sample speech recognition result based on the deep feature sequence of the masked sample audio; the calculation module 28 calculates the difference based on the loss function; wherein the difference is the sum of the contrastive learning difference and the speech recognition difference; the adjustment module 29 adjusts the parameters of the speech recognition network model based on the difference calculated by the loss function and the back propagation algorithm. The sample audio feature matrix used by the device is supervised data. The difference of the sample audio feature matrix is ​​calculated based on the loss function. The difference is the sum of the contrastive learning difference and the speech recognition difference. That is, the contrastive learning task and the speech recognition task are jointly trained, so that the training steps are combined into one. Only one experimental design is needed to train a speech recognition model that meets the requirements. Taking the selection of hyperparameters such as learning rate and number of iterations as an example, the model pre-training and model fine-tuning of the prior art need to design hyperparameters according to the needs, while the training device of the speech recognition model provided by the present disclosure only needs to design hyperparameters once before the start of training, which simplifies the experimental difficulty. At the same time, the accuracy of the model of the prior art mainly depends on the model pre-training to train a large amount of unlabeled data. The sample audio feature matrix used in the device is supervised data, which can improve the accuracy of the trained model.

[0083] Combine the following Figure 2 Here we describe each module in detail.

[0084] Specifically, the sample audio feature matrix in the acquisition sequence module 20 is an 80-dimensional Mel-spectrogram feature, and the sample audio feature matrix is ​​converted by frame by sample audio. Among them, the duration of each frame of the sample audio is 20ms, and the step length is 10ms; the sample audio is sampled at an audio sampling rate of 16kHz. Optionally, the sample audio feature matrix is ​​input into the convolutional neural network layer to obtain a shallow feature sequence of the sample audio.

[0085] Specifically, the mask module 22 is used to randomly extract a sample frame in the sample audio shallow feature sequence and mask it into a sample audio shallow feature sequence e of a given sample. i , from e i In the example above, p percent of sample frames are randomly selected as anchor samples. In the specific implementation, the random masking probability is p = 6.5. In order to further improve the model effect, the features of 2 or 3 adjacent phonemes can be randomly masked on this basis.

[0086] Specifically, the self-attention module 24 is used to obtain the deep feature sequence of the masked sample audio based on the shallow feature sequence of the masked sample audio, which can be used to input the shallow feature sequence of the masked sample audio into the self-attention layer for calculation to obtain the deep feature sequence of the masked sample audio.

[0087] Specifically, the back propagation algorithm in the adjustment module 29 may adopt the existing Adam optimization algorithm. When the difference calculated by the loss function meets the threshold or converges, the adjustment of the speech recognition network model parameters is completed, that is, the speech recognition network model training is completed.

[0088] In an optional embodiment, the mask module 22 is further configured to:

[0089] After randomly extracting sample frames from the shallow feature sequence of the sample audio and performing masking, the sample frames in adjacent frames corresponding to the sample frame of the anchor sample and having the phoneme alignment result with the anchor sample are masked according to the phoneme alignment result corresponding to the sample audio feature matrix.

[0090] Specifically, the anchor sample is the masked sample frame feature.

[0091] The training device of the speech recognition model provided by the embodiment of the present disclosure masks the sample frames of the anchor sample phoneme alignment results in the adjacent frames of the sample frame corresponding to the anchor sample according to the phoneme alignment results corresponding to the sample audio feature matrix, that is, the masking is performed in units of phonemes. The masking strategy is based on the phonemes, and the parameter selection takes into account the length of the speech recognition acoustic modeling unit to obtain a suitable parameter length. The model training will not be difficult due to the loss of more shallow features due to the mask being too long, nor will the difficulty of model comparison learning be reduced due to the mask being too short. The performance of the model trained in this way is better.

[0092] In an optional embodiment, the calculation module 28 further includes:

[0093] A recognition difference calculation unit, used for calculating the speech recognition difference between the sample speech recognition result and the sample text label based on the connection time classification loss function;

[0094] A learning difference calculation unit, used for calculating the contrastive learning difference between the anchor sample and the contrast sample based on the contrastive learning loss function;

[0095] The summing unit is used for summing the speech recognition difference and the contrastive learning difference to obtain the difference.

[0096] Specifically, the sample text label is the real text obtained by manually transcribing the sample audio feature matrix; the comparison sample is the feature in the deep feature sequence of the comparison sample audio obtained by linearly transforming the shallow feature sequence of the sample audio.

[0097] In one embodiment, the comparison samples in the calculation learning difference unit include positive samples and negative samples; wherein the positive sample is a feature in the deep feature sequence of the comparison sample audio that is at the same position as the anchor sample; and the negative sample is a feature in the deep feature sequence of the comparison sample audio that is different from the anchor sample position and phoneme alignment result.

[0098] In the disclosed embodiment, the negative sample selection has features that are different from the anchor sample position and phoneme alignment results, which improves the accuracy of the positive sample and negative sample selection of the anchor sample, thereby improving the convergence effect of the model. In this way, when performing comparative learning, the distance between the anchor sample and the positive sample is reduced, and the distance between the anchor sample and the negative sample is increased. There will be no situation where the anchor sample and the negative sample are the same phoneme in different positions but the distance between the anchor sample and the negative sample needs to be increased, thereby improving the convergence effect of the model.

[0099] In an optional embodiment, the formula of the contrastive learning loss function is:

[0100]

[0101] Among them, M is the number of anchor samples, is the mth anchor sample, is the mth anchor sample positive sample, m∈[1,M], is the sth negative sample in the mth anchor sample, S is the number of negative samples for each anchor sample, s∈[1,S]. In the specific implementation, the number of anchor samples is M=10, the corresponding number of negative samples is set to S=100, and the scale factor T=10.

[0102] The formula of the sim function is:

[0103] sim(a,b)=a Tb / (||a||||b||)

[0104] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the training method of the speech recognition model.

[0105] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the embodiment of the present disclosure is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0106] On the other hand, the present disclosure also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the training method of the speech recognition model provided by the above-mentioned methods.

[0107] On the other hand, the present disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the training methods of the speech recognition models provided above.

[0108] The speech recognition model trained based on the above method can be used in a speech recognition method provided in the embodiment of the present disclosure, referring to Figure 4 , the method comprising:

[0109] Step S401, extracting the frequency spectrum feature matrix corresponding to the audio to be tested in different time intervals;

[0110] Specifically, the extraction method is based on the preset audio frame duration of 20ms, a step size of 10ms, and a spectral feature matrix of 80-dimensional Mel-spectrogram features. The spectral feature matrix is ​​converted by frame through the audio to be tested.

[0111] Step S403: input the frequency spectrum feature matrix into a pre-trained speech recognition model to obtain text as a result of the audio recognition to be tested.

[0112] The speech recognition method provided by the embodiment of the present disclosure obtains text as the recognition result of the audio to be tested by inputting the spectral feature matrix corresponding to the extracted audio to be tested in different time intervals into a pre-trained speech recognition model. The pre-trained speech recognition model here is a model trained in advance according to the above method, and the output result has higher accuracy than the model trained by the prior art, thereby improving the user experience.

[0113] In an optional embodiment, step S403 specifically includes:

[0114] Step S4031, obtaining a shallow feature sequence based on the spectrum feature matrix;

[0115] Specifically, the shallow feature sequence can be obtained by inputting the spectral feature matrix into the convolutional neural network layer.

[0116] Step S4033: based on the shallow feature sequence, obtain a deep feature sequence;

[0117] Specifically, the deep feature sequence can be obtained by inputting the shallow feature sequence into the self-attention layer.

[0118] Step S4035: Obtain a sentence feature vector matrix based on the deep feature sequence;

[0119] Specifically, the sentence feature vector matrix can be obtained by inputting the deep feature sequence into the fully connected network layer.

[0120] Step S4037: Obtain the speech recognition result to be tested based on the sentence feature vector matrix.

[0121] Specifically, the sentence feature vector matrix can be input into the CTC decoder, and the speech recognition output text with the highest decoding probability can be selected as the speech recognition result to be tested. The CTC decoder parameters are related to the connection time classification loss function during training.

[0122] In a specific implementation, the speech recognition model may include a 3-layer convolutional neural network, a 10-layer self-attention module, and a 2-layer fully connected layer.

[0123] The training device for the speech recognition model provided in the embodiment of the present disclosure is described below. The training device for the speech recognition model described below and the training method for the speech recognition model described above can refer to each other.

[0124] Reference Figure 5 The embodiment of the present disclosure provides a speech recognition device, comprising: an extraction module 52 and an acquisition module 54. The extraction module 52 is used to extract the frequency spectrum feature matrix corresponding to the audio to be tested in different time intervals; the acquisition module 54 is used to input the frequency spectrum feature matrix into a pre-trained speech recognition model to obtain a text as the recognition result of the audio to be tested;

[0125] Specifically, the extraction method in the extraction module 52 is based on the preset audio frame duration of 20ms, the step length of 10ms, the spectral feature matrix of 80-dimensional Mel-spectrogram features, and the spectral feature matrix is ​​converted by the audio to be tested frame by frame.

[0126] The speech recognition device provided by the embodiment of the present disclosure inputs the spectral feature matrix corresponding to the audio to be tested extracted by the extraction module 52 in different time intervals into a pre-trained speech recognition model through the acquisition module 54, and obtains text as the recognition result of the audio to be tested. The pre-trained speech recognition model here is a model trained in advance according to the above method, and the output result has higher accuracy than the model trained by the prior art, thereby improving the user experience.

[0127] In an optional embodiment, the acquisition module 54 specifically includes:

[0128] Obtaining a shallow sequence unit, used to obtain a shallow feature sequence based on a spectrum feature matrix;

[0129] Specifically, the shallow feature sequence can be obtained by inputting the spectral feature matrix into the convolutional neural network layer.

[0130] A deep feature sequence acquisition unit is used to acquire a deep feature sequence based on a shallow feature sequence;

[0131] Specifically, the deep feature sequence can be obtained by inputting the shallow feature sequence into the self-attention layer.

[0132] Obtaining a sentence feature unit, for obtaining a sentence feature vector matrix based on a deep feature sequence;

[0133] Specifically, the sentence feature vector matrix can be obtained by inputting the deep feature sequence into the fully connected network layer.

[0134] The recognition result obtaining unit is used to obtain the speech recognition result to be tested based on the sentence feature vector matrix.

[0135] Specifically, the sentence feature vector matrix can be input into the CTC decoder, and the speech recognition output text with the highest decoding probability can be selected as the speech recognition result to be tested. The CTC decoder parameters are related to the connection time classification loss function during training.

[0136] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the speech recognition method.

[0137] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the embodiment of the present disclosure is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0138] On the other hand, the present disclosure also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the speech recognition methods provided by the above methods.

[0139] On the other hand, the present disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the above-mentioned speech recognition methods.

[0140] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0141] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A speech recognition method, characterized in that: include: Extract the frequency spectrum feature matrix corresponding to the audio to be tested in different time intervals; Inputting the frequency spectrum feature matrix into a pre-trained speech recognition model to obtain text as the audio recognition result to be tested; The speech recognition model is obtained by training a plurality of sample spectrum feature matrices and their corresponding sample speech recognition results and sample phoneme alignment results; Furthermore, the sample speech recognition result is a recognition result obtained by using the sample spectrum feature matrix through the speech recognition model, and the sample phoneme alignment result is obtained by inputting the sample spectrum feature matrix into a pre-trained acoustic model; The speech recognition model is trained in the following way: Based on the sample audio feature matrix, obtain the shallow feature sequence of the sample audio; Randomly extracting sample frames from the sample audio shallow feature sequence for masking to obtain a masked sample audio shallow feature sequence; Based on the masked sample audio shallow feature sequence, a masked sample audio deep feature sequence is obtained; Based on the deep feature sequence of the masked sample audio, obtain the sample speech recognition result; Calculate the difference based on the loss function; Adjusting the parameters of the speech recognition network model based on the difference calculated by the loss function and the back propagation algorithm; wherein the difference is the sum of the contrastive learning difference and the speech recognition difference; The randomly extracting sample frames from the shallow feature sequence of the sample audio and performing masking further includes: According to the sample phoneme alignment result corresponding to the sample audio feature matrix, masking the sample frame in the adjacent frames corresponding to the sample frame of the anchor sample and the sample frame aligned with the anchor sample phoneme result; The anchor sample is a masked sample frame feature.

2. The speech recognition method according to claim 1, characterized in that: Inputting the spectrum feature matrix into a pre-trained speech recognition model, and obtaining text as the audio recognition result to be tested comprises: Based on the spectrum feature matrix, obtaining a shallow feature sequence; Based on the shallow feature sequence, a deep feature sequence is obtained; Based on the deep feature sequence, obtaining a sentence feature vector matrix; Based on the sentence feature vector matrix, a speech recognition result to be tested is obtained.

3. A method for training a speech recognition model, comprising: Based on the sample audio feature matrix, obtain the shallow feature sequence of the sample audio; Randomly extracting sample frames from the sample audio shallow feature sequence for masking to obtain a masked sample audio shallow feature sequence; Based on the masked sample audio shallow feature sequence, a masked sample audio deep feature sequence is obtained; Based on the deep feature sequence of the masked sample audio, obtain the sample speech recognition result; Calculate the difference based on the loss function; Adjusting the parameters of the speech recognition network model based on the difference calculated by the loss function and the back propagation algorithm; wherein the difference is the sum of the contrastive learning difference and the speech recognition difference; The randomly extracting sample frames from the shallow feature sequence of the sample audio and performing masking further includes: According to the sample phoneme alignment result corresponding to the sample audio feature matrix, masking the sample frame in the adjacent frames corresponding to the sample frame of the anchor sample and the sample frame aligned with the anchor sample phoneme result; The anchor sample is a masked sample frame feature.

4. The method for training a speech recognition model according to claim 3, characterized in that: Calculating the difference based on the loss function further includes: Calculating the speech recognition difference between the sample speech recognition result and the sample text label based on the connection time classification loss function; Calculating the contrastive learning difference between the anchor sample and the contrast sample based on the contrastive learning loss function; summing the speech recognition difference and the contrastive learning difference to obtain a difference; Among them, the sample text label is a real text obtained in advance by manually listening to the sample audio feature matrix transcription; the comparison sample is the feature in the deep feature sequence of the comparison sample audio obtained by linearly transforming the shallow feature sequence of the sample audio.

5. The method for training a speech recognition model according to claim 4, characterized in that: The comparison samples include: Positive and negative samples; Wherein, the positive sample is a feature in the deep feature sequence of the comparison sample audio having the same position as the anchor sample; The negative sample is a feature in the deep feature sequence of the comparison sample audio that is different from the anchor sample position and phoneme alignment result.

6. The method for training a speech recognition model according to claim 5, characterized in that: The formula of the contrastive learning loss function is: Among them, M is the number of anchor samples, is the mth anchor sample, is the mth anchor sample positive sample, m∈[1,M], is the sth negative sample in the mth anchor sample, S is the number of negative samples for each anchor sample, s∈[1,S], and T is the scale coefficient.

7. A speech recognition device, characterized in that: include: An extraction module is used to extract the frequency spectrum feature matrix corresponding to the audio to be tested in different time intervals; An acquisition module, used for inputting the spectrum feature matrix into a pre-trained speech recognition model to acquire text as the audio recognition result to be tested; The speech recognition model is obtained by training a plurality of sample spectrum feature matrices and their corresponding sample speech recognition results and sample phoneme alignment results; Furthermore, the sample speech recognition result is a recognition result obtained by using the sample spectrum feature matrix through the speech recognition model, and the sample phoneme alignment result is obtained by inputting the sample spectrum feature matrix into a pre-trained acoustic model; The speech recognition device also includes: A sequence acquisition module is used to acquire a shallow feature sequence of a sample audio based on a sample audio feature matrix; A mask module, used for randomly extracting sample frames from the sample audio shallow feature sequence for masking, and obtaining a masked sample audio shallow feature sequence; A self-attention module is used to obtain a deep feature sequence of the masked sample audio based on a shallow feature sequence of the masked sample audio; The result acquisition module is used to obtain the sample speech recognition result based on the deep feature sequence of the masked sample audio; A calculation module for calculating the difference based on the loss function; An adjustment module, used to adjust the parameters of the speech recognition network model based on the difference calculated by the loss function and the back propagation algorithm; wherein the difference is the sum of the contrastive learning difference and the speech recognition difference; The mask module is further used for: After randomly extracting sample frames from the shallow feature sequence of the sample audio and performing masking, masking the sample frames in the adjacent frames corresponding to the sample frame of the anchor sample and the sample frames that are aligned with the anchor sample phonemes according to the sample phoneme alignment result corresponding to the sample audio feature matrix; The anchor sample is a masked sample frame feature.

8. A training device for a speech recognition model, comprising: A sequence acquisition module is used to acquire a shallow feature sequence of a sample audio based on a sample audio feature matrix; A mask module, used for randomly extracting sample frames from the sample audio shallow feature sequence for masking, and obtaining a masked sample audio shallow feature sequence; A self-attention module is used to obtain a deep feature sequence of the masked sample audio based on a shallow feature sequence of the masked sample audio; The result acquisition module is used to obtain the sample speech recognition result based on the deep feature sequence of the masked sample audio; A calculation module for calculating the difference based on the loss function; An adjustment module, used to adjust the parameters of the speech recognition network model based on the difference calculated by the loss function and the back propagation algorithm; Wherein, the difference is the sum of the contrastive learning difference and the speech recognition difference; The mask module is further used for: After randomly extracting sample frames from the shallow feature sequence of the sample audio and performing masking, masking the sample frames in the adjacent frames corresponding to the sample frame of the anchor sample and the phoneme alignment result of the anchor sample is performed according to the phoneme alignment result corresponding to the sample audio feature matrix; The anchor sample is a masked sample frame feature.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the speech recognition method as claimed in claim 1 or 2 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the speech recognition method as claimed in claim 1 or 2 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the speech recognition method as claimed in claim 1 or 2 are implemented.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the training method of the speech recognition model as described in any one of claims 3 to 6 are implemented.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for training a speech recognition model as described in any one of claims 3 to 6 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for training a speech recognition model as described in any one of claims 3 to 6 are implemented.

Images