Speech recognition method and device and storage medium

Abstract

The invention discloses a speech recognition method and device and a storage medium, and the method comprises the steps: obtaining two-dimensional data features distributed on a time axis and a frequency axis, inputting the two-dimensional data features into an acoustic model, carrying out the multi-time feature extraction and aggregation, and outputting a speech recognition sequence result; wherein the acoustic model is formed by connecting a three-layer convolutional neural network, an eight-layer factorized time delay neural network, a multi-head attention mechanism, a two-layer convolutional neural network, a five-layer factorized time delay neural network, a multi-head attention mechanism, a two-layer convolutional neural network and an output layer; taking the output of each layer as the input of the next layer for training until the output layer outputs a sequence result; and converting the sequence result into text data through a decoder to obtain a speech recognition result. According to the embodiment of the invention, by improving the acoustic model structure, redundant information can be filtered out, key features can be extracted, and layer-by-layer accumulation of errors is prevented through multi-layer aggregation extraction, so that the speech recognition effect is improved.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a speech recognition method, device and storage medium. Background Art

[0002] In the prior art, two types of neural networks are usually combined, and the same neural network is superimposed in multiple layers, combined with the attention mechanism to build a hybrid acoustic model. This hybrid acoustic model, especially the baseline acoustic model with a structure of 3-layer CNN (Convolutional Neural Network) + 13-layer TDNNF (Factorized TDNN) + attention mechanism, has a neural network that processes complex data. The fixed length of the hidden layer limits the model's capture and expression of rich information. As the network depth increases, information may be lost or confused, and the resulting errors will accumulate layer by layer, resulting in degradation of the acoustic model recognition performance, thereby reducing the accuracy of the speech recognition results. Summary of the invention

[0003] The present invention provides a speech recognition method, device and storage medium to solve the problems existing in the prior art, reduce the error accumulation generated in the acoustic model training process, and improve the accuracy of speech recognition results.

[0004] To achieve the above object, an embodiment of the present invention provides a speech recognition method, comprising:

[0005] Acquire two-dimensional data features distributed on the time axis and the frequency axis, input the two-dimensional data features into the acoustic model for multiple feature extraction and aggregation, and output the sequence results of speech recognition; wherein the acoustic model is composed of a structure of 3 layers of convolutional neural network connected to 8 layers of factorized time-delay neural network connected to a multi-head attention mechanism connected to 2 layers of convolutional neural network connected to a 5-layer factorized time-delay neural network connected to a multi-head attention mechanism connected to a 2-layer convolutional neural network connected to an output layer, and the output of each layer of the acoustic model is used as the input of the next layer for training until the output layer outputs the sequence results;

[0006] The sequence result is converted into text data by a decoder to obtain a speech recognition result.

[0007] As an improvement of the above solution, the 3-layer convolutional neural network is connected to the 8-layer factorized time-delay neural network to form the first feature extraction unit of the acoustic model, the multi-head attention mechanism is connected to the 2-layer convolutional neural network to form the feature aggregation unit of the acoustic model, and the 5-layer factorized time-delay neural network constitutes the second feature extraction unit of the acoustic model;

[0008] The first feature extraction unit extracts features from the input two-dimensional data features, and outputs a speech feature sequence corresponding to the target segment to the feature aggregation unit;

[0009] The feature aggregation unit filters and aggregates the input speech feature sequence, and outputs the first speech feature to the second feature extraction unit;

[0010] The second feature extraction unit performs parameter learning on the input speech feature and outputs the second speech feature to the feature aggregation unit;

[0011] The feature aggregation unit filters and aggregates the input second speech feature, and outputs the third speech feature to the output layer;

[0012] The output layer trains the input third speech feature and outputs a sequence result of speech recognition.

[0013] As an improvement of the above scheme, the specific formula of the multi-head attention mechanism is:

[0014] MultiHead(Q,K,V)=Concat(head1,...,head i ,...,head h )W O

[0015] Among them, Q represents the query vector, K represents the key vector, V represents the value vector, and h represents the number of heads. i represents the output of the i-th head, W O is the output transformation matrix.

[0016] As an improvement of the above scheme, the output of each head of the multi-head attention mechanism is specifically formulated as follows:

[0017]

[0018] Among them, Q represents the query vector, K represents the key vector, V represents the value vector, and W i Q is the query transformation matrix of the i-th head, is the key transformation matrix of the ith head, is the value transformation matrix of the i-th head, and Attention() is the attention calculation function.

[0019] As an improvement of the above solution, the attention calculation function has the following specific formula:

[0020]

[0021] Among them, Q represents the query vector, K represents the key vector, V represents the value vector, and dk is the dimension of the key vector, and softmax() is a similarity normalization function used to calculate the weight of each key vector. The weight is multiplied by the value vector and then weighted summed to obtain the attention output result.

[0022] As an improvement of the above scheme, the convolutional neural network consists of a data input layer, a convolution layer, an activation function layer and a batch normalization layer;

[0023] The convolution layer filters the input data to extract speech features through a convolution operation, including using a convolution kernel to perform sliding convolution on the input data.

[0024] As an improvement of the above solution, the convolution operation is an operation of multiplying and then adding the values ​​of the corresponding positions of two matrices. The specific formula is:

[0025]

[0026] Among them, k i is the i-th convolution kernel, n is the number of convolution kernels, x i is the i-th image feature, k T is the convolution kernel matrix, X is the image feature matrix, and z is the convolution operation result.

[0027] As an improvement of the above solution, the output layer adopts a cross entropy loss function and a loss function based on maximum mutual information for joint training.

[0028] To achieve the above object, an embodiment of the present invention further provides a speech recognition device, comprising:

[0029] A speech recognition module is used to obtain two-dimensional data features distributed on the time axis and the frequency axis, input the two-dimensional data features into the acoustic model for multiple feature extraction and aggregation, and output the sequence results of speech recognition; wherein the acoustic model is composed of a structure of 3 layers of convolutional neural network connected to 8 layers of factorized time-delay neural network connected to a multi-head attention mechanism connected to 2 layers of convolutional neural network connected to a 5-layer factorized time-delay neural network connected to a multi-head attention mechanism connected to a 2-layer convolutional neural network connected to an output layer, and the output of each layer of the acoustic model is used as the input of the next layer for training until the output layer outputs the sequence results;

[0030] The sequence decoding module is used to convert the sequence result into text data through a decoder to obtain a speech recognition result.

[0031] To achieve the above objectives, an embodiment of the present invention further provides a speech recognition device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the above speech recognition method when executing the computer program.

[0032] To achieve the above objectives, an embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the above speech recognition method.

[0033] To achieve the above objective, an embodiment of the present invention further provides a computer program product, which includes a computer program; wherein, when the computer program is running, the device where the program product is located is controlled to execute the above speech recognition method.

[0034] The implementation of the embodiments of the present invention has the following beneficial effects:

[0035] The speech recognition method, device and storage medium provided by the embodiment of the present invention obtain two-dimensional data features distributed on the time axis and the frequency axis, input the two-dimensional data features into the acoustic model for multiple feature extraction and aggregation, and output the sequence results of speech recognition; the sequence results are converted into text data by a decoder to obtain the speech recognition results; wherein the structure of the acoustic model is composed of 3 layers of CNN + 8 layers of TDNNF + multi-head attention mechanism + 2 layers of CNN + 5 layers of TDNNF + multi-head attention mechanism + 2 layers of CNN + output layer, and the output of each layer of the acoustic model is used as the input of the next layer for training until the output layer outputs the sequence results. By inserting a feature extraction unit composed of a multi-head attention mechanism and a 2-layer CNN in the TDNNF network layer, the structure of the acoustic model is improved, redundant information can be filtered out, key features can be extracted, and through multi-layer aggregation extraction, errors can be prevented from accumulating layer by layer, thereby improving the accuracy of speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of a speech recognition method provided by an embodiment of the present invention;

[0037] Figure 2 is a schematic diagram of an acoustic model structure provided by an embodiment of the present invention;

[0038] Figure 3 is a schematic diagram of a convolution operation process provided by an embodiment of the present invention;

[0039] Figure 4 is a structural block diagram of a speech recognition device provided by an embodiment of the present invention;

[0040] Figure 5 It is a structural block diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] In the description of the present invention, it should be noted that the step numbers in the text are only for the convenience of explanation of specific embodiments and do not serve to limit the order in which the steps are executed. The terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0043] It is known that the existing acoustic models include end-to-end acoustic models, but in the power dispatching communication scenario, data collection is more difficult, resulting in the data volume not meeting the requirements for training end-to-end acoustic models. Therefore, different deep learning network modules are used to build a multi-layer acoustic model structure to achieve speech recognition.

[0044] From the late 20th century to the early 21st century, traditional acoustic models were mainly based on the HMM (Hidden Markov Model) framework, combined with the GMM (Gaussian Mixture Model) to model speech features. This method can capture the statistical characteristics of speech signals to a certain extent, but has limited adaptability to factors such as speech speed, accent, and background noise.

[0045] With the development of deep learning technology, DNN (Deep Neural Networks) has gradually replaced the GMM in the traditional GMM-HMM structure. DNN can automatically extract complex features by learning a large amount of speech data, thereby improving the accuracy and robustness of the acoustic model. DNN mainly includes the following neural network modules:

[0046] (1) CNN (Convolutional Neural Network): It extracts local features by using multiple layers of CNN on the time axis and frequency axis, and uses a smaller receptive field to capture local features in the time domain and frequency domain.

[0047] (2) LSTM (Long Short-Term Memory): LSTM selectively increases or decreases the information in the cell state through its three special gating mechanisms, thereby retaining important information and filtering out unimportant information.

[0048] (3) TDNN (Time Delay Neural Network): The two main features of TDNN are its ability to dynamically adapt to changes in time domain features and its use of fewer parameters. The features of the hidden layer of TDNN depend not only on the input at the current moment, but also on the input at the past and future moments.

[0049] (4) TDNNF (Factorized TDNN): It is a further optimization based on TDNN. It uses factorization technology to discard relatively small singular values ​​to optimize network parameters.

[0050] In recent years, with the development of deep learning, attention mechanism has become a key technology in acoustic models. It enables acoustic models to focus on specific parts of the input speech signal when generating output, thereby enhancing the acoustic model's ability to handle long-distance dependencies. Among them, MHA (Multi-head Attention) extends the attention mechanism to multiple heads, thereby enhancing the acoustic model's attention to different features, which can effectively improve the expressiveness of the acoustic model and help the acoustic model learn more diverse and complex features.

[0051] In the power dispatching system, the construction of the general acoustic model is based on the ordered combination of the above-mentioned DNN modules. Usually, two types of neural networks are combined, and the same neural network is superimposed in multiple layers. At the same time, in order to improve the recognition effect of the acoustic model, an attention mechanism is sometimes added before the final output layer of the acoustic model to form a multi-layer hybrid acoustic model, such as building a baseline acoustic model of 3 layers of CNN + 13 layers of TDNNF + attention mechanism. When the neural network of this hybrid acoustic model processes complex data, the fixed length of the hidden layer limits the model's capture and expression of rich information. As the depth of the network increases, information may be lost or confused, resulting in error accumulation, thereby reducing the accuracy of the speech recognition results.

[0052] See also Figure 1 , Figure 1 : is a flow chart of a speech recognition method provided by an embodiment of the present invention, wherein the speech recognition method comprises steps S1 to S2, which are specifically as follows:

[0053] S1. Obtain two-dimensional data features distributed on the time axis and the frequency axis, input the two-dimensional data features into the acoustic model for multiple feature extraction and aggregation, and output the sequence results of speech recognition; wherein the acoustic model is composed of a structure of 3 layers of convolutional neural network connected to 8 layers of factorized time-delay neural network connected to a multi-head attention mechanism connected to 2 layers of convolutional neural network connected to a 5-layer factorized time-delay neural network connected to a multi-head attention mechanism connected to a 2-layer convolutional neural network connected to an output layer, and the output of each layer of the acoustic model is used as the input of the next layer for training until the output layer outputs the sequence results;

[0054] S2. Convert the sequence result into text data through a decoder to obtain a speech recognition result.

[0055] See also Figure 2 In an embodiment of the present invention, the present invention improves the existing acoustic model by inserting a feature aggregation unit composed of a multi-head attention mechanism + 2 layers of CNN into the multi-layer TDNNF network layer in the baseline acoustic model of 3 layers of CNN + 13 layers of TDNNF + attention mechanism, forming an acoustic model structure of 3 layers of CNN + 8 layers of TDNNF + multi-head attention mechanism + 2 layers of CNN + 5 layers of TDNNF + multi-head attention mechanism + 2 layers of CNN + output layer. By designing a feature aggregation unit composed of a multi-head attention mechanism and 2 layers of CNN, the acoustic model is endowed with the ability to perform effective feature aggregation. When learning the acoustic model parameters of a small data set in the field of power dispatching, the feature aggregation unit uses a multi-head attention mechanism to perform fine sampling of the input data and assign different weights, so that the model can more effectively and dynamically capture and extract the features of important information, and use a filtering network to extract features locally, so as to achieve accurate extraction of speech features, solve the problem of error accumulation in deep neural networks, and improve the expression and processing capabilities of the acoustic model for complex data. This innovative design not only optimizes the capture of key features, but also helps the acoustic model to more fully learn the key features of the speech signal through combination with the CNN layer, thereby enhancing the acoustic model's ability to learn and express speech information.

[0056] In an optional embodiment, the 3-layer convolutional neural network is connected to the 8-layer factorized time-delay neural network to form the first feature extraction unit of the acoustic model, the multi-head attention mechanism is connected to the 2-layer convolutional neural network to form the feature aggregation unit of the acoustic model, and the 5-layer factorized time-delay neural network constitutes the second feature extraction unit of the acoustic model;

[0057] The first feature extraction unit extracts features from the input two-dimensional data features, and outputs a speech feature sequence corresponding to the target segment to the feature aggregation unit;

[0058] The feature aggregation unit filters and aggregates the input speech feature sequence, and outputs the first speech feature to the second feature extraction unit;

[0059] The second feature extraction unit performs parameter learning on the input speech feature and outputs the second speech feature to the feature aggregation unit;

[0060] The feature aggregation unit filters and aggregates the input second speech feature, and outputs the third speech feature to the output layer;

[0061] The output layer trains the input third speech feature and outputs the sequence result of speech recognition. In the embodiment of the present invention, by obtaining the two-dimensional data features on the time axis and the frequency axis and inputting them into the improved acoustic model, speech feature extraction is performed in CNN and TDNNF, feature filtering and aggregation are performed in the multi-head attention mechanism and the 2-layer CNN, speech features are processed layer by layer, and finally the sequence result is output, and the sequence result is converted by using a decoder to realize speech recognition, solve the problem of error accumulation in deep neural networks, and improve the speech recognition effect.

[0062] Exemplarily, two-dimensional data features distributed on the time axis and frequency axis are obtained, and the two-dimensional data features are input into an acoustic model with a structure of 3 layers of CNN + 8 layers of TDNNF + multi-head attention mechanism + 2 layers of CNN + 5 layers of TDNNF + multi-head attention mechanism + 2 layers of CNN + output layer. The first feature extraction unit of 3 layers of CNN + 8 layers of TDNNF is used to extract speech features from the input two-dimensional data features to obtain a speech feature sequence corresponding to the target segment. Since the network is too deep, it will cause error accumulation. Therefore, a feature aggregation unit composed of a multi-head attention mechanism + 2 layers of CNN is added after an acoustic model of a certain depth to filter and aggregate the speech feature sequence. The re-aggregated features are input again into the second feature extraction unit of the 5-layer TDNNF for learning speech feature parameters. After reaching a certain number of layers, the important features are aggregated again through the feature aggregation unit, and the redundant information is filtered to extract more accurate speech features. The sequence results of speech recognition are output through the output layer. The sequence results are a continuous sequence, in which the output at each time point corresponds to the probability distribution of the entire vocabulary. In the context of speech recognition, these probability distributions are usually expressed as probabilities corresponding to various states in the HMM. Each HMM state usually corresponds to a phoneme or part of a phoneme sequence. Through the decoder combined with the language acoustic model and pronunciation dictionary, these probability distributions are converted into the final text output to obtain the speech recognition result, thereby improving the network's ability to express features and speech recognition effect.

[0063] In an optional embodiment, the specific formula of the multi-head attention mechanism is:

[0064] MultiHead(Q,K,V)=Concat(head1,...,head i ,...,head h )W O

[0065] Among them, Q represents the query vector, K represents the key vector, V represents the value vector, and h represents the number of heads. i represents the output of the i-th head, W O is the output transformation matrix.

[0066] In an optional embodiment, the output of each head of the multi-head attention mechanism is specifically formulated as follows:

[0067]

[0068] Among them, Q represents the query vector, K represents the key vector, V represents the value vector, and W i Q is the query transformation matrix of the i-th head, is the key transformation matrix of the ith head, is the value transformation matrix of the i-th head, and Attention() is the attention calculation function.

[0069] In an optional embodiment, the attention calculation function has a specific formula:

[0070]

[0071] Among them, Q represents the query vector, K represents the key vector, V represents the value vector, and d k is the dimension of the key vector, and softmax() is a similarity normalization function used to calculate the weight of each key vector. The weight is multiplied by the value vector and then weighted summed to obtain the attention output result.

[0072] It is understandable that in the field of deep learning, the fixed-length vector dimension of the neural network limits the ability of the acoustic model to capture and express rich information. As the depth of the network increases, information may gradually be lost or confused. In order to solve these problems, the attention mechanism is introduced into the neural network model. In MHA, the self-attention mechanism (Self-Attention) is generally used to calculate attention, where the input of MHA includes the query vector (query), the key vector (key) and the value vector (value).

[0073] In an embodiment of the present invention, MHA is used to aggregate important features. For a given query vector, MHA performs a weighted sum on the key vector, where the weight is calculated by the similarity between the query vector and the key vector, where commonly used similarity calculation methods include dot product or bilinear calculation; the result of the weighted sum is multiplied by the value vector and output. Under the multi-head mechanism of MHA, the input sequence data will be divided into multiple heads, each head performs independent calculations to generate different outputs; these outputs will finally be spliced ​​together to form the final result, by dynamically adjusting the degree of attention of the acoustic model to different input parts, helping to capture and retain key information, and learning more diverse and complex features, thereby enhancing the model's ability to process and express complex data, and effectively addressing the problem of information loss in deep networks.

[0074] Adding an attention mechanism to the deeper neural network layers enables the acoustic model to dynamically focus on the importance of different parts of the input sequence. Through the attention scores generated by dot products and subsequent weighted processing, the acoustic model can more effectively capture and utilize the key information in the input sequence, so that the generated vector has higher information content and pertinence. This improvement improves the performance of neural networks when processing long sequence data and enhances the ability of acoustic models to express complex data.

[0075] The multi-head attention mechanism allows the acoustic model to focus on different parts of multiple positions or features in the input sequence at the same time. By learning multiple sets of attention weights in parallel, each set of weights is used to weight different parts of the input sequence, and finally the multiple sets of weighted results are fused. This enables the acoustic model to more fully capture information at different levels and angles, thereby improving the representation ability. At the same time, multi-head attention also provides more parallel computing capabilities, which can more comprehensively capture the information in the sequence.

[0076] In an optional embodiment, the convolutional neural network consists of a data input layer, a convolution layer, an activation function layer and a batch normalization layer;

[0077] The convolution layer filters the input data to extract speech features through a convolution operation, including using a convolution kernel to perform sliding convolution on the input data.

[0078] It is understandable that adding two layers of CNN after the multi-head attention mechanism filters and extracts features from the output of the multi-head attention mechanism. This filtering function is mainly completed by the convolutional neural network, which consists of a data input layer, a convolutional layer, an activation function layer, and a batch normalization layer, among which the convolutional layer is the most important part of the convolutional neural network. The convolutional layer uses convolution operations to extract the features of the input spectrogram, which can be regarded as a filtering process.

[0079] In an optional embodiment, the convolution operation is an operation of multiplying and then adding the values ​​of corresponding positions of two matrices, and the specific formula is:

[0080]

[0081] Among them, k i is the i-th convolution kernel, n is the number of convolution kernels, x i is the i-th image feature, k T is the convolution kernel matrix, X is the image feature matrix, and z is the convolution operation result.

[0082] It can be understood that the above is a calculation formula for obtaining the result z by performing a convolution operation on the image feature matrix X and the convolution kernel k.

[0083] For example, a convolution kernel is used to perform sliding convolution on the input data. This convolution kernel is a window filter. The convolution kernel is generally a matrix of size a×b. The image feature is also a matrix. The convolution operation process is to multiply and add the corresponding values ​​of the two matrices.

[0084] During the convolution operation, it is necessary to select an appropriate convolution kernel size. Common convolution kernel sizes include 1×1, 3×3, 5×5, etc. The smaller the convolution kernel, the more sufficient the detail features can be extracted.

[0085] The specific convolution operation process is as follows: Figure 3 As shown in the figure, the left side is a 3×3 input feature map matrix, the convolution kernel size is 2×2, the step size is 1, and the padding is 1. After padding, the feature map size becomes 5×5, and the value of the first row and first column in the output result is the sum of the area marked in the input feature and the convolution kernel. That is, (0×0)+(0×1)+(0×2)+(1×3)=3. After that, the marked area in the input feature is moved by 1 (step size) unit each time for calculation, until the entire matrix is ​​traversed and the end is completed. The matrix obtained is the feature pixel value after convolution.

[0086] Assuming that the input image feature size is H×W, the convolution kernel size is k×k, the step size is S, and the padding is P, the size of the feature map generated after convolution is H ′ ×W ′ , the specific formula is:

[0087]

[0088] Among them, H and W are the image feature sizes, H ′ and W ′ They are the sizes of feature maps generated after convolution, k is the convolution kernel size, S is the step size, and P is the padding.

[0089] Based on the above principles, the combination of attention mechanism and CNN can achieve feature aggregation. The feature aggregation unit composed of multi-head attention mechanism and 2-layer CNN can make the acoustic model better adapt to different speech features and recognition requirements through filtering and weight adjustment mechanism, thereby improving the generalization ability and stability of the acoustic model. Since the feature aggregation unit can more effectively capture and emphasize useful information, the learning of redundant features can be reduced during the model training process, thereby shortening the training time and improving the training efficiency.

[0090] Experimental verification shows that the acoustic model structure with feature aggregation units has a lower error rate in speech recognition than the baseline acoustic model. Since the acoustic model can more accurately understand the characteristics of speech signals, in speech recognition tasks, whether in noisy environments or in scenarios with large differences in speech characteristics, the acoustic model with feature aggregation units can achieve better recognition results.

[0091] In an optional embodiment, the output layer is jointly trained using a cross entropy loss function and a maximum mutual information based loss function.

[0092] In an embodiment of the present invention, the output layer uses CE (Cross-Entropy Loss) and a loss function based on maximum mutual information at the same time during training. Through joint training, the occurrence of overfitting is effectively prevented, the generalization ability and stability of the acoustic model are improved, and the model is more accurate and reliable when processing data.

[0093] The speech recognition method provided by the embodiment of the present invention obtains two-dimensional data features distributed on the time axis and the frequency axis, inputs the two-dimensional data features into the acoustic model for multiple feature extraction and aggregation, and outputs the sequence results of speech recognition; the sequence results are converted into text data by a decoder to obtain the speech recognition results; wherein the structure of the acoustic model is composed of 3 layers of CNN + 8 layers of TDNNF + multi-head attention mechanism + 2 layers of CNN + 5 layers of TDNNF + multi-head attention mechanism + 2 layers of CNN + output layer, and the output of each layer of the acoustic model is used as the input of the next layer for training until the output layer outputs the sequence results. By inserting a feature extraction unit composed of a multi-head attention mechanism and 2 layers of CNN in the TDNNF network layer, the structure of the acoustic model is improved, redundant information can be filtered out, key features can be extracted, and through multi-layer aggregation extraction, errors can be prevented from accumulating layer by layer, thereby improving the accuracy of speech recognition.

[0094] See also Figure 4 , Figure 4 : is a structural block diagram of a speech recognition device 10 provided in an embodiment of the present invention, wherein the speech recognition device 10 comprises:

[0095] The speech recognition module 11 is used to obtain two-dimensional data features distributed on the time axis and the frequency axis, input the two-dimensional data features into the acoustic model for multiple feature extraction and aggregation, and output the sequence results of speech recognition; wherein the acoustic model is composed of a structure of 3 layers of convolutional neural network connected to 8 layers of factorized time-delay neural network connected to a multi-head attention mechanism connected to 2 layers of convolutional neural network connected to a 5-layer factorized time-delay neural network connected to a multi-head attention mechanism connected to a 2-layer convolutional neural network connected to an output layer, and the output of each layer of the acoustic model is used as the input of the next layer for training until the output layer outputs the sequence results;

[0096] The sequence decoding module 12 is used to convert the sequence result into text data through a decoder to obtain a speech recognition result.

[0097] Optionally, the 3-layer convolutional neural network is connected to an 8-layer factorized time-delay neural network to form a first feature extraction unit of the acoustic model, the multi-head attention mechanism is connected to a 2-layer convolutional neural network to form a feature aggregation unit of the acoustic model, and the 5-layer factorized time-delay neural network constitutes a second feature extraction unit of the acoustic model;

[0098] The first feature extraction unit extracts features from the input two-dimensional data features, and outputs a speech feature sequence corresponding to the target segment to the feature aggregation unit;

[0099] The feature aggregation unit filters and aggregates the input speech feature sequence, and outputs the first speech feature to the second feature extraction unit;

[0100] The second feature extraction unit performs parameter learning on the input speech feature and outputs the second speech feature to the feature aggregation unit;

[0101] The feature aggregation unit filters and aggregates the input second speech feature, and outputs the third speech feature to the output layer;

[0102] The output layer trains the input third speech feature and outputs a sequence result of speech recognition.

[0103] Optionally, the specific formula of the multi-head attention mechanism is:

[0104] MultiHead(Q,K,V)=Concat(head1,...,head i ,...,head h )W O

[0105] Among them, Q represents the query vector, K represents the key vector, V represents the value vector, and h represents the number of heads.i represents the output of the i-th head, W O is the output transformation matrix.

[0106] Optionally, the output of each head of the multi-head attention mechanism is specifically formulated as follows:

[0107]

[0108] Among them, Q represents the query vector, K represents the key vector, V represents the value vector, and W i Q is the query transformation matrix of the i-th head, is the key transformation matrix of the ith head, is the value transformation matrix of the i-th head, and Attention() is the attention calculation function.

[0109] Optionally, the attention calculation function has the following specific formula:

[0110]

[0111] Among them, Q represents the query vector, K represents the key vector, V represents the value vector, and d k is the dimension of the key vector, and softmax() is a similarity normalization function used to calculate the weight of each key vector. The weight is multiplied by the value vector and then weighted summed to obtain the attention output result.

[0112] Optionally, the convolutional neural network consists of a data input layer, a convolution layer, an activation function layer and a batch normalization layer;

[0113] The convolution layer filters the input data to extract speech features through a convolution operation, including using a convolution kernel to perform sliding convolution on the input data.

[0114] Optionally, the convolution operation is an operation of multiplying and then adding the values ​​of corresponding positions of two matrices, and the specific formula is:

[0115]

[0116] Among them, k i is the i-th convolution kernel, n is the number of convolution kernels, x i is the i-th image feature, k T is the convolution kernel matrix, X is the image feature matrix, and z is the convolution operation result.

[0117] Optionally, the output layer is jointly trained using a cross entropy loss function and a maximum mutual information-based loss function.

[0118] It is worth noting that the working process of each module in the speech recognition device 10 described in the embodiment of the present invention can refer to the working process of the speech recognition method described in the above embodiment, and will not be repeated here.

[0119] The speech recognition device provided by the embodiment of the present invention obtains two-dimensional data features distributed on the time axis and the frequency axis, inputs the two-dimensional data features into the acoustic model for multiple feature extraction and aggregation, and outputs the sequence results of speech recognition; the sequence results are converted into text data by a decoder to obtain the speech recognition results; wherein the structure of the acoustic model is composed of 3 layers of CNN + 8 layers of TDNNF + multi-head attention mechanism + 2 layers of CNN + 5 layers of TDNNF + multi-head attention mechanism + 2 layers of CNN + output layer, and the output of each layer of the acoustic model is used as the input of the next layer for training until the output layer outputs the sequence results. By inserting a feature extraction unit composed of a multi-head attention mechanism and 2 layers of CNN in the TDNNF network layer, the structure of the acoustic model is improved, redundant information can be filtered out, key features can be extracted, and through multi-layer aggregation extraction, errors can be prevented from accumulating layer by layer, thereby improving the accuracy of speech recognition.

[0120] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the above-mentioned speech recognition method.

[0121] An embodiment of the present invention further provides a computer program product, which includes a computer program; wherein, when the computer program is running, the device where the computer program product is located is controlled to execute the above-mentioned speech recognition method.

[0122] See also Figure 5 , Figure 5 1 is a structural block diagram of a speech recognition method, apparatus and storage medium device 20 provided in an embodiment of the present invention, wherein the computer device 20 comprises: a processor 21, a memory 22 and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, the steps in the above speech recognition method embodiment are implemented. Alternatively, when the processor 21 executes the computer program, the functions of each module / unit in the above apparatus embodiments are implemented.

[0123] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the computer device 20.

[0124] The computer device 20 may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will appreciate that the schematic diagram is merely an example of the computer device 20 and does not limit the computer device 20. The computer device 20 may include more or fewer components than shown in the figure, or may combine certain components, or different components. For example, the computer device 20 may also include input and output devices, network access devices, buses, etc.

[0125] The processor 21 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 21 is the control center of the computer device 20, and uses various interfaces and lines to connect various parts of the entire computer device 20.

[0126] The memory 22 can be used to store the computer program and / or module. The processor 21 implements various functions of the computer device 20 by running or executing the computer program and / or module stored in the memory 22 and calling the data stored in the memory 22. The memory 22 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0127] Wherein, if the module / unit integrated in the computer device 20 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor 21. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0128] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A speech recognition method, characterized in that: include: Acquire two-dimensional data features distributed on the time axis and the frequency axis, input the two-dimensional data features into the acoustic model for multiple feature extraction and aggregation, and output the sequence results of speech recognition; wherein the acoustic model is composed of a structure of 3 layers of convolutional neural network connected to 8 layers of factorized time-delay neural network connected to a multi-head attention mechanism connected to 2 layers of convolutional neural network connected to a 5-layer factorized time-delay neural network connected to a multi-head attention mechanism connected to a 2-layer convolutional neural network connected to an output layer, and the output of each layer of the acoustic model is used as the input of the next layer for training until the output layer outputs the sequence results; The sequence result is converted into text data by a decoder to obtain a speech recognition result.

2. The speech recognition method according to claim 1, characterized in that: The 3-layer convolutional neural network is connected to the 8-layer factorized time-delay neural network to form the first feature extraction unit of the acoustic model, the multi-head attention mechanism is connected to the 2-layer convolutional neural network to form the feature aggregation unit of the acoustic model, and the 5-layer factorized time-delay neural network constitutes the second feature extraction unit of the acoustic model; The first feature extraction unit extracts features from the input two-dimensional data features, and outputs a speech feature sequence corresponding to the target segment to the feature aggregation unit; The feature aggregation unit filters and aggregates the input speech feature sequence, and outputs the first speech feature to the second feature extraction unit; The second feature extraction unit performs parameter learning on the input speech feature and outputs the second speech feature to the feature aggregation unit; The feature aggregation unit filters and aggregates the input second speech feature, and outputs the third speech feature to the output layer; The output layer trains the input third speech feature and outputs a sequence result of speech recognition.

3. The speech recognition method according to claim 1, wherein: The specific formula of the multi-head attention mechanism is: MultiHead(Q,K,V)=Concat(head1,...,head i ,...,head h )W O Among them, Q represents the query vector, K represents the key vector, V represents the value vector, and h represents the number of heads. i represents the output of the i-th head, W O is the output transformation matrix.

4. The speech recognition method according to claim 3, characterized in that: The output of each head of the multi-head attention mechanism is as follows: head i =Attention(QW i Q ,KW i K ,VW i V ) Among them, Q represents the query vector, K represents the key vector, V represents the value vector, and W i Q is the query transformation matrix of the i-th head, W i K is the key transformation matrix of the i-th head, W i V is the value transformation matrix of the i-th head, and Attention() is the attention calculation function.

5. The speech recognition method according to claim 4, characterized in that: The attention calculation function has the following specific formula: Among them, Q represents the query vector, K represents the key vector, V represents the value vector, and d k is the dimension of the key vector, and softmax() is a similarity normalization function used to calculate the weight of each key vector. The weight is multiplied by the value vector and then weighted summed to obtain the attention output result.

6. The speech recognition method according to claim 1, characterized in that: The convolutional neural network consists of a data input layer, a convolution layer, an activation function layer and a batch normalization layer; The convolution layer filters the input data to extract speech features through a convolution operation, including using a convolution kernel to perform sliding convolution on the input data.

7. The speech recognition method according to claim 6, characterized in that: The convolution operation is an operation of multiplying and adding the values ​​of corresponding positions of two matrices. The specific formula is: Among them, k i is the i-th convolution kernel, n is the number of convolution kernels, x i is the i-th image feature, k T is the convolution kernel matrix, X is the image feature matrix, and z is the convolution operation result.

8. The speech recognition method according to claim 1, characterized in that: The output layer is jointly trained using a cross entropy loss function and a loss function based on maximum mutual information.

9. A speech recognition device, characterized in that: include: A speech recognition module is used to obtain two-dimensional data features distributed on the time axis and the frequency axis, input the two-dimensional data features into the acoustic model for multiple feature extraction and aggregation, and output the sequence results of speech recognition; wherein the acoustic model is composed of a structure of 3 layers of convolutional neural network connected to 8 layers of factorized time-delay neural network connected to a multi-head attention mechanism connected to 2 layers of convolutional neural network connected to a 5-layer factorized time-delay neural network connected to a multi-head attention mechanism connected to a 2-layer convolutional neural network connected to an output layer, and the output of each layer of the acoustic model is used as the input of the next layer for training until the output layer outputs the sequence results; The sequence decoding module is used to convert the sequence result into text data through a decoder to obtain a speech recognition result.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program; wherein, when the computer program is run, it controls the device where the computer-readable storage medium is located to execute the speech recognition method according to any one of claims 1 to 8.

Images