A speech recognition method and system based on separable attention and block masking

Abstract

The application discloses a speech recognition method and system based on separable attention and block masking, and the method comprises the following steps: constructing a speech recognition model, training the speech recognition model, and using the trained model to perform speech recognition; wherein the speech recognition model comprises a preprocessing module, an embedding layer, an optimization module, and a linear layer; the preprocessing module comprises a transformation alignment module, a blocking module, a block masking module, and an extension module; the transformation alignment module obtains a spectrum graph corresponding to original speech, and aligns the time dimension of all spectrum graphs as a whole; the blocking module divides the spectrum graph into a plurality of block sequences; the block masking module randomly discards part of the block sequences; the extension module extends the remaining block sequences; the optimization module comprises a separable attention layer and a feedforward network layer; the separable attention layer comprises a horizontal attention layer and a vertical attention layer; and the application can reduce the computational complexity of speech recognition, improve the computational efficiency, and improve the speech recognition accuracy.

Description

Technical Field

[0001] This invention relates to a speech recognition method and system, and more particularly to a speech recognition method and system based on separable attention and block masking, belonging to the field of speech recognition. Background Technology

[0002] Speech recognition technology is a technology that converts human language into text commands. It was considered one of the top ten important technological developments in the field of information technology between 2000 and 2010, and plays an important role in applications such as intelligent assistants, voice search, automatic transcription, and voice commands.

[0003] Early speech recognition technologies were primarily based on pattern matching and basic signal processing techniques. However, these methods were limited by their high sensitivity to environmental noise and speaker variations, which restricted performance. Later, statistical methods were introduced, particularly Hidden Markov Models (HMMs), for acoustic feature and language modeling. With the rise of deep learning, speech recognition has achieved significant breakthroughs. For example, Deep Neural Networks (DNNs) have achieved significant performance improvements in acoustic models, while structures such as Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks are used to better model sequential data. Furthermore, the introduction of Transformer architectures, such as self-attention mechanisms, has further improved the accuracy of speech recognition. These methods have improved recognition accuracy to some extent, but are still limited by model complexity and the quality of training data. Summary of the Invention

[0004] Purpose of the invention: In view of the problems existing in the prior art, the purpose of this invention is to provide a speech recognition method and system based on separable attention and block masking that can reduce computational complexity, improve computational efficiency, and improve speech recognition accuracy.

[0005] Technical solution: To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0006] The speech recognition method based on separable attention and block masking described in this invention is characterized by comprising: constructing a speech recognition model, training the speech recognition model, and using the trained model to perform speech recognition;

[0007] The speech recognition model includes a preprocessing module, an embedding layer, an optimization module, and a linear layer;

[0008] The preprocessing module includes a transformation and alignment module, a block segmentation module, a block masking module, and an extension module;

[0009] The transformation and alignment module is used to transform the original speech to obtain the spectrogram corresponding to the original speech, and to use an adaptive pooling layer to align the time dimension of all spectrograms as a whole; the block segmentation module is used to divide the spectrogram into several block sequences; the block masking module is used to randomly discard some block sequences; and the extension module is used to extend the remaining block sequences.

[0010] The optimization module includes a separable attention layer and a feedforward network layer;

[0011] The separable attention layer includes a horizontal attention layer and a vertical attention layer.

[0012] Furthermore, the process of converting the original speech to obtain the corresponding spectrogram, and then using an adaptive pooling layer to align the temporal dimensions of all spectrograms, includes the following steps:

[0013] The original speech is transformed into a two-dimensional time-frequency matrix by using the Short Time Fourier Transform (STFT). The STFT formula is as follows:

[0014]

[0015] Where x[a] is the discrete input signal of the speech, w[] is the window function, and N x Here, is the STFT length, 'a' is the time index of the audio signal, 'k' is the frequency component of the converted audio signal, 'j' is the imaginary unit, 'h' is the frame index, representing the position of the time window of the audio signal, 'R' is a constant, and 'e' is the natural constant; the converted speech data yields the corresponding spectrogram X∈R. T×F , where T represents the size of the time dimension of the spectrum and F represents the size of the frequency dimension of the spectrum;

[0016] Align the spectrograms of different lengths within each batch in the time domain and record the original effective length to obtain... Where L max The duration of the longest speech spectrogram in each batch;

[0017] An adaptive pooling layer is used to align the temporal dimensions of the spectrograms across all batches.

[0018] X pooling =AdaptivePooling(X align )

[0019] The pooled spectrum X is obtained pooling ∈R L×F , where L represents the length of the spectrogram after the adaptive pooling layer.

[0020] Furthermore, the block module will use the pooled spectrum X pooling ∈R L×F The spectrum is divided into p×p blocks, where L represents the length of the spectrum after the adaptive pooling layer, F represents the size of the frequency dimension of the spectrum, and the size of p needs to be divided by L and F to obtain a number of m×n blocks, where m is the number of horizontal blocks and n is the number of vertical blocks.

[0021] Furthermore, the block masking module, specifically, randomly discards a portion of the block sequence by randomly selecting a portion of the block sequence and discarding it, with the randomness being independent of the position; or by randomly discarding several whole columns / rows of blocks extracted from the spectrum.

[0022] Furthermore, the extension module extends each remaining block sequence into a one-dimensional sequence, each one-dimensional sequence having a length of p. 2 ,

[0023] X patch =Extend(Slicing(X) pooling ))

[0024] X pooling The spectrum is the pooled spectrum, and the tensor representation of the stretched spectrum data is as follows: Where m is the number of horizontal block sequences and n is the number of vertical block sequences.

[0025] Furthermore, the embedding layer will extend the spectrogram data sequence matrix X. patch dimension p 2 Extending to d dimensions, where d is a constant,

[0026] X mapping =Linear(X) patch )

[0027] This maps the input data from a low-dimensional space to a higher-dimensional feature space, amplifying the data features and obtaining the expanded spectrogram X. mapping ∈R m×n×d , where m is the number of horizontal block sequences and n is the number of vertical block sequences.

[0028] Furthermore, the separable attention layer includes a horizontal attention layer and a vertical attention layer, both of which have a unified model structure and employ a consistent attention mechanism.

[0029] X f =X mapping +param f

[0030]

[0031]

[0032]

[0033] For the extended spectrum X mapping ∈R m×n×d Add a learnable parameter param to the frequency dimension. f ∈R 1×n×d Get X f ∈R m×n×d After further calculation using longitudinal multi-head attention, the result is obtained. Again Add a learnable parameter `param` to the time dimension. t ∈R m×1×d , obtain X t ∈R m×n×d The output is after lateral multi-head attention calculation. Where d is a constant, m is the number of horizontal block sequences, and n is the number of vertical block sequences;

[0034] The feedforward network layer is used to deepen the network, so that the learning results of the separable attention layer are passed through a feedforward network layer and a normalization layer to obtain X. new ∈R m×n×d The method employs residual calculation to combine the new output with the learning results of the separable attention layer. Add them together to get the final X final ∈R m×n×d .

[0035] Furthermore, the linear layer, for the result X of the optimization module final ∈R m×n×d After linear layer mapping, dimension d is transformed into the original dimension p. 2 And after dimensional transformation, X is obtained. convert ∈R L×F Where L = m × p, F = n × p, d is a constant, m is the number of horizontal block sequences, n is the number of vertical block sequences, L represents the length of the spectrogram after the adaptive pooling layer, and F represents the size of the frequency dimension of the spectrogram. Then, a classifier, i.e., a linear layer, transforms the L dimension into the length, resulting in X. label ∈R length×F , where length represents the total number of unique characters in the dataset.

[0036] Furthermore, the speech recognition model is trained by applying the CTC loss function to the output of the model encoder, calculating the difference between the model's prediction and the real text label, automatically aligning the temporal frame sequence of speech with the corresponding transcribed text sequence during model training, minimizing the CTC loss using stochastic gradient descent, updating the model's weight parameters, and completing the model training.

[0037] Based on the same inventive concept, the present invention also provides a computer system, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is loaded onto the processor, it implements the steps of the speech recognition method based on separable attention and block masking described above.

[0038] Beneficial effects: The speech recognition method based on separable attention and block masking proposed in this invention can more effectively capture the contextual dependencies of long speech sequences in the input speech signal and improve the recognition accuracy. By masking some data blocks, the computational efficiency of the model can be improved without affecting the final recognition accuracy, which has great practical significance for the industrial application of mobile speech recognition. Attached Figure Description

[0039] Figure 1 This is an architecture diagram of an embodiment of the present invention.

[0040] Figure 2 This is a schematic diagram illustrating two methods of block masking in an embodiment of the present invention.

[0041] Figure 3 This is a schematic diagram illustrating the changes in experimental accuracy in an embodiment of the present invention. Detailed Implementation

[0042] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0043] As shown in the attached figure, a speech recognition method based on separable attention and block masking in this embodiment includes: constructing a speech recognition model, training the speech recognition model, and using the trained model for speech recognition; the speech recognition model includes a preprocessing module, an embedding layer, an optimization module, and a linear layer; the specific method includes:

[0044] Step 1: Data preprocessing.

[0045] Step 1.1: Transform each audio sample into a two-dimensional time-frequency matrix by performing a short-time Fourier transform (STFT) on the speech signal, as shown in formula (1).

[0046]

[0047] Where x[a] is the discrete input signal of the speech, w[] is the window function, and N x Here, is the STFT length, 'a' is the time index of the audio signal, 'k' is the frequency component of the converted audio signal, 'j' is the imaginary unit, 'h' is the frame index, representing the position of the time window of the audio signal, 'R' is a constant, and 'e' is the natural constant. After conversion, the speech data yields the corresponding spectrogram data X∈R. T×F Where T represents the size of the time dimension of the spectrum and F represents the size of the frequency dimension of the spectrum.

[0048] Step 1.2: Since the spectrograms of each speech segment in a batch may be inconsistent in the time dimension, it is necessary to align spectrograms of different lengths in the time domain. Using zero-padding, shorter spectrogram data are padded with zeros on the time axis to make them all the same length as the longest spectrogram data in the batch, and the original effective length is recorded, thus obtaining... Where L max The duration of the longest speech spectrogram in each batch;

[0049] Step 1.3: After padding, the time dimension length of the spectrogram within each batch is consistent, but the time dimension length between batches is still different. In order to achieve batch alignment, improve training speed, and simplify the structure of the entire network model, an adaptive pooling layer is used, as shown in Figure (2), to align the time dimension of the spectrograms of all data as a whole, resulting in X. pooling ∈R L×F , where L represents the length of the spectrogram after the adaptive pooling layer.

[0050] X pooling =AdaptivePooling(X align (2)

[0051] Step 2: Divide into blocks. Divide X... pooling ∈R L×F The sequence is divided into p×p blocks, where p is divisible by L and F, resulting in m×n blocks. Here, m is the number of horizontal blocks (time dimension) and n is the number of vertical blocks (frequency dimension). Each block is then extended into a one-dimensional sequence of length p. 2 At this point, the tensor representation of each spectrum data is: The extended spectrogram is then processed through an embedding layer to increase the dimension p of the original data sequence matrix. 2 Extending to d dimensions maps the input data from a low-dimensional space to a higher-dimensional feature space, amplifying the data features and obtaining X. mapping ∈R m×n×dThe above steps can be expressed as a formula.

[0052] X patch =Extend(Slicing(X) pooling (3)

[0053] X mapping =Linear(X) patch (4)

[0054] Step 3: Block Masking. Before the data enters the separable attention layer, a portion of the block sequence is randomly discarded and not included in subsequent calculations. Similar to patchout in image processing, two sampling methods are used to mask the segmented blocks, as shown in Figure (2). The first method randomly selects a portion of the blocks, the randomness being independent of position; the second method randomly selects some frequency units / time frames, i.e., removing several whole columns / rows of extracted blocks from the spectrogram. Block masking reduces the overall data size, effectively standardizes the data training process, and thus reduces the computational complexity of the entire model. The experimental part of this invention uses the first method.

[0055] Step 4: Separable attention layer.

[0056] Step 4.1: Before the data is input into the attention layer, two learnable parameters need to be added, one in the vertical direction and the other in the vertical direction. f ∈R 1×n×d To better learn frequency-related features; a learnable parameter, param, is added in the horizontal direction. t ∈R m×1×d By combining time-domain analysis and frequency-domain analysis, the model can learn at multiple scales and improve recognition performance, thus better learning time-related features.

[0057] Step 4.2: The attention layer is divided into two parts: the vertical attention layer and the horizontal attention layer. Both have a unified model structure and adopt the same attention mechanism. The specific implementation of the attention mechanism is shown in formula (5).

[0058] The attention mechanism in this patent employs a multi-head attention mechanism. Compared to ordinary attention mechanisms, multi-head attention is better able to capture different types of relationships and patterns when processing sequence data, enabling the model to learn more diverse and complex features, thereby improving model performance. Under the multi-head mechanism, the input sequence data is divided into multiple heads, each performing independent computation and producing different outputs. These outputs are then concatenated to form the final output. Based on formula (3), multi-head attention is implemented as shown in formulas (6) and (7).

[0059]

[0060] head i =Attention(QW i Q ,KW i K VW i V (6)

[0061]

[0062]

[0063] Among them, head i W represents the output of the i-th head. i Q W i K W i V Let W be the query, key, and value transformation matrix of the i-th head. O It is the output transformation matrix. The resulting... After passing through a linear layer, the final output is obtained, as shown in formula (8).

[0064] The difference between vertical and horizontal approaches lies in the change of the input data dimension. The former uses the frequency scale as the main body of the attention score matrix, while the latter uses the time scale as the main body of the attention score matrix. This is also the main innovation of this invention.

[0065] In summary, the specific implementation steps of the separable attention layer include: for X mapping ∈R m×n×d Add a learnable parameter param to the frequency dimension. f ∈R 1×n×d Get X f ∈R m×n×d After further calculation using longitudinal multi-head attention, the result is obtained. Again Add a learnable parameter `param` to the time dimension. t ∈R m×1×d , obtain X t ∈R m ×n×d The output is after lateral multi-head attention calculation. As shown in formulas (9), (10), (11), and (12).

[0066] X f =X mapping +param f (9)

[0067]

[0068]

[0069]

[0070] Step 5: Pass the above outputs through a feedforward network layer and a canonical layer respectively to obtain X. new ∈R m×n×d Using residual calculation, the new output is compared with... Add them together to get the final X final ∈R m×n×d The feedforward network layer consists of two linear layers to increase the network depth and better learn data features; the normalization layer is a Layer Norm layer from the PyTorch library.

[0071] Step 6: For X final ∈R m×n×d After linear layer mapping, the dimension d is transformed into the original p. 2 And after dimensional transformation, X is obtained. convert ∈R L×F Where L = m × p, F = n × p, and then a classifier, i.e. a linear layer, is used to transform the L dimension into the length, to obtain X. label ∈R length×F The length represents the total number of unique characters in the dataset, used for subsequent CTC loss calculation and outputting the speech recognition result.

[0072] Step 7: Apply the CTC loss function to the output of the model encoder to calculate the difference between the model's prediction and the real text label. Its main function is to map the input sequence to the output sequence, while considering the alignment relationship between the input and output sequences. Through CTC, the temporal frame sequence of speech and the corresponding transcribed text sequence can be automatically aligned during model training without needing to label the start and end times of each character or phoneme, thus enabling direct classification on the time series and eliminating the complex alignment operations of traditional algorithms.

[0073] Step 8: Use stochastic gradient descent to minimize the CTC loss and update the model's weight parameters.

[0074] Based on the same inventive concept, this embodiment also discloses a computer system, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is loaded onto the processor, it implements the steps of the speech recognition method based on separable attention and block masking described above.

[0075] As attached Figure 1 The flowchart of the method shown in this invention is illustrated below. The method of this invention will be further explained below with reference to specific experimental data. The data used is the AISHELL-1 open-source Chinese speech dataset.

[0076] First, acquire the audio data, perform a short-time Fourier transform on the audio data, using an acquisition rate of 16 kHz, a Hamming window type, a size of 0.02, and an amplitude of 0.01 to obtain the spectrogram.

[0077] Using the Transformer model architecture, an end-to-end model was built using a speech recognition method based on separable attention and block masking as described in this invention. The experiment consisted of 938 batches and 100 epochs, with 128 speech data points in each batch. The learning rate was dynamic. The model was trained using the original AISHELL-1 speech data and corresponding text annotations. The training objective was to minimize the difference between the model output and the true labels. Gradient descent was used to adjust the model parameters, and a validation set was used to adjust hyperparameters and model architecture to avoid overfitting and improve generalization performance. The accuracy variation graph is shown in Figure (3). After 100 epochs, the final learning rate reached 84.52%.

Claims

1. A speech recognition method based on separable attention and block masking, characterized in that, include: Build a speech recognition model, train the speech recognition model, and use the trained model to perform speech recognition; The speech recognition model includes a preprocessing module, an embedding layer, an optimization module, and a linear layer; the linear layer processes the results from the optimization module. After linear layer mapping, dimension d is transformed into the original dimension p. 2 And after dimensional transformation, we get ,in , Where d is a constant, m is the number of horizontal block sequences, n is the number of vertical block sequences, L represents the length of the spectrogram after the adaptive pooling layer, and F represents the size of the frequency dimension of the spectrogram. Then, a classifier, i.e., a linear layer, is used to transform the L dimension into the length, resulting in... , where length represents the total number of unique characters in the dataset; The preprocessing module includes a transformation and alignment module, a block segmentation module, a block masking module, and an extension module. The block masking module randomly discards a portion of the block sequence by randomly selecting a portion of the block sequence and discarding it, with the randomness being independent of the position; or by randomly discarding several whole columns / rows of blocks extracted from the spectrogram. The conversion and alignment module is used to convert the original speech to obtain the spectrogram corresponding to the original speech, and to use an adaptive pooling layer to align the time dimension of all spectrograms as a whole. The segmentation module is used to divide the spectrogram into several block sequences; the block masking module is used to randomly discard some block sequences; the extension module is used to extend the remaining block sequences. The optimization module includes a separable attention layer and a feedforward network layer; The separable attention layer includes a horizontal attention layer and a vertical attention layer, both with a unified model structure and employing a consistent attention mechanism for the expanded spectrogram. Add learnable parameters in the frequency dimension get After further calculation using longitudinal multi-head attention, the result is obtained. And then Add learnable parameters to the time dimension ,get The output is after lateral multi-head attention calculation. , where d is a constant, m is the number of horizontal block sequences, and n is the number of vertical block sequences; The feedforward network layer is used to deepen the network, so that the learning results of the separable attention layer are obtained after passing through a feedforward network layer and a normalization layer respectively. The method employs residual calculation to combine the new output with the learning results of the separable attention layer. Add them together to get the final result. .

2. The speech recognition method based on separable attention and block masking according to claim 1, characterized in that, The process of converting the original speech to obtain the corresponding spectrogram and then using an adaptive pooling layer to align the temporal dimensions of all spectrograms includes the following steps: The original speech is transformed into a two-dimensional time-frequency matrix by using the Short Time Fourier Transform (STFT). The STFT formula is as follows: ； Where x[ [] is the discrete input signal of speech, and w[] is the window function. It is the STFT length. Here, k is the time index of the audio signal, j is the imaginary unit, h is the frame index, representing the position of the audio signal's time window, R is a constant, and e is the natural constant; the converted speech data yields the corresponding spectrogram. , where T represents the size of the time dimension of the spectrum and F represents the size of the frequency dimension of the spectrum; Align the spectrograms of different lengths within each batch in the time domain and record the original effective length to obtain... ,in The duration of the longest speech spectrogram in each batch; An adaptive pooling layer is used to align the temporal dimensions of the spectrograms across all batches. ； The spectrum after pooling is obtained , where L represents the length of the spectrogram after the adaptive pooling layer.

3. The speech recognition method based on separable attention and block masking according to claim 1, characterized in that, The block module will pool the spectrum. Divided into The sequence of blocks of size p, where L represents the length of the spectrogram after the adaptive pooling layer, F represents the frequency dimension of the spectrogram, and the size of p needs to be divided by L and F to obtain a sequence of blocks of size p. The block sequence, where m is the number of horizontal block sequences and n is the number of vertical block sequences.

4. The speech recognition method based on separable attention and block masking according to claim 1, characterized in that, The extension module extends each remaining block sequence into a one-dimensional sequence, with each one-dimensional sequence having a length of p. 2 , ； X pooling The spectrum is the pooled spectrum, and the tensor representation of the stretched spectrum data is as follows: , where m is the number of horizontal block sequences and n is the number of vertical block sequences.

5. The speech recognition method based on separable attention and block masking according to claim 1, characterized in that, The embedding layer will extend the spectrogram data sequence matrix X patch dimension p 2 Extending to d dimensions, where d is a constant, ； This maps the input data from a low-dimensional space to a higher-dimensional feature space, amplifying the data features and obtaining an expanded spectrogram. , where m is the number of horizontal block sequences and n is the number of vertical block sequences.

6. The speech recognition method based on separable attention and block masking according to claim 1, characterized in that, The process involves training the speech recognition model, applying the CTC loss function to the output of the model encoder, calculating the difference between the model's prediction and the real text label, and automatically aligning the temporal frame sequence of the speech with the corresponding transcribed text sequence during model training. The CTC loss is minimized using stochastic gradient descent, the model's weight parameters are updated, and the model training is complete.

7. A computer system, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when loaded onto the processor, implements the steps of the speech recognition method based on separable attention and block masking according to any one of claims 1 to 6.

Images