Linear complexity model architecture for speech recognition

Abstract

The invention discloses and introduces a linear complexity model architecture for speech recognition in the technical field of speech recognition, an AMLP branch uses an MLP to replace an attention mechanism, an attention average pooling layer is added on the basis of the MLP, uniform distribution of attention weights is ensured while the linear complexity is kept, comprehensive global features are obtained, and the accuracy of speech recognition is improved. According to the convolution branch, an enhanced local feature relationship is captured by using a convolution space gating unit, local features and global features are mixed through a channel interacting information with the AMLP branch, and the local features are supplemented for the global features. Advanced fusion is properly carried out during local feature and global feature extraction processing, so that the problem of real-time mutual influence of the global features and the local features is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of speech recognition, and in particular to a linear complexity model architecture for speech recognition. Background Art

[0002] In recent years, a lot of research has focused on improving the structure of Conformer, aiming to further reduce computational overhead and improve recognition performance. For example, Branchformer (Peng et al., 2022) introduces parallel branches to model contextual features of different ranges. One branch uses a convolutional gated multilayer perceptron (cgMLP) to capture local contextual features, and the other branch uses a self-attention mechanism to capture long-range feature dependencies. The Branchformer structure is as follows: Figure 1 As shown, However, this structure has two problems: First, the mainstream mode is to completely separate local features from global features. However, this method of independently extracting local features and global features and then splicing and fusing them does not solve the problem that global features and local features influence each other in real time. Therefore, while extracting local features and global features, this paper appropriately fuses local features and global features in advance during extraction and processing to solve the problem that global features and local features influence each other in real time.

[0003] Second, attention calculations are complex and heavily dependent on hardware. MLP Mixer (Tolstikhin et al., 2021) performs token mixing in the time dimension through a fixed-size MLP, and demonstrates performance comparable to that of the multi-head self-attention mechanism (MHSA) on multiple tasks (Choe et al., 2022). However, given that speech tasks typically involve sequences of variable length, the application of MLP Mixer in speech tasks is limited, resulting in poor performance. Summary of the invention

[0004] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0005] Therefore, the purpose of the present invention is to provide a linear complexity model architecture for speech recognition, which, while extracting local features and global features, appropriately fuses the local features and global features in advance during the extraction process to solve the problem of real-time mutual influence between global features and local features.

[0006] To solve the above technical problems, according to one aspect of the present invention, the present invention provides the following technical solutions: A linear complexity model architecture for speech recognition, comprising: an AMLP branch and a convolution branch; The AMLP branch uses MLP to replace the attention mechanism and adds an attention average pooling layer on the basis of MLP to ensure that the attention weights are evenly distributed while maintaining linear complexity and obtaining comprehensive global features; The convolution branch uses a convolutional spatial gating unit to capture enhanced local feature relationships, and mixes local features with global features through a channel that interacts with the AMLP branch to supplement local features for global features.

[0007] As a preferred solution of the linear complexity model architecture for speech recognition described in the present invention, it also includes a feedforward neural network module, and the feedforward neural network modules are located at both ends of the model architecture.

[0008] As a preferred solution of the linear complexity model architecture for speech recognition described in the present invention, the AMLP branch performs global context modeling on the input information, and the specific steps are as follows: Using two MLPs, W1 and W2 are dynamically generated from the input itself for the feature dimension of the first time, and so on until the feature dimension of the Tth time. The feature weights of all dimensions are described as follows: ; Among them, k∈1,2; For each time step x in the input sequence X t Apply multi-layer perceptron MLP k (x t ) generates weights for each time period, obtains feature weights for all time steps, and then stacks these weights into a weight matrix W with variable height k , thus applicable to variable length sequences; When obtaining the weights, the computational complexity is linearly related to the time length T of the input sequence, and is output through the following formula: ; in, It is nonlinear; In the above formula, the weight matrix MLP2 applies a local transformation to each time step to generate a local contribution, which is then inner-producted with the input sequence of the corresponding time step. The results of these local contributions are summed up globally to obtain an overall representation, and then the global projection matrix is ​​obtained through nonlinear transformation. The result of each local transformation MLP1 (x t) are projected through the global projection matrix and integrated into the final output matrix.

[0009] As a preferred solution of a linear complexity model architecture for speech recognition described in the present invention, the AMLP branch has an AMLP module, the AMLP module includes a hypermixing network and additive attention, the hypermixing network is used to dynamically obtain global information; the additive attention further condenses the global information obtained by the hypermixing network, and finally inputs the result into the Hyper Mixer module of the next layer; The AMLP module uses HyperMixer composed of multi-layer perceptrons to extract global information, and then performs weighted averaging on the output matrix through additive attention to obtain a vector with global context information. , the attention results of linear calculation are further normalized using softmax to obtain the attention weight , the formula is as follows: ; in, is a trainable weight vector, is the scaling factor, the global context information vector The calculation formula is as follows: .

[0010] As a preferred solution of the linear complexity model architecture for speech recognition described in the present invention, the convolution branch performs the following steps: For feature sequences Perform nonlinear transformations; Then, it is sent to the key module linear gate CSGU, and then the feature sequence Cut into two equal feature sequences along the hidden layer dimension ,right Perform layer normalization and use deep convolution to obtain strong local features; Finally, Perform element-by-element product to obtain the final local feature matrix : .

[0011] As a preferred solution of the linear complexity model architecture for speech recognition described in the present invention, the convolution branch adds a convolution space gated feature mixing unit for mixing global and local features on the basis of the convolution space gated unit, so as to mix the local context information from the convolution with the global context information of the AMLP branch; The convolution branch linear calculation is as follows: First, the strong local feature matrix It is concatenated with the global feature matrix AMLP(X) to obtain a new feature matrix that retains two complete information, and then sent to the gated linear unit , and with The feature matrix obtained by element-by-element multiplication is residually connected with the matrix that is not mixed with the global features. The specific formula is as follows: ; ; ; in, and are the same.

[0012] Compared with the prior art, the invention has the following original effects: the HMBformer architecture proposed in the invention is a new architecture with linear time complexity, which aims to eliminate the reliance on the self-attention mechanism in speech recognition and reduce computational complexity. The HMBformer architecture uses multiple linear components to supplement global context information, which can efficiently and fully utilize global and local feature information, thereby achieving better speech recognition performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below in combination with the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them: Figure 1 A diagram of the BranchFormer structure in the prior art provided by the present invention; Figure 2 This is a diagram of the architecture of a linear complexity model for speech recognition innovated by the present invention; Figure 3 An AMLP module structure diagram of a linear complexity model architecture for speech recognition applied by the present invention; Figure 4 A structural diagram of a convolutional spatial gated feature mixing unit of a linear complexity model architecture for speech recognition applied by the present invention; Figure 5 Schematic diagram of word error rate (WER) and parameter size and peak VRAM consumption of AISHELL-1 provided for examples of the present invention. DETAILED DESCRIPTION

[0014] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0015] The linear complexity model architecture for speech recognition provided by the present invention is as follows: Figure 2 shown.

[0016] 1. HMBformer Architecture It mainly consists of two branches, namely the convolution branch and the AMLP branch. The convolution branch uses the convolution gated unit to capture the enhanced local feature relationship. In order to supplement the strong local features obtained by the convolution gated unit, the global features are combined with the local features through the channel of interactive information to supplement the global features for the local features.

[0017] The AMLP branch uses MLP instead of the attention mechanism, which can effectively solve the problem of excessive GPU resource consumption. Although this method can effectively reduce GPU consumption, its recognition performance does not reach the level of MHSA. By adding an attention average pooling layer on the basis of MLP, it can ensure that the attention weights are evenly distributed while maintaining linear complexity and obtain more comprehensive global features. The local features obtained by CSGU in the convolution branch will also mix the local features with the global features through the channel of interactive information to supplement the local features with the global features.

[0018] The global features and local features that have complemented each other, and the local features and global features that have been subsequently processed by their respective branches are then fused again.

[0019] Transformer and its variants usually interleave self-attention mechanisms and feed-forward neural network modules (FFN). Models stacked in random order perform better than interleaved models, and placing specific modules at the top and bottom of the model can improve the performance of the model. In the case of replacing attention with AMLP, in order to maintain consistency with models such as Conformer and EfficientConformer with macaron-style architecture, FFN modules are placed at both ends of the model to increase the capacity of the expected model.

[0020] 2. Global feature modeling branch AMLP In order to solve the problem of excessive self-attention calculation, a multi-layer perceptron (MLP) with a simpler structure and lower computational cost is used to replace multi-head self-attention. Multi-layer perceptrons have been proven in some works (such as MLP-ASR and MLP-BASED) to effectively reduce the computational time complexity. The mixed token multi-layer perceptron (such as MLPMixer, gMLP, etc.) can mix the input feature vectors dimension by dimension. The output of MLPMixer is as follows: (1) in, is nonlinear, is a weight matrix applied to fixed-length input data, which means that the dimension of the input data must be predefined. However, speech signals are long-term series data whose length depends on the actual duration of a speech.

[0021] Figure 2 The left branch of the HMBformer in the above example models the global context of the input information. First, two MLPs are used to dynamically generate W1 and W2 from the input itself for the feature dimension of the first time, and so on until the feature dimension of the Tth time. The feature weights of all dimensions are described as follows: (2) For each time step x in the input sequence X t Apply multi-layer perceptron MLP k (x t ), k∈1,2 can generate weights for their respective time periods. In this way, the feature weights at all time steps can be obtained, and then these weights are stacked into a weight matrix W with variable height k , which is applicable to variable-length sequences.

[0022] At each input time step x t The operations are all performed independently and linearly. The weight matrix in each MLP function is randomly initialized and can be continuously trained and adjusted through the back-propagation algorithm. When obtaining the weights, the computational complexity is linearly related to the time length T of the input sequence. Based on the above, formula (1) is rewritten as: (3) The weight matrix MLP2 applies a local transformation to each time step, generates a local contribution, and then performs an inner product with the input sequence of the corresponding time step. The results of these local contributions are summed up globally to obtain an overall representation, and then a nonlinear transformation is performed to obtain a global projection matrix. The result of each local transformation MLP1 (x t ) is projected through the global projection matrix and integrated into the final output matrix, thereby achieving an effect similar to the weight assignment in the attention mechanism. Since this operation is performed independently at each time step, the contribution of each time step is independent of the length of the entire input sequence and is a constant-level calculation. Therefore, the entire process can maintain a low linear complexity on the input length.

[0023] like Figure 3As shown in the figure, the AMLP branch has an AMLP module, which has two branches: one branch is a hypermixer network (Hyper Mixer), which is used to dynamically obtain global information; the other branch further condenses the global information obtained by the hypermixer network through additive attention, and finally inputs the result to the HyperMixer module of the next layer. This process combines the multi-layer perceptron and the additive attention mechanism to improve the expression ability of global features.

[0024] Since there are silences before and after the speech to be recognized, the speech features have less information at both ends. Fusion of feature information by summing alone will affect the overall features. In order to eliminate the impact of meaningless feature sequences on global feature results, the AMLP module uses HyperMixer composed of multi-layer perceptrons to extract global information, and then performs weighted averaging of the output matrix through additive attention to obtain a vector with global context information. The attention results of linear calculation are further normalized using softmax to obtain the attention weight , the formula is as follows: (4) is a trainable weight vector, is the scaling factor, the global context information vector The calculation formula is as follows: (5) In order to ensure the linear complexity of the model, the element-by-element product method is used to model the direct interaction between the global vector and the feature matrix to handle the nonlinear relationship between the vectors. This method still belongs to the linear complexity operation. Using the element-by-element product of the global vector and the feature matrix twice can ensure that the global feature information extracted by HyperMixer is more concentrated, and the linear complexity of the model is guaranteed while performing effective information extraction.

[0025] The concentration of information by multiple additive attentions may cause some information to be lost, especially when the weights are too concentrated on certain features. Therefore, a parallel structure is adopted to merge information through a simple cascade method. All feature information can be retained and passed to the next layer, ensuring that all potentially useful information can be effectively utilized. Finally, the feature matrix including richer information is sent to HyperMixer to further mix the global information and output a higher-dimensional global feature matrix.

[0026] 3 Early fusion of global and local features 3.1 Convolutional Spatial Gating Unit exist Figure 2In the example, the right branch of HMBformer focuses more on extracting local feature information. Existing studies (such as Conformer and Branchformer) have shown that convolution is effective in improving the expressiveness of models. However, ordinary convolution modules cannot capture complex nonlinear relationships and cannot be dynamically adjusted according to data. Therefore, this paper refers to the convolutional spatial gated unit (CSGU) of Branchformer to convert the feature sequence Enter the module: (6) in, is the channel projection. First, the feature sequence is nonlinearly transformed to make the features richer. Then, it is sent to the key module linear gate CSGU. Next, the feature sequence Cut into two equal feature sequences along the hidden layer dimension ,right Perform layer normalization and use deep convolution to obtain strong local features. Finally, Perform element-by-element product to obtain the final local feature matrix : (7) 3.2 Early Feature Fusion The dual-branch architecture usually extracts local and global features separately, and finally fuses them by direct addition or cascading. This method is highly dependent on the final fusion method. If the fusion is not done properly, the complementary information between global and local features cannot be fully utilized, resulting in the loss of some feature information and poor fusion effect. In order to solve this problem, early feature fusion is performed through the GLU gated hybrid unit, which can dynamically select and filter features according to the input data. The model automatically selects important features and suppresses irrelevant or noise features through the gated hybrid unit, so that local feature information and global feature information complement each other, thereby improving the expressiveness of the model. In addition, through residual connections, the risk of mutual interference between branches can be reduced, making feature information more stably transmitted.

[0027] In the convolutional spatial gated unit, the element-by-element product is directly operated between the features at the corresponding positions without introducing additional dependencies, thereby well preserving the local features extracted by convolution. However, when an element in the feature is close to zero, the result at the corresponding position after the element-by-element product will also be close to zero, which may lead to information loss. In order to effectively avoid information loss, features close to zero values ​​are selectively retained or suppressed by adding gating signals. On the basis of the convolutional spatial gated unit, a gated linear unit that mixes global and local features is added, called the convolutional spatial gated feature mixing unit (CSGUFM), such as Figure 4 shown.

[0028] This module is used to mix the local context information from the convolution with the global context information from the AMLP module. The components of the convolutional spatial gated unit (CSGU) are still linearly transformed, and the total computational complexity is linearly related to T. Although the gated unit (sigmoid nonlinear activation function in GLU) is added, the complexity still maintains a linear relationship, because the calculation of the sigmoid function is a constant time operation for each element, and the overall complexity of the entire module is still mainly affected by the linear transformation. The specific linear calculation formula is expressed as follows: (8) (9) (10) , and are the same. First, the strong local feature matrix It is concatenated with the global feature matrix AMLP(X) to obtain a new feature matrix that retains two complete information. Then, it is sent to the gated linear unit , and with Element-by-element product. The obtained feature matrix is ​​residually connected with the matrix that is not mixed with the global features to form the final local feature matrix of the right branch of HMBformer. Similarly, on the left branch of HMBformer, the global feature matrix AMLP(X) is obtained by the same method to obtain the global feature matrix supplemented with local information. Finally, the global and local feature sequence matrices are cascaded and sent to the deep convolution to extract the final feature matrix.

[0029] like Figure 5As shown, all models achieved competitive recognition rates. It is worth noting that using MHSA to merge global context slightly improved the overall word error rate and greatly reduced the required resources. The general linear model does not have the good recognition results of the model using multi-head attention, but the cost is that the video memory is full. The VRAM required for the HyperMixer version and the MHSA version alone has increased from about 22GB to 32GB (i.e., an increase of 45%). The framework structure of the present invention can effectively reduce the required resources and maintain the recognition accuracy with considerable competitiveness. The HMBformer structure can better realize the fusion of global and local features, complement the missing information, and improve the recognition accuracy. While achieving the same recognition effect as the Conformer model using the attention mechanism, the HMBformer reduces the GPU consumption, and the HMBformer is 0.2% higher in accuracy than the most advanced equivalent architecture equipped with HyperMixer, while maintaining the linear complexity while maintaining the accuracy. In addition, its VRAM consumption is reduced by more than two-thirds compared with the most advanced architecture equipped with a multi-head self-attention mechanism (MHSA), showing lower resource requirements.

[0030] Although the present invention has been described above with reference to the embodiments, various modifications may be made thereto and parts thereof may be replaced by equivalents without departing from the scope of the present invention. In particular, as long as there is no structural conflict, the various features in the embodiments disclosed in the present invention may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A linear complexity model architecture for speech recognition, characterized in that: include: AMLP branch and convolution branch; The AMLP branch uses MLP to replace the attention mechanism and adds an attention average pooling layer on the basis of MLP to ensure that the attention weights are evenly distributed while maintaining linear complexity and obtaining comprehensive global features; The convolution branch uses a convolutional spatial gating unit to capture enhanced local feature relationships, and mixes local features with global features through a channel that interacts with the AMLP branch to supplement local features for global features.

2. A linear complexity model architecture for speech recognition according to claim 1, characterized in that: It also includes a feedforward neural network module, which is located at both ends of the model architecture.

3. A linear complexity model architecture for speech recognition according to claim 1, characterized in that: The AMLP branch performs global context modeling on the input information, and the specific steps are as follows: Using two MLPs, W1 and W2 are dynamically generated from the input itself for the feature dimension of the first time, and so on until the feature dimension of the Tth time. The feature weights of all dimensions are described as follows: ； Among them, k∈1,2; For each time step x in the input sequence X t Apply multi-layer perceptron MLP k (x t ) generates weights for each time period, obtains feature weights for all time steps, and then stacks these weights into a weight matrix W with variable height k , thus applicable to variable length sequences; When obtaining the weights, the computational complexity is linearly related to the time length T of the input sequence, and is output through the following formula: ； in, It is nonlinear; In the above formula, the weight matrix MLP2 applies a local transformation to each time step to generate a local contribution, which is then inner-producted with the input sequence of the corresponding time step. The results of these local contributions are summed up globally to obtain an overall representation, and then the global projection matrix is ​​obtained through nonlinear transformation. The result of each local transformation MLP1 (x t ) are projected through the global projection matrix and integrated into the final output matrix.

4. A linear complexity model architecture for speech recognition according to claim 1, characterized in that: The AMLP branch has an AMLP module, which includes a hypermixing network and additive attention. The hypermixing network is used to dynamically obtain global information; the additive attention further condenses the global information obtained by the hypermixing network, and finally inputs the result into the Hyper Mixer module of the next layer; The AMLP module uses HyperMixer composed of multi-layer perceptrons to extract global information, and then performs weighted averaging on the output matrix through additive attention to obtain a vector with global context information. , the attention results of linear calculation are further normalized using softmax to obtain the attention weight , the formula is as follows: ； in, is a trainable weight vector, is the scaling factor, the global context information vector The calculation formula is as follows: 。 5. A linear complexity model architecture for speech recognition according to claim 1, characterized in that: The convolution branch performs the following steps: For feature sequences Perform nonlinear transformations; Then, it is sent to the key module linear gate CSGU, and then the feature sequence Cut into two equal feature sequences along the hidden layer dimension. Perform layer normalization and use deep convolution to obtain strong local features; Finally, and Perform element-by-element product to obtain the final local feature matrix : 。 6. A linear complexity model architecture for speech recognition according to claim 1, characterized in that: The convolution branch adds a convolutional spatial gating feature mixing unit for mixing global and local features on the basis of the convolutional spatial gating unit, so as to mix the local context information from the convolution with the global context information of the AMLP branch; The convolution branch linear calculation is as follows: First, the strong local feature matrix It is concatenated with the global feature matrix AMLP(X) to obtain a new feature matrix that retains two complete information, and then sent to the gated linear unit , and The feature matrix obtained by element-by-element multiplication is residually connected with the matrix that is not mixed with the global features. The specific formula is as follows: ； ； ； in, and are the same.

Images