Speech recognition method and system based on efficient self-attention mechanism

Abstract

The invention provides a speech recognition method and system based on an efficient self-attention mechanism, and the method comprises the steps: introducing a sparse self-attention mechanism and a linear self-attention mechanism, combining a cross-layer self-attention association mechanism, and designing an efficient self-attention layer technology; the sparse self-attention mechanism is based on the low-rank characteristic of the self-attention matrix, and approximately represents the whole self-attention matrix by reserving the maximum singular value and the corresponding singular vector; the linear self-attention mechanism adopts a ReLU function as a sum function substitution scheme; the layer self-attention association mechanism accelerates the learning process of the deep network by transmitting attention information between different layers; an efficient self-attention layer is integrated in a speech recognition model, weight parameters and a fusion proportion are adjusted, and the model is trained and optimized. According to the method, the technical problems that the capture capability of important audio information and the identification capability of natural language and environmental noise are insufficient, the model complexity is high, the linear calculation efficiency is relatively low, and the deep network learning speed is low are solved.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a speech recognition method and system based on an efficient self-attention mechanism. Background Art

[0002] In speech recognition technology, traditional attention calculation methods suffer from high computational complexity and slow training efficiency. Especially when processing long sequences of data, traditional self-attention mechanisms face challenges such as large memory usage and long computation times, making them difficult to meet the demands of real-time processing. Furthermore, in complex environments, such as those affected by noise interference and accent differences, the recognition accuracy of traditional self-attention mechanisms can significantly decrease, further restricting the application of speech recognition technology.

[0003] The existing invention patent application document, "Single-frame Image Super-resolution Method and System Based on Cross-layer Hybrid Attention Transformer," with publication number CN117173025A, discloses a single-frame image super-resolution method and system based on a cross-layer hybrid attention Transformer. The core of this method is to extract deep image features through a multi-level hybrid attention module, and to capture multi-scale contextual information and channel dependencies using a cross-layer multi-scale non-local attention module and a cross-layer sparse channel attention module, ultimately achieving image super-resolution reconstruction. However, this existing technology has application limitations. This existing solution focuses on the field of image super-resolution, and its technical solution is designed for the spatial and visual characteristics of image data, without addressing the temporal and spectral characteristics of speech signals, nor the noise robustness requirements unique to speech recognition tasks. Furthermore, this existing technology does not consider speech characteristics and noise interference, and does not optimize for noise interference, accent variation, speech rate fluctuations, and other issues in speech signals. Its attention mechanism design lacks adaptability to the dynamic characteristics of speech signals.

[0004] The existing invention patent application document with publication number CN118245969A is titled "A multimodal emotion recognition method based on sparse attention fusion mechanism". The core of this existing technology is to use the sparse attention mechanism to allocate feature weights within the modality, and to realize multimodal information interaction through the bottleneck fusion module and the cross-modal attention mechanism. However, this existing technology has limitations in its application field. It is designed for multimodal emotion recognition tasks. Its technical solution needs to process text, visual and audio data at the same time, and it does not perform deep optimization for a single modality (such as pure speech). At the same time, this existing technology does not deeply optimize for speech characteristics: although the existing solution introduces a sparse attention mechanism, its design goal is multimodal fusion, and it does not fully consider the temporal continuity and spectral sparsity of the speech signal and the recognition difficulties in a noisy environment.

[0005] In summary, the existing technology has technical problems such as insufficient ability to capture important audio information and recognize natural language and environmental noise, high model complexity, low linear computing efficiency, and slow deep network learning speed. Summary of the Invention

[0006] The technical problem to be solved by the present invention is: how to solve the technical problems in the existing technology of insufficient ability to capture important audio information and recognize natural language and environmental noise, high model complexity, low linear calculation efficiency, and slow deep network learning speed.

[0007] The present invention solves the above technical problems by adopting the following technical solutions: a speech recognition method based on an efficient self-attention mechanism includes:

[0008] S1. Preprocess the input speech signal;

[0009] S2. Perform sparse self-attention calculation. Based on the low-rank property of the self-attention matrix, retain the maximum singular value in the self-attention matrix and the singular vector corresponding to the maximum singular value to approximate the entire self-attention matrix. Add a linear mapping layer to map the input matrix to a preset size and obtain a sparse self-attention matrix.

[0010] S3. Perform linear self-attention calculations. The ReLU activation function is used to replace the Softmax layer. When calculating the attention weights, the sparse self-attention matrix is used as an intermediate matrix and multiplied by the query matrix Q to optimize the computational complexity.

[0011] S4, cross-layer self-attention association; wherein, an adaptive weighting module is designed to enable the model to introduce cross-layer self-attention association based on the input feature map, combined with the sparse self-attention matrix and linear self-attention output. The attention weights generated by the shallow network are used to control the learning of the deep network according to the attention weights, and the cross-layer self-attention output is obtained by fusion according to the attention weights.

[0012] S5. Use a large-scale speech dataset to train a speech recognition model that integrates an efficient self-attention layer to obtain an applicable model.

[0013] S6. Use the applicable model to recognize the new speech signal.

[0014] This paper utilizes a highly efficient self-attention mechanism. By introducing sparse and linear self-attention mechanisms, combined with a cross-layer self-attention correlation mechanism, a highly efficient self-attention layer technology is designed. This technology aims to improve the model's ability to capture important information in audio and optimize its ability to discern natural language and ambient noise. This paper integrates a highly efficient self-attention layer into the speech recognition model, replacing the traditional self-attention mechanism. The model is trained and optimized by adjusting weight parameters and fusion ratios.

[0015] The present invention improves the robustness and accuracy of the speech recognition model in complex environments, while reducing computational complexity and memory usage, providing users with a more intelligent and personalized voice interaction experience.

[0016] The present invention preprocesses the input speech signal, including operations such as audio signal acquisition, denoising, framing, and windowing, to improve the quality of the audio signal and provide strong support for subsequent feature extraction and model training.

[0017] This invention focuses on the field of speech recognition. It captures the local temporal dependencies of speech signals through a sparse self-attention mechanism. The linear self-attention mechanism reduces computational complexity to adapt to the long sequence characteristics of speech data. The cross-layer self-attention association mechanism enhances the model's ability to integrate speech context information, directly solving the core challenges in speech recognition tasks.

[0018] This invention focuses on the field of speech recognition. Through the collaborative design of sparse self-attention mechanism and linear self-attention mechanism, it significantly reduces the computational complexity and improves the efficiency of speech signal processing; the cross-layer correlation mechanism enhances the model's ability to capture the hierarchical features of speech, achieving more accurate speech recognition.

[0019] In a more specific technical solution, in S2, the query matrix Q, key matrix K, and value matrix V in the input sequence are calculated; the self-attention matrix is obtained by matrix multiplication of the query matrix Q and the key matrix K;

[0020] Perform singular value decomposition (SVD) on the self-attention matrix to obtain the singular values and the left and right singular vectors corresponding to the singular values;

[0021] According to the size of the singular value, retain the k largest singular values and the left and right singular vectors corresponding to the largest singular value;

[0022] The approximate self-attention matrix is reconstructed using the maximum singular value and the left and right singular vectors corresponding to the maximum singular value;

[0023] Add at least two linear mapping layers to the sparse self-attention mechanism to map the input matrix to a preset size. The query matrix Q and the key matrix K are respectively mapped to lower dimensions through the linear mapping layers. Matrix multiplication is performed to obtain a sparse self-attention matrix.

[0024] The sparse self-attention mechanism of the present invention is based on the low-rank property of the self-attention matrix. It approximates the entire self-attention matrix by retaining the largest singular value and its corresponding singular vector, thereby reducing the complexity of the model.

[0025] Based on the low-rank property of the self-attention matrix, this method approximates the entire self-attention matrix by retaining the largest singular value and its corresponding singular vector, thereby reducing the complexity of the model. Furthermore, two linear mapping layers are added to map the input matrix to a smaller size, further improving the computational speed.

[0026] In a more specific technical solution, the following logic is used to calculate the sparse self-attention matrix:

[0027]

[0028] Among them, E and F are the weight matrices of the linear mapping layer, and d and k are the dimensions after mapping.

[0029] In a more specific technical solution, in S3, the Query matrix Q, Key matrix K, and Value matrix V of the input sequence are linearly transformed respectively;

[0030] Calculate the product of the Key matrix K transpose K^T and the Value matrix V to obtain the intermediate matrix;

[0031] Multiply the query matrix Q by the intermediate matrix to obtain the linear self-attention output;

[0032] Apply the ReLU activation function to the linear self-attention output;

[0033] The linear self-attention output is calculated using the following logic:

[0034]

[0035] The linear self-attention mechanism of the present invention abandons the conventional Softmax layer and adopts the ReLU function as an alternative to the sum function, providing linear computational efficiency.

[0036] The present invention abandons the conventional Softmax layer and adopts the ReLU function as an alternative to the sum function, providing linear computational efficiency. When calculating the attention weight, first calculate the product of K^T and V, and then multiply it by Q, reducing the computational complexity from O(n2 ) is significantly reduced to O(n).

[0037] The present invention effectively suppresses noise interference and retains key speech information through low-rank approximation in the sparse self-attention mechanism; the linear self-attention mechanism uses the ReLU function to replace the traditional Softmax to reduce the excessive impact of noise on the attention weight; the cross-layer association mechanism guides deep learning through shallow attention, improving the model's robustness to dynamic changes in speech signals.

[0038] The sparse self-attention mechanism of the present invention is based on the low-rank characteristics of speech signals and achieves efficient compression by retaining key singular values; the linear self-attention mechanism uses the ReLU function to simplify calculations while maintaining sensitivity to key speech information; the cross-layer association mechanism dynamically adjusts the fusion ratio of shallow and deep attention information through an adaptive weighting module, thereby improving the model's recognition ability in complex speech scenarios.

[0039] In a more specific technical solution, in S4, shallow network attention extraction is performed; in the shallow network of the model, the sparse self-attention matrix and linear self-attention output are calculated respectively to obtain prior information;

[0040] Through attention information transfer, the prior information is transferred to the deep network of the model;

[0041] Design and implement adaptive weighting modules;

[0042] In the deep network of the model, an adaptive weighting module is designed to dynamically adjust the sparse attention weight and linear attention weight according to the input feature map;

[0043] The attention coefficient is calculated by the multi-layer perceptron MLP, and the input feature map is converted into the attention coefficient α. The final attention output will be represented as a weighted combination of sparse attention and linear attention through the attention coefficient α;

[0044] Perform attention fusion; fuse the weighted sparse attention and weighted linear attention output by the adaptive weighting module to obtain cross-layer self-attention output;

[0045] According to the previous layer attention map, according to A conv =Conv(A fused ) applies a two-dimensional convolution operation for mixing; where Aconv is the feature map after the convolution operation;

[0046] Perform fusion maximization operation.

[0047] The cross-layer self-attention association mechanism of the present invention accelerates the learning process of deep networks by transferring attention information between different layers.

[0048] The present invention also designs an adaptive weighting module that enables the model to autonomously combine the outputs of sparse attention and linear attention based on the input feature map. Furthermore, it introduces cross-layer self-attention associations, leveraging the rapidly generated attention weights of shallow networks to guide the learning of deeper networks, accelerating the learning process and reducing training time and computational overhead.

[0049] In a more specific technical solution, the following logic is used to add α times linear attention to the sparse attention to obtain the previous layer attention map A fused :

[0050] A fused =(1-β)*A+β*A prev

[0051] Where β controls the attention map of the current layer and the attention maps of the previous layers.

[0052] In a more specific technical solution, the following logic is used to introduce a weight parameter α to adjust the contribution ratio of the attention map of the difference category in the fusion process to balance local details and global information:

[0053] Afinal=(1-α)*Aconv+α*Afused

[0054] In the formula, α is the weight parameter that controls the fusion of the convolution feature map and the original feature map.

[0055] In a more specific technical solution, in S5, the speech recognition model integrating efficient self-attention layers is trained and optimized by adjusting the weight parameters of sparse self-attention and linear self-attention, as well as the fusion ratio of cross-layer self-attention associations;

[0056] Optimization algorithms such as gradient descent are used to update the network weights to minimize the prediction error.

[0057] In more specific technical solutions, prepare large-scale speech datasets;

[0058] Perform data annotation operations on large-scale speech datasets to obtain real annotation information;

[0059] Initialize a speech recognition model with an efficient self-attention layer.

[0060] The preprocessed speech data is input into the speech recognition model integrated with the efficient self-attention layer for forward propagation to obtain the model output result;

[0061] Calculate the loss function; based on the model output and the real annotation information, calculate the loss function value and prediction error of the speech recognition model integrated with the efficient self-attention layer;

[0062] According to the loss function value, back propagation is performed to calculate the gradient of the network weight;

[0063] Based on the gradient of the network weights, the optimizer is used to update the network weights to minimize the loss function value;

[0064] Repeat the aforementioned forward propagation, loss function calculation, backpropagation, and parameter update operations for iterative training until the speech recognition model with integrated efficient self-attention layer converges or reaches the predetermined training rounds;

[0065] The trained model is evaluated using the validation set, and the speech recognition model with an efficient self-attention layer is fine-tuned based on the evaluation results.

[0066] This paper uses a large-scale speech dataset to train a speech recognition model that integrates an efficient self-attention layer. The model is trained and optimized by adjusting the weight parameters of sparse and linear self-attention, as well as the fusion ratio of cross-layer self-attention associations. Optimization algorithms such as gradient descent are used to update network weights to minimize prediction error.

[0067] In a more specific technical solution, the speech recognition system based on the efficient self-attention mechanism includes:

[0068] A preprocessing module, used to preprocess the input speech signal;

[0069] The sparse self-attention calculation module is used to retain the maximum singular value in the self-attention matrix and the singular vector corresponding to the maximum singular value based on the low-rank property of the self-attention matrix, and approximate the entire self-attention matrix. A linear mapping layer is added to map the input matrix to a preset size to obtain the sparse self-attention matrix. The sparse self-attention calculation module is connected to the preprocessing module.

[0070] The linear self-attention calculation module is used to replace the Softmax layer with the ReLU activation function. When calculating the attention weight, the sparse self-attention matrix is used as the intermediate matrix and multiplied with the query matrix Q to optimize the computational complexity. The linear self-attention calculation module is connected to the sparse self-attention calculation module.

[0071] The cross-layer self-attention association module is used to design an adaptive weighting module. The model introduces cross-layer self-attention association based on the input feature map, combined with the sparse self-attention matrix and the linear self-attention output. The attention weights generated by the shallow network are used to control the learning of the deep network. The cross-layer self-attention output is obtained by fusion based on the attention weights. The cross-layer self-attention association module is connected to the linear self-attention calculation module.

[0072] The model training module is used to train a speech recognition model integrated with an efficient self-attention layer using a large-scale speech dataset to obtain an applicable model. The model training module is connected to the cross-layer self-attention association module.

[0073] The speech signal recognition module is used to use an applicable model to recognize new speech signals. The speech signal recognition module is connected to the model training module.

[0074] The improved principles of the present invention compared to D1 include:

[0075] 1. Sparse self-attention calculation:

[0076] Technical Details: This application leverages the low-rank nature of the self-attention matrix to approximate the entire self-attention matrix by retaining the largest singular value and its corresponding singular vector. In this process, a linear mapping layer is introduced to reduce computational complexity.

[0077] Solution Principle: Speech signals are temporally continuous and spectrally sparse. The sparse self-attention mechanism captures key temporal dependencies and spectral features in speech signals. By retaining the largest singular value, the mechanism focuses on the most discriminative parts of the speech signal, effectively suppressing background noise and irrelevant information. The introduction of the linear mapping layer reduces computational effort, making the mechanism more suitable for real-time speech recognition tasks.

[0078] 2. Linear self-attention calculation:

[0079] Technical details: This application uses the ReLU activation function to replace the traditional Softmax layer to reduce computational complexity.

[0080] Solution: The Softmax layer can cause computational bottlenecks in speech recognition, especially when processing long speech sequences. The linear nature of the ReLU function significantly reduces the computational effort. The ReLU function maintains sensitivity to key information in the speech signal while avoiding the information saturation problem that the Softmax layer can cause.

[0081] 3. Cross-layer self-attention association:

[0082] Technical details: This application designs an adaptive weighting module that uses the attention weights generated by shallow networks to guide deep network learning.

[0083] Solution: Speech signals have hierarchical characteristics. Shallow networks can capture local temporal and spectral information, while deep networks can integrate global context. The cross-layer association mechanism uses adaptive weighting to enable the deep network to focus on key speech segments identified by the shallow network, improving the model's robustness to dynamic changes in speech signals. This mechanism effectively addresses the issue in Reference 1, which failed to consider speech signal characteristics and noise interference. By transferring attention through hierarchical layers, it enhances the model's recognition capabilities in complex speech environments.

[0084] The improved principles of the present invention relative to Reference Document 2 are as follows:

[0085] 1. Sparse self-attention calculation:

[0086] Technical details: Same as above, sparse representation is performed based on the low-rank characteristics of the self-attention matrix.

[0087] Solution: While Comparative Document 2 uses a sparse attention mechanism, it doesn't optimize for the characteristics of speech signals and noise interference. The sparse self-attention mechanism in this application more accurately captures key information in speech signals by retaining the largest singular value, effectively suppressing noise. This mechanism demonstrates greater efficiency and accuracy in speech recognition tasks, directly addressing the shortcomings of Comparative Document 2 in terms of application areas and speech characteristic optimization.

[0088] 2. Linear self-attention calculation:

[0089] Technical details: Same as above, using ReLU activation function to replace the Softmax layer.

[0090] Solution principle:

[0091] Comparative Document 2 does not optimize for the long sequence characteristics of speech signals, which may result in high computational complexity and memory usage. The linear self-attention mechanism in this application reduces computational complexity through the ReLU function, making the model more suitable for processing long speech sequences. This improvement improves the real-time performance and scalability of the model, which is of great significance in speech recognition tasks.

[0092] 3. Cross-layer self-attention correlation:

[0093] Technical details: Same as above, design an adaptive weighting module for cross-layer attention transfer.

[0094] Solution principle:

[0095] Comparative Document 2 does not fully consider the hierarchical features and dynamic changes of speech signals. The cross-layer association mechanism of this application uses adaptive weighting to enable the deep network to focus on the key speech segments recognized by the shallow network. This mechanism enhances the robustness of the model to the dynamic changes of speech signals and improves the recognition accuracy in complex speech environments. At the same time, the cross-layer association mechanism also promotes the model's capture of the hierarchical features of speech, which helps to improve the overall performance of speech recognition.

[0096] Compared with the prior art, the present invention has the following advantages:

[0097] This paper utilizes a highly efficient self-attention mechanism. By introducing sparse and linear self-attention mechanisms, combined with a cross-layer self-attention correlation mechanism, a highly efficient self-attention layer technology is designed. This technology aims to improve the model's ability to capture important information in audio and optimize its ability to discern natural language and ambient noise. This paper integrates a highly efficient self-attention layer into the speech recognition model, replacing the traditional self-attention mechanism. The model is trained and optimized by adjusting weight parameters and fusion ratios.

[0098] The present invention improves the robustness and accuracy of the speech recognition model in complex environments, while reducing computational complexity and memory usage, providing users with a more intelligent and personalized voice interaction experience.

[0099] The present invention preprocesses the input speech signal, including operations such as audio signal acquisition, denoising, framing, and windowing, to improve the quality of the audio signal and provide strong support for subsequent feature extraction and model training.

[0100] The sparse self-attention mechanism of the present invention is based on the low-rank property of the self-attention matrix. It approximates the entire self-attention matrix by retaining the largest singular value and its corresponding singular vector, thereby reducing the complexity of the model.

[0101] Based on the low-rank property of the self-attention matrix, this method approximates the entire self-attention matrix by retaining the largest singular value and its corresponding singular vector, thereby reducing the complexity of the model. Furthermore, two linear mapping layers are added to map the input matrix to a smaller size, further improving the computational speed.

[0102] The linear self-attention mechanism of the present invention abandons the conventional Softmax layer and adopts the ReLU function as an alternative to the sum function, providing linear computational efficiency.

[0103] The present invention abandons the conventional Softmax layer and adopts the ReLU function as an alternative to the sum function, providing linear computational efficiency. When calculating the attention weight, first calculate the product of K^T and V, and then multiply it by Q, reducing the computational complexity from O(n2 ) is significantly reduced to O(n).

[0104] The cross-layer self-attention association mechanism of the present invention accelerates the learning process of deep networks by transferring attention information between different layers.

[0105] The present invention also designs an adaptive weighting module that enables the model to autonomously combine the outputs of sparse attention and linear attention based on the input feature map. Furthermore, it introduces cross-layer self-attention associations, leveraging the rapidly generated attention weights of shallow networks to guide the learning of deeper networks, accelerating the learning process and reducing training time and computational overhead.

[0106] This paper uses a large-scale speech dataset to train a speech recognition model that integrates an efficient self-attention layer. The model is trained and optimized by adjusting the weight parameters of sparse and linear self-attention, as well as the fusion ratio of cross-layer self-attention associations. Optimization algorithms such as gradient descent are used to update network weights to minimize prediction error.

[0107] The present invention solves the technical problems existing in the prior art, such as insufficient ability to capture important audio information and recognize natural language and environmental noise, high model complexity, low linear calculation efficiency, and slow deep network learning speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] Figure 1 Schematic diagram of the basic steps of the speech recognition method based on the efficient self-attention mechanism according to Example 1 of the present invention;

[0109] Figure 2 This is an information distribution diagram of the feature matrix of Example 1 of the present invention;

[0110] Figure 3 Schematic diagram of sparse attention data stream processing according to embodiment 1 of the present invention;

[0111] Figure 4 This is an example diagram of linear layer calculation in Example 1 of the present invention;

[0112] Figure 5 Schematic diagram of integrated linear self-attention data stream processing according to embodiment 1 of the present invention;

[0113] Figure 6 This is an example diagram of cross-layer self-attention association in Example 1 of the present invention. DETAILED DESCRIPTION

[0114] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0115] Example 1

[0116] like Figure 1 As shown, the speech recognition method based on the efficient self-attention mechanism provided by the present invention includes the following basic steps:

[0117] S1. Perform data preprocessing;

[0118] Preprocess the input speech signal; specifically, the preprocessing operations include but are not limited to: audio signal acquisition, denoising, framing, and windowing to improve the quality of the audio signal and provide strong support for subsequent feature extraction and model training.

[0119] Specifically, during the data preprocessing process, audio signal collection is performed; specifically, the original voice signal is obtained through an audio collection device such as a microphone;

[0120] Perform denoising; specifically, use filtering algorithms, including but not limited to: Wiener filtering and Kalman filtering, to remove background noise from the audio signal and improve the signal-to-noise ratio of the speech signal;

[0121] Perform frame processing; specifically, cut the continuous speech signal into short frames, and the length of each frame can be set to, for example, 20 to 30 milliseconds, so as to perform subsequent time-frequency analysis.

[0122] Perform windowing processing; specifically, perform a windowing operation on each frame of speech signal, and the windowing operation includes but is not limited to: Hamming window and Hanning window to reduce spectrum leakage and improve the accuracy of spectrum analysis.

[0123] S2. Perform sparse self-attention calculations. Specifically, based on the low-rank nature of the self-attention matrix, the entire self-attention matrix is approximated by retaining the largest singular value and its corresponding singular vector, thereby reducing the complexity of the model. Simultaneously, two linear mapping layers are added to map the input matrix to a smaller size, further improving the computational speed.

[0124] like Figure 2 and Figure 3As shown, in the sparse self-attention calculation process of this embodiment, the self-attention matrix is calculated; specifically, the Query matrix (Q), Key matrix (K), and Value matrix (V) of the input sequence are calculated. Then, the self-attention matrix is obtained by matrix multiplication of Q and K;

[0125] Perform singular value decomposition (SVD); specifically, perform singular value decomposition on the self-attention matrix to obtain singular values and corresponding left and right singular vectors;

[0126] Keep the main components; specifically, keep the largest k singular values and their corresponding left and right singular vectors based on the size of the singular values. Typically, k is much smaller than the dimension n of the self-attention matrix.

[0127] Reconstruct the approximate matrix; specifically, use the retained k singular values and their corresponding left and right singular vectors to reconstruct the approximate self-attention matrix;

[0128] Set up a linear mapping layer; specifically, Figure 4 As shown, to further reduce computational complexity, this embodiment adds two linear mapping layers to the sparse self-attention mechanism to map the input matrix to a smaller size. Specifically, the query matrix (Q) and the key matrix (K) are mapped to a lower dimension through two linear mapping layers respectively; matrix multiplication is then performed to obtain the sparse self-attention matrix.

[0129] In this embodiment, the calculation process of the sparse self-attention mechanism can be expressed by the following formula:

[0130]

[0131] Among them, E and F are the weight matrices of the linear mapping layer, and d and k are the dimensions after mapping.

[0132] S3, perform linear self-attention calculation;

[0133] like Figure 5 As shown in this embodiment, the conventional Softmax layer is abandoned and the ReLU function is used as an alternative to the sum function to provide linear computational efficiency. When calculating the attention weight, the product of K^T and V is first calculated, and then multiplied by Q, reducing the computational complexity from O(n 2 ) is significantly reduced to O(n).

[0134] In the linear self-attention calculation process of this embodiment, a linear transformation is performed; specifically, the Query matrix (Q), Key matrix (K), and Value matrix (V) of the input sequence are linearly transformed respectively;

[0135] Calculate the product of K^T and V; specifically, calculate the product of the transpose of the Key matrix (K^T) and the Value matrix (V) to obtain an intermediate matrix;

[0136] Calculate the product of Q and the intermediate matrix; specifically, multiply the query matrix (Q) with the intermediate matrix obtained in step 2 to obtain the output of linear self-attention;

[0137] Set and apply the ReLU activation function; specifically, apply the ReLU activation function to the output of the linear self-attention to introduce nonlinearity and enhance the expressiveness of the model;

[0138] Compared with the traditional Softmax self-attention mechanism, the linear self-attention mechanism avoids the complex Softmax operation and reduces the computational complexity from O(n 2 ) is significantly reduced to O(n), thus greatly improving the computational efficiency. The computational process of the linear self-attention mechanism can be expressed by the following formula:

[0139]

[0140] S4, cross-layer self-attention correlation;

[0141] like Figure 6 As shown, in this embodiment, an adaptive weighting module is designed to enable the model to autonomously combine the outputs of sparse attention and linear attention based on the input feature map. At the same time, cross-layer self-attention associations are introduced, and the attention weights quickly generated by the shallow network are used to guide the learning of the deep network, accelerating the learning process of the deep network and reducing training time and computational overhead.

[0142] In the cross-layer self-attention association process of this embodiment, shallow network attention extraction is performed; specifically, in the shallow network of the model, the outputs of sparse self-attention and linear self-attention are calculated respectively;

[0143] Perform attention information transfer; specifically, the sparse self-attention and linear self-attention outputs calculated by the shallow network are passed to the deep network of the model as prior information;

[0144] Design and apply an adaptive weighting module; specifically, in the deep network of the model, design an adaptive weighting module that dynamically adjusts the weights of sparse attention and linear attention based on the input feature map, realizing the complementary advantages of the two attention mechanisms. The calculation process of the attention coefficient can be implemented through a multi-layer perceptron (MLP), which converts the input feature map into an attention coefficient α. The final attention output will be represented by α as a weighted combination of sparse attention and linear attention, that is, sparse attention plus α times linear attention, that is:

[0145] A fused =(1-β)*A+β*A prev

[0146] Among them, Aprev is the attention map of the previous layer, and β controls the attention map of the current layer and the attention maps of the previous layers.

[0147] Perform attention fusion; specifically, fuse the weighted sparse attention and weighted linear attention outputs of the adaptive weighting module to obtain the final cross-layer self-attention output. The final attention output is a weighted combination of sparse attention and linear attention.

[0148] The weighted attention map will be A conv =Conv(A fused ) applies a two-dimensional convolution operation for further mixing. Where Aconv is the feature map after the convolution operation.

[0149] A fusion maximization operation is performed; specifically, a two-dimensional convolution operation captures spatial dependencies and semantic associations while mixing weighted attention weights, supporting the generation of more representative attention maps. After the convolution step, the attention maps of each data type are weighted and combined separately. During speech signal processing, new attention maps are generated through convolution operations, capturing deeper spatial and semantic features; while the original attention maps retain the global information of the initial features. To ensure the maximum effect of feature fusion, a second weight parameter α is introduced as shown in the following formula to adjust the contribution ratio of the new and old attention maps in the fusion process.

[0150] Afinal=(1-α)*Aconv+α*Afused

[0151] Here, α is the weight parameter that controls the fusion of the convolutional feature map with the original feature map. The goal of this weighted combination is to balance local details and global information so that the final drawing attention map can fully reflect the semantic features at different levels.

[0152] S5, model training and optimization;

[0153] In this example, a speech recognition model integrated with an efficient self-attention layer was trained using a large-scale speech dataset. The model was trained and optimized by adjusting the weight parameters of sparse and linear self-attention, as well as the fusion ratio of cross-layer self-attention associations. Optimization algorithms such as gradient descent were used to update the network weights to minimize prediction error.

[0154] In the model training and optimization operations of this embodiment, a data set is prepared; specifically, a large-scale speech data set is prepared, covering speech data in various scenarios, various accents, and various noise environments, to improve the generalization ability of the model;

[0155] Perform data annotation operations; specifically, accurately annotate speech datasets, including but not limited to: text transcription of speech signals, speaker identity, and emotional state;

[0156] Performing model initialization operations, specifically initializing the speech recognition model integrated with an efficient self-attention layer, including but not limited to: initializing network weights and setting optimizer parameters;

[0157] Perform forward propagation operation; specifically, input the preprocessed speech data into the model, perform forward propagation, and obtain the output result of the model;

[0158] Calculate the loss function; specifically, based on the model's output and the actual annotation information, calculate the model's loss function value to measure the model's prediction error;

[0159] Perform back propagation operations; specifically, perform back propagation based on the loss function value to calculate the gradient of the network weights;

[0160] Perform parameter update operations; specifically, based on the gradient of the network weights, use an optimizer such as stochastic gradient descent (SGD) or Adam to update the network weights to minimize the loss function value;

[0161] Perform iterative model training; in this embodiment, the steps of forward propagation, loss function calculation, back propagation, and parameter update are repeated to iteratively train the model until the model converges or reaches a predetermined number of training rounds;

[0162] Model evaluation and tuning: In this embodiment, the trained model is evaluated using a validation set, and the model is tuned based on the evaluation results. The aforementioned tuning operations include but are not limited to: adjusting the learning rate, regularization coefficient, and attention weight parameters to further improve the performance of the model.

[0163] S6. After the model training is completed, the trained model can be used to recognize new speech signals;

[0164] In the operations of sound classification and speech recognition, voice signal input is performed: the voice signal to be recognized is input into the trained model;

[0165] Perform feature extraction; specifically, the model automatically extracts features from the input speech signal. The features to be extracted include but are not limited to: sparse self-attention features and linear self-attention features;

[0166] Perform attention fusion operation; specifically, the model fuses the extracted sparse self-attention features and linear self-attention features to obtain the final speech feature representation;

[0167] Perform classification or recognition; specifically, the model performs classification or recognition based on the fused speech feature representation and outputs the text transcription result or classification label of the speech signal.

[0168] In summary, by combining sparse self-attention and linear self-attention, the present invention significantly reduces the computational complexity of the model, improves the ability to process long sequence data, makes the model more suitable for real-time processing tasks, improves computational efficiency, and reduces computational complexity.

[0169] The present invention adopts a sparse self-attention mechanism to reduce the number of attention weights that need to be stored and calculated, thereby reducing memory usage, making the model more efficient when processing long speech inputs, reducing the consumption of memory resources, reducing memory usage, and being suitable for real-time processing.

[0170] The present invention adopts a cross-layer self-attention association mechanism to enhance the robustness of the model, so that the model can still maintain a high recognition accuracy when facing complex environments. Especially in complex environments such as noise interference and accent differences, the efficient self-attention mechanism of the present invention can still maintain stable performance, strengthen the model robustness, and improve the recognition accuracy.

[0171] In complex environments, the efficient self-attention mechanism of the present invention can significantly improve the performance of the speech recognition model, provide users with a more intelligent and personalized voice interaction experience, and improve the performance of the speech recognition model.

[0172] The present invention solves the technical problems existing in the prior art, such as insufficient ability to capture important audio information and recognize natural language and environmental noise, high model complexity, low linear calculation efficiency, and slow deep network learning speed.

[0173] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A speech recognition method based on an efficient self-attention mechanism, characterized in that: The method comprises: S1. Preprocess the input speech signal; S2. Perform sparse self-attention calculation; based on the low-rank property of the self-attention matrix, retain the maximum singular value in the self-attention matrix and the singular vector corresponding to the maximum singular value to approximate the entire self-attention matrix; add a linear mapping layer to map the input matrix to a preset size to obtain a sparse self-attention matrix; S3. Perform linear self-attention calculation; wherein, the Softmax layer is replaced by the ReLU activation function; when calculating the attention weight, the sparse self-attention matrix is used as an intermediate matrix, and the intermediate matrix is multiplied by the query matrix Q to adjust the computational complexity; S4, cross-layer self-attention association; wherein, an adaptive weighting module is designed to enable the model to introduce cross-layer self-attention association based on the input feature map, the sparse self-attention matrix, and the linear self-attention output, and use the attention weights generated by the shallow network to control the learning of the deep network according to the attention weights, and fuse the cross-layer self-attention output according to the attention weights; S5. Use a large-scale speech dataset to train a speech recognition model that integrates an efficient self-attention layer to obtain an applicable model. S6. Using the applicable model, recognize a new speech signal.

2. The speech recognition method based on the efficient self-attention mechanism according to claim 1 is characterized in that: In S2, the query matrix Q, the key matrix K, and the value matrix V in the input sequence are calculated; and a self-attention matrix is obtained by matrix multiplication of the query matrix Q and the key matrix K. Performing singular value decomposition (SVD) on the self-attention matrix to obtain singular values and left and right singular vectors corresponding to the singular values; According to the size of the singular value, retain k maximum singular values and the left and right singular vectors corresponding to the maximum singular values; Reconstructing an approximate self-attention matrix using the maximum singular value and the left and right singular vectors corresponding to the maximum singular value; Add at least two linear mapping layers to the sparse self-attention mechanism to map the input matrix to the preset size; wherein the Query matrix Q and the Key matrix K are respectively mapped to a lower dimension through each linear mapping layer; perform matrix multiplication operation to obtain the sparse self-attention matrix.

3. The speech recognition method based on the efficient self-attention mechanism according to claim 2, characterized in that: The sparse self-attention matrix is calculated using the following logic: Where E and F are the weight matrices of the linear mapping layer, and d and k are the dimensions after mapping.

4. The speech recognition method based on the efficient self-attention mechanism according to claim 1, characterized in that: In S3, linear transformations are performed on the Query matrix Q, the Key matrix K, and the Value matrix V of the input sequence respectively; Calculate the product of the transposed K^T of the Key matrix K and the Value matrix V to obtain the intermediate matrix; Multiply the query matrix Q by the intermediate matrix to obtain a linear self-attention output; Applying the ReLU activation function to the linear self-attention output; The linear self-attention output is calculated using the following logic:

5. The speech recognition method based on the efficient self-attention mechanism according to claim 1, characterized in that: In S4, shallow network attention extraction is performed; in the shallow network of the model, the sparse self-attention matrix and the linear self-attention output are respectively calculated to obtain prior information; The prior information is transferred to the deep network of the model through attention information transfer; Design and implement adaptive weighting modules; In the deep network of the model, an adaptive weighting module is designed to dynamically adjust the sparse attention weight and linear attention weight according to the input feature map; The input feature map is converted into an attention coefficient α by calculating the attention coefficient through a multi-layer perceptron (MLP). The final attention output is represented as a weighted combination of sparse attention and linear attention by the attention coefficient α. Perform attention fusion; The weighted sparse attention and weighted linear attention output by the adaptive weighting module are fused to obtain the cross-layer self-attention output. According to the previous layer attention map, according to A conv =Conv(A fused ) applies a two-dimensional convolution operation for mixing; where Aconv is the feature map after the convolution operation; Perform fusion maximization operation.

6. The speech recognition method based on the efficient self-attention mechanism according to claim 5, characterized in that: Using the following logic, add α times linear attention to the sparse attention to obtain the attention map A of the previous layer fused : A fused =(1-β)*A+β*A prev Where β controls the attention map of the current layer and the attention maps of the previous layers.

7. The speech recognition method based on the efficient self-attention mechanism according to claim 5, characterized in that: Using the following logic, we introduce a weight parameter α to adjust the contribution ratio of the attention map of the difference category in the fusion process to balance local details and global information: Afinal=(1-α)*Aconv+α*Afused In the formula, α is the weight parameter that controls the fusion of the convolution feature map and the original feature map.

8. The speech recognition method based on the efficient self-attention mechanism according to claim 1, characterized in that: In S5, the speech recognition model of the integrated efficient self-attention layer is trained and optimized by adjusting the weight parameters of sparse self-attention and linear self-attention, as well as the fusion ratio of cross-layer self-attention associations; Optimization algorithms such as gradient descent are used to update the network weights to minimize the prediction error.

9. The speech recognition method based on the efficient self-attention mechanism according to claim 8, characterized in that: Preparing the large-scale speech dataset; Performing a data annotation operation on the large-scale speech data set to obtain real annotation information; Initializing the speech recognition model integrating the efficient self-attention layer; The preprocessed speech data is input into the speech recognition model with the integrated efficient self-attention layer, and forward propagation is performed to obtain the model output result; Calculate the loss function; Calculating the loss function value and prediction error of the speech recognition model integrated with the efficient self-attention layer based on the model output result and the real annotation information; According to the loss function value, back propagation is performed to calculate the gradient of the network weight; Based on the gradient of the network weights, the optimizer is used to update the network weights to minimize the loss function value; Repeat the aforementioned forward propagation, loss function calculation, backpropagation, and parameter update operations to perform iterative training until the speech recognition model with the integrated efficient self-attention layer converges or reaches a predetermined training round; The trained model is evaluated using the validation set, and the speech recognition model with the integrated efficient self-attention layer is tuned based on the evaluation results.

10. A speech recognition system based on an efficient self-attention mechanism, characterized by: The system comprises: A preprocessing module, used to preprocess the input speech signal; A sparse self-attention calculation module is used to retain the maximum singular value in the self-attention matrix and the singular vector corresponding to the maximum singular value based on the low-rank property of the self-attention matrix, so as to approximately represent the entire self-attention matrix; a linear mapping layer is added to map the input matrix to a preset size to obtain a sparse self-attention matrix, and the sparse self-attention calculation module is connected to the preprocessing module; A linear self-attention calculation module is used to replace the Softmax layer with a ReLU activation function; when calculating the attention weight, the sparse self-attention matrix is used as an intermediate matrix, and the intermediate matrix is multiplied by the query matrix Q to optimize the computational complexity. The linear self-attention calculation module is connected to the sparse self-attention calculation module; A cross-layer self-attention association module is used to design an adaptive weighting module so that the model introduces cross-layer self-attention association based on the input feature map, the sparse self-attention matrix, and the linear self-attention output. The attention weights generated by the shallow network are used to control the learning of the deep network according to the attention weights. The cross-layer self-attention output is obtained by fusion according to the attention weights. The cross-layer self-attention association module is connected to the linear self-attention calculation module. A model training module is used to train a speech recognition model integrated with an efficient self-attention layer using a large-scale speech dataset to obtain an applicable model, wherein the model training module is connected to the cross-layer self-attention association module; The speech signal recognition module is used to use the applicable model to recognize new speech signals, and the speech signal recognition module is connected to the model training module.

Images