A low-complexity efficient attention method, device, apparatus and storage medium

Abstract

The application discloses a low-complexity efficient attention method, device, equipment and storage medium. A compression vector s is introduced, two-step global interaction attention is used, and floating point calculation is only Dxd and dxD times in each use. When an input sequence is particularly long, the calculation complexity is not affected by the length of the input sequence, is only related to the embedding dimension, the occupation of the calculation resource can be reduced, the calculation cost of the network is further reduced, and the technical problems that a current speech recognition method based on a self-attention mechanism has large memory consumption and long calculation time when facing a large-scale input sequence are solved.

Description

Technical Field

[0001] This application relates to the field of attention mechanism technology, and in particular to a low-complexity, high-efficiency attention method, apparatus, device and storage medium. Background Technology

[0002] In the field of speech recognition, attention mechanisms have become an important technique for improving model performance. Attention mechanisms include, but are not limited to, self-attention mechanisms, sparse attention mechanisms, and hierarchical attention mechanisms, among which:

[0003] Sparse attention reduces computation by selectively focusing on key parts of a sequence, but this approach may result in the loss of some important information because information in non-focused regions is ignored; furthermore, determining which information is "key" may require additional computation or rely on heuristics, which may introduce bias.

[0004] Hierarchical attention reduces complexity by processing information at different levels, but this approach may introduce inconsistencies in representation learning between different levels, affecting the overall performance of the model.

[0005] Approximate attention reduces computation by approximating calculations, but this approximation may sacrifice some accuracy, thus affecting the quality of the final recognition result.

[0006] Through self-attention, the model can capture the correlation between any two positions in the input sequence, which is crucial for understanding the structure and semantics of speech signals.

[0007] However, the self-attention mechanism requires computing and storing an n×n relevance score matrix, where n is the length of the sequence. This means that for a sequence of length n, the model needs to store n^2 relevance scores. For example, for a sequence of length 1000, the model needs to store up to 10 billion elements, which is a huge memory consumption.

[0008] Meanwhile, calculating the correlation score matrix also requires O(n^2) time, meaning that the computational cost increases quadratically with the sequence length. For a sequence of length 1000, the model needs to perform approximately 10 billion correlation calculations, which is unacceptable in real-time applications.

[0009] Therefore, this self-attention-based speech recognition method faces challenges in terms of memory consumption and computation time when dealing with large-scale input sequences. To address this issue, there is a need to propose a more efficient method to reduce space and time complexity, thereby improving the performance and real-time capabilities of speech recognition systems. Summary of the Invention

[0010] This application provides a low-complexity, high-efficiency attention method, apparatus, device, and storage medium, which solves the technical problems of large memory consumption and long computation time in existing speech recognition methods based on self-attention mechanisms when dealing with large-scale input sequences.

[0011] In view of this, the first aspect of this application provides a low-complexity, efficient attention method, the method comprising:

[0012] Step S1: Obtain the voice input sequence x;

[0013] Step S2: Perform a linear transformation on the speech input sequence x to obtain the query vector q, key vector k, value vector v, and compression vector s, where... ;

[0014] Step S3: Perform a quadratic linear transformation on the query vector q, the key vector k, the value vector v, and the compressed vector s to generate feature maps in different spaces. , , as well as ,in , ;

[0015] Step S4: Perform the following based on the softmax function. The two-step global interaction attention weight calculation yields the first attention weight value and the second attention weight value, respectively;

[0016] Step S5: Map the first attention weight value and the second attention weight value to the feature of the value vector v. Perform weighted output to obtain the final output result.

[0017] Optionally, step S2 specifically includes:

[0018] The speech input sequence x is subjected to a linear transformation to obtain the query vector q, key vector k, value vector v, and compression vector s, specifically:

[0019] , , , ;

[0020] in , , as well as , , , as well as These are the weight matrices for query, key, value, and compression, respectively.

[0021] Optionally, step S3 specifically includes:

[0022] The query vector q, the key vector k, the value vector v, and the compressed vector s are subjected to quadratic linear transformations to generate feature maps in different spaces. , , as well as , specifically:

[0023] ;

[0024] in: , , , and , Represented as Attention space of size , ;

[0025] Optionally, step S4 specifically includes:

[0026] Based on the softmax function, respectively Two-step global interaction attention weight calculation:

[0027] ;

[0028] ;

[0029] in, This is the scaling factor.

[0030] A second aspect of this application provides a low-complexity, high-efficiency attention device, the device comprising:

[0031] Acquisition unit, used to acquire speech input sequence x;

[0032] The first transformation unit is used to perform a linear transformation on the speech input sequence x to obtain a query vector q, a key vector k, a value vector v, and a compression vector s, wherein... ;

[0033] The second transformation unit is used to perform quadratic linear transformations on the query vector q, the key vector k, the value vector v, and the compression vector s to generate feature maps in different spaces. , , as well as ,in , ;

[0034] The computational unit is used to perform calculations based on the softmax function. The two-step global interaction attention weight calculation yields the first attention weight value and the second attention weight value, respectively;

[0035] Output unit, configured to map the first attention weight value and the second attention weight value to the feature of the value vector v. Perform weighted output to obtain the final output result.

[0036] Optionally, the first transformation unit is specifically used for:

[0037] The speech input sequence x is subjected to a linear transformation to obtain the query vector q, key vector k, value vector v, and compression vector s, specifically:

[0038] , , , ;

[0039] in , , as well as , , , as well as These are the weight matrices for query, key, value, and compression, respectively.

[0040] Optionally, the second transformation unit is specifically used for:

[0041] The query vector q, the key vector k, the value vector v, and the compressed vector s are subjected to quadratic linear transformations to generate feature maps in different spaces. , , as well as , specifically:

[0042] ;

[0043] in: , , , and , Represented as Attention space of size , ;

[0044] Optionally, the computing unit is specifically used for:

[0045] Based on the softmax function, respectively Two-step global interaction attention weight calculation:

[0046] ;

[0047] ;

[0048] in, This is the scaling factor.

[0049] A third aspect of this application provides a low-complexity, high-efficiency attention device, the device comprising a processor and a memory:

[0050] The memory is used to store program code and transmit the program code to the processor;

[0051] The processor is configured to execute, according to instructions in the program code, the steps of the low-complexity, high-efficiency attention method as described in the first aspect above.

[0052] A fourth aspect of this application provides a computer-readable storage medium for storing program code for performing the method described in the first aspect above.

[0053] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0054] This application provides a low-complexity, high-efficiency attention method, apparatus, device, and storage medium. By introducing a compressed vector s, it uses a two-step global interactive attention mechanism, but the floating-point computation in each use is only a fraction of a second. and Secondly, when the input sequence is particularly long, the computational complexity is not affected by the length of the input sequence, but only by the embedding dimension. This can reduce the consumption of computing resources and further reduce the computational cost of the network. It solves the technical problem of large memory consumption and long computation time in existing speech recognition methods based on self-attention mechanism when facing large-scale input sequences. Attached Figure Description

[0055] Figure 1 This is a flowchart of a low-complexity, high-efficiency attention method in the embodiments of this application;

[0056] Figure 2 This is a schematic diagram of the structure of a low-complexity, high-efficiency attention device in an embodiment of this application;

[0057] Figure 3 This is a schematic diagram of the structure of a low-complexity, high-efficiency attention device in an embodiment of this application;

[0058] Figure 4 This is a diagram illustrating the computational process of the self-attention mechanism.

[0059] Figure 5 This is a diagram illustrating the computational process of low-complexity, high-efficiency attention in this invention. Detailed Implementation

[0060] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0061] This application designs a low-complexity, high-efficiency attention method, device, equipment, and storage medium, which solves the technical problems of large memory consumption and long computation time in existing speech recognition methods based on self-attention mechanisms when facing large-scale input sequences.

[0062] For easier understanding, please refer to Figure 1 , Figure 1 This is a flowchart of a low-complexity, high-efficiency attention method in the embodiments of this application, such as... Figure 1 As shown, specifically:

[0063] Step S1: Obtain the voice input sequence x;

[0064] It should be noted that, firstly, an audio signal needs to be recorded or acquired, which is usually an analog signal that needs to be converted into a digital signal by an analog-to-digital converter (ADC).

[0065] Preprocessing digital audio signals may include steps such as noise reduction, normalization, and silence removal to improve the accuracy of subsequent processing.

[0066] The continuous audio signal is divided into shorter frames (typically between 20 and 40 milliseconds), each frame containing a short duration of audio signal.

[0067] Apply a window function (such as a Hamming window) to each frame to reduce the impact of discontinuities at frame boundaries.

[0068] Perform a Fast Fourier Transform (FFT) on the windowed signal to transform it from the time domain to the frequency domain.

[0069] Features are extracted from frequency domain signals. Common speech features include Mel frequency cepstral coefficients (MFCCs), Mel spectral energy features (MFBs), and Mel spectral dynamic features (MFSDs).

[0070] The extracted features are organized into a sequence in chronological order, with the features of each frame forming a sequence element, and the entire speech segment is converted into a feature sequence.

[0071] Preferably, VAD is used to detect and remove silent portions, retaining only frames containing speech.

[0072] Preferably, the start and end points of the speech are determined for further processing.

[0073] Preferably, post-processing of the feature sequence may include feature normalization, denoising, and feature enhancement.

[0074] Step S2: Perform a linear transformation on the speech input sequence x to obtain the query vector q, key vector k, value vector v, and compression vector s, where... ;

[0075] It should be noted that for query, key, value, and compression, the speech input sequence x undergoes four different linear transformations, namely weight matrices, to generate the query vector q, key vector k, value vector v, and compression vector s. D is the embedding dimension, which can be determined based on the need to reduce computational complexity.

[0076] Step S3: Perform quadratic linear transformations on the query vector q, key vector k, value vector v, and compressed vector s respectively to generate feature maps in different spaces. , , as well as ,in , ;

[0077] Step S4: Perform the following based on the softmax function. The two-step global interaction attention weight calculation yields the first attention weight value and the second attention weight value, respectively;

[0078] It should be noted that since the raw audio data is converted to the frequency domain, the range acceptable to humans in the frequency domain is very limited. Although adjustments are made using Mel-spectral coefficients and other methods during data preprocessing, a large amount of redundant information still exists in the input features. Therefore, within a certain range, the input audio features exhibit scale similarity, meaning that using higher-dimensional or lower-dimensional representations has little impact on the final prediction.

[0079] like Figure 4 As shown, Figure 4 This is a diagram illustrating the computational process of a self-attention mechanism. For a traditional self-attention mechanism, the input... , and After linear transformation, feature maps Q, K, and V in different spaces are obtained, as shown in the following formulas:

[0080]

[0081]

[0082] Here, , , ,and ; Represented as The size of the attention space facilitates the capture of attention features in different spaces; scaling factor It is used to reduce due to Dot product calculation The impact of functions.

[0083] Here The operation requires Floating-point calculations require a significant amount of time and computing resources.

[0084] Because we added a compression vector s, s after the linear transformation Here This allows the traditional one-step attention mechanism to be transformed into a two-step global interactive attention mechanism, but the floating-point calculations in each use are only a fraction of the total. and Secondly, when the input sequence is particularly long, i.e. Traditional self-attention mechanisms face serious computational problems, while the technical solution of this application is not affected by the length of the input sequence. The computational complexity of the attention mechanism is only related to the embedding dimension under arbitrary length, and for compressed vectors, the computational cost of the network can be further reduced.

[0085] Step S5: Map the first attention weight value and the second attention weight value to the feature of the value vector v. Perform weighted output to obtain the final output result.

[0086] It should be noted that the calculated attention weights are used to sum the weighted vector v to generate the final output. This weighted sum represents the contribution of each element in the sequence to the output, where the weights are determined by their relevance to the query.

[0087] Preferably, in the Transformer architecture, this process can be replicated multiple times, forming a multi-head attention mechanism, where each head learns a different representation of the sequence. Each head has its own... , , as well as .

[0088] After calculating self-attention, residual connections and layer normalization are typically used to help stabilize the training process and allow the model to learn deeper representations.

[0089] Following the self-attention layer, there is usually a feedforward network (typically consisting of a linear layer followed by a non-linear activation function) to further process the output of the self-attention layer.

[0090] Furthermore, step S2 specifically includes:

[0091] The speech input sequence x is transformed by a single linear transformation to obtain the query vector q, key vector k, value vector v, and compression vector s. Specifically:

[0092] , , , ;

[0093] in , , as well as , , , as well as These are the weight matrices for query, key, value, and compression, respectively.

[0094] It should be noted that for query, key, value, and compression, the speech input sequence x undergoes four different linear transformations, namely weight matrices, to generate the query vector q, key vector k, value vector v, and compression vector s. D is the embedding dimension, which can be determined based on the need to reduce computational complexity.

[0095] These transformations can be expressed as:

[0096] , , , ;

[0097] in , , as well as These are the weight matrices for query, key, value, and compression algorithms, respectively.

[0098] Furthermore, step S3 specifically includes:

[0099] The query vector q, key vector k, value vector v, and compressed vector s are subjected to quadratic linear transformations to generate feature maps in different spaces. , , as well as , specifically:

[0100] ;

[0101] in: , , , and , Represented as Attention space of size , ;

[0102] Furthermore, step S4 specifically includes:

[0103] Based on the softmax function, respectively Two-step global interaction attention weight calculation:

[0104] ;

[0105] ;

[0106] in, This is the scaling factor.

[0107] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of a low-complexity, high-efficiency attention device in an embodiment of this application, as shown below. Figure 2 As shown, specifically:

[0108] Acquisition unit 201 is used to acquire the speech input sequence x;

[0109] It should be noted that, firstly, an audio signal needs to be recorded or acquired, which is usually an analog signal that needs to be converted into a digital signal by an analog-to-digital converter (ADC).

[0110] Preprocessing digital audio signals may include steps such as noise reduction, normalization, and silence removal to improve the accuracy of subsequent processing.

[0111] The continuous audio signal is divided into shorter frames (typically between 20 and 40 milliseconds), each frame containing a short duration of audio signal.

[0112] Apply a window function (such as a Hamming window) to each frame to reduce the impact of discontinuities at frame boundaries.

[0113] Perform a Fast Fourier Transform (FFT) on the windowed signal to transform it from the time domain to the frequency domain.

[0114] Features are extracted from frequency domain signals. Common speech features include Mel frequency cepstral coefficients (MFCCs), Mel spectral energy features (MFBs), and Mel spectral dynamic features (MFSDs).

[0115] The extracted features are organized into a sequence in chronological order, with the features of each frame forming a sequence element, and the entire speech segment is converted into a feature sequence.

[0116] Preferably, VAD is used to detect and remove silent portions, retaining only frames containing speech.

[0117] Preferably, the start and end points of the speech are determined for further processing.

[0118] Preferably, post-processing of the feature sequence may include feature normalization, denoising, and feature enhancement.

[0119] The first transformation unit 202 is used to perform a linear transformation on the speech input sequence x to obtain a query vector q, a key vector k, a value vector v, and a compression vector s, wherein... ;

[0120] It should be noted that for query, key, value, and compression, the speech input sequence x undergoes four different linear transformations, namely weight matrices, to generate the query vector q, key vector k, value vector v, and compression vector s. D is the embedding dimension, which can be determined based on the need to reduce computational complexity.

[0121] The second transformation unit 203 is used to perform quadratic linear transformations on the query vector q, key vector k, value vector v, and compression vector s to generate feature maps in different spaces. , , as well as ,in , ;

[0122] Computation unit 204 is used to perform calculations based on the softmax function. The two-step global interaction attention weight calculation yields the first attention weight value and the second attention weight value, respectively;

[0123] It should be noted that since the raw audio data is converted to the frequency domain, the range acceptable to humans in the frequency domain is very limited. Although adjustments are made using Mel-spectral coefficients and other methods during data preprocessing, a large amount of redundant information still exists in the input features. Therefore, within a certain range, the input audio features exhibit scale similarity, meaning that using higher-dimensional or lower-dimensional representations has little impact on the final prediction.

[0124] like Figure 4 As shown, Figure 4 This is a diagram illustrating the computational process of a self-attention mechanism. For a traditional self-attention mechanism, the input... , and After linear transformation, feature maps Q, K, and V in different spaces are obtained, as shown in the following formulas:

[0125]

[0126]

[0127] Here, , , ,and ; Represented as The size of the attention space facilitates the capture of attention features in different spaces; scaling factor It is used to reduce due to Dot product calculation The impact of functions.

[0128] Here The operation requires Floating-point calculations require a significant amount of time and computing resources.

[0129] Because we added a compression vector s, s after the linear transformation Here This allows the traditional one-step attention mechanism to be transformed into a two-step global interactive attention mechanism, but the floating-point calculations in each use are only a fraction of the total. and Secondly, when the input sequence is particularly long, i.e. Traditional self-attention mechanisms face serious computational problems, while the technical solution of this application is not affected by the length of the input sequence. The computational complexity of the attention mechanism is only related to the embedding dimension under arbitrary length, and for compressed vectors, the computational cost of the network can be further reduced.

[0130] Output unit 205 is used to map the first attention weight value and the second attention weight value to the feature of the value vector v. Perform weighted output to obtain the final output result.

[0131] It should be noted that the calculated attention weights are used to sum the weighted vector v to generate the final output. This weighted sum represents the contribution of each element in the sequence to the output, where the weights are determined by their relevance to the query.

[0132] Preferably, in the Transformer architecture, this process can be replicated multiple times, forming a multi-head attention mechanism, where each head learns a different representation of the sequence. Each head has its own... , , as well as .

[0133] After calculating self-attention, residual connections and layer normalization are typically used to help stabilize the training process and allow the model to learn deeper representations.

[0134] Following the self-attention layer, there is usually a feedforward network (typically consisting of a linear layer followed by a non-linear activation function) to further process the output of the self-attention layer.

[0135] Furthermore, the first transformation unit 202 is specifically used for:

[0136] The speech input sequence x is transformed by a single linear transformation to obtain the query vector q, key vector k, value vector v, and compression vector s. Specifically:

[0137] , , , ;

[0138] in , , as well as , , , as well as These are the weight matrices for query, key, value, and compression, respectively.

[0139] It should be noted that for query, key, value, and compression, the speech input sequence x undergoes four different linear transformations, namely weight matrices, to generate the query vector q, key vector k, value vector v, and compression vector s. D is the embedding dimension, which can be determined based on the need to reduce computational complexity.

[0140] These transformations can be expressed as:

[0141] , , , ;

[0142] in , , as well as These are the weight matrices for query, key, value, and compression algorithms, respectively.

[0143] Furthermore, the second transformation unit 203 is specifically used for:

[0144] The query vector q, key vector k, value vector v, and compressed vector s are subjected to quadratic linear transformations to generate feature maps in different spaces. , , as well as , specifically:

[0145] ;

[0146] in: , , , and , Represented as Attention space of size , ;

[0147] Furthermore, the computing unit 204 is specifically used for:

[0148] Based on the softmax function, respectively Two-step global interaction attention weight calculation:

[0149] ;

[0150] ;

[0151] in, This is the scaling factor.

[0152] This application also provides another low-complexity, high-efficiency attention device, such as... Figure 3 As shown, for ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. The terminal can be any terminal device including mobile phones, tablets, personal digital assistants (PDAs), point-of-sale (POS) terminals, in-vehicle computers, etc. Taking a mobile phone as an example:

[0153] Figure 3 This is a block diagram illustrating a portion of the structure of a mobile phone related to the terminal provided in the embodiments of this application. (Reference) Figure 3The mobile phone includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090, etc. Those skilled in the art will understand that... Figure 3 The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0154] The following is combined with Figure 3 A detailed introduction to each component of a mobile phone:

[0155] The RF circuit 1010 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with the processor 1080; additionally, it transmits uplink data to the base station. Typically, the RF circuit 1010 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, the RF circuit 1010 can also communicate wirelessly with networks and other devices. The aforementioned wireless communications may use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0156] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various mobile phone functions and data processing by running the software programs and modules stored in the memory 1020. The memory 1020 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 1020 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0157] The input unit 1030 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 1031), and drive the corresponding connection devices according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 1080, and can also receive and execute commands sent by the processor 1080. In addition, the touch panel 1031 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1031, the input unit 1030 may also include other input devices 1032. Specifically, other input devices 1032 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0158] The display unit 1040 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 1040 may include a display panel 1041, which may optionally be configured as a Liquid Crystal Display (LCD), Organic Light-Emitting Diode (OLED), or similar display panel 1041. Furthermore, a touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it transmits the information to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides corresponding visual output on the display panel 1041 based on the type of touch event. Although in Figure 3 In this embodiment, the touch panel 1031 and the display panel 1041 are two separate components to realize the input and output functions of the mobile phone. However, in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.

[0159] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 1041 according to the ambient light level, and the proximity sensor can turn off the display panel 1041 and / or backlight when the phone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, taps), etc. Other sensors that may be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0160] The audio circuit 1060, speaker 1061, and microphone 1062 provide an audio interface between the user and the mobile phone. The audio circuit 1060 converts the received audio data into electrical signals and transmits them to the speaker 1061, where the speaker 1061 converts them into sound signals for output. On the other hand, the microphone 1062 converts the collected sound signals into electrical signals, which are then received by the audio circuit 1060, converted into audio data, and then processed by the processor 1080 before being transmitted via the RF circuit 1010 to, for example, another mobile phone, or the audio data can be output to the memory 1020 for further processing.

[0161] WiFi is a short-range wireless transmission technology. Through the WiFi module 1070, mobile phones can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 3 The WiFi module 1070 is shown, but it is understood that it is not an essential component of a mobile phone and can be omitted as needed without changing the essence of the invention.

[0162] The processor 1080 is the control center of the mobile phone, connecting various parts of the phone through various interfaces and lines. It executes software programs and / or modules stored in the memory 1020 and calls data stored in the memory 1020 to perform various functions and process data, thereby providing overall monitoring of the phone. Optionally, the processor 1080 may include one or more processing units; preferably, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 1080.

[0163] The mobile phone also includes a power supply 1090 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 1080 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0164] Although not shown, mobile phones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0165] In this embodiment of the application, the processor 1080 included in the terminal also has the following functions:

[0166] Step S1: Obtain the voice input sequence x;

[0167] It should be noted that, firstly, an audio signal needs to be recorded or acquired, which is usually an analog signal that needs to be converted into a digital signal by an analog-to-digital converter (ADC).

[0168] Preprocessing digital audio signals may include steps such as noise reduction, normalization, and silence removal to improve the accuracy of subsequent processing.

[0169] The continuous audio signal is divided into shorter frames (typically between 20 and 40 milliseconds), each frame containing a short duration of audio signal.

[0170] Apply a window function (such as a Hamming window) to each frame to reduce the impact of discontinuities at frame boundaries.

[0171] Perform a Fast Fourier Transform (FFT) on the windowed signal to transform it from the time domain to the frequency domain.

[0172] Features are extracted from frequency domain signals. Common speech features include Mel frequency cepstral coefficients (MFCCs), Mel spectral energy features (MFBs), and Mel spectral dynamic features (MFSDs).

[0173] The extracted features are organized into a sequence in chronological order, with the features of each frame forming a sequence element, and the entire speech segment is converted into a feature sequence.

[0174] Preferably, VAD is used to detect and remove silent portions, retaining only frames containing speech.

[0175] Preferably, the start and end points of the speech are determined for further processing.

[0176] Preferably, post-processing of the feature sequence may include feature normalization, denoising, and feature enhancement.

[0177] Step S2: Perform a linear transformation on the speech input sequence x to obtain the query vector q, key vector k, value vector v, and compression vector s, where... ;

[0178] It should be noted that for query, key, value, and compression, the speech input sequence x undergoes four different linear transformations, namely weight matrices, to generate the query vector q, key vector k, value vector v, and compression vector s. D is the embedding dimension, which can be determined based on the need to reduce computational complexity.

[0179] Step S3: Perform quadratic linear transformations on the query vector q, key vector k, value vector v, and compressed vector s respectively to generate feature maps in different spaces. , , as well as ,in , ;

[0180] Step S4: Perform the following based on the softmax function. The two-step global interaction attention weight calculation yields the first attention weight value and the second attention weight value, respectively;

[0181] It should be noted that since the raw audio data is converted to the frequency domain, the range acceptable to humans in the frequency domain is very limited. Although adjustments are made using Mel-spectral coefficients and other methods during data preprocessing, a large amount of redundant information still exists in the input features. Therefore, within a certain range, the input audio features exhibit scale similarity, meaning that using higher-dimensional or lower-dimensional representations has little impact on the final prediction.

[0182] like Figure 4 As shown, Figure 4 This is a diagram illustrating the computational process of a self-attention mechanism. For a traditional self-attention mechanism, the input... , and After linear transformation, feature maps Q, K, and V in different spaces are obtained, as shown in the following formulas:

[0183]

[0184]

[0185] Here, , , ,and ; Represented as The size of the attention space facilitates the capture of attention features in different spaces; scaling factor It is used to reduce due to Dot product calculation The impact of functions.

[0186] Here The operation requires Floating-point calculations require a significant amount of time and computing resources.

[0187] Because we added a compression vector s, s after the linear transformation Here This allows the traditional one-step attention mechanism to be transformed into a two-step global interactive attention mechanism, but the floating-point calculations in each use are only a fraction of the total. and Secondly, when the input sequence is particularly long, i.e. Traditional self-attention mechanisms face serious computational problems, while the technical solution of this application is not affected by the length of the input sequence. The computational complexity of the attention mechanism is only related to the embedding dimension under arbitrary length, and for compressed vectors, the computational cost of the network can be further reduced.

[0188] Step S5: Map the first attention weight value and the second attention weight value to the feature of the value vector v. Perform weighted output to obtain the final output result.

[0189] It should be noted that the calculated attention weights are used to sum the weighted vector v to generate the final output. This weighted sum represents the contribution of each element in the sequence to the output, where the weights are determined by their relevance to the query.

[0190] Preferably, in the Transformer architecture, this process can be replicated multiple times, forming a multi-head attention mechanism, where each head learns a different representation of the sequence. Each head has its own... , , as well as .

[0191] After calculating self-attention, residual connections and layer normalization are typically used to help stabilize the training process and allow the model to learn deeper representations.

[0192] Following the self-attention layer, there is usually a feedforward network (typically consisting of a linear layer followed by a non-linear activation function) to further process the output of the self-attention layer.

[0193] Furthermore, step S2 specifically includes:

[0194] The speech input sequence x is transformed by a single linear transformation to obtain the query vector q, key vector k, value vector v, and compression vector s. Specifically:

[0195] , , , ;

[0196] in , , as well as , , , as well as These are the weight matrices for query, key, value, and compression, respectively.

[0197] It should be noted that for query, key, value, and compression, the speech input sequence x undergoes four different linear transformations, namely weight matrices, to generate the query vector q, key vector k, value vector v, and compression vector s. D is the embedding dimension, which can be determined based on the need to reduce computational complexity.

[0198] These transformations can be expressed as:

[0199] , , , ;

[0200] in , , as well as These are the weight matrices for query, key, value, and compression algorithms, respectively.

[0201] Furthermore, step S3 specifically includes:

[0202] The query vector q, key vector k, value vector v, and compressed vector s are subjected to quadratic linear transformations to generate feature maps in different spaces. , , as well as , specifically:

[0203] ;

[0204] in: , , , and , Represented as Attention space of size , ;

[0205] Furthermore, step S4 specifically includes:

[0206] Based on the softmax function, respectively Two-step global interaction attention weight calculation:

[0207] ;

[0208] ;

[0209] in, This is the scaling factor.

[0210] This application also provides a computer-readable storage medium for storing program code that executes any one of the implementations of the low-complexity, high-efficiency attention method described in the foregoing embodiments.

[0211] This application provides a low-complexity, high-efficiency attention method, apparatus, device, and storage medium. By introducing a compressed vector s, it uses two-step global interactive attention, but the floating-point calculations in each use are only a few. and Secondly, when the input sequence is particularly long, the computational complexity is not affected by the length of the input sequence, but only by the embedding dimension. This can reduce the consumption of computing resources and further reduce the computational cost of the network. It solves the technical problem of large memory consumption and long computation time in existing speech recognition methods based on self-attention mechanism when facing large-scale input sequences.

[0212] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0213] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0214] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0215] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0216] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0217] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0218] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0219] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A low-complexity, high-efficiency attention method, characterized in that, include: Step S1: Obtain the voice input sequence x; Step S2: Perform a linear transformation on the speech input sequence x to obtain the query vector q, key vector k, value vector v, and compression vector s, where... ; Step S3: Perform a quadratic linear transformation on the query vector q, the key vector k, the value vector v, and the compressed vector s to generate feature maps in different spaces. , , as well as ,in , ; Step S4: Perform the following based on the softmax function. The two-step global interaction attention weight calculation yields the first attention weight value and the second attention weight value, respectively; Step S5: Map the first attention weight value and the second attention weight value to the feature of the value vector v. Perform weighted output to obtain the final output result.

2. The low-complexity, high-efficiency attention method according to claim 1, characterized in that, Step S2 specifically includes: The speech input sequence x is subjected to a linear transformation to obtain the query vector q, key vector k, value vector v, and compression vector s, specifically: ， ， ， ； in , , as well as , , , as well as These are the weight matrices for query, key, value, and compression, respectively.

3. The low-complexity, high-efficiency attention method according to claim 2, characterized in that, Step S3 specifically includes: The query vector q, the key vector k, the value vector v, and the compressed vector s are subjected to quadratic linear transformations to generate feature maps in different spaces. , , as well as , specifically: ； in: , , , and , Represented as Attention space of size , .

4. The low-complexity, high-efficiency attention method according to claim 3, characterized in that, Step S4 specifically includes: Based on the softmax function, respectively Two-step global interaction attention weight calculation: ； ； in, This is the scaling factor.

5. A low-complexity, high-efficiency attention device, characterized in that, include: Acquisition unit, used to acquire speech input sequence x; The first transformation unit is used to perform a linear transformation on the speech input sequence x to obtain a query vector q, a key vector k, a value vector v, and a compression vector s, wherein... ; The second transformation unit is used to perform quadratic linear transformations on the query vector q, the key vector k, the value vector v, and the compression vector s to generate feature maps in different spaces. , , as well as ,in , ; The computational unit is used to perform calculations based on the softmax function. The two-step global interaction attention weight calculation yields the first attention weight value and the second attention weight value, respectively; Output unit, configured to map the first attention weight value and the second attention weight value to the feature of the value vector v. Perform weighted output to obtain the final output result.

6. The low-complexity, high-efficiency attention device according to claim 5, characterized in that, The first transformation unit is specifically used for: The speech input sequence x is subjected to a linear transformation to obtain the query vector q, key vector k, value vector v, and compression vector s, specifically: ， ， ， ； in , , as well as , , , as well as These are the weight matrices for query, key, value, and compression, respectively.

7. The low-complexity, high-efficiency attention device according to claim 6, characterized in that, The second transformation unit is specifically used for: The query vector q, the key vector k, the value vector v, and the compressed vector s are subjected to quadratic linear transformations to generate feature maps in different spaces. , , as well as , specifically: ； in: , , , and , Represented as Attention space of size , .

8. The low-complexity, high-efficiency attention device according to claim 7, characterized in that, The computing unit is specifically used for: Based on the softmax function, respectively Two-step global interaction attention weight calculation: ； ； in, This is the scaling factor.

9. A low-complexity, high-efficiency attention device, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the low-complexity, high-efficiency attention method according to any one of claims 1-4, based on instructions in the program code.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the low-complexity, high-efficiency attention method according to any one of claims 1-4.

Images