Speech recognition Transform multi-head attention dynamic information interaction analysis method
By analyzing the attention head connections of the Transformer model, a cross-layer temporal correlation model is constructed, which solves the problem of insufficient dynamic modeling of the collaborative relationship between attention heads and improves the transparency of the model and speech recognition performance.
Patent Information
- Application Number
- CN202511077999.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-01
- Publication Date
- 2025-11-18
AI Technical Summary
Existing Transformer models lack the ability to dynamically model the collaborative relationships between attention heads in speech recognition tasks, resulting in a lack of cross-layer temporal dynamic feature capture and an inability to effectively analyze the collaborative relationships between attention heads.
By analyzing the attention weight matrix of the attention heads in the Transformer model, a sequence of correlation matrices is constructed and clustered to identify key attention connection patterns. By combining self-attention attribution, attention head pruning, KL divergence, and global connection weight contribution, the information flow mechanism between attention heads is revealed.
It improves the interpretability and transparency of the model, optimizes the model structure, enhances the accuracy of speech recognition, and promotes cross-domain applications.
Smart Images
Figure CN120977304A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of speech recognition and model interpretability, and specifically relates to a speech recognition Transformer multi-head attention dynamic information interaction analysis method. Background Technology
[0002] As a deep learning architecture based on attention mechanisms, the Transformer model has demonstrated significant technical advantages in fields such as Natural Language Processing (NLP), Computer Vision, and Speech Recognition. The core of the Transformer speech recognition model is the multi-head attention mechanism. The Transformer consists of multiple layers of encoders and decoders, with each layer containing multiple attention heads. These multiple attention heads achieve distributed capture of semantic features, acoustic features, and contextual relationships in the input speech sequence through self-attention operations.
[0003] While Transformers perform exceptionally well in automatic speech recognition tasks, research on the collaborative mechanisms between attention heads is limited. Due to the continuous nature of speech signals, the collaborative relationships between attention heads dynamically evolve over time. However, current techniques for analyzing these relationships still have limitations in their dynamic modeling capabilities, and further research is needed to capture the collaborative relationships between attention heads during temporal changes. Furthermore, the long-distance dependency characteristics of speech signals require modeling through the transmission of temporal information across attention heads. However, existing attention head analysis methods are limited to local feature analysis between layers and fail to establish temporal correlation models between attention heads across layers, resulting in a lack of capture of cross-layer temporal dynamic features. Summary of the Invention
[0004] The purpose of this invention is to provide a method for analyzing the dynamic information interaction of multi-head attention in speech recognition Transformer, so as to further analyze the collaborative working mechanism of attention heads within the model, thereby contributing to the research on cross-layer temporal dynamic feature capture and improving the accuracy of speech recognition.
[0005] To achieve the above objectives, the present invention employs the following technical solution:
[0006] A speech recognition Transformer multi-head attention dynamic information interaction analysis method includes:
[0007] Step 1: Acquire the audio and preprocess it;
[0008] Step 2: Obtain the Transformer model to be analyzed;
[0009] Step 3, input the pre-processed audio, obtain the attention weight matrix of each attention head of each layer of the Transformer model; at each sampling time point, the attention weight matrix of each attention head at the time is intercepted in a preset size, thereby obtaining the attention weight interception matrix sequence of each attention head after interception;
[0010] Step 4, for each sampling time point, calculate the correlation matrix of the attention weight interception matrix of all attention heads in the Transformer model at the sampling time point, to obtain a correlation matrix sequence;
[0011] Step 5, cluster and classify all correlation matrices in the correlation matrix sequence, and each classification result is taken as a dynamic connection mode; in all dynamic connection modes, based on the sampling time of the correlation matrix and the speech time period in the audio, determine the key attention connection mode;
[0012] Step 6, for each key attention connection mode, average all correlation matrices contained therein to obtain an average correlation matrix and determine a strong connection;
[0013] Step 7, determine the strong connection corresponding to other audio; take the common strong connection in the strong connection corresponding to all audio as the stable strong connection of the Transformer model;
[0014] Step 8, after identifying the stable strong connection, perform interaction analysis on the attention heads.
[0015] Further, the preprocessing includes downsampling to adjust the sampling rate and segmentation processing.
[0016] Further, at the first sampling time point, the first A*A size attention weight interception matrix w1 is obtained by intercepting from the upper left corner of the attention weight matrix; at each subsequent sampling time point, non-repeated interception is performed along the diagonal line of the attention weight matrix W in turn, thereby obtaining a series of attention weight interception matrices w1, w2, … corresponding to each sampling time point.
[0017] Further, the K-means clustering method is used to cluster and classify all correlation matrices in the correlation matrix sequence;
[0018] The silhouette coefficient of the classification result is calculated after clustering; if the silhouette coefficient is less than a set threshold, the number of cluster classifications is adjusted and re-clustered.
[0019] Further, in all dynamic connection modes, based on the sampling time of the correlation matrix and the speech time period in the audio, the key attention connection mode is determined, including:
[0020] The sampling time corresponding to the correlation matrix included in each dynamic connection mode is counted, and a speech time period in the audio input into the Transformer model is analyzed; the degree of correspondence between the sampling time corresponding to the correlation matrix included in each dynamic connection mode and the speech time period is judged, and if the degree of correspondence is greater than a preset value, the dynamic connection mode is considered as a key attention connection mode.
[0021] Further, each element in the average correlation matrix represents an attention connection score of a corresponding attention head; the top N corresponding attention heads with the highest attention connection scores are selected as strong connections.
[0022] Further, the interaction analysis of the attention heads includes self-attention attribution analysis, attention head pruning, KL divergence, and global connection weight contribution.
[0023] Further, the analysis of the self-attention attribution analysis, the attention head pruning, the KL divergence, and the global connection weight contribution includes:
[0024] Self-attention attribution; the self-attention attribution between the attention weight matrices of any two attention heads in the Transformer model at the same time is calculated, which is used to evaluate the sensitivity of the attention connection between the attention weight matrices;
[0025] Attention head pruning; by pruning the attention head corresponding to the stable strong connection and other attention heads respectively, the importance of each attention head in the recognition process is evaluated by combining the word error rate of the final speech recognition result of the Transformer model on the audio after each pruning;
[0026] KL divergence; the KL divergence of the attention head corresponding to the stable strong connection is calculated; the higher the KL divergence, the more the attention head focuses on the local; otherwise, the attention head focuses more on the global;
[0027] Global connection weight contribution; the attention weight matrix of the attention head corresponding to the stable strong connection is obtained, and the proportion of the matrix in the weighted sum of all Transformer model attention weight matrices is taken as the global connection weight contribution; the higher the global connection weight contribution, the greater the role of the attention head in the global.
[0028] A terminal device includes a processor, a memory, and a computer program stored in the memory; when the processor executes the computer program, the speech recognition Transformer multi-head attention dynamic information interaction analysis method is implemented.
[0029] A computer-readable storage medium, the medium storing a computer program; the computer program is executed by a processor to implement the voice recognition Transformer multi-head attention dynamic information interaction analysis method.
[0030] Compared with the prior art, the present application has the following technical characteristics:
[0031] 1. Improve the model interpretability: by analyzing the attention connection between each attention head in the Transformer model, the present application identifies the stable strong connection in the model, can explain the information flow, reveals the internal connection mechanism, makes the decision-making process of the model more transparent, and breaks through the "black box" problem in the prior art.
[0032] 2. Optimize model performance: through attention connection analysis, the present application can identify the weak connection of each attention head in the model in a specific task, and then perform attention head pruning and other structural optimization on the model, identify the potential deficiencies of the model in a specific task, improve the model structure, reduce the model parameters, and finally improve the accuracy of speech recognition.
[0033] 3. Promote cross-domain application: the research method of the present application combines the dynamic functional connectivity analysis method of human brain based on functional magnetic resonance imaging (fMRI) and the attention head of the Transformer model, promotes the combination of cross-domain multidisciplinary. This research method can also provide an explainable analysis method for other Transformer models (such as large language model GPT, DeepSeek, etc.), helping to explain the internal information flow and information transmission of large models in different tasks. BRIEF DESCRIPTION OF DRAWINGS
[0034] Figure 1 The present application is a method flowchart;
[0035] Figure 2 The present application is a framework diagram of the method;
[0036] Figure 3 The present application is a flowchart of constructing the timing connection matrix. DETAILED DESCRIPTION
[0037] The multi-head attention mechanism of the Transformer model in the prior art can efficiently model the phonetic temporal dependence and context association, but the cooperative working mechanism of the internal attention heads is still unclear. In view of the problems of insufficient explainability between the attention heads, lack of temporal features, and missing cross-layer information capture in the prior art, the present application performs time domain dynamic analysis on the multi-head attention connection, thereby realizing the explanation of the information flow between the attention heads and indicating how the model makes decisions on the input speech features. Inspired by the dynamic functional connectivity analysis method in functional magnetic resonance imaging (fMRI), the present application proposes a method for constructing a temporal attention head connection matrix, analyzing the information flow mechanism between the attention heads, explaining the key rules of information interaction between the attention heads, improving the model transparency, and providing a theoretical basis for performance optimization of speech recognition, thereby realizing the cross-domain application of the model.
[0038] The present application provides a speech recognition Transformer multi-head attention dynamic information interaction analysis method, which models and analyzes the temporal relationship of the multi-head attention connection matrix, and the technical scheme is as follows:
[0039] Step 1, obtaining audio and preprocessing it; wherein, the preprocessing includes downsampling to adjust the sampling rate, and segmenting processing; the preprocessing can reduce memory consumption and improve processing efficiency, and the segmenting processing can ensure that the audio can be processed in batches, thereby improving the audio and model processing efficiency.
[0040] Step 2, obtaining the Transformer model to be analyzed.
[0041] In this embodiment, the Transformer model to be analyzed is a speech recognition model Wav2Vec 2.0; this model can extract speech features from raw speech signals and support pre-training on unlabeled raw speech data, so it can efficiently complete the speech recognition task.
[0042] Step 3, after deploying the Transformer model, input the preprocessed audio, and obtain the attention weight matrix W of each attention head of each layer of the Transformer model through forward propagation; at each sampling time point, for the attention weight matrix W of each attention head at that time, the attention weight matrix W is intercepted with a preset size A*A, then an attention weight interception matrix w is obtained at each sampling time point, thereby obtaining the sequence of attention weight interception matrices w1, w2, … of each attention head after interception; the value of A can be set according to actual needs; in this embodiment, the intercepted matrix size is 75*75, and the interval of the sampling time points is 1.5s.
[0043] Wherein, at the first sampling time point, the first A*A size attention weight interception matrix w1 is obtained by intercepting from the upper left corner of the attention weight matrix W; at each subsequent sampling time point, the non-repeated interception is sequentially performed along the diagonal line of the attention weight matrix W, thereby obtaining a series of attention weight interception matrices w1, w2, … corresponding to each sampling time point.
[0044] In the interception process of the original attention weight matrix W of each attention head in the present scheme, since the scale of W is often large, only the main information part thereof, i.e., the relevant attention weight data on the diagonal line, is intercepted. The sequence of the attention weight interception matrices obtained by the above extraction reflects how the attention is allocated between different time points and different attention heads when the model processes the speech input.
[0045] Step 4, for each sampling time point, the correlation matrix of the attention weight interception matrix w of all attention heads in the Transformer model at the sampling time point is calculated; the sequence of the correlation matrix is obtained as a time sequence connection matrix, thereby simulating the multi-head attention connection mode of the dynamic functional connection of the human brain; wherein the elements in the correlation matrix are the mean values of the corresponding attention weights.
[0046] The sequence of the correlation matrix reflects the dynamic changes of the connection between the attention heads. By analyzing the sequence of the correlation matrix, the path of the information flow can be observed, and the potential dependency relationship between the attention heads can be identified.
[0047] Step 5, the K-means clustering method is used to cluster and classify all the correlation matrices in the sequence of the correlation matrices, and each classification result is taken as a dynamic connection mode; in all the dynamic connection modes, the sampling time corresponding to the correlation matrix contained in each dynamic connection mode is counted, and the speech time period (blank, noise, speech time period, etc. are contained in the audio) in the audio input into the Transformer model is analyzed; it is judged that the corresponding degree of the sampling time corresponding to the correlation matrix contained in each dynamic connection mode and the speech time period, such as the corresponding degree greater than a preset value, then the dynamic connection mode is considered as a key attention connection mode.
[0048] Wherein, the silhouette coefficient of the classification result is calculated after clustering; such as the silhouette coefficient is less than a set threshold value (for example, 0.5), indicating that the clustering effect is not ideal, at this time the number of clustering classification needs to be adjusted, and then re-clustering is performed to realize the optimization of the clustering classification process.
[0049] The corresponding degree refers to the proportion of the sampling time falling into the speech time period. For example, a certain dynamic connection mode contains 30 correlation matrices corresponding to 30 sampling times. If 28 of the 30 sampling times fall into the speech time period, which is greater than the preset value of 90%, the dynamic connection mode is considered to be a key attention connection mode. The key attention connection mode can reflect the attention connection therein.
[0050] Step 6, for each key attention connection mode, first average all the correlation matrices contained therein to obtain an average correlation matrix; each element in the average correlation matrix represents the attention connection score of the corresponding attention head; select the top N corresponding attention heads with the highest attention connection scores as strong connections; wherein the top N may be set to the top 5% of the attention connection scores.
[0051] Step 7, for other audios, each audio can obtain the corresponding strong connection by the same method as steps 3 to 6; the common strong connection in the strong connections corresponding to all audios is the stable strong connection of the Transformer model; the stable strong connection can reflect the cross-layer strong dependence relationship existing in the model.
[0052] Step 8, after identifying the stable strong connection, analyze it in combination with self-attention attribution analysis, attention head pruning, KL divergence and global connection weight contribution.
[0053] (1) Self-attention attribution; calculate the self-attention attribution between the attention weight matrices of any two attention heads in the Transformer model at the same time, for evaluating the sensitivity of the attention connection between the attention weight matrices.
[0054] (2) Attention head pruning; by pruning the attention heads corresponding to the stable strong connection and other attention heads respectively, in combination with the word error rate (WER) of the final speech recognition result of the Transformer model after each pruning, the importance of each attention head in the recognition process is evaluated.
[0055] (3) KL divergence; calculate the KL divergence of the attention heads corresponding to the stable strong connection; the higher the KL divergence, the more the attention head focuses on the local; on the contrary, the attention head pays more attention to the global.
[0056] (4) Global connection weight contribution; obtain the attention weight matrix of the attention heads corresponding to the stable strong connection, and take the proportion of the matrix in the weighted sum of all Transformer model attention weight matrices as the global connection weight contribution; the higher the global connection weight contribution, the greater the role of the attention head in the global.
[0057] Therefore, the combination of KL divergence and global connection weight contribution can evaluate the attention characteristics of each stable strong connection corresponding attention head in the model.
[0058] Through the above steps, the timing correlation analysis model between the cross-layer attention heads is constructed, the connection between the attention heads in the Transformer model is reflected, the information interaction relationship between the attention heads is revealed, and the transparency and interpretability of the model are improved.
[0059] Embodiment:
[0060] In an embodiment of the present application, the audio sample is a natural voice story audio in the Narratives dataset, the sampling rate is adjusted to 8 kHz by downsampling the voice, and the input audio is segmented, and the audio is divided into multiple segments every 20 s to ensure that the data can be processed in batches.
[0061] The Transformer model to be analyzed adopts a wav2vec2_base_960h model, which has a 12-layer structure with 12 attention heads per layer, a total of 144 attention heads; the model can obtain the attention weight matrix of each attention head through forward propagation.
[0062] When extracting the attention weight interception matrix, the size is set to 75*75, and the interval of the sampling time point is 1.5 s; after subsequent processing, there are two key attention connection patterns obtained using the natural voice story audio.
[0063] After identifying the key attention connection pattern, the attention connection is visualized and the strong connection is determined; by processing other audios, the stable strong connection is finally determined as: L5H2—L6H11—L7H6—L8H3—L9H9—L10H6 (wherein L represents the layer number, H represents the head number, and L5H2 represents the fifth layer second attention head).
[0064] It is found that the self-attention attribution scores of the attention heads contained in the stable strong connection are all located within the top 20% of all attention head attribution scores, which proves the importance of the stable strong connection; by pruning the attention heads in the model, the word error rate (WER) of the speech recognition result is analyzed to further evaluate the importance of the stable strong connection in the recognition process, ensuring the reliability of the finding; through the analysis of the KL divergence and the global connection weight contribution value, it is found that the KL divergence values of the stable strong connection are all higher than 0.7, located in the top 30% of the total, while the global connection weight contribution values are all located in the last 37.5% of the total, indicating that the stable strong connection has more local attention characteristics relative to the global.
[0065] The above examples are only used to illustrate the technical solutions of the present application, but not limit the same; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some technical features can be replaced by equivalent ones; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.
Claims
1. A speech recognition Transformer multi-head attention dynamic information interaction analysis method, characterized in that, include: Step 1: Acquire the audio and preprocess it; Step 2: Obtain the Transformer model to be analyzed; Step 3: Input the preprocessed audio and obtain the attention weight matrix of each attention head in each layer of the Transformer model; at each sampling time point, truncate the attention weight matrix of each attention head at that time with a preset size to obtain the truncated attention weight matrix sequence of each attention head. Step 4: For each sampling time point, calculate the correlation matrix of the attention weight truncation matrix of all attention heads in the Transformer model at that sampling time point to obtain the correlation matrix sequence; Step 5: Cluster and classify all correlation matrices in the correlation matrix sequence, with each classification result serving as a dynamic connection pattern; among all dynamic connection patterns, determine the key attention connection pattern based on the sampling time of the correlation matrix and the speech time segment in the audio. Step 6: For each key attention connection pattern, average all the correlation matrices contained therein to obtain the average correlation matrix and determine the strong connections. Step 7: Determine the strong connections corresponding to other audio; use the common strong connections among the strong connections corresponding to all audio as the stable strong connections of the Transformer model; Step 8: After identifying stable strong connections, perform interaction analysis of attention heads.
2. The speech recognition Transformer multi-head attention dynamic information interaction analysis method according to claim 1, characterized in that, Preprocessing includes downsampling to adjust the sampling rate and segmented processing.
3. The speech recognition Transformer multi-head attention dynamic information interaction analysis method according to claim 1, characterized in that, At the first sampling time point, the attention weight matrix is truncated from the top left corner to obtain the first A*A size attention weight truncated matrix w1; at each subsequent sampling time point, non-repeating truncations are performed along the diagonal of the attention weight matrix W to obtain a series of attention weight truncated matrices w1, w2, ... corresponding to each sampling time point.
4. The speech recognition Transformer multi-head attention dynamic information interaction analysis method according to claim 1, characterized in that, The K-means clustering method is used to cluster and classify all correlation matrices in the correlation matrix sequence; After clustering, calculate the silhouette coefficient of the classification results; if the silhouette coefficient is less than the set threshold, adjust the number of clusters and re-cluster.
5. The speech recognition Transformer multi-head attention dynamic information interaction analysis method according to claim 1, characterized in that, Among all dynamic connectivity patterns, key attention connectivity patterns are determined based on the sampling time of the correlation matrix and the speech time segment in the audio, including: The sampling time corresponding to the correlation matrix contained in each dynamic connection pattern is statistically analyzed, and the speech time segment in the audio input to the Transformer model is analyzed. The degree of correspondence between the sampling time corresponding to the correlation matrix contained in each dynamic connection pattern and the speech time segment is determined. If the degree of correspondence is greater than a preset value, the dynamic connection pattern is considered to be a key attention connection pattern.
6. The speech recognition Transformer multi-head attention dynamic information interaction analysis method according to claim 1, characterized in that, Each element in the average correlation matrix represents the attention connection score of the corresponding attention head; the top N attention heads with the highest to lowest attention connection scores are selected as strong connections.
7. The speech recognition Transformer multi-head attention dynamic information interaction analysis method according to claim 1, characterized in that, Interaction analysis of attention heads is performed, including analysis combining self-attention attribution analysis, attention head pruning, KL divergence, and global connectivity weight contribution.
8. The speech recognition Transformer multi-head attention dynamic information interaction analysis method according to claim 7, characterized in that, The analysis combines self-attention attribution analysis, attention head pruning, KL divergence, and global connectivity weight contributions, including: Self-attention attribution; calculates the self-attention attribution between the attention weight matrices of any two attention heads in the Transformer model at the same time, and is used to evaluate the sensitivity of attention connections between attention weight matrices; Attention head pruning: By pruning the attention heads corresponding to stable strong connections and other attention heads separately, and combining the word error rate of the final speech recognition result of the Transformer model after each pruning, the importance of each attention head in the recognition process is evaluated. KL divergence; calculates the KL divergence of the attention head corresponding to a stable strong connection; the higher the KL divergence, the more the attention head focuses on the local; conversely, the lower the KL divergence, the more the attention head focuses on the global. Global connection weight contribution: Obtain the attention weight matrix of the attention head corresponding to the stable strong connection, and take the proportion of this matrix to the weighted sum of the attention weight matrices of all attention heads of the Transformer model as the global connection weight contribution; the higher the global connection weight contribution, the greater the global effect of the attention head.
9. A terminal device, comprising a processor, a memory, and a computer program stored in the memory; characterized in that, When the processor executes a computer program, it implements the speech recognition Transformer multi-head attention dynamic information interaction analysis method according to any one of claims 1-8.
10. A computer-readable storage medium storing a computer program; characterized in that, When the computer program is executed by the processor, it implements the speech recognition Transformer multi-head attention dynamic information interaction analysis method according to any one of claims 1-8.