Audio large language model cross-modal information flow tracking method based on attention ablation
Patent Information
- Application Number
- CN202611209558.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-11
- Publication Date
- 2026-10-27
AI Technical Summary
[0006]本发明的目的是为解决现有音频大语言模型可解释性分析缺乏系统性方法的问题，提供基于注意力消融的音频大语言模型跨模态信息流追踪方法，对声学与文本模态之间的注意力路径进行选择性干预，定量追踪不同层次中两种模态对最终预测的贡献，得到音频大语言模型内部的模态融合机制与层次化信息处理规律
[0038]1、本发明公开的基于注意力消融的音频大语言模型跨模态信息流追踪方法，在不同基础语言模型骨干的音频大语言模型上重复执行分析流程，对比不同架构下模态消融效应的幅度分布与峰值层位置的异同；通过对声学与文本模态之间的注意力路径进行选择性干预，定量追踪不同层次中两种模态对最终预测的贡献，在关键词识别、情感分类和问答推理三类任务下，分别执行声学-末位阻断、文本-末位阻断、末位自阻断以及声学-文本阻断四类消融模式，构建逐层模态贡献剖面，得到音频大语言模型内部的模态融合机制与层次化信息处理规律：声学表示主导低层、文本表示主导高层、跨模态注入集中于中间层的层次化组织结构；分析基础语言模型骨干层数差异对模态敏感层绝对位置的影响，以及多阶段融合整体模式的保留性，为音频大语言模型的结构理解与功能优化提供了理论依据与工具支撑。
Smart Images
Figure CN122991187A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method for cross-modal information flow tracking in audio large language models based on attention ablation, and belongs to the field of audio large language models. Background Technology
[0002] Large-scale language models have made significant progress in numerous tasks in recent years, but their powerful performance is hampered by a clear black-box effect, prompting extensive research into their internal mechanisms. For large-scale language models focused on pure text, existing work has systematically explored neuronal behavior, attention flow, and modular task specialization. Similarly, the structural design and modal alignment mechanisms of large video-language models have received initial attention. However, the internal workings of large audio language models remain largely unexplored.
[0003] Large audio language models typically consist of an efficient audio encoder and a text-based large language model. During the input phase, the acoustic representation extracted by the audio encoder is concatenated with the text embeddings and fed into the language model to enable joint reasoning between audio and text. Compared to pure text models, acoustic representations play a crucial role in multimodal understanding; therefore, studying how acoustic representations propagate and interact within the model is essential for revealing the internal mechanisms of audio-language reasoning.
[0004] A few works have attempted to explore the relationship between the internal state and external behavior of large multimodal language models, involving information storage in model parameters, latent signals in output distribution, localization and evolution of non-textual features, and pruning of redundant input tokens. However, the information flow mechanism between acoustic and textual modalities in large audio language models still lacks a systematic understanding. In particular, during autoregressive inference, the core questions that remain to be explored are: at which layers and in what ways does acoustic information fuse with textual representations to generate the final prediction?
[0005] Existing work on the interpretability of large audio language models has the following shortcomings: First, it lacks an analysis framework specifically for audio modalities; second, it lacks a refined tracking method for cross-modal attention paths, making it impossible to quantitatively characterize the hierarchical distribution of acoustic information propagation to text representation; and third, it lacks systematic experimental verification covering speech tasks with different cognitive complexities, making it difficult to fully reveal the dynamic changes in modality dependence of the model under different tasks. Summary of the Invention
[0006] The purpose of this invention is to address the lack of systematic methods for interpretability analysis of existing large audio language models. It provides a cross-modal information flow tracking method for large audio language models based on attention ablation, which selectively intervenes in the attention path between acoustic and textual modalities, quantitatively tracks the contribution of the two modalities to the final prediction at different levels, and obtains the modal fusion mechanism and hierarchical information processing rules within the large audio language model.
[0007] The objective of this invention is achieved through the following technical solution:
[0008] The technical features of this invention are as follows: For the input audio-text pair, the acoustic feature sequence extracted by the audio encoder is concatenated with the text embedding sequence to form a unified input sequence, which is then fed into an audio large language model composed of L layers of Transformer decoders. In each Transformer layer, the causal mask matrix in the masking multi-head attention module of each Transformer layer is modified. Under three tasks—keyword recognition, sentiment classification, and question answering reasoning—the invention systematically performs the first type of ablation acoustic-end blocking, selectively blocking attention connections between specific token position sets and blocking attention connection paths from acoustic tokens; performs the second type of ablation text-end blocking, blocking all attention connection paths from text token positions to the final sequence position; and performs the third type of ablation end blocking. Self-blocking at position one block blocks the self-attention connection path from the final sequence position to itself, and self-blocking at the last position one block to the final prediction position, thus truncating the corresponding information flow. The contribution of the blocked information path to the final prediction is quantitatively measured by the relative change rate of the prediction probability of the correct answer token before and after the ablation intervention. In the cross-modal information flow analysis, a fourth type of ablation acoustic-text blocking is further performed to block the attention path from the acoustic token to the text token position, tracing the injection process of acoustic information into the text flow. By systematically applying ablation interventions with fixed window sizes in different layer intervals, a layer-by-layer modal contribution profile is constructed to obtain the complementary division of labor between acoustic representation and text representation in the hierarchical dimension and the injection mechanism of cross-modal information flow. Finally, the modal fusion mechanism and hierarchical information processing rules within the audio large language model are obtained.
[0009] This invention relates to a cross-modal information flow tracking method for large audio language models based on attention ablation, which is implemented through the following steps:
[0010] S1. Extraction and construction of acoustic representations for multimodal inputs
[0011] S1.1 For each input sample's original audio waveform, perform Mel spectrum transformation to generate a Mel spectrum diagram. After processing by the audio encoder and lightweight projection module, generate several d-dimensional acoustic feature vectors, which constitute an acoustic feature sequence.
[0012] S1.2 processes the text prompts containing task descriptions and options using a large language model word segmenter to generate a text feature sequence;
[0013] S1.3 Inserts a special speech placeholder into the text prompt. Before forward propagation, the position of the special speech placeholder is replaced with an acoustic feature vector. This allows the acoustic feature sequence formed by S1.1 and the text feature sequence formed by S1.2 to be spliced and fused together to form a unified input sequence, which is then fed into the audio large language model composed of an L-layer Transformer decoder. The acoustic token is located before the text token, and the acoustic token occupies the first part of the unified input sequence while the text token occupies the second part of the unified input sequence.
[0014] S2. Introducing an in-layer attention mask intervention mechanism
[0015] Based on the unified input sequence formed in step S1, the masked multi-head attention module in the Transformer layer of the decoder architecture is the channel for information flow between tokens. The causal mask matrix is modified as follows: on the basis that the standard autoregressive causal mask ensures that the token at position t can only pay attention to the token at position no greater than t, an intra-layer attention mask is introduced. For the source token position set S and the target token position set T to be blocked, the mask positions that satisfy the condition that the source position belongs to S and the target position belongs to T are additionally set to negative infinity, so as to accurately truncate the corresponding information flow without modifying any model parameters.
[0016] S3. Setting the ablation window and layer-by-layer intervention plan
[0017] To balance the sensitivity and stability of the intervention effect, an ablation scheme with a fixed ablation window size of k layers is adopted: For each target layer, the attention mask intervention described in step S2 is applied simultaneously within a continuous window range of k layers centered on the target layer, and the intervention effect is measured by calculating the relative change rate of the predicted probability before and after ablation according to equation (1):
[0018] Relative rate of change of predicted probability The calculation method is as follows:
[0019] (1)
[0020] in, This indicates the predicted probability after ablation intervention. This indicates the predicted probability before ablation intervention;
[0021] By systematically sliding intervention windows across all Transformer layers, a complete layer-by-layer modal contribution profile is constructed, thereby providing a hierarchical basis for ablation mode analysis in step S4 and cross-modal information flow tracing in step S5, in order to identify acoustic-dominant layer intervals and text-dominant layer intervals, as well as key layer locations for cross-modal information injection.
[0022] S4. Definition and Execution of Three Types of Ablation Modes
[0023] Based on the layer-by-layer intervention in step S3, four ablation modes are executed respectively: acoustic-terminal blocking, text-terminal blocking, terminal self-blocking, and acoustic-text blocking, to construct a layer-by-layer modal contribution profile.
[0024] Three ablation modes are defined, each with a corresponding information flow intervention target. Specifically, the three ablation modes are: The first is the acoustic-last-position blocking mode, which sets the acoustic token position set as the source set and the final sequence position as the target set, blocking all attention connections from acoustic token positions to the final sequence position, and quantifying the direct contribution of the acoustic modality to the final prediction; the second is the text-last-position blocking mode, which sets the text token position set as the source set and the final sequence position as the target set, blocking all attention connections from text token positions to the final sequence position, and quantifying the direct contribution of the text modality to the final prediction; the third is the last-position self-blocking mode, which sets the final sequence position as both the source and target sets, blocking the self-attention connections from the final sequence position to itself, and quantifying the direct contribution of the final position token itself to the prediction. By performing layer-by-layer intervention on each of the three ablation modes and recording the relative change rate of the prediction probability, the corresponding acoustic-last-position blocking ablation curve, text-last-position blocking ablation curve, and last-position self-blocking ablation curve are obtained.
[0025] In the layer-by-layer analysis of the three ablation modes, the acoustic representation-dominant layer intervals showed a concentrated acoustic-terminal blocking ablation effect with a significantly larger amplitude than the text-terminal blocking ablation effect; the text representation-dominant layer intervals showed a text-terminal blocking ablation effect that continuously increased with layer depth; and the terminal self-blocking ablation effect was close to zero in all layers.
[0026] The three ablation modes can be executed independently on the same model and the same input sample, and the resulting layer-by-layer intervention curves can be compared with each other; as the complexity of the task increases, a modal balance mechanism is realized that adaptively switches between two modal inference modes.
[0027] S5. Conduct special tracking of cross-modal information flow.
[0028] To reveal the propagation mechanism of acoustic information to text representation, an acoustic-text blocking mode is introduced based on the analysis in step S4. Acoustic-text blocking fourth-type ablation is performed, and the propagation law of acoustic information to text representation is analyzed. The specific method is as follows:
[0029] S5.1 In each Transformer layer, set all acoustic token location sets as the source set and all text token location sets as the target set, while blocking all attention connections from acoustic tokens to text tokens, and preserving the self-attention paths within acoustic tokens and text tokens.
[0030] S5.2 By performing the fourth type of acoustic-text blocking ablation intervention layer by layer and recording the relative change rate of the predicted probability corresponding to each target layer, and then connecting the relative change rates of the predicted probability of each target layer into a change curve according to the layer number order, the main occurrence layer interval of acoustic information injection into text representation and the decay law of the injection effect with the increase of layer depth are identified, and the propagation process of cross-modal information flow is quantitatively tracked; combined with the acoustic-terminal blocking curve obtained in step S4, the hierarchical organization structure of audio-text information fusion in the audio large language model is comprehensively judged.
[0031] S6. Multi-task system comparison experiment
[0032] The multi-task approach includes three categories: keyword recognition, sentiment classification, and question-answering reasoning. The analysis process described above is systematically executed based on steps S1 to S5 for each of these three tasks.
[0033] In the keyword recognition task, the specific requirements are that the model performs low-level perceptual alignment between the semantic content of the input speech segment and the given text description, and analyzes the hierarchical contribution of acoustic cues in word recognition.
[0034] In the emotion classification task, the specific requirements are that the model attribute the emotion of speech based on multimodal cues, and comprehensively utilize acoustic and textual information to analyze the joint contribution of acoustic and textual cues.
[0035] In the question-answering reasoning task, the model is specifically required to perform detailed identification of audio events and combine text semantics to complete multi-step reasoning.
[0036] Based on steps S1-S6, by comparing the amplitude distribution, peak layer position, and cross-layer distribution differences of the layer-by-layer ablation curves under the three types of tasks, a cross-modal information flow tracking method for audio large language models based on attention ablation is realized, revealing the internal modal fusion mechanism and hierarchical information processing rules of the audio large language model. Specifically, in the keyword recognition task, acoustic-terminal blocking ablation leads to a significant decrease in prediction probability, indicating that the prediction results of this task mainly depend on acoustic modal information. As the task complexity increases, the model's dependence on acoustic and textual information gradually becomes more balanced, and the amplitudes of the two types of modal ablation effects gradually approach each other and are distributed across a wider range of network layers. In the question-answering reasoning task, the injection effect of acoustic information into text representation extends further to deeper network layers, indicating that complex semantic reasoning tasks require more sufficient audio-text information interaction.
[0037] Beneficial effects
[0038] 1. This invention discloses a method for tracking cross-modal information flow in audio large language models based on attention ablation. The analysis process is repeatedly executed on audio large language models with different basic language model backbones, comparing the amplitude distribution and peak layer positions of modal ablation effects under different architectures. By selectively intervening in the attention path between acoustic and text modalities, the contribution of the two modalities to the final prediction at different levels is quantitatively tracked. Under three tasks—keyword recognition, sentiment classification, and question answering reasoning—four ablation modes are executed: acoustic-last blocking, text-last blocking, last self-blocking, and acoustic-text blocking. A layer-by-layer modal contribution profile is constructed to obtain the modal fusion mechanism and hierarchical information processing rules within the audio large language model: a hierarchical organizational structure where acoustic representation dominates the lower layers, text representation dominates the higher layers, and cross-modal injection is concentrated in the middle layers. The influence of differences in the number of backbone layers of the basic language model on the absolute position of the modality-sensitive layer, as well as the retention of the overall pattern of multi-stage fusion, is analyzed, providing theoretical basis and tool support for the structural understanding and functional optimization of audio large language models.
[0039] 2. The attention ablation-based method for tracking cross-modal information flow in audio large language models disclosed in this invention requires no modification or retraining of model parameters and can be applied in a plug-and-play manner to any audio large language model based on the Transformer decoder architecture, demonstrating high versatility and practicality.
[0040] 3. The attention-based ablation-based method for tracing cross-modal information flow in audio large language models disclosed in this invention, in the layer-by-layer analysis of three ablation modes, shows that in layers dominated by acoustic representation, the acoustic-last-position blocking ablation effect is concentrated and significantly larger than the text-last-position blocking ablation effect; in layers dominated by text representation, the text-last-position blocking ablation effect continuously increases with layer depth; the last-position self-blocking ablation effect is close to zero in all layers, and the self-contribution of confirming the final position token is negligible; a modal balancing mechanism that adaptively switches from single acoustic modality-driven to balanced multimodal inference as task complexity increases; and a cross-modal propagation path of acoustic information injected into the text flow through intermediate layers; providing direct guidance for subsequent research such as model design, hierarchical pruning, and multimodal information localization and editing. Attached Figure Description
[0041] Figure 1 This is a flowchart of the cross-modal information flow tracking method for audio large language models based on attention ablation according to the present invention; Detailed Implementation
[0042] To better illustrate the purpose and advantages of the present invention, the invention will be further described below in conjunction with the accompanying drawings and examples.
[0043] Example
[0044] This embodiment uses the LLaMA-Omni series model as the object of analysis to illustrate the method of cross-modal information flow tracking and analysis of audio large language models. LLaMA-Omni consists of a Whisper-large-v3 audio encoder and an LLaMA-3.1-8B-Instruct language model, with a total of 32 Transformer layers, a hidden layer dimension of 4096, 32 attention heads, and a feedforward layer dimension of 14336. For example... Figure 1 As shown, this is achieved through the following steps:
[0045] S1. Extraction and construction of acoustic representations for multimodal inputs
[0046] S1.1 For each input sample, the original audio waveform is processed by an audio encoder after Mel-frequency spectral transform, and an acoustic feature sequence is generated through a lightweight linear projection layer. A special speech placeholder is inserted into the text prompt, and before forward propagation, the position of this special speech placeholder is replaced with an acoustic feature vector, achieving the concatenation and fusion of acoustic and textual features to form a unified input sequence.
[0047] S1.2 processes the text prompts containing task descriptions and options using a large language model word segmenter to generate a text feature sequence;
[0048] S1.3 inserts a special speech placeholder in the text prompt. Before forward propagation, the position of this special speech placeholder is replaced with an acoustic feature vector, thus concatenating and fusing the acoustic feature sequence constructed in S1.1 and the text feature sequence constructed in S1.2 to form a unified input sequence that is fed into the 32-layer Transformer stack of the model. The acoustic token is placed before the text token.
[0049] S2. Based on the unified input sequence formed in step S1, an in-layer attention mask intervention mechanism is introduced.
[0050] Based on the unified input sequence formed in step S1, the masked multi-head attention module in the Transformer layer of the decoder architecture serves as the channel for information flow between tokens. While the standard autoregressive causal mask ensures that the token at position t can only focus on tokens at positions no greater than t, an intra-layer attention mask is introduced. For the source token position set S and the target token position set T to be blocked, the mask positions satisfying that the source position belongs to S and the target position belongs to T are additionally set to negative infinity. This accurately truncates the corresponding information flow without modifying any model parameters. After softmax normalization, the attention weights on the blocked path approach zero, and the corresponding value vector contribution is completely eliminated.
[0051] This intervention is applied only during inference, achieved by hooking the mask tensor of the Transformer attention module in the Python layer. The specific modifications are dynamically injected during each forward propagation, without affecting the model parameters or the computational flow of other layers.
[0052] S3. Based on the attention mask intervention mechanism in step S2, set the ablation window and the layer-by-layer intervention scheme.
[0053] To balance the sensitivity and stability of the intervention effect, this embodiment sets the fixed ablation window size to 9 layers. For each target layer, which includes layers 0 to 31, the attention mask intervention described in step S2 is applied simultaneously within a continuous window range of 9 layers centered on the target layer, and the intervention effect is measured by calculating the relative change rate of the predicted probability before and after ablation according to formula (2).
[0054] Relative rate of change of predicted probability The calculation method is as follows:
[0055] (2)
[0056] in, This indicates the predicted probability after ablation intervention. This indicates the predicted probability before ablation intervention;
[0057] This embodiment systematically evaluated eight window sizes for k: 1, 3, 5, 7, 9, 11, 13, and 15. The results showed that when the window was too small (k not exceeding 5), the intervention effect was weak and too localized, failing to reveal the characteristic steep drop pattern of the intermediate layer. When the window was too large (k not less than 11), the curve was overly smoothed, and the interlayer difference signal was submerged. K equal to 9 achieved the optimal balance between the two, clearly presenting the steep drop characteristics of the acoustic-terminal blocking ablation effect in the 10th to 20th layer interval, and was confirmed as the optimal parameter setting.
[0058] S4. Based on the layered intervention plan in step S3, define and implement three types of ablation modalities.
[0059] Three core ablation modes are defined, each corresponding to different information flow intervention targets. The three modes—acoustic-terminal blocking mode, text-terminal blocking mode, and terminal self-blocking mode—can be executed independently on the same model and the same input sample. The resulting layer-by-layer intervention curves can be compared with each other, revealing the complementary relationship and synergistic mechanism between the two modes in the hierarchical dimension.
[0060] This embodiment independently performs three ablation modes on each sample of the three types of tasks. Taking the keyword recognition task, GSC dataset, 1050 samples as an example: acoustic-terminal blocking ablation causes an average decrease of about 100% in the prediction probability of each layer, indicating that the acoustic mode is extremely important in the final prediction of this task; text-terminal blocking ablation causes an average decrease of about 80%, indicating that the text mode is also indispensable; the change caused by terminal self-blocking ablation is almost zero, indicating that the direct contribution of the final sequence position token to its own prediction is negligible.
[0061] The results of layer-by-layer analysis of the three ablation modes reveal that: in layers dominated by acoustic representation, the acoustic-terminal blocking ablation effect is concentrated and significantly larger than the text-terminal blocking ablation effect, which is concentrated in the lower and middle layers; in layers dominated by text representation, the text-terminal blocking ablation effect continuously increases with layer depth, clearly showing a hierarchical structure where acoustics dominates the lower layers and text dominates the higher layers; the terminal self-blocking ablation effect is close to zero in all layers, confirming that the self-contribution of the final position token is negligible.
[0062] In the sentiment classification task (MELD dataset, 460 samples) and the question-answering reasoning task (Clotho-AQA dataset, 1000 samples), the amplitudes of the acoustic-terminal blocking ablation effect and the text-terminal blocking ablation effect were reduced to approximately 20% and 30% respectively, and were more evenly distributed across the layers, reflecting the model's balanced dependence on the two modalities under more complex tasks.
[0063] S5. Based on the ablation results of step S4, conduct a special tracking of cross-modal information flow.
[0064] To reveal the propagation mechanism of acoustic information to text representation, an acoustic-text blocking mode was introduced based on the analysis in step S4. Interventions were then applied sequentially to each target layer according to the layer-by-layer window settings in step S3, and the propagation pattern of acoustic information to text representation was analyzed. Experimental results show that in all three tasks, the acoustic-text blocking ablation effect peaked near layers 8 to 15 in the intermediate layers. The peak reduction was approximately 18% for keyword recognition, 12% for sentiment classification, and 30% for question-answering reasoning. After layer 20, the ablation effect rapidly decreased to near zero, indicating that the cross-modal information injection from acoustics to text mainly occurs in the intermediate layers, and the direct influence of acoustic information on text representation in higher layers has essentially disappeared.
[0065] Therefore, this indicates that acoustic information does not directly affect the final prediction, but rather gradually permeates the text flow through the cross-modal attention path in the intermediate layer, providing a semantic basis for subsequent high-level pure text reasoning. The acoustic-text blocking effect in question-answering reasoning tasks extends to a deeper level, confirming that more complex tasks require more persistent audio-text interaction mechanisms.
[0066] S6. Based on the analysis results of steps S4 and S5, conduct a multi-task systematic comparative experiment.
[0067] Based on steps S1 to S5, the above analysis process was systematically executed on three tasks: keyword recognition, sentiment classification, and question-answering reasoning. By comparing the amplitude, peak layer position, and cross-layer distribution differences of the layer-by-layer ablation curves under the three tasks, the dynamic change law of modality dependence of the model under different cognitive loads was obtained: In the keyword recognition task with the simplest cognitive requirements, acoustic-terminal blocking ablation leads to a near 100% collapse of the prediction probability, reflecting a strong dependence on the acoustic modality; as the task complexity increases, the model gradually transitions to balanced multimodal reasoning, and the ablation effect amplitudes of the acoustic modality and the multimodal modality tend to be similar and distributed in a wider layer interval; in the question-answering reasoning task, the cross-modal injection effect extends to a deeper level, reflecting the demand of complex semantic reasoning for persistent audio-text fusion.
[0068] Thus, based on steps S1 to S6, the method for tracing cross-modal information flow in audio large language models based on attention ablation is completed.
[0069] This embodiment further replicates the analysis of the question-answering reasoning task on LLaMA-Omni2, where Qwen2.5-7B-Instruct replaces LLaMA-3.1-8B-Instruct as the base language model, comprising 28 Transformer layers. Comparative results show that the text-terminal blocking ablation curves of the two architectures exhibit a highly similar overall trend, with text ablation effects significantly greater than acoustic ablation effects, confirming the shared modality dependency pattern between the two models. The absolute positions of the modality-sensitive layer intervals shift slightly due to differences in the number of backbone layers—the LLaMA-Omni peak occurs in layers 12 to 22, while the LLaMA-Omni2 peak occurs in layers 8 to 18—but the overall pattern of hierarchical multi-stage fusion is preserved in both architectures, demonstrating the cross-architecture universality of the revealed mechanism.
[0070] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for tracing cross-modal information flow in audio large language models based on attention ablation, characterized in that, This is achieved through the following steps: S1. Extraction and construction of acoustic representations for multimodal inputs S1.1 For each input sample's original audio waveform, perform Mel spectrum transformation to generate a Mel spectrum diagram. After processing by the audio encoder and lightweight projection module, generate several d-dimensional acoustic feature vectors, which constitute an acoustic feature sequence. S1.2 processes the text prompts containing task descriptions and options using a large language model word segmenter to generate a text feature sequence; S1.3 Inserts a special speech placeholder into the text prompt. Before forward propagation, the position of the special speech placeholder is replaced with an acoustic feature vector. This allows the acoustic feature sequence formed by S1.1 and the text feature sequence formed by S1.2 to be spliced and fused together to form a unified input sequence, which is then fed into the audio large language model composed of an L-layer Transformer decoder. The acoustic token is located before the text token, and the acoustic token occupies the first part of the unified input sequence while the text token occupies the second part of the unified input sequence. S2. Introducing an in-layer attention mask intervention mechanism Based on the unified input sequence formed in step S1, in the Transformer layer of the decoder architecture, the multi-head attention module is the channel for information flow between tokens. The causal mask matrix is modified as follows: on the basis that the standard autoregressive causal mask ensures that the token at position t can only pay attention to the token at position no greater than t, an intra-layer attention mask is introduced to intervene. For the source token position set S and the target token position set T to be blocked, the mask positions that satisfy the condition that the source position belongs to S and the target position belongs to T are additionally set to negative infinity, thus truncating the corresponding information flow. S3. Setting the ablation window and layer-by-layer intervention plan To balance the sensitivity and stability of the intervention effect, an ablation scheme with a fixed ablation window size of k layers was adopted: For each target layer, attention mask intervention was applied simultaneously within a continuous k-layer window centered on the target layer, and the intervention effect was measured by calculating the relative change rate of the predicted probability before and after ablation according to equation (1): Relative rate of change of predicted probability The calculation method is as follows: （1） in, This indicates the predicted probability after ablation intervention. This indicates the predicted probability before ablation intervention; By systematically sliding the intervention window across all Transformer layers, a complete layer-by-layer modal contribution profile is constructed, thereby providing a hierarchical basis for the ablation mode analysis in step S4 and the cross-modal information flow tracing in step S5, in order to identify the acoustic-dominant layer interval and the text-dominant layer interval, as well as the key layer locations for cross-modal information injection. S4. Definition and Implementation of Three Types of Ablation Modes Based on the layer-by-layer intervention in step S3, four ablation modes are executed respectively: acoustic-terminal blocking, text-terminal blocking, terminal self-blocking, and acoustic-text blocking, to construct a layer-by-layer modal contribution profile. Three core ablation patterns are defined, each corresponding to a different information flow intervention target. The three ablation patterns are as follows: The first type of ablation mode is the acoustic-end blocking mode, which sets the acoustic token position set as the source set and the final sequence position as the target set, blocks all attention connections from the acoustic token position to the final sequence position, and quantifies the acoustic modality; The second type of ablation mode is the text-end blocking mode, which sets the text token position set as the source set and the final sequence position as the target set, blocks all attention connections from the text token position to the final sequence position, and quantifies the text modality; The third type of ablation mode is the last-position self-blocking mode, which sets the final sequence position as both the source set and the target set, blocks the self-attention connection from the final sequence position to itself, and verifies the contribution of the final position token itself to the prediction. By intervening in the three types of ablation modes layer by layer and recording the relative change rate of the prediction probability, the corresponding acoustic-last-position blocking ablation curve, text-last-position blocking ablation curve, and last-position self-blocking ablation curve are obtained. S5, Specialized Tracking of Cross-Modal Information Flow To reveal the propagation mechanism of acoustic information to text representation, an acoustic-text blocking mode is introduced based on the analysis in step S4. Acoustic-text blocking fourth-type ablation is performed, and the propagation law of acoustic information to text representation is analyzed. The specific method is as follows: S5.1 In each Transformer layer, set all acoustic token location sets as the source set and all text token location sets as the target set, while blocking all attention connections from acoustic tokens to text tokens, and preserving the self-attention paths within acoustic tokens and text tokens. S5.2 By performing the fourth type of acoustic-text blocking ablation intervention layer by layer and recording the relative change rate of the predicted probability, the relative change rate of the predicted probability of each target layer is connected into a change curve according to the layer number order. The main occurrence layer interval of acoustic information injection into text representation and the decay law of the injection effect with the increase of layer depth are identified to quantitatively track the propagation process of cross-modal information flow. Combined with the acoustic-terminal blocking ablation curve obtained in step S4, the hierarchical organization structure of audio-text information fusion in the audio big language model is comprehensively judged. S6. Conduct a multi-task system comparison experiment. The analysis process described in steps S1 to S5 is systematically executed under the three tasks of keyword recognition, sentiment classification, and question-answering reasoning: In the keyword recognition task, the specific requirements are that the model performs low-level perceptual alignment between the semantic content of the input speech segment and the given text description, and analyzes the hierarchical contribution of acoustic cues in word recognition. In the emotion classification task, the specific requirements are that the model attribute the emotion of speech based on multimodal cues, and comprehensively utilize acoustic and textual information to analyze the joint contribution of acoustic and textual cues. In the question-answering reasoning task, the model is specifically required to perform detailed identification of audio events and combine text semantics to complete multi-step reasoning. Thus far, based on steps S1 to S6, by comparing the amplitude distribution, peak layer position, and cross-layer distribution differences of the layer-by-layer ablation curves under the three types of tasks, a cross-modal information flow tracking method for audio large language models based on attention ablation is realized, and the internal modal fusion mechanism and hierarchical information processing rules of audio large language models are obtained.
2. The method for cross-modal information flow tracking of large audio language models based on attention ablation as described in claim 1, characterized in that: In the layer-by-layer analysis of the three ablation modes in S4, the acoustic representation-dominant layer intervals show a concentrated acoustic-terminal blocking ablation effect with a significantly larger amplitude than the text-terminal blocking ablation effect; the text representation-dominant layer intervals show a text-terminal blocking ablation effect that continuously increases with layer depth; the terminal self-blocking ablation effect is close to zero in each layer; the three ablation modes can be executed independently on the same model and the same input sample, and the resulting layer-by-layer intervention curves can be compared with each other; As task complexity increases, the model's reliance on modal information gradually transitions from being driven by a single acoustic modality to a balanced collaborative reasoning of acoustic and textual modalities.
3. The method for cross-modal information flow tracking of large audio language models based on attention ablation as described in claim 1, characterized in that: The modality-dependent dynamic change pattern of the model under different cognitive loads in S6 is as follows: In the keyword recognition task, acoustic-terminal blocking ablation leads to a significant decrease in prediction probability, indicating that the prediction results of this task mainly depend on acoustic modal information. As task complexity increases, the model's dependence on acoustic and textual information gradually becomes more balanced, and the magnitudes of the ablation effects of the two modalities gradually approach each other and are distributed across a wider range of network layers. In question-answering reasoning tasks, the injection effect of acoustic information into textual representations extends further to deeper network layers, indicating that complex semantic reasoning tasks require more sufficient audio-text information interaction.