An asynchronous sparse pulse fusion system and method for multi-modal dialogue sentiment recognition and a storage medium

Abstract

The present application relates to the technical field of emotion recognition and natural language processing, and discloses an asynchronous sparse pulse fusion method and system for multi-modal dialogue emotion recognition, aiming to solve the technical problems of insufficient single-modal key emotional clue capture, difficulty in balancing cross-modal synchronous enhancement and asynchronous fine-grained alignment in the prior art. The present application constructs an asynchronous sparse pulse fusion unified framework, obtains three-modal context features through a single-modal context encoding module, enhances single-modal key emotional features through an asynchronous sparse pulse self-attention module, realizes cross-modal consistency enhancement through a multi-modal pulse synchronous fusion module, and finally outputs the emotion recognition result after completing local asynchronous alignment through an asynchronous pulse guided cross-modal attention module. The present application greatly improves the accuracy and scene robustness of multi-modal dialogue emotion recognition, and can be widely applied to intelligent human-computer interaction, mental health monitoring and other scenes.

Description

Technical Field

[0001] This invention relates to the fields of emotion recognition and natural language processing, and in particular to an asynchronous sparse impulse fusion system, method and storage medium for multimodal dialogue emotion recognition. Background Technology

[0002] With the development of artificial intelligence, human-computer interaction, and affective computing, accurately understanding human emotions from natural dialogue has become a crucial issue in intelligent information processing. Dialogue emotion recognition aims to identify the emotional state corresponding to each utterance by combining contextual information within the dialogue. With the widespread application of multimodal data, researchers have begun to combine text, audio, and visual information for emotion modeling, leveraging the complementarity between different modalities to improve recognition performance.

[0003] In existing multimodal dialogue emotion recognition methods, the typical technical approach is to first encode each modality independently, and then combine context modeling and cross-modal interaction to complete emotion classification.

[0004] However, the above-mentioned technical approach still faces several challenges in practical applications.

[0005] Firstly, within a single modality, key emotional changes are usually concentrated in only a few locations, and dense temporal interactions can easily introduce redundant dependencies, thereby weakening the ability to model high-response locations.

[0006] Secondly, in cross-modal interactions, most existing methods assume that different modalities can be aligned relatively directly, or use a unified global fusion approach to handle modal relationships, rarely explicitly distinguishing between synchronous and asynchronous responses. This makes it difficult for the model to simultaneously achieve both enhanced consistency at critical moments and fine-grained alignment under local time offsets, even though it can establish intermodal connections.

[0007] In summary, existing technologies for multimodal dialogue emotion recognition lack a unified technical framework capable of simultaneously addressing sparse dynamic modeling of key emotion locations, synchronous enhancement of multimodal key moments, and asynchronous cross-modal fine-grained alignment under local time offsets. These issues constitute a pressing technical challenge that requires solutions from those skilled in the art. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of existing technologies and address the deficiencies of current multimodal dialogue emotion recognition methods in terms of dynamic modeling of key emotions in single-modality, enhancement of multimodal synchronization consistency, and fine-grained local asynchronous alignment. This invention provides an asynchronous sparse pulse fusion system, method, and storage medium for multimodal dialogue emotion recognition. By introducing a pulse-driven sparse temporal modeling mechanism, a multimodal pulse synchronization detection and enhancement mechanism, and a pulse-guided local asynchronous cross-modal alignment mechanism within a unified framework, it achieves effective capture of key emotion changes within a single modality, collaborative enhancement of key moments between multiple modalities, and fine-grained cross-modal alignment under local time offsets, thereby improving the accuracy and robustness of multimodal dialogue emotion recognition.

[0009] To achieve the above objectives, this application proposes an asynchronous sparse impulse fusion system for multimodal dialogue emotion recognition, comprising:

[0010] The unimodal context encoding module is used to perform linear mapping and context modeling on the text modal features, audio modal features and visual modal features of the input multimodal dialogue data, respectively, to obtain text context features, audio context features and visual context features;

[0011] An asynchronous sparse spurious self-attention module is used to sequentially perform linear mapping, layer normalization, and spurious neuron transformation on the text context features, audio context features, and visual context features to generate corresponding query vectors Query, Key vectors, and Value vectors. The module performs multi-head partitioning on the query vectors Query, Key vectors, and Value vectors, and calculates the multi-head spurious self-attention score. At each query position, the top k connections with the highest attention response are retained, while the remaining low-response connections are disabled. The event intensity at the current time step is estimated based on the average response of the query vector Query spurs in the sequence dimension. An event gating coefficient is generated based on the event intensity, and the attention score at the same time step is scaled overall using the event gating coefficient. The scaled attention score and the Value vector are weighted and aggregated, and the output is projected to obtain the spurious response. The spurious response is then injected back into the original unimodal context features to output the enhanced unimodal spurious feature.

[0012] The multimodal pulse synchronization fusion module is used to perform single-modal pulse event detection on the enhanced single-modal pulse features, construct multimodal synchronization strength, generate a shared synchronization gate based on the multimodal synchronization strength, perform consistency recalibration on the enhanced single-modal pulse features through the shared synchronization gate, and output synchronized enhanced multimodal features.

[0013] An asynchronous pulse-guided cross-modal attention module is used to take any one of the synchronously enhanced multimodal features as the query modality, and perform cross-modal attention interactions with the other two modalities under local time window constraints to obtain cross-modal aligned text features, audio features, and visual features.

[0014] The classification module is used to fuse the cross-modal aligned text features, audio features, and visual features to output the sentiment category prediction result of the current dialogue utterance.

[0015] As a further solution, the asynchronous sparse pulse self-attention module is configured to perform the following operations:

[0016] The input text context features, audio context features, and visual context features are sequentially subjected to linear mapping, layer normalization, and spiking neuron transformation to generate corresponding query vectors Query, Key vectors, and Value vectors.

[0017] The query vector, key vector, and value vector are divided into multiple heads, and the multi-head impulse self-attention score is calculated.

[0018] At each query location, retain the top k connections with the highest attention response and disable the remaining low-response connections;

[0019] The event intensity at the current time step is estimated based on the average response of the query vector Query impulse in the sequence dimension. An event gating coefficient is generated based on the event intensity, and the attention score at the same time step is scaled as a whole using the event gating coefficient.

[0020] The scaled attention score and value vector Value are weighted and aggregated, and the impulse response is obtained after output projection. The impulse response is then injected back into the original unimodal context features to output the enhanced unimodal impulse features.

[0021] As a further solution, the event gating coefficient is calculated using the following formula:

[0022]

[0023] In the formula, For the first The event gating coefficient of the time step, The lower bound parameter for gating. For the Sigmoid function, To learn the linear weight parameters, For the first The intensity of events at each time step.

[0024] As a further solution, the multimodal pulse synchronization fusion module is configured to perform the following operations:

[0025] Based on the pulse changes at adjacent positions in the enhanced single-modal pulse features, the event intensities corresponding to the text modal pulse features, audio modal pulse features, and visual modal pulse features are calculated respectively.

[0026] An adaptive impulse detection threshold is constructed based on the in-sample statistics of the event intensity of each modality. The event intensity of each modality is mapped to the corresponding adaptive impulse detection threshold to obtain the single-mode impulse response of each modality.

[0027] Multimodal synchronization pulse intensity is constructed using the k-of-3 synchronization rule for the single-modal impulse responses of text, audio, and visual modalities;

[0028] Based on the multimodal synchronization pulse intensity, a shared synchronization gate is generated by mapping with the Sigmoid function and combining it with the synchronization modulation coefficients;

[0029] The shared synchronization gating is applied simultaneously to the enhanced text modality features, audio modality features, and visual modality features to perform consistent recalibration of the three modality features and output synchronized enhanced multimodal features.

[0030] As a further solution, the asynchronous pulse-guided cross-modal attention module is configured to perform the following operations:

[0031] Using any one of the synchronously enhanced multimodal features as the query modality, and taking the other two modalities as source modalities, cross-modal attention interaction is performed to obtain a dual-path cross-modal enhanced representation. In calculating the cross-modal attention score, the impulse event detector is reused to obtain the impulse intensity of each source modality, and the attention connection at the filling position is masked by the effective speech mask. Local time window constraints, time distance attenuation terms, and guidance bias determined by the impulse intensity of the source modality are introduced.

[0032] The dual-path cross-modal enhancement representations are fused to obtain the cross-modal correction representation;

[0033] By using learnable residual gating, the cross-modal corrected representation is fused with the original features of the query modality to output the cross-modal aligned target features.

[0034] As a further solution, the event intensity of each modality is mapped to the corresponding adaptive impulse detection threshold to obtain the single-mode impulse response of each modality:

[0035]

[0036] In the formula, Represents the single-mode impulse response for each mode. These correspond to the three modalities of text, audio, and video, respectively. The value represents the variation amplitude of features between adjacent frames, and T is the temperature parameter;

[0037] in, The adaptive pulse detection threshold for the m-th mode is expressed by the following formula:

[0038]

[0039] In the formula, It is a mean function. Let m be the event intensity sequence. For threshold coefficient, This is the Sigmoid function.

[0040] As a further solution, the k-of-3 synchronization rule is used to construct the multimodal synchronization impulse intensity for the single-modal impulse responses of text, audio, and visual modalities:

[0041]

[0042] In the formula, This represents the intensity of the multimodal synchronization pulse at time step t. Represents the text modal impulse response. Indicates the audio modal impulse response. This represents the video modal impulse response, where k is the preset synchronization threshold in the k-of-3 synchronization rule. To synchronize temperatures;

[0043] Based on the multimodal synchronization pulse intensity, a shared synchronization gate is generated by mapping using the Sigmoid function and combining it with the synchronization modulation coefficients. The calculation process is as follows:

[0044]

[0045]

[0046] In the formula, For the shared synchronization gating at time step t, These are the synchronization modulation coefficients. For learnable upper bound parameters of synchronous gating, This is the Sigmoid function.

[0047] As a further solution, the cross-modal corrected representation is fused with the original features of the query modality using learnable residual gating to output the cross-modal aligned target features. Specifically, it includes:

[0048]

[0049] In the formula, This represents the original characteristics of the query modality. This represents the cross-modal correction representation;

[0050] λ is the residual gating coefficient, which satisfies: ;

[0051] In the formula, For the Sigmoid function, These are learnable residual gating parameters.

[0052] On the other hand, the present invention also provides an asynchronous sparse impulse fusion method for multimodal dialogue emotion recognition, comprising the following steps:

[0053] We acquire text modal features, audio modal features, and visual modal features from multimodal dialogue data, and perform linear mapping and context modeling respectively to obtain text context features, audio context features, and visual context features.

[0054] The text context features, audio context features, and visual context features are sequentially linearly mapped, layer normalized, and transformed by spiking neurons to generate corresponding query vectors (Query), key vectors (Key), and value vectors (Value). Multi-head partitioning is performed on the query vectors (Query), key vectors (Key), and value vectors (Value), and multi-head spiking self-attention scores are calculated. At each query position, the top k connections with the highest attention responses are retained, while the remaining low-response connections are disabled. The event intensity at the current time step is estimated based on the average response of the query vector spiking on the sequence dimension. An event gating coefficient is generated based on the event intensity, and the attention scores at the same time step are scaled overall using the event gating coefficient. The scaled attention scores and value vectors (Value) are weighted and aggregated, and the resulting impulse response is obtained after output projection. The impulse response is then injected back into the original unimodal context features to output enhanced unimodal impulse features.

[0055] The enhanced single-mode pulse features are subjected to single-mode pulse event detection, a multi-mode synchronization strength is constructed, and a shared synchronization gating is generated based on the multi-mode synchronization strength. The enhanced single-mode pulse features are then recalibrated for consistency through the shared synchronization gating, and the synchronized enhanced multi-mode features are output.

[0056] Using any one of the synchronously enhanced multimodal features as the query modality, cross-modal attention interactions are performed with the other two modalities under local time window constraints to obtain cross-modal aligned text features, audio features, and visual features.

[0057] The cross-modal aligned text features, audio features, and visual features are fused to output the sentiment category prediction result of the current dialogue utterance.

[0058] On the other hand, according to embodiments of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium storing a computer program configured to implement the steps of the asynchronous sparse impulse fusion method for multimodal dialogue emotion recognition described above when invoked by a processor.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] This invention constructs a unified asynchronous sparse pulse fusion framework. Through the progressive collaboration of three core modules—asynchronous sparse pulse self-attention, multimodal pulse synchronous fusion, and asynchronous pulse-guided cross-modal attention—it solves the technical bottlenecks of single-modal redundancy dependency, insufficient synchronous enhancement, and difficulty in asynchronous alignment in existing multimodal dialogue emotion recognition.

[0061] At the unimodal level, by retaining only the top k connections with the highest responses at each query location and introducing event-intensity-based gating modulation, noise interference from fully connected self-attention is effectively filtered out. This allows the model to accurately focus on a few but crucial emotional change locations in the dialogue, significantly improving the representation quality of unimodal emotional dynamics. At the multimodal synchronization level, by explicitly detecting consistent responses of different modalities at key moments at the impulse level, a shared synchronization gating is generated using adaptive thresholds and k-of-3 synchronization rules. This achieves synergistic enhancement across modal key moments without directly mixing modal features, strengthening the complementarity of multimodal information at key nodes of emotional expression. At the asynchronous alignment level, by simultaneously introducing a temporal distance decay term and a source modal impulse guiding bias within a local time window, cross-modal attention is subject to the dual constraints of temporal proximity and impulse saliency. Combined with learnable residual gating for adaptive adjustment of the correction amplitude, this captures intermodal temporal shifts while avoiding excessive perturbations, achieving fine-grained and robust asynchronous cross-modal alignment.

[0062] In summary, this invention, through a unified architecture driven by pulses throughout the entire process, possesses excellent generalization and adaptation capabilities in real and complex dialogue scenarios, and can be widely applied to various dialogue emotion recognition scenarios such as intelligent customer service, emotional companionship, and public opinion analysis. Attached Figure Description

[0063] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0064] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0065] Figure 1 A schematic diagram of the structure of an asynchronous sparse pulse fusion system for multimodal dialogue emotion recognition provided by the present invention;

[0066] Figure 2 This is a schematic diagram of the asynchronous sparse pulse fusion framework provided by the present invention;

[0067] Figure 3 A schematic diagram of the asynchronous sparse pulse self-attention module provided by the present invention;

[0068] Figure 4 This is a schematic diagram of the structure of the multimodal pulse synchronization fusion module provided by the present invention;

[0069] Figure 5 A schematic diagram of the structure of the asynchronous pulse-guided cross-modal attention module provided by the present invention;

[0070] Figure 6 A performance comparison chart of the asynchronous sparse pulse self-attention module provided by the present invention with different sparsity rates and event gating configurations on the MELD dataset;

[0071] Figure 7 A schematic diagram illustrating the sensitivity analysis of synchronization fusion parameters of the multimodal pulse synchronization fusion module provided by this invention;

[0072] Figure 8 A schematic diagram illustrating the sensitivity analysis of the core parameters of the multimodal pulse synchronization fusion module provided by this invention on the IEMOCAP dataset;

[0073] Figure 9 A schematic diagram illustrating the sensitivity analysis of the core parameters of the asynchronous pulse-guided cross-modal attention module provided by this invention on the MELD dataset;

[0074] Figure 10 The flowchart illustrates the steps of an asynchronous sparse impulse fusion method for multimodal dialogue emotion recognition provided by this invention. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0076] Example 1

[0077] like Figure 1 As shown, this embodiment of the invention proposes an asynchronous sparse impulse fusion system for multimodal dialogue emotion recognition, comprising:

[0078] The unimodal context encoding module is used to perform linear mapping and context modeling on the text modal features, audio modal features and visual modal features of the input multimodal dialogue data, respectively, to obtain text context features, audio context features and visual context features;

[0079] An asynchronous sparse spurious self-attention module is used to sequentially perform linear mapping, layer normalization, and spurious neuron transformation on the text context features, audio context features, and visual context features to generate corresponding query vectors Query, Key vectors, and Value vectors. The module performs multi-head partitioning on the query vectors Query, Key vectors, and Value vectors, and calculates the multi-head spurious self-attention score. At each query position, the top k connections with the highest attention response are retained, while the remaining low-response connections are disabled. The event intensity at the current time step is estimated based on the average response of the query vector Query spurs in the sequence dimension. An event gating coefficient is generated based on the event intensity, and the attention score at the same time step is scaled overall using the event gating coefficient. The scaled attention score and the Value vector are weighted and aggregated, and the output is projected to obtain the spurious response. The spurious response is then injected back into the original unimodal context features to output the enhanced unimodal spurious feature.

[0080] The multimodal pulse synchronization fusion module is used to perform single-modal pulse event detection on the enhanced single-modal pulse features, construct multimodal synchronization strength, generate a shared synchronization gating based on the multimodal synchronization strength, and perform consistency recalibration on text modal features, audio modal features and visual modal features through the shared synchronization gating to output synchronized enhanced multimodal features;

[0081] An asynchronous pulse-guided cross-modal attention module is used to take any one of the synchronously enhanced multimodal features as the query modality, and perform cross-modal attention interactions with the other two modalities under local time window constraints to obtain cross-modal aligned text features, audio features, and visual features.

[0082] The classification module is used to fuse the cross-modal aligned text features, audio features, and visual features to output the sentiment category prediction result of the current dialogue utterance.

[0083] It should be noted that, in order to address the technical problems existing in the prior art, an Asynchronous Sparse Pulse Fusion (ASPF) framework is proposed. This framework jointly models single-modal key dynamic responses, multimodal synchronous enhancements, and asynchronous cross-modal fine-grained alignment under a unified structure, thereby more fully characterizing multimodal emotional changes in dialogue.

[0084] like Figure 2 As shown, Figure 2 The orange arrows represent the text modal information flow, the yellow arrows represent the audio modal information flow, and the blue arrows represent the visual modal information flow. In this embodiment, single-modal context encoding is first performed on the text, audio, and visual modalities respectively, corresponding to... Figure 2 The semantic modeling stage in the process. Specifically, for the i-th utterance, the text input, audio input, and visual input are extracted by the text feature extractor, audio feature extractor, and video feature extractor, respectively, to obtain initial unimodal features, denoted as […]. , and Here, the superscripts t, a, and v represent text, audio, and visual modalities, respectively. After the initial features of the three modalities are extracted, they are projected onto a unified latent space dimension through their respective linear mapping layers. Furthermore, a Dialogue Recurrent Neural Network (DialogueRNN) is used to model the utterance-level contextual dependencies within each modality. Specifically, the audio and visual branches obtain contextual representations through DialogueRNN, while the text branch further incorporates DialogueRNN for contextual encoding on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset. The linearly projected representation is retained on the Multimodal Emotion Lines Dataset (MELD). These processes yield the contextual features of the three modalities, which serve as input for the subsequent dynamic modeling stage.

[0085] Then enter Figure 2The dynamic modeling stage is shown. This stage includes an asynchronous sparse impulse self-attention module, a multimodal impulse synchronization fusion module, and an asynchronous impulse-guided cross-modal attention module. First, text, audio, and visual contextual features are input into the asynchronous sparse impulse self-attention module to model asynchronous temporal relationships and enhance sparse impulse attention within each modality. After passing through this module, the multimodal impulse synchronization fusion module further synchronizes and fuses the asynchronous dynamic pattern relationships between different modalities, obtaining impulse state representations corresponding to the three modalities, denoted as follows: , and ,in, , and Let represent the modal impulse responses of the i-th utterance in text, audio, and visual modes, respectively.

[0086] After obtaining the pulse state representations for the three modalities, this embodiment further guides the cross-modal attention module through asynchronous pulses. , and Cross-modal synchronous interaction and fusion are performed to enhance complementary emotional information between different modalities. After processing by this module, updated text, audio, and visual modal features are obtained, denoted as follows: , and Finally, By concatenating the features, we obtain the multimodal joint features of the i-th utterance. The data is then input into a classifier for sentiment classification, and the corresponding sentiment category is output.

[0087] For specific implementation details, please refer to [link / reference]. Figure 3 After obtaining the unimodal contextual features, an Asynchronous Sparse Self-Attention (ASA) module is used. This module simultaneously introduces sparse connection filtering and time-step event gating into the spurious self-attention mechanism to enhance the model's responsiveness to key sentiment changes. For any modality's contextual features, the ASA module first transforms them using linear mapping, layer normalization (LN), and Leaky Integrate-and-Fire (LIF) neurons to generate corresponding query vectors (Query, Key, and Value). Subsequently, the Query, Key, and Value are multi-headed, and the spurious self-attention score is calculated independently for each attention head.

[0088] like Figure 3As shown, the attention score matrix is ​​calculated first. ;

[0089] In the formula, Q is the query matrix and K is the key matrix.

[0090] Then, the sparse attention score is obtained through Topk masking. M is the Topk mask matrix;

[0091] To reduce redundant interference caused by low-response connections, the asynchronous sparse impulse self-attention module retains only the top k connections with the highest attention response at each query position, and masks the remaining low-response connections.

[0092] The number of connections k to be retained is determined by the sequence length L and the preset sparsity ρ, satisfying the following formula:

[0093]

[0094] In the formula, L is the temporal sequence length of the modal features, and ρ is the preset sparsity rate.

[0095] This formula ensures that, under extremely sparse settings (ρ close to 0), the value of k is at least 1, guaranteeing that at least one valid connection is retained at each query position. This sparsity operation allows attention computation to focus on truly important temporal positions, effectively suppressing the noise interference that is prevalent in fully connected self-attention.

[0096] It should be noted that the Asynchronous Sparse Pulse Self-Attention (ASA) module is based on a time-step event intensity-gated modulation mechanism. It estimates the event intensity of the current time step based on the average response of the Query pulse in the sequence dimension. The event intensity reflects the pulse activity level in the overall sequence at the current moment.

[0097] The Query impulses are averaged and processed through a linear layer. Based on the estimated event intensity, event gating coefficients are generated.

[0098]

[0099] In the formula, For the first The event gating coefficient of the time step, This is the lower limit parameter for gating (pulse bias parameter). For the Sigmoid function, To learn the weight parameters, For the first The intensity of events at each time step.

[0100] The event gating coefficients are used to scale the attention score maps at the same time step as a whole. The effect is to adaptively enhance the response of key impulse event locations based on event intensity while preserving the original attention structure, and at the same time suppress interference at low response locations.

[0101] Among them, the sparsified attention score Gating modulation includes: ,

[0102] In the formula, This is the sparse attention score matrix after gating modulation. This represents the event gating coefficient.

[0103] After gating modulation is completed, the asynchronous sparse pulse self-attention module performs weighted aggregation on the scaled attention score and value vector Value, and obtains the impulse response after output projection transformation;

[0104] The formula for calculating the impulse response is as follows: V is the value matrix, and S is the scaling factor.

[0105] The impulse response is processed by linear transformation, layer normalization, and LIF at the output terminal to output enhanced features.

[0106] Finally, the impulse response is used as a dynamic enhancement weight and injected back into the original unimodal context feature to form the enhanced unimodal impulse feature.

[0107] like Figure 4 As shown, after the asynchronous sparse pulse self-attention module completes the key dynamic enhancement within a single modality, this invention introduces a multi-modal pulse synchronization fusion module (MPSF). Taking three modal pulse features spk_t (text), spk_a (audio), and spk_v (video) as input, MPSF explicitly detects the consistent pulse response of different modalities at key moments through a pulse event detector, and performs synchronous enhancement of the text, audio, and visual features through a shared gating mechanism. The core technical process of the multi-modal pulse synchronization fusion module includes three key steps: single-modal pulse event detection, multi-modal synchronization intensity construction, and shared gating generation.

[0108] In the single-modal impulse event detection stage, the input three-modal impulse features spk_t (text), spk_a (audio), and spk_v (video) are processed by an impulse event detector. The event intensity of each modality is calculated based on the impulse changes at adjacent positions, thus obtaining the text modal impulse response s. t Audio modal impulse response s aand video modal impulse response s v :

[0109]

[0110] In the formula, These correspond to the three modalities of text, audio, and video, respectively. The value represents the variation amplitude of features between adjacent frames, and T is the temperature parameter;

[0111] Subsequently, MPSF constructs an adaptive impulse detection threshold based on the in-sample statistics of the intensity of each modal event:

[0112]

[0113] In the formula, The adaptive pulse detection threshold for the m-th mode is... It is a mean function. Let m be the event intensity sequence. For threshold coefficient, This is the Sigmoid function.

[0114] When obtaining the single-mode impulse response of each mode, if the preset detection window parameter w is greater than zero, the multimodal impulse synchronization fusion module MPSF performs a local maximum pooling operation on the single-mode impulse response in the time dimension to preserve the continuous impulse event response.

[0115] After obtaining the text, audio, and visual single-modal impulse responses, MPSF fuses the event scores of the three channels and constructs a multimodal synchronization impulse intensity c using the k-of-3 synchronization rule. This intensity c is used to characterize the consistent activation level of the three single-modal impulse responses at time step t.

[0116]

[0117] In the formula, This represents the intensity of the multimodal synchronization pulse at time step t. Represents the text modal impulse response. Indicates the audio modal impulse response. This represents the video modal impulse response, where k is the preset synchronization threshold in the k-of-3 synchronization rule. To synchronize temperatures;

[0118] Based on the multimodal synchronization pulse intensity, a shared synchronization gate is generated by mapping using the Sigmoid function and combining it with the synchronization modulation coefficients. The calculation process is as follows:

[0119]

[0120]

[0121] In the formula, These are the synchronization modulation coefficients. For learnable upper bound parameters of synchronous gating, For the Sigmoid function, For the shared synchronization gating at time step t.

[0122] To ensure numerical stability and eliminate global bias, shared synchronization gating is used. Perform mean normalization over time steps:

[0123]

[0124] in, Indicates the initial gating The average value is calculated across the discourse dimension (i.e., all time steps in a sample) to ensure the scale stability of the gating.

[0125] Finally, the generated shared synchronization gate will be... Simultaneously applied to text, audio, and visual modalities, it achieves consistent recalibration of multimodal features and outputs synchronously enhanced multimodal features:

[0126]

[0127] Through the above process, the Multimodal Pulse Synchronization Fusion (MPSF) module does not directly mix the original feature representations of the three modes. Instead, it first estimates the synchronization response relationship between different modes at the pulse level, and then performs consistency recalibration of the representation in the form of shared gating. This achieves cross-modal collaborative enhancement at critical moments and strengthens the complementarity and consistency of multimodal information at key nodes of emotional expression.

[0128] like Figure 5 As shown, after completing the synchronous pulse fusion of the multimodal pulse synchronous fusion module, the asynchronous pulse-guided cross-modal attention module (APCA) is further adopted to model the local asynchronous interaction between text, audio and video modalities.

[0129] The asynchronous pulse-guided cross-modal attention module uses the synchronously enhanced multimodal features as input to construct independent processing branches for text, audio, and visual modalities respectively. (See also...) Figure 5Each branch consists of n stacked APCA attention layers, each layer containing APCA attention computation, layer normalization, and a feedforward network. For each branch, APCA uses any text, audio, or video modality from the synchronously enhanced multimodal features as the query modality, and performs cross-modal attention interaction with the other two modalities as source modalities to obtain a dual-path cross-modal enhanced representation. Finally, it outputs the three-path corrected text features F. t Audio features F a Video features F v .

[0130] Unlike conventional cross-modal attention mechanisms, the Asynchronous Impulse Guided Cross-Modal Attention (APCA) module introduces a triple modulation mechanism when calculating the cross-modal attention score. First, APCA reuses the aforementioned impulse event detector to extract impulse intensity signals from the text modality impulse feature spk_t, the audio modality impulse feature spk_a, and the visual modality impulse feature spk_v, respectively, to obtain the text modality impulse response s. t Text modal impulse response s a Text modal impulse response s v This serves as an auxiliary signal for subsequent guidance bias. Based on this, the first modulation layer uses a local time window constraint, limiting the attention calculation to a preset time window near the query position. The second layer is a time distance attenuation term, which weights the source modality positions within the window based on their time distance from the query position, with greater distance resulting in lower weight. The third layer is a guidance bias determined by the source modality impulse intensity, using the impulse intensity of the source modality at the corresponding time step as an additional bias term added to the attention score, making cross-modal attention more focused on positions where the source modality impulse response is significant. Simultaneously, when calculating the attention score, an effective utterance mask is used to mask attention connections at fill positions. It should be noted that the impulse intensity participates in the attention calculation as a key bias on the source modality side, rather than directly affecting the query modality, thus ensuring the asymmetry and directionality of cross-modal interaction.

[0131] After completing the dual-path cross-modal attention interaction, the asynchronous pulse-guided cross-modal attention module (APCA) fuses the obtained dual-path cross-modal enhanced representations to obtain the cross-modal corrected representation. The fusion method can employ operations such as element-wise addition, concatenation, or attention-weighted fusion.

[0132] To control the impact of cross-modal information on the original modal representation and avoid excessive perturbation caused by cross-modal correction, APCA further introduces a learnable residual gating mechanism. The residual gating coefficient λ in the residual gating is calculated using the following formula:

[0133]

[0134] In the formula, λ is the residual gating coefficient, and σ(*) is the Sigmoid function. These are learnable residual gating parameters.

[0135] Based on the residual gating coefficient, the cross-modal corrected representation and the original features of the query modality are fused in the following manner to output the cross-modal aligned target features. :

[0136] In the formula, This represents the original characteristics of the query modality. This represents the cross-modal correction representation.

[0137] For text, audio, and visual modalities, the asynchronous impulse-guided cross-modal attention module (APCA) constructs three parallel processing branches respectively:

[0138] Text branch: Using text as the query modality, it interacts with audio and video modalities, and outputs the corrected text features F. t ;

[0139] Audio branch: Using audio as the query modality, it interacts with text and video modalities, and outputs the corrected audio features F. a ;

[0140] Video branch: Using video as the query modality, it interacts with text and audio modalities, and outputs the corrected video features F. v .

[0141] After processing by the Asynchronous Impulse Guided Cross-Modal Attention (APCA) module, three cross-modal aligned modal feature representations are obtained. Subsequently, the classification module concatenates these three features along the feature dimensions to form a fused feature vector containing integrated information from text, audio, and vision. This fused feature vector is fed into the classifier, and after linear transformation and nonlinear activation, it outputs the prediction results of the current dialogue utterance in multiple preset sentiment categories.

[0142] During model training, the entire ASPF framework undergoes end-to-end joint optimization in the order of context encoding, ASA, MPSF, APCA, and classifier. The model first obtains textual, audio, and visual context features through a single-modal context encoding module. Then, it sequentially undergoes sparse impulse self-attention enhancement (ASA), synchronous impulse fusion (MPSF), and asynchronous cross-modal alignment (APCA). Finally, the classifier outputs the sentiment prediction result. The training loss function uses a classification loss, and auxiliary loss terms can be introduced to further guide the model to learn impulse representations with clear emotional dynamics.

[0143] Through the above design, the ASPF framework provided in this embodiment can jointly model single-modal key dynamic responses, multimodal synchronous enhancement, and asynchronous cross-modal alignment under a unified structure, thereby more fully characterizing multimodal emotional changes in dialogue and improving the performance of multimodal dialogue emotion recognition.

[0144] Example 2

[0145] This embodiment, based on embodiment 1, verifies the effectiveness and advancement of the ASPF framework on public datasets through specific experiments.

[0146] The experiment uses two publicly available benchmark datasets commonly used in the field of conversational multimodal emotion recognition for evaluation. The data statistics according to the official division are shown in Table 1. Both datasets provide three modalities: text, audio, and video, and organize the data in a "dialogue-discourse" format, preserving speaker annotations and dialogue boundaries.

[0147] The first dataset is MELD (Multimodal EmotionLines Dataset), derived from TV drama scenes. MELD features shorter dialogues and more frequent turns, encompassing seven emotion categories: neutral, surprise, fear, sadness, joy, disgust, and anger. The second dataset is IEMOCAP (Interactive Emotional Dyadic Motion Capture), derived from two-person dialogue scenarios. IEMOCAP exhibits more nuanced and natural emotional expression with longer dialogue spans, covering six emotion categories: happiness, sadness, neutral, anger, excitement, and frustration. The two datasets are significantly complementary in terms of emotion distribution, modal quality, and dialogue style: MELD focuses on emotion switching in multi-turn, rapid dialogues, while IEMOCAP emphasizes emotional evolution in deep two-person interactions. The combination of the two datasets can comprehensively evaluate the generalization ability and robustness of the Asynchronous Sparse Impulse Fusion Framework (ASPF) across different dialogue scenarios.

[0148] Table 1: Statistics for MELD and IEMOCAP

[0149]

[0150] In the data preprocessing stage, text modalities are aligned by utterance and encoded by a pre-trained language model, audio modalities extract commonly used acoustic features, and visual modalities are represented at the frame level from face and expression regions. The three modal features are standardized after utterance-level temporal alignment and uniformly projected to the same dimension using linear mapping. Missing modalities or locations are marked using a masking mechanism to ensure consistency between training and evaluation.

[0151] In terms of experimental setup, the model input consists of features from three modalities: text, speech, and vision. To facilitate subsequent joint modeling, the initial features of the three modalities are first passed through independent linear mapping layers, projected uniformly into a 256-dimensional representation space, and then fed into the temporal coding and cross-modal interaction module for processing. The core hyperparameters of the model are configured as follows:

[0152] The multimodal interaction module has 6 layers, a representation dimension of 256, 4 multi-head attention heads, and a feedforward hidden layer dimension of 1024. The pulse encoder uses a 2-layer structure with a time step of 32, a membrane time constant of 10.0, and a discharge threshold of 1.0. During training, the optimizer uses the Adaptive Moment Estimation (Adam) algorithm with an initial learning rate of 1×10⁻⁶. -4 The weight decay factor is set to 1×10. -5 The total number of training rounds was set to 100, and the learning rate scheduling strategy adopted was a platform scheduling algorithm with reduced learning rate. To ensure the reproducibility of the experimental results, the random seed was uniformly set to 2023.

[0153] The evaluation metrics used are weighted F1 score (W-F1) and accuracy. Weighted F1 score is used as the main performance evaluation metric because it takes into account the differences in sample size and classification performance of each category.

[0154] To verify the overall performance of the present invention, ASPF was compared with several representative dialogue emotion recognition methods.

[0155] As shown in Table 2, experimental results on the MELD dataset demonstrate that the ASPF framework proposed in this invention achieves a weighted F1 score of 66.81, achieving the best performance among all compared methods. Specifically, ASPF improves upon Spiking Emotion Recognition (SpikEmo), which also incorporates pulse modeling, by 0.89 (SpikEmo's score is 65.92); it improves upon Multimodal Emotion Recognition (MultiEMO) by 2.38 (MultiEMO's score is 64.43); and it improves upon Multimodal Dynamic Fusion Network (MM-DFN) by 7.35 (MM-DFN's score is 59.46). Looking at the F1 scores across various categories, ASPF achieved 79.93 in the neutral category, 26.67 in the fear category, 42.11 in the sadness category, 65.50 in the joy category, 32.76 in the disgust category, and 54.79 in the anger category. ASPF achieved the best or second-best individual scores in multiple categories, which fully demonstrates that ASPF not only leads in overall performance but also has a strong ability to discriminate fine-grained emotions across various categories.

[0156] As shown in Table 3, the ASPF framework also demonstrates significant advantages on the IEMOCAP dataset, achieving a weighted F1 score of 73.16. Compared to existing methods, ASPF improves upon SpikEmo by 1.66 (SpikEmo is 71.50), MultiEmo by 4.83 (MultiEmo is 68.33), Contextualized Graph-based Multimodal Emotion Recognition (COGMEN) by 5.56 (COGMEN is 67.60), and MM-DFN by 4.98 (MM-DFN is 68.18).

[0157] Looking at the F1 scores across different categories, ASPF achieved 63.44 for happiness, 82.91 for sadness, 75.23 for neutrality, 65.89 for anger, 78.73 for excitement, and 67.02 for frustration. Compared to the MELD dataset, ASPF achieved a more significant performance improvement on IEMOCAP, indicating that ASPF has a stronger ability and adaptability to model the more complex and nuanced emotional dynamics in two-person dialogue scenarios.

[0158] Table 2: Experimental results on the MELD dataset

[0159]

[0160] Table 3: Experimental results on the IEMOCAP dataset

[0161]

[0162] To comprehensively analyze the impact of different modal inputs on the performance of the ASPF framework, this embodiment further sets up comparative experiments on single-modal, bimodal, and trimodal inputs. As shown in Table 4, the experimental results demonstrate that the trimodal setting (simultaneously using text, audio, and visual features) achieves optimal performance on both datasets. In the single-modal setting, the text modality achieves a weighted F1 score of 60.31 on MELD and 65.92 on IEMOCAP, significantly outperforming the audio modality (37.23 on MELD, 45.33 on IEMOCAP) and the visual modality (30.97 on MELD, 29.34 on IEMOCAP) on both datasets. This result fully demonstrates that semantic information remains the most important criterion for dialogue sentiment recognition. In the bimodal setting, the weighted F1 scores of the text-based combination (text + audio, denoted as T+A, and text + visual, denoted as T+V) on both datasets reached 63 to 69, significantly outperforming the combination consisting only of visual and audio (V+A, only 39.45 on MELD and only 50.10 on IEMOCAP). This further confirms that the text modality plays a dominant role in the overall recognition, while audio and visual modalities mainly play important supplementary and enhancing roles. The trimodal fusion further improved upon the optimal bimodal combination on both datasets, validating the effectiveness of the multimodal information fusion strategy within the ASPF framework.

[0163] Table 4: Comparison of different modal settings

[0164]

[0165] In another specific implementation, we maintained complete consistency in the overall model framework and experimental settings on the MELD and IEMOCAP datasets. We removed the ASA (Asynchronous Sparse Impulse Self-Attention Module), MPSF (Multimodal Impulse Synchronization Fusion Module), and APCA (Asynchronous Impulse Guided Cross-Modal Attention Module), respectively, using weighted F1 as the evaluation metric. The experimental results are shown in Table 5. Compared to the complete ASPF model, removing any core module leads to a stable decrease in performance, strongly demonstrating that all three modules play an irreplaceable positive role in improving overall performance.

[0166] Specifically, on the MELD dataset, removing the ASA module decreased the weighted F1 score from 66.81 to 66.24, removing the MPSF module decreased it to 66.30, and removing the APCA module decreased it to 66.48. On the IEMOCAP dataset, the performance degradation was even more significant: removing the ASA module decreased the weighted F1 score from 73.16 to 70.80 (a decrease of 2.36), removing the MPSF module decreased it to 72.31 (a decrease of 0.85), and removing the APCA module decreased it to 72.23 (a decrease of 0.93). In-depth analysis of the ablation results revealed that the performance degradation caused by removing the ASA module was most significant on both datasets, especially on IEMOCAP where the degradation reached 2.36. This indicates that the asynchronous sparse impulse self-attention mechanism can more effectively model temporal dependencies in dialogue and improve the quality of unimodal representations, making it the most core contributing component of the entire ASPF framework. The removal of the MPSF module also resulted in a stable and consistent performance degradation, indicating that the impulse event detection and synchronization fusion mechanism helps capture shared emotional cues across different modalities, thereby enhancing the collaborative expressive power of multimodal information. The removal of the APCA module also caused a performance decrease, demonstrating that impulse-guided asynchronous cross-modal attention interaction is necessary for achieving fine-grained cross-modal alignment. This necessity is particularly evident in dialogue scenarios where cross-modal emotional cues are not fully synchronized. Cross-dataset comparisons also revealed an important pattern: the performance degradation on IEMOCAP was generally greater than on MELD, indicating that in two-person deep dialogue scenarios, the synergistic effect among the three core modules of ASPF is more crucial for multimodal emotion modeling, and none can be omitted.

[0167] Table 5: Comparison after removing each module

[0168]

[0169] Furthermore, to analyze the impact of key parameter settings on model performance, this embodiment conducts sensitivity analysis on the relevant parameters of the ASA, impulse event detection, multimodal synchronous fusion, and APCA modules, respectively.

[0170] Figure 6 This is a performance comparison chart on the MELD dataset for different sparsity rates and event gating configurations of the Asynchronous Sparse Impulse Self-Attention (ASA) module provided by this invention. Figure 6 In the comparative experiment, for the ASA module of this invention, when the sparsity parameter sparse_topk is set to 0.5 and the event gating function is enabled, the model achieves the best F1 score on the MELD dataset in this experiment, verifying the synergistic effect between appropriate sparsity and event-aware gating.

[0171] Figure 7This diagram illustrates the sensitivity analysis of synchronization fusion parameters for the multimodal pulse synchronization fusion (MPSF) module provided by this invention. The concentric rings represent the F1 scores of the module on the MELD dataset. Arrows point to each ring and are labeled with their corresponding values, from the inside out: 66.2, 66.4, 66.6, and 66.8. Figure 7 (a) shows the impact of the threshold coefficient α on model performance. Figure 7 (b) To detect the effect of temperature T on model performance, Figure 7 (c) shows the impact of the time window w on model performance.

[0172] Experimental results show that, for the core parameters of the pulse event detection component in the MPSF module, when the threshold coefficient α is set to 2.5, the detection temperature T is set to 0.35, and the time window W is set to 1, the model performance reaches the optimal level, achieving the best cross-modal synchronization enhancement effect.

[0173] Figure 8 This is a schematic diagram illustrating the sensitivity analysis of the core parameters of the Multimodal Pulse Synchronization Fusion (MPSF) module provided by this invention on the IEMOCAP dataset. Figure 8 (a) is the upper limit of the synchronization gate. The impact curve on F1 score on the IEMOCAP dataset. Figure 8 (b) shows the impact curve of the synchronization rule k-of-3 on the F1 score on the IEMOCAP dataset. Figure 8 (c) represents the synchronous temperature. The impact curve on F1 score on the IEMOCAP dataset.

[0174] Experimental results show that when the upper limit of the synchronization gate is reached... Use 0.25, synchronization rule k-of-3 is set to 2, synchronization temperature The model performance reaches its peak when the value is 0.35.

[0175] Figure 9 A schematic diagram illustrating the sensitivity analysis of the core parameters of the Asynchronous Pulse Guided Cross-Modal Attention (APCA) module provided by this invention on the MELD dataset; wherein, Figure 9 (a) shows the effect of pulse bias β on the F1 score on the MELD dataset. Figure 9 (b) shows the effect curve of the local time window Δt on the F1 score on the MELD dataset.

[0176] Experimental results show that for the APCA module, the asynchronous cross-modal alignment effect is optimal when the pulse bias β is set to 0.011 and the local time window Δt is set to 3.

[0177] In summary, the experimental results of this embodiment demonstrate that the ASPF proposed in this invention can jointly model single-modal key dynamic responses, multimodal synchronous enhancement, and asynchronous cross-modal alignment within a unified framework, thereby more fully characterizing multimodal emotional changes in dialogue and improving the performance of multimodal dialogue emotion recognition. Experimental comparisons, modal combination experiments, ablation experiments, and parameter sensitivity analyses all indicate that the three modules, ASA, MPSF, and APCA, can form an effective synergy, thereby jointly improving the overall performance of the model.

[0178] Example 3

[0179] On the other hand, embodiments of the present invention also provide an asynchronous sparse impulse fusion method for multimodal dialogue emotion recognition, which includes the following steps:

[0180] We acquire text modal features, audio modal features, and visual modal features from multimodal dialogue data, and perform linear mapping and context modeling respectively to obtain text context features, audio context features, and visual context features.

[0181] The text context features, audio context features, and visual context features are sequentially linearly mapped, layer normalized, and transformed by spiking neurons to generate corresponding query vectors (Query), key vectors (Key), and value vectors (Value). Multi-head partitioning is performed on the query vectors (Query), key vectors (Key), and value vectors (Value), and multi-head spiking self-attention scores are calculated. At each query position, the top k connections with the highest attention responses are retained, while the remaining low-response connections are disabled. The event intensity at the current time step is estimated based on the average response of the query vector spiking on the sequence dimension. An event gating coefficient is generated based on the event intensity, and the attention scores at the same time step are scaled overall using the event gating coefficient. The scaled attention scores and value vectors (Value) are weighted and aggregated, and the output is projected to obtain the impulse response. The impulse response is then injected back into the original unimodal context features to output the enhanced unimodal impulse features.

[0182] The enhanced single-mode pulse features are subjected to single-mode pulse event detection, a multi-mode synchronization strength is constructed, and a shared synchronization gating is generated based on the multi-mode synchronization strength. The enhanced single-mode pulse features are then recalibrated for consistency through the shared synchronization gating, and the synchronized enhanced multi-mode features are output.

[0183] Using any one of the synchronously enhanced multimodal features as the query modality, cross-modal attention interactions are performed with the other two modalities under local time window constraints to obtain cross-modal aligned text features, audio features, and visual features.

[0184] The cross-modal aligned text features, audio features, and visual features are fused to output the sentiment category prediction result of the current dialogue utterance.

[0185] It should be noted that the method provided in this embodiment belongs to the same inventive concept as the asynchronous sparse pulse fusion system described in Embodiments 1 and 2 above. Their technical principles, module correspondences, and technical effects correspond to each other. The following describes each step in detail with reference to the specific execution process. The execution of the method described in this embodiment does not depend on specific hardware devices. It can be executed by general computing devices such as computers or servers with data processing capabilities, or by specialized embedded systems or neural network acceleration chips.

[0186] Through the above-described methodological steps, this application systematically solves three key technical challenges in existing multimodal dialogue emotion recognition: The asynchronous sparse impulse self-attention step effectively suppresses redundant dependencies in fully connected self-attention, enhancing the ability to capture key emotional change locations; the multimodal impulse synchronization fusion step explicitly models the synchronous response relationship of multimodal responses at key moments, strengthening cross-modal collaborative expression; and the asynchronous impulse-guided cross-modal attention step achieves fine-grained asynchronous cross-modal alignment guided by impulses under local time window constraints, taking into account the temporal offset between modalities. These three progressive and synergistic steps enable the model to more fully characterize the dynamic changes of multimodal emotions in dialogue, achieving superior recognition performance compared to existing methods on two publicly available benchmark datasets.

[0187] This embodiment also provides a computer-readable storage medium, which can be a read-only memory (ROM), random access memory (RAM), disk, optical disk, flash memory, or other media with data storage capabilities, and stores a computer program thereon. When this computer program is configured to be called by a processor, it implements all the steps of the asynchronous sparse impulse fusion method described in this invention, specifically including: context encoding of text, audio, and visual modal features; enhancing single-modal key emotional dynamics through sparse connection filtering and event-gated modulation; achieving multimodal synchronization consistency recalibration through adaptive impulse detection threshold detection and k-of-3 synchronization rule generation of shared synchronization gating; performing impulse-guided cross-modal attention interaction within a local time window and completing asynchronous alignment using learnable residual gating; and finally, fusing the three aligned features to output the emotional category prediction result.

[0188] By storing the above methods in the form of computer programs on a readable medium, the complete processing flow of the ASPF framework can be easily loaded and executed by general-purpose computing devices, servers, or embedded systems, which lowers the implementation threshold of the technical solution and facilitates its deployment and promotion in multimodal dialogue emotion recognition scenarios.

[0189] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An asynchronous sparse impulse fusion system for multimodal dialogue emotion recognition, characterized in that, include: The unimodal context encoding module is used to perform linear mapping and context modeling on the text modal features, audio modal features and visual modal features of the input multimodal dialogue data, respectively, to obtain text context features, audio context features and visual context features; The asynchronous sparse spiking self-attention module is used to sequentially perform linear mapping, layer normalization, and spiking neuron transformation on the text context features, audio context features, and visual context features to generate corresponding query vectors, key vectors, and value vectors; multi-head partitioning is performed on the query vectors, key vectors, and value vectors, and multi-head spiking self-attention scores are calculated. At each query position, retain the top k connections with the highest attention response and mask the remaining low-response connections, including obtaining the sparsity attention score through a Topk mask. , M is the Topk mask matrix; the event intensity at the current time step is estimated based on the average response of the query vector Query impulse in the sequence dimension, the event intensity reflects the impulse activity level in the overall sequence at the current moment, an event gating coefficient is generated based on the event intensity, and the sparse attention score at the same time step is scaled as a whole by the event gating coefficient; the scaled attention score and the value vector Value are weighted and aggregated, and the impulse response is obtained after output projection, the impulse response is injected back into the original single-modal context features, and the enhanced single-modal impulse features are output; The multimodal pulse synchronization fusion module is used to perform single-modal pulse event detection on the enhanced single-modal pulse features, construct multimodal synchronization strength, generate a shared synchronization gate based on the multimodal synchronization strength, perform consistency recalibration on the enhanced single-modal pulse features through the shared synchronization gate, and output synchronized enhanced multimodal features. The multimodal pulse synchronization fusion module is configured to perform the following operations: Based on the pulse changes at adjacent positions in the enhanced single-modal pulse features, the event intensities corresponding to the text modal pulse features, audio modal pulse features, and visual modal pulse features are calculated respectively. An adaptive impulse detection threshold is constructed based on the in-sample statistics of the event intensity of each modality. The event intensity of each modality is mapped to the corresponding adaptive impulse detection threshold to obtain the single-mode impulse response of each modality. Multimodal synchronization pulse intensity is constructed using the k-of-3 synchronization rule for the single-modal impulse responses of text, audio, and visual modalities; Based on the multimodal synchronization pulse intensity, a shared synchronization gate is generated by mapping with the Sigmoid function and combining it with the synchronization modulation coefficients; The shared synchronization gate is applied simultaneously to the enhanced text modal features, audio modal features, and visual modal features to perform consistent recalibration of the three modal features and output synchronized enhanced multimodal features; The process involves mapping the event intensity of each mode to the corresponding adaptive pulse detection threshold to obtain the single-mode impulse response of each mode. In the formula, Represents the single-mode impulse response of each mode. These correspond to the three modalities of text, audio, and video, respectively. The value represents the variation amplitude of features between adjacent frames, and T is the temperature parameter; in, The adaptive pulse detection threshold for the m-th mode is expressed by the following formula: In the formula, It is a mean function. Let m be the event intensity sequence. For threshold coefficient, For the Sigmoid function; Multimodal synchronization impulse strengths are constructed using the k-of-3 synchronization rule for the single-modal impulse responses of text, audio, and visual modalities: In the formula, This represents the intensity of the multimodal synchronization pulse at time step t. Represents the text modal impulse response. Indicates the audio modal impulse response. This represents the video modal impulse response, where k is the preset synchronization threshold in the k-of-3 synchronization rule. To synchronize temperatures; Based on the multimodal synchronization pulse intensity, a shared synchronization gate is generated by mapping using the Sigmoid function and combining it with the synchronization modulation coefficients. The calculation process is as follows: In the formula, For the shared synchronization gating at time step t, These are the synchronization modulation coefficients. For learnable upper bound parameters of synchronous gating, For the Sigmoid function; An asynchronous pulse-guided cross-modal attention module is used to take any one of the synchronously enhanced multimodal features as the query modality, and perform cross-modal attention interactions with the other two modalities under local time window constraints to obtain cross-modal aligned text features, audio features, and visual features. The classification module is used to fuse the cross-modal aligned text features, audio features, and visual features to output the sentiment category prediction result of the current dialogue utterance.

2. The asynchronous sparse impulse fusion system for multimodal dialogue emotion recognition according to claim 1, characterized in that, The event gating coefficient is calculated using the following formula: In the formula, For the first The event gating coefficient of the time step, The lower bound parameter for gating. For the Sigmoid function, To learn the linear weight parameters, For the first The intensity of events at each time step.

3. The asynchronous sparse impulse fusion system for multimodal dialogue emotion recognition according to claim 1, characterized in that, The asynchronous pulse-guided cross-modal attention module is configured to perform the following operations: Using any one of the synchronously enhanced multimodal features as the query modality, and taking the other two modalities as source modalities, cross-modal attention interaction is performed to obtain a dual-path cross-modal enhanced representation. In calculating the cross-modal attention score, the impulse event detector is reused to obtain the impulse intensity of each source modality, and the attention connection at the filling position is masked by the effective speech mask. Local time window constraints, time distance attenuation terms, and guidance bias determined by the impulse intensity of the source modality are introduced. The dual-path cross-modal enhancement representations are fused to obtain the cross-modal correction representation; By using learnable residual gating, the cross-modal corrected representation is fused with the original features of the query modality to output the cross-modal aligned target features.

4. The asynchronous sparse impulse fusion system for multimodal dialogue emotion recognition according to claim 3, characterized in that, The method utilizes learnable residual gating to fuse the cross-modal corrected representation with the original features of the query modality, outputting the cross-modal aligned target features. Specifically, it includes: In the formula, This represents the original characteristics of the query modality. This represents the cross-modal correction representation; λ is the residual gating coefficient, which satisfies: ; In the formula, For the Sigmoid function, These are learnable residual gating parameters.

5. An asynchronous sparse impulse fusion method for multimodal dialogue emotion recognition, characterized in that, Includes the following steps: We acquire text modal features, audio modal features, and visual modal features from multimodal dialogue data, and perform linear mapping and context modeling respectively to obtain text context features, audio context features, and visual context features. The text context features, audio context features, and visual context features are sequentially subjected to linear mapping, layer normalization, and spiking neuron transformation to generate corresponding query vectors Query, Key vectors, and Value vectors; the query vectors Query, Key vectors, and Value vectors are then subjected to multi-head partitioning, and multi-head spiking self-attention scores are calculated. At each query position, retain the top k connections with the highest attention response and mask the remaining low-response connections, including obtaining the sparsity attention score through a Topk mask. , M is the Topk mask matrix; the event intensity at the current time step is estimated based on the average response of the query vector Query impulse in the sequence dimension, the event intensity reflects the impulse activity level in the overall sequence at the current moment, an event gating coefficient is generated based on the event intensity, and the attention score at the same time step is scaled as a whole by the event gating coefficient; the scaled attention score and the value vector Value are weighted and aggregated, and the impulse response is obtained after output projection; the impulse response is injected back into the original single-modal context features, and the enhanced single-modal impulse features are output. The enhanced single-mode pulse features are subjected to single-mode pulse event detection, a multi-mode synchronization strength is constructed, and a shared synchronization gating is generated based on the multi-mode synchronization strength. The enhanced single-mode pulse features are then recalibrated for consistency through the shared synchronization gating, and the synchronized enhanced multi-mode features are output. The multimodal pulse synchronization fusion module is configured to perform the following operations: Based on the pulse changes at adjacent positions in the enhanced single-modal pulse features, the event intensities corresponding to the text modal pulse features, audio modal pulse features, and visual modal pulse features are calculated respectively. An adaptive impulse detection threshold is constructed based on the in-sample statistics of the event intensity of each modality. The event intensity of each modality is mapped to the corresponding adaptive impulse detection threshold to obtain the single-mode impulse response of each modality. Multimodal synchronization pulse intensity is constructed using the k-of-3 synchronization rule for the single-modal impulse responses of text, audio, and visual modalities; Based on the multimodal synchronization pulse intensity, a shared synchronization gate is generated by mapping with the Sigmoid function and combining it with the synchronization modulation coefficients; The shared synchronization gate is applied simultaneously to the enhanced text modal features, audio modal features, and visual modal features to perform consistent recalibration of the three modal features and output synchronized enhanced multimodal features; The process involves mapping the event intensity of each mode to the corresponding adaptive pulse detection threshold to obtain the single-mode impulse response of each mode. In the formula, Represents the single-mode impulse response of each mode. These correspond to the three modalities of text, audio, and video, respectively. The value represents the variation amplitude of features between adjacent frames, and T is the temperature parameter; in, The adaptive pulse detection threshold for the m-th mode is expressed by the following formula: In the formula, It is a mean function. Let m be the event intensity sequence. For threshold coefficient, For the Sigmoid function; Multimodal synchronization impulse strengths are constructed using the k-of-3 synchronization rule for the single-modal impulse responses of text, audio, and visual modalities: In the formula, This represents the intensity of the multimodal synchronization pulse at time step t. Represents the text modal impulse response. Indicates the audio modal impulse response. This represents the video modal impulse response, where k is the preset synchronization threshold in the k-of-3 synchronization rule. To synchronize temperatures; Based on the multimodal synchronization pulse intensity, a shared synchronization gate is generated by mapping using the Sigmoid function and combining it with the synchronization modulation coefficients. The calculation process is as follows: In the formula, For the shared synchronization gating at time step t, These are the synchronization modulation coefficients. For learnable upper bound parameters of synchronous gating, For the Sigmoid function; Using any one of the synchronously enhanced multimodal features as the query modality, cross-modal attention interactions are performed with the other two modalities under local time window constraints to obtain cross-modal aligned text features, audio features, and visual features. The cross-modal aligned text features, audio features, and visual features are fused to output the sentiment category prediction result of the current dialogue utterance.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program configured to, when invoked by a processor, implement the steps of the asynchronous sparse impulse fusion method for multimodal dialogue emotion recognition as described in claim 5.

Images