A multi-modal emotion dynamic recognition method and system based on emotion prototype attention and a large language model

Abstract

The application relates to the technical field of multi-modal emotion recognition, in particular to a multi-modal emotion dynamic recognition method and system based on emotion prototype attention and a large language model. The method comprises multi-modal emotion input coding and unified sequence representation based on acquired multi-modal emotion data, including multi-modal data time alignment and quality evaluation, multi-modal data coding, unified feature projection and sequence construction, and cross-modal consistency constraint; an emotion prototype-based causal tracking attention mechanism is constructed based on the unified sequence representation, including emotion prototype initialization, query key and value mapping and contribution heat map calculation, key clue screening and contribution score normalization, sparse supervision training and time continuity constraint, and key event segment screening; the application enables a user to clearly see the basis for model emotion judgment, and improves the explainability and reliability of emotion reasoning.

Description

Technical Field

[0001] This invention relates to the field of multimodal emotion recognition technology, and in particular to a multimodal emotion dynamic recognition method and system based on emotion prototype attention and large language model. Background Technology

[0002] With the development of applications such as artificial intelligence, human-computer interaction, virtual assistants, and mental health support, the ability of machines to accurately identify, interpret, and respond to human emotions has become a crucial capability of intelligent systems. Human emotions are typically manifested simultaneously in facial expressions, body movements, tone of voice, semantic content, and contextual events, and are characterized by implicitness, latency, and contextual dependence. A single modality or a single time slice cannot fully reflect the true emotional state. Therefore, accurately identifying the correlations between emotions in continuous time series and interpretably modeling emotional states and their changes is one of the core tasks of multimodal emotion recognition systems.

[0003] Existing multimodal emotion recognition methods mostly model the task as a closed-class classification problem, directly outputting fixed labels such as happy, angry, sad, and surprised. Although deep learning and multimodal large language models have improved emotion recognition and natural language description capabilities, most methods still focus on static result generation, lacking the ability to dynamically model emotions from their generation, development, to their turning points, and also struggling to pinpoint which specific video, audio, or text cues triggered the emotion judgment and when. Furthermore, different individuals express emotions significantly differently; the same silence, smile, or low tone may have different meanings for different users, making general models prone to misinterpretation.

[0004] While existing research has attempted to utilize multimodal large language models to achieve a unified understanding of video, speech, and text, and generate relatively fluent emotion interpretations, these interpretations often lack clear chains of evidence. Models may generate seemingly reasonable descriptions based on the overall context, but they cannot trace emotion judgments back to specific time segments, specific modal contributions, and specific triggering events. In scenarios such as mental health support and educational companionship, emotion judgment results must not only be accurate but also be able to be manually verified and tracked; otherwise, they cannot support highly reliable applications.

[0005] Furthermore, real-world applications often encounter issues such as video occlusion, audio noise, speech-to-text errors, missing text, and asynchronous multimodal sampling, making emotion understanding tasks more complex. Users may not directly express their emotions but rather indirectly convey their psychological state through changes in tone of voice, facial micro-expressions, duration of silence, word choice, or dialogue rhythm. Especially during prolonged interactions, emotional changes often accumulate gradually from multiple subtle cues, making it difficult to accurately determine their true meaning by observing a single frame or sentence of text. This places higher demands on the model's temporal memory capabilities, modal robustness, and cross-modal attribution capabilities.

[0006] Therefore, current technologies still lack a unified technical solution capable of explicitly locating emotional trigger evidence in multimodal input, maintaining continuous emotional memory, and personalizing it based on a small number of individual samples. If emotional cue tracking, dynamic emotional trajectory generation, and personalized emotional interpretation can be combined, and the final output can be traced back to specific time segments and specific modal evidence, the credibility, interpretability, and engineering application value of emotional intelligence systems in scenarios such as educational companionship, mental health assistance, and virtual human interaction can be significantly improved. Summary of the Invention

[0007] To address the aforementioned problems, this invention provides a multimodal emotion dynamic recognition method and system based on emotion prototype attention and a large language model.

[0008] Firstly, the present invention provides a multimodal emotion dynamic recognition method based on emotion prototype attention and a large language model, which adopts the following technical solution: A multimodal emotion dynamic recognition method based on emotion prototype attention and large language model includes: Acquire multimodal sentiment data; Multimodal sentiment input encoding and unified sequence representation based on acquired multimodal sentiment data, including multimodal data time alignment and quality assessment, multimodal data encoding, unified feature projection and sequence construction, and cross-modal consistency constraints; A causal tracking attention mechanism based on emotion prototypes is constructed based on unified sequence representation, including emotion prototype initialization, query key and value mapping and contribution heatmap calculation, key clue screening and contribution score normalization, sparse supervised training and temporal continuity constraints, and key event fragment screening. Based on the causal tracking attention mechanism, a contextual memory unit and an emotional trajectory model are constructed, including the construction of emotional history memory vectors, gated memory updates, emotional trajectory generation, and structured reporting. Personalized target adaptation based on contextual memory and emotional trajectory includes the introduction of individual expression samples, low-rank adaptation based on LoRA, and personalized training targets and interpretation generation. Robust reasoning and joint optimization output, including modality missing state modeling, modality missing robust training, end-to-end joint optimization, and result visualization output.

[0009] Secondly, a multimodal emotion dynamic recognition system based on emotion prototype attention and a large language model includes: The data acquisition module is configured to acquire multimodal sentiment data; The unified sequence module is configured to perform multimodal sentiment input encoding and unified sequence representation based on the acquired multimodal sentiment data, including multimodal data time alignment and quality assessment, multimodal data encoding, unified feature projection and sequence construction, and cross-modal consistency constraints. The attention module is configured to build a causal tracking attention mechanism based on emotion prototypes based on unified sequence representation, including emotion prototype initialization, query key and value mapping and contribution heatmap calculation, key clue filtering and contribution score normalization, sparse supervised training and temporal continuity constraints, and key event fragment filtering. The emotion trajectory module is configured to construct contextual memory units and emotion trajectory models based on a causal tracking attention mechanism, including emotional history memory vector construction, gated memory updating, emotion trajectory generation, and structured reporting. The target adaptation module is configured to perform personalized target adaptation based on contextual memory and emotional trajectory, including the introduction of individual expression samples, low-rank adaptation based on LoRA, and personalized training targets and interpretation generation. The output module is configured to provide robust inference and joint optimization output, including modality missing state modeling, modality missing robust training, end-to-end joint optimization, and result visualization output.

[0010] Thirdly, the present invention provides a computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device of the multimodal emotion dynamic recognition method based on emotion prototype attention and a large language model.

[0011] Fourthly, the present invention provides a terminal device, including a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, the instructions being adapted to be loaded and executed by the processor to provide a multimodal emotion dynamic recognition method based on emotion prototype attention and a large language model.

[0012] In summary, the present invention has the following beneficial technical effects: Compared to existing multimodal emotion recognition methods, which suffer from technical bottlenecks such as insufficient ability to dynamically model emotion over time, lack of evidence to support emotion interpretation, poor model recognition capabilities, and lack of personalized adaptation for specific users, this invention addresses the intelligent emotion recognition needs of "multimodal emotion dynamic reasoning" by constructing a multimodal emotion dynamic modeling and reliable reasoning system based on a large language model, which has the following beneficial effects.

[0013] First, by constructing a trimodal collaborative embedding mechanism of "text + speech + image," the system addresses the semantic alignment and structural differences among multimodal data, effectively integrating the comprehensive feature representations that dynamically change across the three modalities. Second, through the CTA causal tracking attention module, emotion recognition is transformed from traditional black-box classification into an evidence localization process based on emotional prototypes. The system can output key emotion trigger moments, corresponding emotion prototypes, multimodal contribution rates, and contribution scores, allowing users to clearly see the basis for the model's emotion judgments and improving the interpretability and credibility of emotion inference.

[0014] Finally, by connecting the current emotion judgment with historical emotion states through the contextual memory GRU module, an emotion arc with temporal continuity can be generated. Compared with methods that only make isolated judgments on short segments, this invention can better express the dynamic evolution of emotions from calm, hesitation, anxiety to calm, and reduce misjudgments caused by instantaneous noise. In addition, through the LoRA personalized context adaptation module, the low-rank adaptation parameters can be trained with only a small number of samples while freezing the main capabilities of the basic large language model, thus quickly learning the expression habits of the target object. At the same time, this invention still has strong robustness in modality-deficient scenarios. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of a multimodal emotion dynamic recognition method based on emotion prototype attention and large language model according to Embodiment 1 of the present invention; Figure 2 This is a schematic diagram comparing the performance of various methods in Embodiment 1 of the present invention on different evaluation indicators; Figure 3 This is a schematic diagram comparing the ablation experiment results of each core module in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram illustrating the impact of the key clue contribution threshold on clue localization performance in Embodiment 1 of the present invention; Figure 5 This is a schematic diagram comparing the comprehensive scores of various methods under different modal loss conditions in Embodiment 1 of the present invention; Figure 6 This is a schematic diagram comparing the scores of various methods under different numbers of personalized training samples in Embodiment 1 of the present invention. Detailed Implementation

[0016] The present invention will be further described in detail below with reference to the accompanying drawings.

[0017] Example 1 Reference Figure 1 This embodiment of a multimodal emotion dynamic recognition method based on emotion prototype attention and a large language model includes: (1) Multimodal emotion input encoding and unified sequence representation module 1) Multimodal data time alignment and quality assessment In multimodal sentiment reasoning tasks, video, audio, and text correspond to different data sampling methods and sentiment expression granularities. The video modality reflects facial expressions, eye movements, head movements, and posture information; the audio modality reflects tone, volume, speech rate, pauses, and acoustic tension; and the text modality reflects semantic content, emotional vocabulary, negation structures, and contextual events. Because the three modalities differ in sampling frequency, temporal boundaries, and information density, directly concatenating features can easily lead to temporal misalignment and inconsistent modal expressions. Therefore, this module first performs temporal alignment and segment quality assessment on the original multimodal data.

[0018] Define the original video frame sequence, audio segment sequence, and text transcription sequence as follows: , in, Indicates the number of uniform time windows. Indicates the first Video frames or video clips within a time window Indicates the corresponding audio segment. This indicates the corresponding text segment. The system uses a unified time window. Based on this, different modes are mapped to the same time step to form a combined input: , To handle occlusion, noise, and transcription errors in real-world scenarios, this module sets a modal availability mask for each time step: , in, , , These indicate whether the video, audio, and text modalities are valid. The system further calculates the quality score for each modality. This quality score is used to measure video clarity, audio signal-to-noise ratio, and text transcription confidence. This quality score will be used in subsequent weight adjustments during the fusion process to prevent low-quality modalities from having an excessive impact on sentiment assessment.

[0019] 2) Video visual modality coding This submodule handles video clips vt Extracting visual emotion features using a visual encoder: , in, This represents the visual feature extraction function, implemented by the visual Transformer encoder; Indicates the first The system generates visual feature vectors for each time window. To reduce interference from lighting, occlusion, and pose changes, the system can perform face region detection, key point normalization, and low-quality frame filtering before encoding, retaining only visual regions that contribute significantly to emotional expression.

[0020] 3) Audio Modality Coding This submodule deals with audio segment a t Perform acoustic feature extraction: , in, This represents the audio encoding function. It represents acoustic features that contain information about pitch, energy, rhythm, speech rate, and pauses.

[0021] 4) Text modality coding and semantic feature extraction This submodule will handle text fragments. t Input text segmenter and text encoder extract semantic features of the text: , in, This represents a text encoding function. Indicates the first Semantic sentiment features within a time window.

[0022] 5) Unified Feature Projection and Sequence Construction Because the video, audio, and text encoders output different dimensions, this module sets up separate linear projection layers to map the three modalities to a unified feature space. Let k∈{v,a,l} represent the modality type, then the projection process is as follows: , in, and They represent the first The projection matrix and bias terms of the modality. This is used to eliminate differences in modal scales. Then, a fusion input is generated by combining the modal availability mask and the quality score. , To enable the model to perceive temporal location and modal missing states, this module adds a temporal location encoding P. t and mask embedding E(M) tFinally, a unified multimodal sequence representation is obtained: , The unified input sequence consisting of all time steps is: , in, This represents a unified feature dimension. The sequence simultaneously contains visual representation, acoustic variation, textual semantics, temporal location, and modal availability information, providing a stable input for the subsequent causal tracking attention module.

[0023] 6) Cross-modal consistency constraints To enhance the consistency of different modalities in the emotion semantic space, this module introduces cross-modal consistency loss during the training phase. Let z be the normalized features of any two effective modalities at the same time step. i and z j Then the consistency loss is defined as: , in, Represents a set of modes. Cosine similarity is used to represent this constraint. This constraint encourages different modalities expressing the same emotional semantics within the same time window to approach each other in the feature space, while preserving modal conflict information for subsequent interpretation module judgment. Through the above processing, this module forms a unified multimodal input representation for emotion reasoning tasks, solving problems such as asynchronous time, inconsistent scale, unstable quality, and missing modalities in multi-source data.

[0024] (2) Causal Tracking Attention Module (CTA) Based on Emotional Prototypes 1) Initialization of emotional prototypes In multimodal sentiment reasoning, emotion triggering cues are often distributed across different time segments and modalities. To enable the model to proactively retrieve key evidence related to specific emotions, this module first constructs a set of learnable sentiment prototype vectors to represent the triggering patterns of different basic emotions in the multimodal feature space.

[0025] The emotional prototype matrix is ​​defined as follows: , in, Indicates the number of emotional archetypes. This represents the prototype vector dimension. Each emotional prototype corresponds to a type of emotional triggering pattern, such as anger, sadness, surprise, anxiety, etc. The emotional prototype matrix is ​​not a fixed rule, but is automatically learned during training based on video, audio, and textual evidence, enabling large models to search for key triggering cues around different emotional categories. To avoid emotional prototypes learning only single-modal features, this module requires prototypes to receive triggering information from visual, audio, and text simultaneously during training.

[0026] , in, Separate regularization terms for prototypes. This is the upper bound of the similarity between prototypes. represents the norm constraint coefficient. This formula is used to prevent different emotional prototypes from collapsing into the same direction. For emotional fragments that exhibit inconsistent performance across different modalities, the system does not rely solely on a single modality, thus providing a stable prior for the emotional category to be explained later.

[0027] 2) Query key and value mapping and contribution heatmap calculation After obtaining the emotional prototype, this module uses the emotional prototype as the query vector and the unified multimodal sequence as the key and value to construct a causal tracking attention mechanism. Let the unified multimodal sequence be X, then the query Q, key K, and value V are represented as follows: , in, For modal availability and quality status, This is used to inject modality missing information into the key vector, enabling attention computation to distinguish between normal and low-quality evidence. Subsequently, the system calculates a heatmap of sentiment contribution. , in, Indicates the first The time slice for the first The contribution of emotion archetypes, For emotional prior bias. As a time-location correction term, this formula generates a heatmap of emotional contribution that can trace back the triggering conditions.

[0028] In the query, key, and value mapping process, the sentiment prototype undertakes the active retrieval function, while the multimodal sequence undertakes the evidence-carrying function. The query vector represents "what kind of emotional evidence needs to be found," the key vector represents "what emotional cues are contained in each time segment," and the value vector stores the modal features of the corresponding time segment. To further illustrate the relative role of each modality in this emotion judgment, the modal contribution weights are calculated: , in, This indicates the relative contribution of visual, audio, or textual modalities to the emotion judgment. This represents the quality of the corresponding modality; this formula is used to form a three-dimensional evidence matrix of "time-emotion-modality".

[0029] The contribution heatmap is not only used to calculate attention weights but also preserved as an interpretable intermediate result. For each emotion prototype, the contribution heatmap provides a distribution of trigger intensity along the time axis, showing the process of emotion generation, intensification, and attenuation in a video or dialogue. Unlike traditional classification models that only output final labels, this heatmap can illustrate the source of evidence for the model's judgment, thereby improving its verifiability in high-confidence applications.

[0030] 3) Screening of key clues and normalization of contribution scores The goal of key cue screening is to retain a small number of truly emotionally triggering segments within long-term time series. Directly inputting all time segments into a large language model would not only increase the inference burden but also potentially expose the model to irrelevant segments, leading to unfounded explanations. To avoid overly dispersed attention distribution, this module first combines the emotion contribution heatmap with a weighted modality contribution to obtain a comprehensive trigger score: , in, Indicates the first Emotions in the first The overall trigger score over a time segment This represents the contribution coefficient of the emotional archetype. Represents the modal contribution weight. This represents the corresponding modal weighting coefficient. When the quality of a certain mode is poor, its contribution is... and The contribution scores are reduced collectively. Subsequently, the system normalizes and corrects the contribution scores using an adaptive threshold. , in, The selection threshold is adaptively changed according to the emotional archetype. and These represent the mean and standard deviation of the contribution scores for each modality. This formula is used to adaptively filter key cues based on the contribution distribution of different emotional modalities. When the video is occluded or the audio contains significant noise, the system reduces the corresponding contribution based on the modality quality score to avoid the erroneous amplification of emotional cues due to a single modality anomaly. The final structured evidence set is then formed. The system retains emotion category, temporal location, dominant modality, normalized contribution score, and original modality features for subsequent memory updates.

[0031] , 4) Sparse supervised training and continuous temporal constraints The goal of sparse supervised training is to compress irrelevant time segments while preserving genuine trigger evidence. To this end, this submodule first utilizes focus loss to enhance the learning of a few key cues, enabling the model to prioritize genuine trigger segments even in long sequences with highly imbalanced positive and negative samples. The loss function is expressed as: , in, As a key clue monitoring label, This is the model's predicted probability that this time segment belongs to a key clue; this formula alleviates the class imbalance caused by the scarcity of key triggering segments and the abundance of ordinary segments by increasing the weight of positive samples.

[0032] After obtaining the probability of key clues, this submodule further introduces a sparse term to limit the number of activations for each type of emotion prototype, and a time-continuous term to constrain the variation range of adjacent contribution scores. These are expressed as follows: , in, For the first The maximum number of key fragments that can be retained for a given emotion. and is the sparsity penalty coefficient; this formula is used to suppress background fragment activation, so that the model retains only a small amount of high-value evidence.

[0033] The aforementioned sparsity constraint addresses the problem of scarce emotional trigger fragments. Without this constraint, the contribution heatmap is prone to generating low-intensity activations on a large number of background fragments, leading to subsequent reports citing too much irrelevant evidence. Introducing the maximum fragment count constraint ensures that the system retains only fragments with high contribution and strong matching to the emotional prototype. The temporal continuity constraint prevents abrupt jumps in the contribution heatmap when there are no significant changes in evidence; its calculation method is as follows: , in, The system dynamically adjusts based on the differences in features between adjacent segments: smoothing is enhanced when the differences between adjacent inputs are small, while jumps in contribution scores are allowed when there are significant changes in adjacent inputs. Through this constraint, the system maintains sensitivity to genuine emotional shifts while preserving the stability of the evidence trajectory. The comprehensive loss function is defined as: , in, Used to strengthen the monitoring of key clues Used to suppress irrelevant fragments These three elements are used to maintain the continuity of temporal evidence; together, they enable the CTA module to output focused, stable, and interpretable emotional trigger evidence.

[0034] 5) Screening of key event segments The goal of key event segment screening is to organize key emotional event contribution points exceeding a threshold into readable and verifiable evidence. First, key events with contribution scores exceeding the threshold (set to 0.1) are extracted. For the same emotional archetype, if multiple key emotional event time points are consecutive in time or the interval is less than a preset threshold δ, they are merged into event segments, and the start and end positions of the segments are determined based on boundary thresholds. , in, For critical event thresholds, and These represent the start and end times of the event segment, respectively. This formula is used to determine the boundaries of event segments from the high-contribution sequence, merging consecutive trigger points into a single emotional event instead of outputting discrete frames. After determining the boundaries, the system then calculates segment-level contribution scores to measure the importance of the event in the overall emotional reasoning: , in, For the first The, the The contribution intensity of emotional event fragments To integrate modal quality, This formula is used to characterize modal contribution differences. It assigns higher explanatory weight to segments with a clear dominant modality and high quality. By merging key event segments, the system no longer outputs isolated frame numbers, but instead outputs event-level evidence with start and end boundaries, emotional prototypes, dominant modality, and contribution strength, supporting the report generation module in organizing the description according to the format of "time interval - triggering evidence - emotional change".

[0035] (3) Contextual memory and emotional trajectory generation module 1) Construction of Emotional Historical Memory Vectors The goal of constructing the emotional history memory vector is to organize the current multimodal state, key CTA cues, and the previous emotional summary into a continuously updated contextual state input to the large model. When emotion needs to rely on preceding events for judgment, the system first performs time-decayed aggregation on high-contribution evidence: , in, A summary of key clues to be written into memory. For the evidence mapping function, the exponential decay term is used to weaken the impact of distant events, so that memory writing is mainly driven by high-contribution, recent-moment evidence.

[0036] After obtaining a summary of key clues, the system breaks down immediate fluctuations and stable backgrounds into short-term and long-term memories, forming a combined emotional memory vector: , in, It consists of short-term and long-term components. The short-term component preserves the emotional fluctuations of the most recent period, while the long-term component preserves the stable emotional background formed by high-confidence events. This is the normalized projection function.

[0037] Subsequently, the system uses the current multimodal state, key clue summary, and previous emotional state as memory input to ensure that the current judgment can inherit the historical context: , in, Indicates the current multimodal state. This represents the current key clue vector. This represents a summary of the emotional state at the previous moment; through this structure, subsequent modules can avoid isolated judgments in continuous dialogues or long videos.

[0038] Through the above processing, emotional memory is no longer a simple stacking of all historical fragments, but a compressed state driven by key cues. This state can retain both recent emotional fluctuations and long-term stable background, providing a unified contextual input for subsequent gating updates and model report generation.

[0039] 2) Gating memory update The goal of gated memory updates is to choose between new evidence and historical states. When the current fragment contains a clear triggering clue, the system increases the proportion of new state writes; when the current fragment is only background information, the system retains more historical memory. The memory write ratio is increased when a strong triggering clue is detected by the CTA, while weak evidence fragments do not easily overwrite historical states.

[0040] For each memory state m t At every moment With input feature z t The GRU update process is as follows: Update Gate: , in, Use the Sigmoid activation function to ensure that the output value is between [0,1]. The input weight matrix (applied to the current input) ); The hidden state weight matrix (applied to the hidden state at the previous time step) ); To update the gate bias term; To update the gate vector, we determine how much past information to retain in the hidden state.

[0041] Reset Door: , in ， ， Similar to the update gate, but with different weights and biases than the reset gate; To reset the gate vector, control how the current input is fused with past memories.

[0042] Candidate state: , in, This represents the Hadamard product (element-by-element multiplication). ， This represents the corresponding input and hidden layer weights; Bias terms representing candidate states; The candidate hidden state is the "proposed update value" generated by combining the current input with past information.

[0043] Final status update: , in, Indicates the current time step The hidden state; the formula incorporates past hidden states (through ) and the current candidate state (via (Weighted); After this operation, the gate is updated. In effect, it controls the degree of "forgetting" and "updating" of memory. Through this selective update mechanism, the system can distinguish between short-term noise and genuine emotional changes: it quickly updates the emotional state when strong evidence arrives, and maintains historical state stability when evidence is insufficient or modality quality is low. This output directly serves as the memory input for the next moment's reasoning, avoiding the disruption of emotional trajectory caused by single-frame noise.

[0044] 3) Emotional Trajectory Generation and Structured Reporting The goal of emotion trajectory generation and structured reporting is to transform continuous memory states into readable and verifiable emotion evolution results using a large-scale model. When generating emotion description reports, the large language model is limited by evidence-constrained templates, and can only cite key cues, memory states, and personalized adaptation information. , in, For evidence-constrained large language model generators, This is a constraint on the report template; the formula is used to reduce unfounded interpretations and enable the output to be traced back to... Key evidence. Ultimately, the system constructs an emotional arc in chronological order, recording the emotional state, reasons for change, and confidence level at each stage: , in, Indicates emotional state. Indicate the reason for the change in emotion. This indicates the confidence level. The output of the large model is not just the current emotion label, but a trajectory record that includes stage emotion, reasons for change, confidence level, and evidence citations. It can describe the continuous process of change from calm, doubt, anxiety to relief, and makes it easy for human reviewers to check the model's inference chain in chronological order.

[0045] (4) Personalized context adaptation module 1) Introduction of individual expression samples The goal of introducing individual expression samples is to establish a learnable expression baseline for the target audience to dynamically adjust the base model. When different users express the same emotion with different speaking speeds, volumes, facial expressions, or word choices, the system first constructs a personalized sample set: , in, For users A personalized sample set Record the baseline of expression, including audio, visuals, vocabulary, and speech rate; this formula is used to explicitly write individual expression habits into the adaptation process.

[0046] After the sample set is established, the system further calculates the historical expression center of users under different emotion categories to measure whether the current expression deviates from the individual baseline: , in, Indicates user In emotions The historical expression center below, This structure measures the deviation of the current expression from the historical baseline, reducing misinterpretations of individual expression habits by the general model. The personalization module does not require samples to cover all emotion categories; instead, it prioritizes recording common expression habits, triggering methods, and human feedback results, providing individual priors for subsequent low-rank adaptation. When judging the current emotion, the system compares not only general emotion features but also the deviation of the current expression from the user's historical patterns, thereby reducing misjudgments of specific expression habits.

[0047] 2) Low-rank adaptation based on LoRA The goal of LoRA-based low-rank adaptation is to learn individual user differences without updating the main parameters of the base large language model. For target user i, the system freezes the base weights and trains only the low-rank increment matrix: , in, For target users The low-rank adapted rank, Scaling factor; freeze And only training , It can achieve fine-tuning based on individual user habits with a small sample size.

[0048] When the number of personalized samples changes, the system can dynamically adjust the low-rank value and calculate the adapted output accordingly: , This formula dynamically determines the low-rank value based on the number of personalized samples. This reduces the risk of overfitting when there are few samples and improves individual expressive power when there are many samples. The system can load independent adapters for each user, enabling rapid switching in multi-user scenarios. It saves lightweight incremental parameters for different users, allowing for fast personalized inference in multi-user scenarios.

[0049] 4) Personalized training objectives and interpretation generation The goal of personalized training and interpretation generation is to maintain general emotion understanding capabilities while ensuring that the output interpretation conforms to the expression habits of the target audience. The system first constrains the sample expression embedding to be close to the corresponding user's historical expression center: , in, This embeds the sample expression close to the center of the user's historical expression of the corresponding emotion. Limit adapter parameter offsets to prevent overfitting with few samples.

[0050] Subsequently, the system incorporates the generated loss, expression regularity alignment, low-rank regularization, and general model deviation constraints into the personalized training objective: , in, The personalized model is constrained to avoid deviating excessively from the general model; this formula strikes a balance between personalization enhancement and general sentiment understanding. After fine-tuning without introducing additional calibration terms, the model organizes text in the order of "current evidence - individual baseline - personalized conclusion" when generating interpretations, thereby reducing misjudgments due to individual differences. Personalized interpretations retain the constraints of current evidence while incorporating the user's historical expression baseline. The final result can indicate whether a certain expression is a general sentiment signal or an anomalous change relative to the user's own baseline, thus improving the credibility of the interpretation.

[0051] (5) Modality-deficient robust reasoning and joint output module 1) Modeling of missing modal states The goal of modal missing state modeling is to distinguish between valid and invalid evidence when video, audio, or text is incomplete. The system first constructs a modal state vector that includes both usability and quality scores: , in, Simultaneously record the existence and quality of the mode. This represents the currently available set of modalities. During the fusion phase, the system only performs normalized weighting on available and quality-compliant modalities to form a robust representation with no awareness of missing features. , in, For missing modalities, the fusion weights are automatically removed from or reduced in the normalized denominator when a modality is missing or its quality degrades. For basic fusion scenarios, the system can also directly construct fusion vectors based on the available modality set. , Through the above processing, the model can fully utilize the remaining modalities while also identifying low-quality or missing sources in the report, preventing missing information from being mistakenly used as normal evidence in reasoning. The final fusion result automatically reduces the weight of anomalous modalities and provides missing or low-quality indications in the results.

[0052] 3) Robust training for modal loss The goal of modality-depleted robust training is to actively simulate incomplete inputs such as occlusion, noise, transcription errors, or modality closure during the training phase, enabling the model to maintain stable judgments using the remaining modalities during deployment. The system first constructs degenerate samples and constrains the representations under complete and missing inputs to be consistent: , in, Indicates the first The system employs degenerate operators for modal masking, occlusion, noise perturbation, or transcription perturbation; this formula requires that the sentiment representation under missing input closely approximates the complete input. In addition to ensuring consistent final sentiment representation, the system also constrains the contribution heatmaps under missing modalities to still locate the same key events. , in, The contribution heatmap under the constraint-missing mode remains consistent with the contribution heatmap under the complete mode, preventing the model from turning to irrelevant background evidence under missing conditions. For the basic training process of single-modal masking, robust constraints can... This training method reduces the model's over-reliance on a single modality, allowing the system to still output stable results by combining the remaining modalities and historical memory even when any of the video, audio, or text modalities are missing.

[0053] 3) End-to-end joint optimization The goal of end-to-end joint optimization is to enable encoding, attribution, memory, personalization, and robust output to converge collaboratively under the same objective. The system construction includes the total loss of each core module: , in, The module (1) is used to constrain cross-modal consistency. Module (2) is used to constrain the location of key clues. Module (4) is used to constrain personalized interpretations. The module (5) is used to constrain the stability under the condition of missing modes. This formula incorporates key clue localization, cross-modal consistency, personalized adaptation, missing robustness and confidence calibration into a unified optimization. Each module is no longer trained separately with isolated targets, but is jointly optimized under a unified loss.

[0054] 4) Visual output of results The goal of visualizing the output is to transform the model's inference results into readable, verifiable, and continuously updatable output records. The system first saves a complete evidence cache of the inference process: , in, A record of evidence for a reasoning exercise, containing Input identifier, Key clues Contribute heatmaps, Memory state, Personalized adapter labeling Emotional arc and The report text creates a traceable record for each reasoning step, which is used to instantly display the emotional trajectory and key evidence, thus forming a continuous optimization loop.

[0055] Experimental verification To systematically verify the performance advantages of the method of this invention in multimodal emotional dynamic attribution and personalized reasoning tasks, a multimodal emotional reasoning experimental dataset was constructed. The dataset consists of three modalities: ① Video modality: including facial expressions, head posture, gaze changes, and action clips, used to reflect overt emotional states; ② Audio modality: including speech rate, pitch, energy, pauses, and acoustic prosody, used to identify emotional intensity and tone changes; ③ Text modality: including automatically transcribed sentences, contextual dialogues, and semantic events, used to model the causes of emotional triggers. Four core task information categories are labeled in the dataset: ① Emotion category; ② Key trigger time segments; ③ Continuous emotional trajectory; ④ Personalized emotional interpretation. A total of 80 multimodal observation samples were constructed, of which 60 were used for training and 20 for testing, ensuring coverage of different characters, scenarios, and quality conditions across different modalities.

[0056] To test the system's ability to locate clues, model dynamic trajectories, and provide personalized interpretations in complex real-world scenarios, the following six mainstream comparison methods were set up: ①TFN: a typical tensor fusion multimodal emotion recognition method; ②MISA: a multimodal model based on modality invariance and modality-specific representation decomposition; ③MultimodalTransformer: a sequence model using cross-modal attention for fusion; ④Video-LLaVA: a general video multimodal large model; ⑤AffectGPT: a large language model method for emotion understanding; ⑥The method of this invention: introducing CTA causal tracking attention, contextual memory, LoRA personalized adaptation, and modality-deficient robust inference mechanism.

[0057] All methods were evaluated under the same training, validation, and test sets. Test perturbation conditions included: no perturbation (0.0): complete video, audio, and text; moderate perturbation (0.5): random addition of audio noise, video occlusion, and minor text transcription errors; and high perturbation (1.0): missing mixed modalities, severe occlusion, speech noise, and context truncation. Evaluation metrics included: ① Traceability: the ability to track emotional cues, measuring the model's accuracy in locating key emotional trigger moments; ② ArcGeneration: the ability to generate emotional trajectories, measuring the consistency between the generated emotional arc and the description of actual emotional changes; ③ Personalization: the ability to personalize emotional descriptions, measuring the model's adaptation to the target audience's expression habits; and ④ Comprehensivescore: a comprehensive score used to measure overall emotional understanding ability.

[0058] Table 1 Comparison of data for different methods under various indicators Traditional multimodal emotion recognition methods such as TFN and MISA typically only output overall emotion labels, failing to provide key trigger fragments and individualized explanations, thus exhibiting significant shortcomings in traceability and personalization tasks. While mPLUG-Owl2 can fuse multimodal features, it lacks an explicit evidence localization mechanism; EmoVIT possesses strong emotion recognition capabilities but does not specifically model the causal chain of emotion triggers; AffectGPT outperforms general large models in emotion interpretation, but its clue tracking score is still significantly lower than the method of this invention.

[0059] The experimental results are shown in Table 1. Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 As shown, the method of this invention achieves a score of 0.632 in the Traceability task, significantly higher than AffectGPT's 0.325, indicating that the CTA causal tracking attention module can effectively locate emotional trigger evidence; it achieves a score of 0.781 in the ArcGeneration task, indicating that the contextual memory module can maintain continuous emotional states; and it achieves a score of 0.823 in the Personalization task, indicating that the LoRA personalization adaptation mechanism can learn the target object's expression habits; the overall score reaches 0.745, the highest among all methods.

[0060] From Table 1 and Figure 2 It can be seen that traditional fusion methods and general multimodal large models have certain limitations in multimodal sentiment reasoning tasks. Although TFN, MISA, and mPLUG-Owl2 can fuse video, audio, and text features, their outputs mainly focus on overall sentiment classification and lack explicit localization of key trigger segments. Therefore, their traceability scores are significantly lower than the method of this invention. EmoVIT has strong sentiment recognition capabilities, but it does not specifically model the causal chain of emotions and individual expression differences, resulting in insufficient personalized interpretation capabilities. AffectGPT has certain advantages in sentiment description, but its cue tracking score is still significantly lower than the method of this invention. Figure 2 The radar chart further shows that the method of the present invention forms an expanded profile in all four dimensions of emotion cue tracking, emotion trajectory generation, personalized description and comprehensive score, indicating that its overall performance is more balanced.

[0061] Combination Figure 3 and Figure 4It is evident that each core module of this invention makes a clear contribution to the final performance. Ablation experiments show that the Traceability score drops most significantly after removing the CTA module, indicating that the causal tracking attention mechanism is key to achieving dynamic emotional attribution; the ArcGeneration score drops after removing the context memory module, indicating that historical emotional states play an important role in generating continuous emotional arcs; the Personalization score drops significantly after removing the Personalization adapter, indicating that the LoRA low-rank adaptation parameters can effectively learn the target object's expression habits. Contribution threshold experiments further show that when τ=0.10, precision, recall, and F1 score achieve a good balance; a threshold that is too low will introduce too many irrelevant fragments, while a threshold that is too high will miss real triggering cues. Therefore, this embodiment preferably uses τ=0.10 as the key cue screening threshold.

[0062] also, Figure 5 and Figure 6 The stability of this invention under complex application conditions was verified. Modal missingness experiments showed that, under conditions where text, audio, and video were missing respectively, the overall score decrease of the method of this invention was less than that of AffectGPT, indicating that the system does not overly rely on a single modality. The CTA module can relocate valid emotional cues from the remaining modalities, and the context memory module can use historical states to compensate for insufficient current evidence. Few-sample personalization experiments showed that as the number of personalized training samples increased from 5 to 50, the Personalization score of the method of this invention continuously improved; even using only 20 personalized samples, it still maintained a high fit, proving that the emotional prototype prior and the LoRA low-rank fitting mechanism can quickly learn the target object's expression patterns under small-sample conditions.

[0063] In summary, the method of this invention not only achieves optimal results in emotional cue localization, dynamic emotional trajectory generation, and personalized emotional interpretation, but also maintains strong stability under conditions of modality missing and limited sample fitting. Experimental results fully demonstrate that this invention is suitable for multimodal emotional intelligence applications requiring high reliability, high interpretability, and high personalization, and possesses significant application value.

[0064] Example 2 This embodiment provides a multimodal emotion dynamic recognition system based on emotion prototype attention and large language model.

[0065] A computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device, the aforementioned multimodal emotion dynamic recognition method based on emotion prototype attention and a large language model.

[0066] A terminal device includes a processor and a computer-readable storage medium, the processor being configured to implement various instructions; the computer-readable storage medium being configured to store multiple instructions adapted for loading and execution by the processor of the multimodal emotion dynamic recognition method based on emotion prototype attention and a large language model.

[0067] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A multi-modal emotion dynamic recognition method based on emotion prototype attention and a large language model, characterized in that, include: Acquire multimodal sentiment data; Multimodal sentiment input encoding and unified sequence representation based on acquired multimodal sentiment data, including multimodal data time alignment and quality assessment, multimodal data encoding, unified feature projection and sequence construction, and cross-modal consistency constraints; A causal tracking attention mechanism based on emotion prototypes is constructed based on unified sequence representation, including emotion prototype initialization, query key and value mapping and contribution heatmap calculation, key clue screening and contribution score normalization, sparse supervised training and temporal continuity constraints, and key event fragment screening. Based on the causal tracking attention mechanism, a contextual memory unit and an emotional trajectory model are constructed, including the construction of emotional history memory vectors, gated memory updates, emotional trajectory generation, and structured reporting. Personalized target adaptation based on contextual memory and emotional trajectory includes the introduction of individual expression samples, low-rank adaptation based on LoRA, and personalized training targets and interpretation generation. Robust reasoning and joint optimization output, including modality missing state modeling, modality missing robust training, end-to-end joint optimization, and result visualization output.

2. The multimodal emotion dynamic recognition method based on emotion prototype attention and large language model according to claim 1, characterized in that, The multimodal data time alignment and quality assessment includes first performing timeline unification and segment quality assessment on the original multimodal data, defining the original video frame sequence, audio segment sequence, and text transcription sequence as follows: in, Indicates the number of uniform time windows. Indicates the first Video frames or video clips within a time window Indicates the corresponding audio segment. Representing corresponding text segments to unify time windows Based on this, different modalities are mapped to the same time step to form a combined input; to handle occlusion, noise, and transcription errors in real-world scenarios, a modal availability mask is set for each time step: in, , , This indicates whether the video, audio, and text modalities are valid, and further calculates the quality score for each modality. The method is used to measure video clarity, audio signal-to-noise ratio, and text transcription confidence. The multimodal data encoding includes extracting visual emotion features from block pairs of video segments vt using a visual encoder; extracting acoustic features from audio segments at; inputting text segments lt into a text segmenter and a text encoder to extract text semantic features; then setting linear projection layers to map the three modalities to a unified feature space; subsequently combining modal availability masks and quality scores to generate a fused input; and adding temporal position encoding Pt and mask embedding E(Mt) to finally obtain a unified multimodal sequence representation.

3. The multimodal emotion dynamic recognition method based on emotion prototype attention and large language model according to claim 2, characterized in that, The initialization of the emotional prototype includes first constructing a learnable emotional prototype vector to represent the triggering patterns of different basic emotions in the multimodal feature space. The emotional prototype matrix is ​​defined as follows: in, Indicates the number of emotional archetypes. The prototype vector dimension is represented; to avoid the emotional prototype learning only single-modal features, during training, the prototype is required to simultaneously receive trigger information from visual, audio, and text; after obtaining the emotional prototype, it is used as the query vector, and the unified multimodal sequence is used as the key and value to construct a causal tracking attention mechanism; then, the emotion contribution heatmap is calculated: in, Indicates the first The time slice for the first The contribution of emotion archetypes, For emotional prior bias. For time position correction; in the query, key and value mapping process, the emotion prototype undertakes the active retrieval function, and the multimodal sequence undertakes the evidence carrying function; to further illustrate the relative role of each modality in the emotion judgment, the modality contribution weight is calculated; for each type of emotion prototype, the contribution heatmap gives the trigger intensity distribution along the time axis, which is used to show the process of emotion generation, enhancement and weakening in a certain video or dialogue.

4. The multimodal emotion dynamic recognition method based on emotion prototype attention and large language model according to claim 3, characterized in that, The key clue screening and contribution score normalization process includes, to avoid distracting attention distribution, first combining the emotion contribution heatmap and modal contribution with weights to obtain a comprehensive trigger score. in, Indicates the first Emotions in the first The overall trigger score over a time segment This represents the contribution coefficient of the emotional archetype. Indicates the modal contribution weight. This represents the corresponding modality weight coefficient; subsequently, an adaptive threshold is used to normalize and correct the contribution score. When the video is occluded or the audio contains noise, the corresponding contribution is reduced based on the modality quality score to avoid the erroneous amplification of emotional cues due to single modality anomalies, ultimately forming a structured evidence set. The model retains emotion category, temporal location, dominant modality, normalized contribution score, and original modality features for subsequent memory updates. Sparse supervised training with continuous temporal constraints, including using focus loss to enhance the learning of a few key cues, allows the model to prioritize true trigger segments in long sequences with extremely imbalanced positive and negative samples. Its loss function is expressed as: in, As a key clue monitoring label, The model predicts the probability that a time segment belongs to a key clue. After obtaining the key clue probability, a sparse term is introduced to limit the number of activations of each type of emotional prototype, and a temporal continuity term is introduced to constrain the variation range of adjacent contribution scores. The temporal continuity constraint is used to avoid abrupt jumps in the contribution heatmap when there is no obvious change in evidence, and to retain sensitivity to real emotional transitions while maintaining the stability of the evidence trajectory. The goal of key event segment screening is to organize the contribution points of key emotional events that exceed the threshold into readable and verifiable evidence. By filtering out key events with contribution scores exceeding the threshold, for the same emotional prototype, if multiple key emotional event time points are temporally consecutive or the interval is less than the preset threshold δ, they are merged into event segments. The start and end positions of the segments are first determined according to the boundary threshold. After determining the boundary, the segment-level contribution score is calculated to measure the importance of the event in the entire emotional reasoning.

5. The multimodal emotion dynamic recognition method based on emotion prototype attention and large language model according to claim 4, characterized in that, The construction of the emotional history memory vector includes inputting the current multimodal state, CTA key clues, and the previous emotional summary into a continuously updated contextual state into the large model. When the emotion needs to rely on preceding events for judgment, high-contribution evidence is first converged through time decay, represented as follows: in, A summary of key clues to be written into memory. For evidence mapping function, For emotional archetype index, The total number of emotional archetypes. For the current time step, No later than Historical timeline For the first Emotional archetypes in the first The contribution score of each time step after modal quality correction and normalization. , and The first Video, audio, and text feature vectors at time steps For the first Emotional prototype vectors The semicolon in the text indicates vector concatenation. The time decay coefficient is greater than 0. The time decay weight is used to assign a smaller memory write weight to historical evidence that is further removed from the current moment; after obtaining the key clue summary, the immediate fluctuations and stable background are separated into short-term memory and long-term memory, forming a combined emotional memory vector: in, It consists of short-term and long-term components. The short-term component preserves the emotional fluctuations of the most recent period, while the long-term component preserves the stable emotional background formed by high-confidence events. For the normalized projection function, As of the date The short-term memory component of a time step, As of the date The long-term memory component of a time step. Index for key historical events For the first A fragment of a key historical emotional event, For the event Fragment-level contribution score, To filter for high-confidence events, This means only those with a contribution score higher than [a certain value] will be retained. Historical events, In order to make the event The event feature mapping function encoded into the memory space. This means that event features are weighted by their contribution scores; then, the current multimodal state, key clue summary, and previous emotional state are used as memory inputs to ensure that the current judgment can inherit the historical context. in, Indicates the current multimodal state. This represents the current key clue vector. This represents a summary of the emotional state at the previous moment.

6. The multimodal emotion dynamic recognition method based on emotion prototype attention and large language model according to claim 5, characterized in that, The gated memory update includes increasing the memory write ratio when the CTA detects a strong triggering cue, and updating the memory state mt at each time step. Given input features zt, the update process of GRU is as follows: Update Gate: in, Use the Sigmoid activation function; The input weight matrix; The hidden state weight matrix; To update the gate bias term; To update the gate vector; Reset Door: in , , Similar to the update gate; To reset the gate vector, Candidate state: in, Represents the Hadamard product; , This represents the corresponding input and hidden layer weights; Bias terms representing candidate states; This is the candidate hidden state; Final status update: in, Indicates the current time step The hidden state; Emotion trajectory generation and structured reporting, including the generation of emotion description reports, involves the large language model being constrained by evidence-based templates, referencing key cues, memory states, and personalized adaptation information, represented as follows: ， in, For evidence-constrained large language model generators, To constrain the report template, an emotional arc was ultimately created in chronological order, recording the emotional state, reasons for change, and confidence level for each stage: ， in, Indicates emotional state. Indicate the reason for the change in emotion. Indicates the confidence level.

7. The multimodal emotion dynamic recognition method based on emotion prototype attention and large language model according to claim 6, characterized in that, The introduction of individual expression samples includes dynamically adjusting the base model to establish a learnable expression baseline. When different users express the same emotion with different speaking speeds, volumes, facial expressions, or word choices, a personalized sample set is first constructed. After the sample set is established, the historical expression center of the user under different emotion categories is further calculated to measure whether the current expression deviates from the individual baseline. in, Indicates user In emotions The historical expression center below, Used to measure the degree of deviation of current expression from historical baseline. For users In emotions The following historical sample set, For historical sample index, For the first The sentiment labels of each sample This indicates that only the label "emotion" is counted. The sample, For the first Multimodal expression features of historical samples To express the feature mapping function, This is the current sample to be identified. For the multimodal expression features of the current sample, Here, is the L2 norm, used to calculate the Euclidean distance; to learn individual user differences without updating the main parameters of the base large language model, for target user i, the base weights are frozen and only the low-rank increment matrix is ​​trained: in, For target users The low-rank adapted rank, Scaling factor; freeze And only training , With a small number of samples, fine-tuning is achieved based on individual user habits; when the number of personalized samples changes, the low-rank value is dynamically adjusted and the adapted output is calculated accordingly. in, For users The low-rank value of LoRA, and These are the lower and upper limits of the rank value, respectively. The mapping coefficient from sample size to rank value. For users A personalized sample set It is a logarithmic function. and Used to limit the dynamic rank within a preset range For the personalized adapted output representation, The base model weight matrix is ​​frozen. This is the LoRA scaling factor. and For users The corresponding trainable low-rank matrix, This is a low-rank increment relative to the basic weights. The input is the feature vector; to maintain general sentiment understanding capabilities while ensuring the output interpretation aligns with the target audience's expression habits, the sample expression embedding is constrained to be close to the corresponding user's historical expression center: in, This embeds the sample expression close to the center of the user's historical expression of the corresponding emotion. Limit adapter parameter offsets to prevent overfitting with few samples; subsequently, incorporate generative loss, expression regularity alignment, low-rank regularization, and general model deviation constraints into the personalized training objective: in, The customized model for each constraint should not deviate excessively from the general model. For the total loss of personalized training, Generate loss for personalized emotion description, The alignment loss between the current expression and the user's historical expression center. For low-rank adapter regularization loss, , and These are the non-negative weight coefficients corresponding to the sub-loss. Indexing by emotion categories, To input multimodal samples, For personalized models in input Predicting Sentiment The probability, This represents the predicted probability corresponding to the general model.

8. The multimodal emotion dynamic recognition method based on emotion prototype attention and large language model according to claim 7, characterized in that, The modality missing state modeling includes, to distinguish between valid and invalid evidence when video, audio, or text is incomplete, first constructing a modality state vector that simultaneously includes usability and quality scores: in, Simultaneously record the existence and quality of the mode. This represents the currently available set of modalities. During the fusion phase, only available modalities that meet the quality standards are normalized and weighted to form a robust representation that is aware of missing features. in, For the missing perception fusion weights, For the first Robust fusion representation with awareness of missing time steps For the current sample index, For time step index, For the sample The currently available and quality-compliant modal set. and For modal indexing, For the first The feature projection matrix of the modality, For the first The first time step The feature vector of the modality, For the learnable score vector of the corresponding modality, Indicates matrix transpose. For the first Modal availability and quality state vectors at each time step. This represents the concatenation of the modal feature vector and the modal state vector. It is an exponential function; when a modality is missing or its quality degrades, its weight is automatically removed from or reduced in the normalized denominator; for basic fusion scenarios, the fusion vector is directly constructed based on the set of available modalities. Robust training for missing modalities includes constructing degenerate samples and constraining the representations under complete and missing inputs to be consistent. in, Indicates the first Degenerate operators for modal masking, occlusion, noise perturbation, or transcription perturbation. For complete multimodal input, For the set of modalities participating in robust training, To control the random degradation parameters of shielding ratio, shielding range, noise intensity, or transcription perturbation mode, For the first The incomplete input obtained after the class modality performs degradation. Based on parameters For input The Modal classes perform degenerate operations. This is the emotion representation extraction function for the model. and The emotion representations are for complete input and degenerate input, respectively. To represent the loss of consistency, It is a squared L2 norm; in addition to ensuring consistency in the final sentiment representation, it also constrains the contribution heatmaps under the missing modalities to still locate the same key events: in, The contribution heatmap for the mode with missing constraints is consistent with the contribution heatmap for the complete mode. Contribute heatmap generation functions to the CTA module. and The complete input and the first Contribution heatmap of modal degradation inputs, For degenerate modal indexes, For the set of modes participating in the consistency constraints, The Kullback-Leibler divergence is used to ensure that the overall distribution of the contribution heatmaps is consistent under both inputs. To select the one with the highest contribution value Operators for key positions and their fractions, where To retain the preset quantity, It is a norm 1, used to constrain the differences at the key positions with the highest contribution, and to prevent the model from turning to irrelevant background evidence under missing conditions.

9. A multimodal emotion dynamic recognition method based on emotion prototype attention and a large language model according to claim 8, characterized in that, The end-to-end joint optimization includes constructing a total loss comprising each core module to ensure that encoding, attribution, memory, personalization, and robust output converge collaboratively under the same objective, expressed as: in, Used to constrain cross-modal consistency. Used to constrain the location of key clues. Used to constrain personalized interpretations This is used to constrain stability under conditions of missing modes; finally, to transform the model inference results into readable, verifiable, and continuously updatable output records, a complete evidence cache of the inference is first saved: in, A record of evidence for a reasoning exercise, containing Input identifier, Key clues Contribute heatmaps, Memory state, Personalized adapter labeling Emotional arc and The report text creates a traceable record for each reasoning step, which is used to instantly display the emotional trajectory and key evidence, thus forming a continuous optimization loop.

10. A multimodal emotion dynamic recognition system based on emotion prototype attention and a large language model, executing the multimodal emotion dynamic recognition method based on emotion prototype attention and a large language model as described in claim 1, characterized in that, include: The data acquisition module is configured to acquire multimodal sentiment data; The unified sequence module is configured to perform multimodal sentiment input encoding and unified sequence representation based on the acquired multimodal sentiment data, including multimodal data time alignment and quality assessment, multimodal data encoding, unified feature projection and sequence construction, and cross-modal consistency constraints. The attention module is configured to build a causal tracking attention mechanism based on emotion prototypes based on unified sequence representation, including emotion prototype initialization, query key and value mapping and contribution heatmap calculation, key clue filtering and contribution score normalization, sparse supervised training and temporal continuity constraints, and key event fragment filtering. The emotion trajectory module is configured to construct contextual memory units and emotion trajectory models based on a causal tracking attention mechanism, including emotional history memory vector construction, gated memory updating, emotion trajectory generation, and structured reporting. The target adaptation module is configured to perform personalized target adaptation based on contextual memory and emotional trajectory, including the introduction of individual expression samples, low-rank adaptation based on LoRA, and personalized training targets and interpretation generation. The output module is configured to provide robust inference and joint optimization output, including modality missing state modeling, modality missing robust training, end-to-end joint optimization, and result visualization output.

Images