A youth emotion recognition method and system based on a bidirectional attention mechanism and multi-level fusion

Abstract

The application discloses a kind of based on bidirectional attention mechanism and multi-level fusion adolescent emotion recognition method and system, it is related to artificial intelligence and affective computing technical field.The method includes: obtaining and preprocessing the audio and video data of adolescent, extracts initial feature vector;Characteristics are input into the multi-level fusion model based on the architecture of Transformer, sequentially early, mid and late fusion are carried out by bidirectional cross attention mechanism, to capture subtle signs, semantic conflict and decision weighting respectively;Dynamic weighting is carried out to anti-interference by introducing gate mechanism to fusion characteristics;Finally, output emotion category.Corresponding system contains data acquisition, feature processing, dynamic gate and output module.The application solves the problem of inconsistent expression of adolescent emotion and modal heterogeneity by multi-level progressive fusion and dynamic loss optimization, significantly improves the recognition accuracy, robustness and generalization ability of complex emotion.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and affective computing, specifically to a method and system for adolescent emotion recognition based on a bidirectional attention mechanism and multi-level fusion. Background Technology

[0002] Mental health is the cornerstone of adolescents' all-round development. However, influenced by factors such as family environment and academic pressure, adolescent mental health problems are becoming increasingly prominent, but only a very small number of patients receive professional intervention. With the development of artificial intelligence, technologies that identify emotions by analyzing external features such as facial expressions and tone of voice have made it possible to conduct seamless and convenient mental health screening.

[0003] Most existing emotion recognition technologies are based on modeling adult data. However, adolescent emotions are characterized by strong volatility, immature expression, and inconsistencies between inner feelings and outward expressions (such as "saying one thing and meaning another"), making adult models perform poorly on adolescents. Multimodal fusion technology, by integrating information from text, speech, and vision, can improve recognition robustness and is a current research focus. Existing multimodal fusion methods mainly include early fusion, mid-term fusion, and late fusion. Early fusion (such as feature splicing) can preserve fine-grained correlations but is susceptible to modal heterogeneity and noise interference. Late fusion (such as decision-level weighting) is robust but may lose low-level interaction information between modalities. Recently, cross-attention mechanisms based on the Transformer architecture have been used for multimodal emotion recognition, realizing intermodal interaction. However, existing methods are mostly limited to single-stage or single-scale feature interactions, making it difficult to fully capture the dynamic evolution and inconsistent representations of adolescent emotions at different feature levels (from instinctive micro-reactions to high-level semantics).

[0004] Therefore, existing technologies lack an efficient and robust emotion recognition scheme that can effectively handle the specificity of adolescents' emotional expression, especially the inconsistency between modalities. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for adolescent emotion recognition based on a bidirectional attention mechanism and multi-level fusion, so as to overcome the shortcomings of existing technologies, such as poor adaptability to adolescents, insufficient accuracy in recognizing complex and inconsistent emotions, and weak model robustness. Through a progressive, bidirectional interactive multi-level fusion architecture, combined with a dynamically optimized training strategy, the invention achieves accurate capture of complex emotions in adolescents.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A method for adolescent emotion recognition based on a bidirectional attention mechanism and multi-level fusion, the method being executed by a computing device, includes the following steps: Multimodal data acquisition and preprocessing: Acquire audio and video data of adolescents, and preprocess them separately to extract the corresponding initial audio feature vectors and initial video feature vectors; Multi-level feature fusion: The initial audio feature vector and the initial video feature vector are input into a multi-level feature fusion model based on the Transformer architecture. Through a bidirectional cross-attention mechanism, early fusion, mid-term fusion and late-term fusion are performed sequentially to output multi-level fused features. Among them, the early fusion is used to capture subtle emotional signs in shallow features, the mid-term fusion is used to process intermediate semantic features and identify inconsistencies between modalities, and the late-term fusion is used to perform interactive weighting based on the prediction confidence of each modality at the decision level. Dynamic gating and weighting: A gating mechanism is introduced into the multi-level fusion features after late fusion. The gating coefficients are dynamically calculated based on the quality of the audio and video data to weight the multi-level fusion features and output the final fusion feature vector. Emotion recognition output: The final fused feature vector is mapped to a probability distribution of a preset emotion category to complete the recognition of adolescent emotions.

[0007] Furthermore, the bidirectional cross-attention mechanism includes a first attention path that guides the initial audio feature vector based on the initial video feature vector, and a second attention path that guides the initial video feature vector based on the initial audio feature vector.

[0008] Furthermore, the multi-level feature fusion model is obtained through training, and the composite loss function used in the training is a linear weighted sum of the main loss and the auxiliary loss, wherein the main loss is the label smooth cross-entropy loss.

[0009] Furthermore, the auxiliary losses include modality consistency loss, feature alignment loss, and center loss.

[0010] Furthermore, the modal consistency loss is calculated based on the audio feature vector and video feature vector during the training process of the multi-level feature fusion model, and is used to constrain the cosine similarity matrix of the two to approximate the identity matrix. The feature alignment loss is calculated based on the audio feature vector and video feature vector during the training process, and is used to constrain the mean vector and standard deviation vector of their feature distribution to be similar. The center loss, calculated based on the sample features in the training batch, is used to constrain the sample features of the same emotion category to cluster towards the feature center of that category.

[0011] Furthermore, the training also includes dynamic weight adjustment, specifically: Monitor the performance metrics of the multi-level feature fusion model during the training process; When the performance metric does not exceed the historical best value within a predetermined number of consecutive training rounds, the weight coefficient of the auxiliary loss is increased according to a preset rule.

[0012] Furthermore, the preset rule refers to setting different upward adjustment ranges and weight caps for different auxiliary losses.

[0013] Another objective of this invention is to provide a adolescent emotion recognition system based on a bidirectional attention mechanism and multi-level fusion, for implementing the aforementioned adolescent emotion recognition method based on a bidirectional attention mechanism and multi-level fusion, the system comprising: The multimodal data acquisition module is used to acquire and preprocess audio and video data of adolescents, and output initial audio feature vectors and initial video feature vectors. A bidirectional attention feature processing module, connected to the multimodal data acquisition module, is used to receive the initial feature vector and perform multi-level feature fusion through its early fusion unit, mid-term fusion unit and late fusion unit; The dynamic gating weighting module, connected to the late fusion unit in the bidirectional attention feature processing module, is used to dynamically weight the fused features based on data quality and output the final fused feature vector. The emotion analysis output module, connected to the dynamic gating weighting module, is used to output the emotion category based on the final fused feature vector.

[0014] Furthermore, the system also includes: The composite loss optimization training module is connected to the bidirectional attention feature processing module and the dynamic gating weighting module during the training phase. It is used to calculate the composite loss function and perform dynamic weight adjustment to optimize the parameters of the bidirectional attention feature processing module and the dynamic gating weighting module.

[0015] Furthermore, the composite loss optimization training module includes: The main and auxiliary loss calculation unit is configured to calculate the main loss and auxiliary loss in the composite loss function. The main loss is the label smoothing cross-entropy loss, and the auxiliary loss includes modality consistency loss, feature alignment loss and center loss. The dynamic weight adjustment unit is configured to dynamically adjust the weight coefficients of the auxiliary loss according to changes in the system's performance metrics during training.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. Significantly improves the accuracy of identifying complex and inconsistent emotions in adolescents. The multi-level feature fusion structure employed in this invention, through progressive bidirectional interaction in the early, middle, and late stages, enables in-depth analysis of emotional signals from multiple levels, including instinctive micro-reactions, semantic conflicts, and decision confidence. This dynamic fusion strategy allows the model to transform "intermodal inconsistency," which is considered noise in traditional techniques, into effective emotion discrimination features, thereby achieving more accurate identification of complex and compound emotions commonly found in adolescents, such as "saying one thing but meaning another" and "suppressed anger." On a specific dataset for adolescent emotion recognition, the model of this invention demonstrates significantly higher recognition accuracy than existing mainstream models.

[0017] 2. It exhibits strong robustness and anti-interference capabilities. By introducing a gating mechanism to dynamically evaluate and weight the contributions of different modalities, and combining modality consistency constraints and feature distribution alignment constraints in the composite loss function, this invention effectively improves the stability of the system in practical applications. When the quality of a certain modality (such as audio due to environmental noise or video due to partial occlusion) degrades, the system can automatically suppress the negative impact of low-quality modalities, relying on high-quality modalities and the cooperative relationships between modalities to maintain accurate judgment, significantly reducing the misjudgment rate.

[0018] 3. High emotion recognition accuracy and high inter-class discrimination. Addressing the dual tasks of emotion classification and intensity assessment, this invention introduces a center loss function, forcing feature vectors of the same emotion category to cluster compactly in space, effectively improving the prediction accuracy of emotion recognition. Simultaneously, this mechanism increases the distance between feature centers of different emotion categories, enhancing the model's ability to distinguish easily confused emotions, thereby comprehensively improving the prediction accuracy of emotion classification.

[0019] 4. The model exhibits fast convergence speed and strong generalization ability, effectively avoiding local optima. The label-smoothed cross-entropy loss and dynamic weight adjustment strategy employed in this invention work synergistically, not only mitigating overfitting risks but also providing multi-objective collaborative supervision for model training. Particularly during the training plateau, the dynamic weight adjustment mechanism based on validation set performance feedback automatically strengthens auxiliary loss constraints, forcing the model to mine more robust feature representations, thereby effectively escaping local optima, accelerating the convergence process, and ultimately improving the model's generalization performance on unknown data. Ablation experiments demonstrate that the complete technical solution of this invention significantly outperforms model variants containing only some innovative points, proving the comprehensive performance improvement brought about by the synergy of various technical means. Attached Figure Description

[0020] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a system architecture diagram of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0022] This embodiment provides a method for adolescent emotion recognition based on a bidirectional attention mechanism and multi-level fusion. First, refer to... Figure 1 The diagram illustrates the overall flowchart of the method described in this invention. This method is executed by a computing device with data processing capabilities, such as a server equipped with a GPU, a high-performance workstation, or a dedicated edge computing device. The core of this method lies in accurately identifying the complex emotions unique to adolescents through a novel multi-level progressive fusion architecture and dynamic optimization strategy. The specific process of this method and the model training process are described in detail below.

[0023] I. Main Process of the Method 1. Multimodal data acquisition and preprocessing: Acquire audio and video data of adolescents, and preprocess them separately to extract the corresponding initial audio feature vectors and initial video feature vectors.

[0024] The computing device collects raw audio and video data of teenagers through a camera and microphone. The data in this embodiment can be acquired in various ways, for example, by constructing a dataset by collecting performance clips of teenagers from publicly available film and television works. Specifically, multiple films and television series with teenagers as their theme can be selected, covering genres such as education, romance, crime, and family. Audio and video clips with obvious emotional characteristics are extracted using video editing tools, ensuring that the clips meet the following quality requirements: the speaker has a clear facial expression; the dialogue scene has low noise and clear speech; the speaker has a clear and identifiable emotional tendency. For the video data, face detection and key point localization are first performed, followed by face alignment to eliminate the influence of pose and position changes, and then video frame sequences are extracted. For the audio data, preprocessing such as pre-emphasis, frame segmentation, and windowing is performed to reduce environmental noise interference. Next, pre-trained deep convolutional neural networks (e.g., using a ResNet-50 model pre-trained on a large face dataset to extract video features and a VGGish model to extract audio features) are used to extract features from the preprocessed video and audio frames, respectively, resulting in initial audio and video feature vectors of 1024 dimensions each. This step provides standardized, high-quality input features for subsequent processing.

[0025] 2. Multi-level feature fusion: The initial audio feature vector and the initial video feature vector are input into a multi-level feature fusion model based on the Transformer architecture. Through a bidirectional cross-attention mechanism, early fusion, mid-term fusion and late-term fusion are performed sequentially to output multi-level fused features. Among them, the early fusion is used to capture subtle emotional signs in shallow features, the mid-term fusion is used to process intermediate semantic features and identify inconsistencies between modalities, and the late-term fusion is used to perform interactive weighting based on the prediction confidence of each modality at the decision level. This is the core step of the invention. The initial feature vectors of audio and video obtained in step 1 are input into a multi-level feature fusion model based on the Transformer architecture. This model is not a simple single fusion, but is designed with three cascaded fusion levels: early, middle, and late. Each level uses a bidirectional cross-attention mechanism for deep interaction, thereby achieving a comprehensive analysis of emotional signals from micro to macro and from instinct to decision.

[0026] (1) Early Fusion: This layer focuses on processing shallow features from the neural network. The bidirectional cross-attention mechanism is specifically manifested here as two independent attention paths. In the first path, the initial video feature vector is used as the query, and the initial audio feature vector is used as the key and value, calculating the guiding attention weight of the video features on the audio features. In the second path, the roles are reversed, with the audio features as the query and the video features as the key and value. Through this bidirectional interaction, the model can capture those fleeting, instinctive reactions that teenagers themselves may not even be aware of, such as subtle twitches in the corners of the eyes (micro-expressions) or brief, unintentional tremors in the tone of voice (micro-voice). These subtle signs captured by early fusion provide the first layer of key clues for recognizing real emotions, outputting detailed early fusion features.

[0027] (2) Mid-level fusion: Early fusion features are fed into this level for deeper semantic encoding. Here, the role of the bidirectional cross-attention mechanism is to dynamically evaluate and integrate the mid-level semantic information conveyed by the audio and video modalities. When adolescents exhibit "saying one thing but meaning another" emotional expressions, such as saying "I'm fine" in a calm voice but with facial muscles revealing tension and grievance, the model can keenly identify this semantic inconsistency through the bidirectional attention weight calculation of mid-level fusion. It does not simply deny or average a certain modality, but encodes this conflict itself as a high-weight emotional signal. This processing mechanism greatly enhances the ability to identify the inherent contradictions in adolescents' emotions and outputs mid-level fusion features with more semantic information.

[0028] (3) Late Fusion: This level operates at the high-level abstract feature level, focusing more on fusion at the decision-making level. First, the audio and visual modal branches each generate preliminary probability distributions for the seven emotion categories through independent classifier layers. Subsequently, the bidirectional attention mechanism plays the role of an "intelligent arbitrator." It does not fuse mechanically, but interacts and weights based on the prediction confidence of each modality. For example, when the system identifies the emotion of "anger," if the visual modality (such as clenching teeth or frowning) gives a very high confidence, while the audio modality (such as a steady tone) has a low confidence due to environmental noise or individual expression habits, the late fusion mechanism will automatically give the visual modality a higher decision weight. This confidence-based game mechanism enables the model to intelligently handle the uncertainty between modalities, thereby accurately identifying complex compound emotions such as "suppressed anger" and outputting late fusion features for the final decision.

[0029] 3. Dynamic Gating Weighting: A gating mechanism is introduced into the multi-level fusion features after late-stage fusion. The gating coefficients are dynamically calculated based on the quality of the audio and video data to weight the multi-level fusion features and output the final fusion feature vector.

[0030] To further enhance the system's robustness in complex real-world application scenarios, a gating mechanism is introduced after obtaining the late-stage fusion features. This mechanism can be a lightweight neural network, for example, consisting of a fully connected layer and a sigmoid activation function. The gating network takes the late-stage fusion features and / or the original audio and video feature vectors as input, automatically learns and outputs a set of gating coefficients (typically between 0 and 1) to evaluate the relative quality or reliability of the audio and video modalities in the current sample. For example, when the video modality suffers quality degradation due to insufficient lighting or partial occlusion, the coefficients calculated for the video features by the gating mechanism are automatically reduced; conversely, if the audio modality is subjected to transient strong noise interference, its corresponding coefficients are also suppressed. Subsequently, these coefficients are used to dynamically weight the late-stage fusion features, thereby highlighting the contribution of high-quality modalities and suppressing the negative impact of low-quality modalities. This mechanism significantly improves the system's stability and adaptability under non-ideal conditions, outputting a final fusion feature vector with stronger anti-interference capabilities.

[0031] 4. Emotion Recognition Output: The final fused feature vector is mapped to a probability distribution of a preset emotion category to complete the recognition of adolescent emotions.

[0032] The dynamically weighted final fused feature vector is input into the final task-specific layer. This layer typically includes a fully connected layer and a Softmax activation function, used to map the features to a probability distribution of seven basic emotions (e.g., joy, sadness, anger, fear, surprise, disgust, and neutral). Finally, the computing device presents the identified emotion category (the one with the highest probability) to the end user through a graphical user interface, API, or other means, completing the entire emotion recognition and quantification process. Figure 2 The complete system architecture diagram of the multi-level feature fusion model is shown.

[0033] II. Model Training Process Next, the training process of the multi-level feature fusion model will be described in detail. This training strategy is the key to ensuring that the model of the present invention achieves excellent performance.

[0034] Before model training begins, a well-labeled multimodal emotion dataset for teenagers needs to be prepared. This dataset should contain a large number of audio and video samples from teenagers, each labeled with a real emotion category. To monitor the training process and prevent overfitting, the dataset needs to be divided into three mutually exclusive subsets: a training set, a validation set, and a test set. The training set is used to directly calculate the loss and update the model parameters using gradient descent. The validation set does not participate in parameter updates; its role is to evaluate the model's generalization performance after each training epoch and to serve as the basis for triggering dynamic weight adjustments, early stopping, and other strategies. The test set is used to finally evaluate the model's performance, simulating real-world application scenarios. The subsequent model training and optimization processes are all based on this standard data partitioning process. Model training relies on a carefully designed composite loss function and its dynamic adjustment strategy.

[0035] 1. The overall composite loss function is a linearly weighted sum of the main loss and auxiliary loss, and its expression is as follows: in, It is the main loss. , , The weight parameters are used to assist in the loss.

[0036] (1) Calculation of main loss: The main loss adopts the label smoothing cross-entropy loss, and its expression is as follows: Where C is the number of emotion categories (C=7), Let i be the probability of the i-th type of emotion predicted by the model. The target distribution after label smoothing. Smoothing coefficient ( =0.1). Introducing label smoothing effectively alleviates the overfitting of the model to the training labels and enhances its generalization ability.

[0037] (2) Auxiliary loss calculation: The auxiliary loss includes three items, which together constrain the model to learn more robust features.

[0038] 1) Modality Consistency Loss: Calculated based on the audio feature vector 'a' and video feature vector 'v' obtained during training, it constrains their alignment in the semantic space. Its expression is: Where sim(a,v) is the cosine similarity matrix between the two, and I is the identity matrix. This loss ensures that even if one mode is disturbed, the other mode can still guide the model to make the correct judgment through consistency constraints.

[0039] 2) Feature Alignment Loss: Also based on audio and video feature vectors, it is used to narrow down their distribution in the feature space. Its expression is: in, These are the mean vectors of the audio and video features, respectively. These are the standard deviation vectors of the two modalities, respectively. This loss, through the superposition of the two MSE terms, simultaneously constrains the central location and dispersion of the features, promoting feature isomorphism between modalities and improving the system's adaptability in cases of missing or impaired modalities.

[0040] 3) Center Loss: Based on the features of samples in the training batch. The calculation, used to improve intra-class compactness and inter-class differentiation, is expressed as: Where N is the batch sample size. Let be the fused feature vector of the i-th sample. For the first The loss function generates a feature center vector for each emotion category, which is dynamically updated during training (with an update rate of 0.9). This loss function clusters the features of samples with the same emotion, thereby significantly improving the prediction accuracy of emotion recognition and reducing confusion between similar emotions.

[0041] 2. Dynamic weight adjustment The specific process of dynamic weight adjustment is as follows: monitoring the performance indicators of the multi-level feature fusion model during training; when the performance indicators do not exceed the historical best value within a predetermined number of consecutive training rounds, increasing the weight coefficient of the auxiliary loss according to preset rules. The preset rules refer to setting different increase magnitudes and weight caps for different auxiliary losses.

[0042] During training, the model's recognition accuracy on the independent validation set is continuously monitored. If the accuracy fails to surpass the historical best value within three consecutive training epochs, a dynamic adjustment mechanism for the loss function weights is triggered. Specifically, at the beginning of training, the weights of the auxiliary loss are set to relatively small initial values ​​(e.g., ...). =0.003, =0.001, =0.002), to ensure the dominance of the primary classification task. After triggering, the consistency loss weight increases by 5% (maximum 0.05), the feature alignment loss weight also increases by 5% (maximum 0.03), and the centering loss weight increases by 2% (maximum 0.02). Meanwhile, to avoid optimization imbalance caused by unlimited weight growth, each loss weight has a clear upper limit, ensuring that the auxiliary losses always serve the primary classification task rather than dominate the training process. When the validation set accuracy improves and updates to the optimal value, the weight adjustment counter is reset to 0, pausing weight adjustment and allowing the model to learn stably with the current weight configuration.

[0043] This feedback adjustment mechanism based on validation set performance stagnation essentially relies on a step-by-step adjustment strategy to proactively strengthen constraints on cross-modal feature association, inter-modal feature distribution matching, and intra-class feature clustering during the training plateau. This is equivalent to adaptively introducing stronger guidance signals when the model gets stuck in a local optimum, forcing the model to mine deeper and more robust feature representations, thereby breaking through the training bottleneck. This mechanism can adaptively balance feature learning and classification objectives, ultimately effectively improving the model's generalization ability.

[0044] III. Model Performance Verification Experiment To verify the effectiveness of the method described in this invention, the following comparative experiments were conducted.

[0045] 1. Experimental setup To comprehensively evaluate the performance of this invention, two representative emotion recognition benchmark datasets were selected for the experiment: The RAVDESS dataset contains emotion data in audio and video modalities and is widely used in general emotion recognition research. MERA-S dataset: An audio and video emotion recognition dataset for teenagers, which better reflects the specificity of emotional expression among teenagers.

[0046] The experimental environment consisted of a server equipped with an NVIDIA Tesla V100 GPU and PyTorch 1.9.0. Model training used the Adam optimizer with an initial learning rate of 1e-4 and a batch size of 32. The model constructed in this experiment according to the method described in this invention is denoted as the HiFusion-MER model.

[0047] 2. Comparison Model Settings To fully verify the innovativeness and effectiveness of this invention, two sets of comparative experiments were set up: (1) Ablation experiment comparison: To verify the independent contribution and synergistic effect of the core modules (multi-level fusion architecture and composite loss function) in this invention, the following ablation comparison model was set up: The MFC-Loss-only model replaces the multi-level fusion architecture in the HiFusion-MER model with a fusion network based on the basic Transformer, retaining only the composite loss function described in this invention for training.

[0048] HLF-only model: The composite loss function in the HiFusion-MER model is replaced with the basic cross-entropy loss, retaining only the multi-level fusion architecture described in this invention.

[0049] Base model: Replace the multi-level fusion architecture and composite loss function in the HiFusion-MER model with the baseline component (basic Transformer fusion + cross-entropy loss).

[0050] (2) Comparison with mainstream models: The present invention is compared with the currently recognized classic and mainstream models in the field, including: Swin3D-Tiny: A mainstream video recognition model based on 3D Transformer.

[0051] ConvNeXt-Tiny: A mainstream model based on modern 2D CNN design.

[0052] MViT-v2: The state-of-the-art (SOTA) model based on multi-scale visual Transformer proposed at CVPR 2022.

[0053] R(2+1)D-18: A classic baseline model for 3D video understanding.

[0054] ResNet50: A widely used baseline model for 2D image classification.

[0055] 3. Experimental Results and Analysis Under the same experimental conditions, the emotion recognition accuracy of each model on the above dataset is shown in Tables 1 and 2.

[0056] (1) As can be seen from the ablation experiment results in Table 1, in the specific implementation case of this embodiment: Comparing the complete HiFusion-MER model with the MFC-Loss only model, the accuracy improved from 64.85% to 67.96% on the MERA-S dataset and from 77.78% to 89.68% on the RAVDESS dataset, demonstrating that the multi-level fusion architecture proposed in this invention makes a significant contribution to the model performance.

[0057] Compared with the HLF-only model, the accuracy of the HiFusion-MER model improved from 63.60% to 67.96% on the MERA-S dataset and from 72.91% to 89.68% on the RAVDESS dataset, demonstrating that the composite loss function and dynamic training strategy proposed in this invention make a significant contribution to the model performance.

[0058] The complete HiFusion-MER model outperforms its ablation variant, demonstrating a synergistic effect between the multi-level fusion architecture and the composite loss function, which together constitute the complete technical solution of this invention.

[0059] Table 1 Comparison of Emotion Recognition Accuracy in Ablation Experiments

[0060] (2) As shown in Table 2, the HiFusion-MER model proposed in this invention achieves significantly higher recognition accuracy on both datasets than the listed mainstream and state-of-the-art models. Particularly on the MERA-S dataset for adolescents, the model of this invention shows a significant absolute improvement in accuracy compared to the second-best performing model. This result verifies the effectiveness and advancement of the technical solution of this invention in capturing the specific emotions of adolescents.

[0061] Table 2 Comparison of emotion recognition accuracy with mainstream models:

[0062] This embodiment provides a youth emotion recognition system based on a bidirectional attention mechanism and multi-level fusion, used to implement the aforementioned youth emotion recognition method based on a bidirectional attention mechanism and multi-level fusion. The system includes: The multimodal data acquisition module is used to acquire and preprocess audio and video data of adolescents, and output initial audio feature vectors and initial video feature vectors. A bidirectional attention feature processing module, connected to the multimodal data acquisition module, is used to receive the initial feature vector and perform multi-level feature fusion through its early fusion unit, mid-term fusion unit and late fusion unit; The dynamic gating weighting module, connected to the late fusion unit in the bidirectional attention feature processing module, is used to dynamically weight the fused features based on data quality and output the final fused feature vector. The emotion analysis output module, connected to the dynamic gating weighting module, is used to output the emotion category based on the final fused feature vector.

[0063] In addition, the system further includes a composite loss optimization training module, which is connected to the bidirectional attention feature processing module and the dynamic gating weighting module during the training phase. This module calculates the composite loss function and performs dynamic weight adjustment to optimize the parameters of the bidirectional attention feature processing module and the dynamic gating weighting module. The composite loss optimization training module includes: The main and auxiliary loss calculation unit is configured to calculate the main loss and auxiliary loss in the composite loss function. The main loss is the label smoothing cross-entropy loss, and the auxiliary loss includes modality consistency loss, feature alignment loss and center loss. The dynamic weight adjustment unit is configured to dynamically adjust the weight coefficients of the auxiliary loss according to changes in the system's performance metrics during training.

[0064] Embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0065] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0066] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0067] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0068] Contents not described in detail in this specification are prior art known to those skilled in the art. It is hereby indicated that the above description is intended to help those skilled in the art understand this invention, but does not limit the scope of protection of this invention. Any equivalent substitutions, modifications, improvements, or simplifications of the above descriptions that do not depart from the essential content of this invention fall within the scope of protection of this invention.

Claims

1. A method for adolescent emotion recognition based on bidirectional attention mechanism and multi-level fusion, characterized in that, The method is executed by a computing device and includes the following steps: Multimodal data acquisition and preprocessing: Acquire audio and video data of adolescents, and preprocess them separately to extract the corresponding initial audio feature vectors and initial video feature vectors; Multi-level feature fusion: The initial audio feature vector and the initial video feature vector are input into a multi-level feature fusion model based on the Transformer architecture. Through a bidirectional cross-attention mechanism, early fusion, mid-term fusion and late-term fusion are performed sequentially to output multi-level fused features. Among them, the early fusion is used to capture subtle emotional signs in shallow features, the mid-term fusion is used to process intermediate semantic features and identify inconsistencies between modalities, and the late-term fusion is used to perform interactive weighting based on the prediction confidence of each modality at the decision level. Dynamic gating and weighting: A gating mechanism is introduced into the multi-level fusion features after late fusion. The gating coefficients are dynamically calculated based on the quality of the audio and video data to weight the multi-level fusion features and output the final fusion feature vector. Emotion recognition output: The final fused feature vector is mapped to a probability distribution of a preset emotion category to complete the recognition of adolescent emotions.

2. The adolescent emotion recognition method based on bidirectional attention mechanism and multi-level fusion according to claim 1, characterized in that, The bidirectional cross-attention mechanism includes a first attention path that guides the initial audio feature vector based on the initial video feature vector, and a second attention path that guides the initial video feature vector based on the initial audio feature vector.

3. The adolescent emotion recognition method based on bidirectional attention mechanism and multi-level fusion according to claim 1, characterized in that, The multi-level feature fusion model is obtained through training. The training uses a composite loss function that is a linear weighted sum of the main loss and the auxiliary loss, wherein the main loss is the label smoothing cross-entropy loss.

4. The adolescent emotion recognition method based on bidirectional attention mechanism and multi-level fusion according to claim 3, characterized in that, The auxiliary losses include modality consistency loss, feature alignment loss, and center loss.

5. The adolescent emotion recognition method based on bidirectional attention mechanism and multi-level fusion according to claim 4, characterized in that: The modality consistency loss is calculated based on the audio feature vector and video feature vector during the training process of the multi-level feature fusion model, and is used to constrain the cosine similarity matrix of the two to approximate the identity matrix. The feature alignment loss is calculated based on the audio feature vector and video feature vector during the training process, and is used to constrain the mean vector and standard deviation vector of their feature distribution to be similar. The center loss, calculated based on the sample features in the training batch, is used to constrain the sample features of the same emotion category to cluster towards the feature center of that category.

6. The adolescent emotion recognition method based on bidirectional attention mechanism and multi-level fusion according to claim 3, characterized in that, The training also includes dynamic weight adjustment, specifically: Monitor the performance metrics of the multi-level feature fusion model during the training process; When the performance metric does not exceed the historical best value within a predetermined number of consecutive training rounds, the weight coefficient of the auxiliary loss is increased according to a preset rule.

7. The adolescent emotion recognition method based on bidirectional attention mechanism and multi-level fusion according to claim 6, characterized in that, The preset rule refers to setting different upward adjustment ranges and weight caps for different auxiliary losses.

8. A system for recognizing adolescent emotions based on a bidirectional attention mechanism and multi-level fusion, used to implement the adolescent emotion recognition method based on a bidirectional attention mechanism and multi-level fusion as described in any one of claims 1-7, characterized in that, The system includes: The multimodal data acquisition module is used to acquire and preprocess audio and video data of adolescents, and output initial audio feature vectors and initial video feature vectors. A bidirectional attention feature processing module, connected to the multimodal data acquisition module, is used to receive the initial feature vector and perform multi-level feature fusion through its early fusion unit, mid-term fusion unit and late fusion unit; The dynamic gating weighting module, connected to the late fusion unit in the bidirectional attention feature processing module, is used to dynamically weight the fused features based on data quality and output the final fused feature vector. The emotion analysis output module, connected to the dynamic gating weighting module, is used to output the emotion category based on the final fused feature vector.

9. A youth emotion recognition system based on bidirectional attention mechanism and multi-level fusion according to claim 8, characterized in that, The system also includes: The composite loss optimization training module is connected to the bidirectional attention feature processing module and the dynamic gating weighting module during the training phase. It is used to calculate the composite loss function and perform dynamic weight adjustment to optimize the parameters of the bidirectional attention feature processing module and the dynamic gating weighting module.

10. A youth emotion recognition system based on a bidirectional attention mechanism and multi-level fusion as described in claim 9, characterized in that, The composite loss optimization training module includes: The main and auxiliary loss calculation unit is configured to calculate the main loss and auxiliary loss in the composite loss function. The main loss is the label smoothing cross-entropy loss, and the auxiliary loss includes modality consistency loss, feature alignment loss and center loss. The dynamic weight adjustment unit is configured to dynamically adjust the weight coefficients of the auxiliary loss according to changes in the system's performance metrics during training.

Images