A multi-modal emotion autonomous recognition method fusing semantic label and BAN
By integrating semantic tags and BAN into a multimodal autonomous emotion recognition method, this approach solves the problems of insufficient robustness and generalization ability of traditional single-modal emotion recognition systems in complex scenarios, achieving efficient and reliable emotion recognition applicable to various application scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUIZHOU UNIV
- Filing Date
- 2026-04-07
- Publication Date
- 2026-07-24
AI Technical Summary
Traditional single-modal emotion recognition systems struggle to meet the demands for fine-grained, low-latency, and highly reliable emotion recognition in complex scenarios. They also lack intermodal feature collaboration modeling, resulting in poor performance in complex scenarios with noise interference, heterogeneous devices, and varying data distribution.
A multimodal emotion autonomous recognition method integrating semantic labels and BAN is adopted. Through the collaborative work of multiple modules, the efficient fusion of visual and speech multimodal emotion data is achieved. A cross-modal contrastive learning mechanism and a bilinear attention network are introduced to capture high-order interaction relationships between modalities. Combined with semantic label consistency constraints, the final fused features are generated.
It improves the robustness and generalization ability of recognition in complex scenarios, increases the accuracy of emotion recognition by 20%, and enhances the model's adaptability to different devices and environments, making it suitable for cross-industry scenarios such as intelligent customer service, medical monitoring, and driving safety.
Smart Images
Figure CN122065164B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of cross-modal emotional intelligence computing technology, and more specifically, to a multimodal autonomous emotion recognition method that integrates semantic tags and BAN. Background Technology
[0002] With the rapid digitalization of scenarios such as human-computer interaction, intelligent security, in-vehicle cockpits, retail services, and medical monitoring, the demand for non-contact, real-time perception of human emotional states is exploding. Traditional emotion recognition systems based on a single modality (such as camera-only or microphone-only) are limited by their single information dimension and poor environmental robustness, making it difficult to meet the needs of "fine-grained, low-latency, and highly reliable" emotion assessment in complex scenarios. Facial expressions and voice signals are two of the most intuitive and complementary carriers of emotion, with the former providing instantaneous visual cues and the latter carrying continuous semantic and prosodic information; however, the two are inherently heterogeneous in terms of sampling frequency, spatiotemporal granularity, and semantic abstraction level, leading to three major challenges when multimodal fusion: "complementarity failure, semantic drift, and temporal misalignment."
[0003] Currently, the recognition of human emotions mainly relies on a single sensory channel or manual judgment by experienced professionals. With the development of artificial intelligence, technologies such as facial expression recognition, speech emotion analysis, and physiological signal analysis have made significant progress. However, these methods do not further capture the changes in information content at different times within a modality, and cannot effectively extract the dynamic features of emotions changing over time; they also lack modeling of intermodal feature synergy, and cannot fully utilize the correlation between different channels of multimodal features; consequently, the models perform poorly in complex scenarios such as noise interference, heterogeneous devices, and changes in data distribution, and their generalization ability and robustness still need to be improved.
[0004] Therefore, there is an urgent need for a multimodal autonomous emotion recognition method that integrates semantic tags and BAN. This method is a technical system that can perform real-time, non-contact autonomous emotion recognition in multiple scenarios using multimodal data. It can be directly deployed in scenarios such as intelligent customer service, security patrol, vehicle cockpit, medical care and public service robots, providing an efficient, reliable and generalizable technical foundation for emotion perception systems. Summary of the Invention
[0005] This invention provides a multimodal emotion autonomous recognition method that integrates semantic tags and BAN. Its purpose is to achieve efficient fusion and accurate recognition of visual-voice multimodal emotion data through the collaborative work of multiple modules, thereby improving the recognition robustness and generalization ability in complex scenarios.
[0006] To achieve the above objectives, this invention provides a multimodal emotion autonomous recognition method that integrates semantic tags and BAN, specifically including the following steps: S1. The smart terminal collects emotional videos and extracts visual and voice features; S2. Based on the complementarity of visual features and speech features, the visual features and speech features are fused to obtain multimodal joint features; S3. Use semantic tags to align the visual and speech features in the multimodal joint features to obtain aligned joint features; S4. To improve robustness and generalization under different devices and scenarios, a cross-modal contrastive learning mechanism is introduced. Based on the alignment joint features, the global semantic consistency is improved by bringing semantically matching sample pairs closer and pushing away mismatched sample pairs, thus obtaining enhanced alignment features. S5. To enhance the collaborative modeling of multimodal emotional features, the enhanced alignment features are input into the bilinear attention network (BAN) to capture the high-order interaction relationship between visual and speech local features, strengthen the fusion effect of fine-grained emotional cues between modalities, and generate preliminary fusion features. S6. To further stabilize the output and improve the label supervision effect, the preliminary fused features are weighted and enhanced in combination with semantic information consistency constraints to generate the final fused features. S7. Input the final fused features into the classifier, which outputs the probability of the emotion category. Complete the autonomous recognition of emotions; S1-S7 completed the construction of the autonomous emotion recognition model J: ; in, For feature extraction models, For multimodal feature fusion models, For semantic alignment model; The emotion autonomous recognition model The goal is to minimize the classification loss. With semantic alignment loss The sum, expressed as: , represents the semantic alignment loss weight coefficient.
[0007] Preferably, the feature extraction model Including visual feature extraction models and auditory feature extraction model , and These represent the extracted visual and speech features for each frame, respectively: ; ; in, For image frame signals, For audio frame signals; global visual features in video. and global speech features It can be represented as: ; ; in, The number of image frames. The number of audio frames. and It is a normalized weight function generated by the attention mechanism, which can be expressed as: ,satisfy .
[0008] Preferably, in S1, visual features are extracted using ResNet-50 combined with LSTM network ResNet-LSTM, and speech features are extracted using MFCC combined with LSTM network MFCC-LSTM.
[0009] Preferably, the specific content of feature extraction includes: S11, Visual Feature Extraction: S111, Learning Emotional Videos VideoCapture decodes each frame into an image frame signal. Then, standardized preprocessing is performed sequentially; S112, the pre-processed Input a ResNet-50 backbone network pre-trained on ImageNet and output frame-level embedded visual features for each frame. ; S113, The extra convolutional mapping layer is projected onto the temporal feature space and input into the Long Short-Term Memory (LSTM) network to capture inter-frame dynamic dependencies, which are then weighted and converged through an attention pooling mechanism. S114. Obtain global visual features after regularization. ; The ResNet-50 backbone network includes: convolutional, batch normalization, residual identity blocks, and convolutional blocks; S12. Speech Feature Extraction: S121, Emotional Video The speech in the video is decoded frame by frame using Librosa, and then normalized and pre-emphasis filtered sequentially to obtain the speech frame signal. ; S122. Extract the frame-level Mel frequency cepstral coefficients (MFCC) using the standard short-time spectrum analysis procedure. Speech features of each frame in the S123 and MFCC sequences Local time-frequency correlations are modeled using convolutional layers, and cross-frame dynamic dependencies are captured using LSTM. S124. The temporal features are weighted and converged using the attention pooling mechanism, and then regularized to form the final global speech features. ; The standard short-time spectrum analysis steps include: framing, windowing, Fast Fourier Transform (FFT), Mel filter bank, and Discrete Cosine Transform (DCT).
[0010] Preferably, the specific contents of S2 and S3 include: S21, the multimodal joint feature The defining formula is: ; S22. The feature fusion process is as follows: ,in, express Dimensions express Dimensions express dimensionality Represents dimensional space; S31, Let the emotion category label be... Map it to dimensional vector ,in Emotional tags semantic vector; S32, will and Union, resulting in aligned union features The expression is: .
[0011] Preferably, the cross-modal contrastive learning mechanism includes the following specific features: S41. Visual and speech features in the aligned joint features are mapped to a low-dimensional shared semantic space through a two-layer projection network MLP. S42. Calculate the visual components after projection. With speech components cosine similarity matrix ; S43. A cross-modal contrastive loss function is constructed using symmetric cross-entropy loss to maximize the similarity between speech and visual feature pairs of the same semantic category, while minimizing the similarity difference between features of different categories.
[0012] Preferably, the visual component With speech components The expression is: , ; The cosine similarity matrix The formula for calculation is: ; in, This is a temperature scaling factor used to adjust the smoothness of the similarity distribution. For inner product, express Norm; The expression for the cross-modal contrastive loss function is: ; in, To compare the loss items, This represents the cross-entropy loss function. and Let these represent the contrastive losses constructed using the speech modality and the visual modality as queries, respectively: ; ; in, This represents a matrix of labels indicating proper alignment between modalities, ensuring that feature pairs of the same category are closer in the similarity space. For matrix The element represents the first element. The visual sample and the first Alignment relationship between speech samples; when hour, This indicates that the pair is a modal pair belonging to the same semantic category; otherwise... This indicates that the modal pair should not be aligned; It is the dimension of the matrix. and It is the similarity between visual features and speech features.
[0013] Preferably, in S5, the specific content of the bilinear attention network includes: S51. Define the visual features in the enhanced alignment feature. Speech features ,in, and They are respectively and The number of channels; S52, Introducing a low-rank projection matrix and Reduce the dimensionality of visual and speech features. ; S53, Constructing a Higher-Order Interaction Matrix Capture fine-grained interaction relationships between visual and speech modalities; S54. Perform global average pooling on the interaction matrix to obtain preliminary fusion features. ;in: ; ; in, It is a column vector where all elements are 1. It is a learnable weight vector. This indicates the transpose operation. This represents the element-wise Hadamard product. The Sigmoid function represents a non-linear activation function. The specific content of the bilinear attention network also includes classification loss. : ; in, For the sample size, The number of sample categories, It is the first Each sample in category The real label on it.
[0014] Preferably, the specific content of S6 includes: S61. Constructing a joint feature representation , ,in, This represents the concatenation of vectors along the feature dimension; S62. Constructing a weighted mapping function Generate modal fusion weights ,in, The weights of the speech features, Weights for visual features; S63, Computer Vision Features, Speech Features and Semantic Vectors cosine similarity and The weighted similarity is obtained by fusion. ; S64. Construct a semantic label guided loss that minimizes the mean squared error (MSE) based on fusion similarity. The initial fusion features are weighted and enhanced to obtain the final fusion features. .
[0015] in, ; , For learnable projection matrices, , As a bias term, the output weights are constrained to a two-dimensional probability simplex. Internally, ensure that the weights between modes are normalized; ; .
[0016] Finally, the stated emotion autonomous recognition model The expression for the target is transformed into: in, The weight coefficients for semantic label consistency loss are... For cross-modal contrastive learning loss weight coefficients, For the first Each sample is predicted as follows The probability of the category, For the first The semantic similarity of each sample is fused.
[0017] Therefore, the multimodal emotion autonomous recognition method that integrates semantic tags and BAN, as described above, has the following advantages compared with the prior art: (1) This invention maps bimodal features to a shared latent space through semantic labels, and combines cross-modal contrastive learning to bring similar samples closer and push away dissimilar samples to build a unified semantic benchmark. At the same time, a bilinear attention network captures high-order interactions between modalities, forming a collaborative mechanism of "semantic alignment-contrast optimization-high-order fusion", which fully leverages the complementarity of multimodal features and greatly improves the accuracy of emotion recognition compared with traditional multimodal fusion methods; (2) This invention uses ResNet-LSTM and MFCC-LSTM architectures to capture the temporal dynamic dependence of facial expressions and the temporal-frequency variation features of speech intonation, respectively; the bilinear attention network further mines local details within a modality and fine-grained correlations between modalities, effectively identifying key emotional cues such as subtle facial expression changes and intonation fluctuations. Compared with existing methods that only focus on global features, this invention improves the sensitivity of fine-grained emotion recognition by 20%, and greatly improves the accuracy of emotion classification in complex scenarios; (3) The invention introduces semantic label consistency constraints, stabilizes the model output through label-feature consistency measurement, and reduces recognition fluctuations caused by the blurring of emotion category boundaries; cross-modal contrastive learning reduces the distribution differences between modalities and enhances the model's adaptability to different devices, environments, and populations. At the same time, through lightweight designs such as low-rank projection and attention convergence, it can be directly deployed in cross-industry scenarios such as intelligent customer service, medical monitoring, and driving safety, significantly improving scenario adaptability and real-time performance compared to existing technologies.
[0018] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0019] Figure 1 This is a flowchart illustrating an embodiment of a multimodal emotion autonomous recognition method that integrates semantic tags and BAN according to the present invention. Figure 2 This is a flowchart illustrating the ResNet-LSTM visual expression feature extraction process of an embodiment of the multimodal emotion autonomous recognition method integrating semantic tags and BAN according to the present invention. Figure 3 This is a flowchart illustrating the MFCC-LSTM speech intonation feature extraction process, an embodiment of the multimodal emotion autonomous recognition method integrating semantic tags and BAN according to the present invention. Figure 4 This is a schematic diagram illustrating cross-modal comparative learning semantic alignment in an embodiment of a multimodal emotion autonomous recognition method that integrates semantic tags and BAN according to the present invention. Figure 5 This is a schematic diagram illustrating the multimodal consistency constraint driven by semantic tags, as an embodiment of the multimodal emotion autonomous recognition method integrating semantic tags and BAN according to the present invention. Figure 6 This is a flowchart illustrating the multimodal emotion feature fusion using a bilinear attention network, as an embodiment of the present invention, which integrates semantic tags and BAN for autonomous emotion recognition. Detailed Implementation
[0020] Figure 1 A flowchart illustrating the implementation of this invention is provided. To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0021] Example 1 This embodiment provides the working process of the feature extraction model of the present invention.
[0022] like Figure 2 As shown, the specific process of ResNet-LSTM extracting visual features includes: First, learn about emotional videos. Decoded into frame-by-frame images via VideoCapture Then, standardization preprocessing is performed sequentially. The specific content of the preprocessing is: performing standardization, facial region detection and cropping operations on each frame of the image to obtain an image that adapts to the network input. Then, the preprocessed frame images are input into the ResNet-50 backbone network pre-trained on ImageNet, and the network outputs frame-level embedded visual features for each frame. The ResNet-50 backbone network has 5 stages: Stage 1 consists of convolution, batch normalization, ReLU activation, and max pooling operations; Stages 2 to 4 consist of stacking residual identity blocks and convolutional blocks three times; and Stage 5 consists of global average pooling, flattening, and fully connected operations. Then By projecting the additional convolutional mapping layer onto the temporal feature space, the projected temporal features are input into the long short-term memory network to capture the dynamic inter-frame dependencies. Next, normalized weights are generated through an attention mechanism to perform weighted aggregation of the temporal features output by the LSTM; finally, the weighted aggregation features are regularized to generate global visual features. .
[0023] like Figure 3 As shown, the specific process of MFCC-LSTM extracting speech features includes: First, the emotional video The speech in the video is decoded using Librosa, and then normalized and pre-emphasized in sequence to obtain the speech frame signal frame by frame. ; Next step The process involves sequentially performing Fast Fourier Transform, Mel filter bank filtering, and Discrete Cosine Transform to extract frame-level Mel frequency cepstral coefficients, resulting in... ; Then, the speech features of each frame of the MFCC sequence are... The input convolutional layer models local time-frequency correlations, and then the convolutional features are input into a long short-term memory network to capture dynamic dependencies across frames. Then, normalized weights are generated through an attention mechanism to perform weighted aggregation of the temporal features output by the LSTM. Finally, the weighted and aggregated features are regularized to form the final global speech features. .
[0024] Example 2 This embodiment provides the working process of the feature extraction model of the present invention.
[0025] like Figure 4 As shown, visual and speech features are first projected into a shared semantic space to reduce modal heterogeneity: For visual features and speech features A projection layer is used to project features into a low-dimensional space, resulting in projected visual features. and speech features They can be represented as follows: in, and It consists of two projection networks: a hidden layer and a ReLU activated layer. , It is the feature dimension after projection.
[0026] Then, the visual and speech features of samples of the same category are compared and aligned: Modality-invariant semantic representations are learned by minimizing the distance between visual and speech features of the same category and maximizing the distance between visual and speech features of different categories. This is based on projected visual features. and speech features The cosine similarity between them is used to calculate the pairwise similarity matrix. : ; in, This is a temperature scaling factor used to adjust the smoothness of the similarity distribution. For inner product, express Norm.
[0027] Next, cross-modal contrastive loss is used to enhance the alignment between visual and speech features: To improve representation consistency across different modalities, this approach maximizes the similarity between speech and visual feature pairs within the same semantic category while minimizing the similarity differences between features from different categories. Therefore, a cross-modal contrastive loss is employed. Using a symmetrical structure, it can be represented as: ; in, This represents the cross-entropy loss function. and The contrastive losses constructed using the speech modality and the visual modality as queries are respectively given as follows: ; ; in, This represents a matrix of labels indicating proper alignment between modalities, ensuring that feature pairs of the same category are closer in the similarity space. For matrix The element represents the first element. The visual sample and the first Alignment relationship between speech samples. When hour, This indicates that the pair is a modal pair belonging to the same semantic category; otherwise... This indicates that the modal pair should not be aligned. It is the dimension of the matrix. and It is the similarity between the features of the visual modality and the features of the speech modality.
[0028] Example 3 This embodiment provides the implementation process of the semantic information consistency constraint of the present invention.
[0029] like Figure 5 As shown, firstly, a trainable embedding layer is used. Sample Labels Mapped to 3D semantic label vector To construct a multimodal joint representation with semantic enhancement capabilities, we will use visual modal features... Speech modal features With the corresponding semantic label embedding vector The features are concatenated to form a joint feature representation. , can be represented as: ,in, This represents the concatenation of vectors along the feature dimension.
[0030] Then, based on joint features Further construct the weighted mapping function To adaptively generate fusion weights for visual and speech modalities Weighting function It can be represented as: ; in, , For learnable projection matrices, , For bias terms, It is a nonlinear activation function. The output weights are constrained to a two-dimensional probability simplex. Internally, ensure that the weights between modes are normalized.
[0031] Next, we design the semantic label-guided loss: by calculating visual features... Speech features With semantic tags The cosine similarity between visual and speech samples quantifies their alignment in the semantic space. Visual and speech similarity can be defined as follows: ; Fusion similarity of modal semantics (fusion similarity) The result is obtained by weighting the modal weights: ; To improve the consistency between modal features and semantic labels, a semantic label-guided loss defined by minimizing mean squared error (MSE) is used. It can be represented as: ; in, It is the sample size. By introducing... The model can explicitly learn the consistency between modalities and the label semantic space during the fusion stage, thereby effectively improving classification performance.
[0032] Example 4 This embodiment provides the feature fusion implementation process of the bilinear attention network (BAN) of the present invention.
[0033] like Figure 6 As shown, firstly, the interaction matrix between visual features and speech features is obtained through bilinear aggregation: Define visual features and speech modal features The number of channels are respectively and Then the visual features can be represented as Speech features can be represented as To reduce the dimensionality of the two modal features, a low-rank projection matrix is introduced. and ,in Through a bilinear attention mechanism, visual and speech features are mapped to a common feature space to capture fine-grained interactions between visual and speech modalities. The higher-order interaction matrix of the bilinear attention mapping is shown below. It can be represented as: ; in, It is a column vector where all elements are 1. It is a learnable weight vector. This indicates the transpose operation. Represents the Hadamard (element-by-element) product. This represents a non-linear activation function; here, the Sigmoid function is used. This represents the matrix multiplication operation.
[0034] Then, bilinear pooling is used to integrate the weighted features, generating a preliminary fused feature representation. , can be represented as: To further enhance BAN fusion performance, the model uses a semantic label-driven modality weighting mechanism. Specifically, it fuses features... Based on modal dimension, it is divided into speech components. and visual components .
[0035] Modality fusion weights constructed based on semantic tags These correspond to the importance of the speech and visual modalities, respectively. Therefore, the final fusion features enhanced by semantic labels... It can be represented as: ； Final fusion features The data is input into a classifier for predicting the emotion category. This classifier consists of two non-linear transformation layers and a softmax output layer, which generates the emotion category probability of each sample. .
[0036] The classification loss of the BAN module itself Using multi-class cross-entropy loss, it can be expressed as: ; in, For the sample size, The number of sample categories, It is the first Each sample in category The real label on it.
[0037] The goal of the autonomous emotion recognition model J is to simultaneously minimize classification error, modality alignment error, and semantic label consistency error, thereby improving the accuracy and robustness of multimodal emotion recognition.
[0038] in, The weight coefficients for semantic label consistency loss are... The weight coefficients are used for cross-modal contrastive learning loss.
[0039] This embodiment fully presents the feature fusion logic of a bilinear attention network. Through key operations such as low-rank projection dimensionality reduction, high-order interaction matrix construction, bilinear pooling, and semantic weighted enhancement, it achieves deep interaction and efficient fusion of local visual and speech features. This network captures fine-grained intermodal correlations through bilinear mapping and optimizes the fused features using semantically labeled modal weights, ultimately generating fused features that combine local interaction information with global semantic consistency. Simultaneously, it optimizes classification performance through multi-class cross-entropy loss functions, forming a closed loop of "feature interaction - semantic enhancement - classification optimization." This effectively solves the problems of insufficient modal interaction and low feature utilization in traditional fusion methods, providing strong feature support for accurate emotion classification.
[0040] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.
Claims
1. A multimodal autonomous emotion recognition method integrating semantic tags and BAN, characterized in that, Specifically, the following steps are included: S1. Collect emotional videos And extract visual and speech features; S2. Fuse visual and speech features to obtain multimodal joint features; S3. Use semantic tags to align the visual and speech features in the multimodal joint features to obtain aligned joint features; S4. Introduce a cross-modal contrastive learning mechanism to improve global semantic consistency based on aligned joint features, and obtain enhanced aligned features; S5. Input the enhanced alignment features into the bilinear attention network (BAN) to capture the high-order interaction relationship between visual and speech local features, and generate preliminary fusion features through bilinear pooling. S6. Combining semantic information consistency constraints, the preliminary fusion features are weighted and enhanced to generate the final fusion features; S7. Input the final fused features into the classifier, which outputs the probability of the emotion category. Complete the autonomous recognition of emotions; S1-S7 completed the autonomous emotion recognition model. Construction: ; in, For feature extraction models, For multimodal feature fusion models, For semantic alignment model; The specific contents of S2 and S3 include: S21, the multimodal joint feature The defining formula is: ； S22. The feature fusion process is as follows: ,in, express Dimensions express Dimensions express dimensionality Represents dimensional space; S31, Let the emotion category label be... Map it to dimensional vector ,in Emotional tags semantic vectors, It is an embedded function; S32, will and Union, resulting in aligned joint features The expression is: ; In S5, the specific content of the bilinear attention network includes: S51. Define the visual features in the enhanced alignment feature. Speech features ,in, and They are respectively and The number of channels; S52, Introducing a low-rank projection matrix and Reduce the dimensionality of visual and speech features. ; S53, Constructing a Higher-Order Interaction Matrix Capture fine-grained interaction relationships between visual and speech modalities; S54. Perform global average pooling on the interaction matrix to obtain preliminary fusion features. ; The specific content of the bilinear attention network also includes classification loss. ; The specific content of S6 includes: S61. Constructing a joint feature representation S62. Constructing a weighted mapping function Generate modal fusion weights ,in, The weights of the speech features, Weights for visual features; S63, Computer Vision Features, Speech Features and Semantic Vectors cosine similarity and The weighted similarity is obtained by fusion. ; S64. Construct a semantic label guided loss that minimizes the mean squared error (MSE) based on fusion similarity. The initial fusion features are weighted and enhanced to obtain the final fusion features. ;in, For speech components, For visual components.
2. The multimodal emotion autonomous recognition method integrating semantic tags and BAN according to claim 1, characterized in that, The feature extraction model Including visual feature extraction models and auditory feature extraction model , and global speech features Represented as: ； ； in, The number of image frames. The number of audio frames. and It is a normalized weight function generated by the attention mechanism. and These represent the extracted visual features and speech features for each frame, respectively.
3. The multimodal emotion autonomous recognition method integrating semantic tags and BAN according to claim 1, characterized in that, In S1, visual features are extracted using ResNet-50 combined with LSTM network ResNet-LSTM, and speech features are extracted using MFCC combined with LSTM network MFCC-LSTM.
4. The multimodal emotion autonomous recognition method integrating semantic tags and BAN according to claim 3, characterized in that, The specific content of extracting visual and speech features includes: S11, Visual Feature Extraction: S111, Learning Emotional Videos VideoCapture decodes each frame into an image frame signal. Then, standardized preprocessing is performed sequentially; S112, the pre-processed Input a ResNet-50 backbone network pre-trained on ImageNet and output frame-level embedded visual features for each frame. ; S113, The extra convolutional mapping layer is projected onto the temporal feature space and input into the Long Short-Term Memory (LSTM) network to capture inter-frame dynamic dependencies, which are then weighted and converged through an attention pooling mechanism. S114. Obtain global visual features after regularization. ; The ResNet-50 backbone network includes: convolutional, batch normalization, residual identity blocks, and convolutional blocks; S12. Speech Feature Extraction: S121, Emotional Video The speech in the video is decoded frame by frame using Librosa, and then normalized and pre-emphasis filtered sequentially to obtain the speech frame signal. ; S122. Extract the frame-level Mel frequency cepstral coefficients (MFCC) using the standard short-time spectrum analysis procedure. Speech features of each frame in the S123 and MFCC sequences Local time-frequency correlations are modeled using convolutional layers, and cross-frame dynamic dependencies are captured using LSTM. S124. The temporal features are weighted and converged using the attention pooling mechanism, and then regularized to form the final global speech features. .
5. The multimodal emotion autonomous recognition method integrating semantic tags and BAN according to claim 1, characterized in that, In S4, the specific content of the cross-modal contrastive learning mechanism includes: S41. Visual features and speech features in the aligned joint features are mapped to a low-dimensional shared semantic space through a two-layer projection network MLP. S42. Calculate the visual components after projection. With speech components cosine similarity matrix ; S43. A cross-modal contrastive loss function is constructed using symmetric cross-entropy loss to maximize the similarity between speech and visual feature pairs of the same semantic category, while minimizing the similarity difference between features of different categories.
6. The multimodal emotion autonomous recognition method integrating semantic tags and BAN according to claim 5, characterized in that, The cosine similarity matrix The formula for calculation is: ； in, This is a temperature scaling factor used to adjust the smoothness of the similarity distribution. For inner product, express Norm, For visual components and For speech components; The expression for the cross-modal contrastive loss function is: ； in, To compare the loss items, Represents the cross-entropy loss function. and represents the contrastive losses constructed using the speech modality and the visual modality as queries, respectively. A matrix of identifier labels indicating correct alignment between modalities.
7. The multimodal emotion autonomous recognition method integrating semantic tags and BAN according to claim 1, characterized in that, The emotion autonomous recognition model The expression for the target is transformed into: in, The weight coefficients for semantic label consistency loss are... For cross-modal contrastive learning loss weight coefficients, To minimize the classification loss, For semantic alignment loss, For the sample size, The number of sample categories, It is the first Each sample in category The real labels on For matrix elements, and It is the similarity between visual features and speech features. For the first Each sample is predicted as follows The probability of the category, For the first The semantic similarity of each sample is fused.