Image-text multi-modal emotion classification method based on cross-modal and double-width learning

Abstract

The invention discloses an image-text multi-mode emotion classification method based on cross-mode and double-width learning, and belongs to the technical field of image-text processing emotion classification. The image-text multi-mode emotion classification method based on cross-mode and double-width learning comprises the following steps: S01, single-mode coding: extracting features from a text mode and an image mode; s02, cross-modal interaction feature extraction: extracting modal unified features, text specific features, visual specific features and modal interaction features; s03, performing feature fusion: performing fusion on multi-modal features, and performing fusion on the features by adopting double-width learning; and S04, emotion prediction: obtaining an emotion classification result through cooperative training. The detection result of the image-text multi-modal emotion classification method is obviously superior to that of an existing related comparison model, and the problems that in an existing emotion classification method, fine-grained features in modals are not sufficiently extracted, and a cross-modal interaction fusion mechanism is low in efficiency are solved.

Description

Technical Field

[0001] This invention belongs to the field of image and text processing emotion classification technology, and in particular relates to an image and text multimodal emotion classification method based on cross-modality and dual-broad learning (CMDBL). Background Technology

[0002] With the widespread adoption of social networks, online platforms, and smart devices, user-generated images, videos, and accompanying text and voice data have exploded. Against this backdrop, multimodal emotion classification technology, capable of comprehensively utilizing multiple information sources, has become a key task in the fields of natural language processing and human-computer interaction. Compared to unimodal emotion classification that relies solely on text, multimodal emotion classification aims to integrate information from multiple modalities, including text, visual (such as images and video frames), and voice, to more comprehensively and accurately identify users' emotional states or sentiment tendencies. This technology has broad application prospects in areas such as public opinion monitoring, intelligent customer service, online education, precision recommendation, and medical and health auxiliary diagnosis, and has therefore attracted widespread attention from academia and industry.

[0003] To address this task, existing technologies generally employ deep learning-based approaches. Early research focused primarily on designing specific neural network architectures to process data from different modalities separately and attempting to fuse them. Specifically, existing methods can be broadly categorized as follows: frameworks based on multi-task learning, methods based on graph neural networks, models based on convolutional neural networks, architectures based on Transformers, and models incorporating multi-head attention mechanisms. These methods have improved the performance of emotion classification to some extent, demonstrating the effectiveness of multimodal fusion.

[0004] However, during their in-depth research and practice, the inventors discovered that a core challenge remains unresolved in existing technologies: how to efficiently and collaboratively extract deep features within and between modes. Specifically:

[0005] Insufficient intramodal feature extraction: Existing methods often fail to fully extract the fine-grained information highly correlated with emotion inherent in a single modality (such as text, image, or speech) when extracting features independently. The robustness and discriminativeness of intramodal features still need to be improved.

[0006] Inefficient intermodal feature interaction and fusion: Complex complementary and correlated relationships exist between information from different modalities. Current technologies lack effective solutions for achieving efficient semantic alignment between modalities, capturing cross-modal collaborative interaction information, and dynamically weighing the contributions of different modalities in specific contexts. Simple feature concatenation or shallow attention mechanisms cannot achieve true "comprehensive understanding," leading to information fusion bottlenecks and limiting further improvements in model accuracy and robustness.

[0007] Cross-modal technologies are considered a key path to address the aforementioned challenges, aiming to overcome the limitations of single-modal perception through mechanisms such as information complementarity, semantic alignment, dynamic weighting, and robust optimization. Although there is a consensus in this direction, existing technologies have not yet provided a mature solution that can systematically and simultaneously optimize intra-modal feature representation and inter-modal feature interaction within a unified framework.

[0008] Therefore, there is an urgent need for an innovative multimodal emotion classification method to overcome the shortcomings of existing technologies and achieve more accurate and robust emotion recognition. Summary of the Invention

[0009] This invention provides a text-image multimodal emotion classification method based on cross-modal and dual-width learning, aiming to address the shortcomings in intra-modal feature extraction and inter-modal feature fusion, which limit the accuracy and robustness of emotion classification.

[0010] The technical solution of this invention is implemented as follows:

[0011] A multimodal emotion classification method for images and text based on cross-modal and dual-width learning includes the following steps:

[0012] S01. Single-modal coding: Responsible for extracting features from text modalities and image modalities; for text modalities, a BERT pre-trained model is used for encoding; the image modalities include emoticons in images and text, for emoticons, a PIXEL pre-trained model is used for encoding, for images, a Deeplabv3 pre-trained model is used to extract foreground image features and background image features, and finally a CLIP pre-trained model is used to uniformly encode the image modal features;

[0013] S02. Cross-modal interaction feature extraction: Extract modality-unified features (MUF), text-specific features (TSF), visual-specific features (VSF), and modality-interactive features (MIF). MLP-Transformer is used to extract MUF, TSF, and VSF features, while Cross-Attention is used to extract MIF features. Multi-kernel Maximum Mean Discrepancy (MK-MMD) is used to measure the distribution differences between modalities.

[0014] S03. Feature Fusion: Responsible for fusing multimodal features, using double-width learning to fuse these features;

[0015] S04. Sentiment Prediction: Sentiment classification is achieved through collaborative training and loss optimization based on fusion features.

[0016] The single-modal encoding in step S01 is to obtain image and text features for multimodal fusion, and different encoders are used to extract features from the image and text respectively.

[0017] In step S01, for the text modality, the BERT pre-trained model is used to extract text features H. t , means as follows:

[0018]

[0019] in, This represents the input text. express The parameter N t D represents the number of samples in the input text. t This indicates the dimension of the BERT output.

[0020] Since text often contains emojis, emojis are encoded using a pre-trained PIXEL model, specifically represented as follows:

[0021]

[0022] in, N represents the parameter of PIXEL. e D represents the number of emojis. v1 This indicates the dimension of the PIXEL output.

[0023] Then, since the background features of the image can affect emotion recognition, the DeepLabv3 model is used to analyze the foreground image features. and background image features Extraction is performed, specifically as follows:

[0024]

[0025] in, This represents the original input image. N represents the parameters of DeepLabv3. v D represents the number of input images. v2 This indicates the dimension of the DeepLabv3 output.

[0026] Finally, the CLIP pre-trained model is used to uniformly encode the emojis, foreground images, and background images of the image modality. Specifically, it is expressed as follows:

[0027]

[0028] in, The original image is the input. for Extracted background image features for Extracted foreground image features, for The parameter N v =max{N t , N e , N v}, D v =max{D v1 D v2 D v}

[0029] The cross-modal interaction feature extraction in step S02 aims to effectively acquire interaction feature information in the text modality and the image modality. Based on the domain adaptation concept, the text modality and the image modality are regarded as two domains with different feature distributions, and their common features and unique features are extracted simultaneously.

[0030] In step S02, by mapping the text modality and image modality to feature subspaces with smaller distribution distances, the potential commonalities between images and text at the emotional and motivational levels are extracted to obtain domain-shared features. These shared features can characterize the inherent consistency of emotional expression among different modalities. At the same time, since each modality has different emotional expression methods, it is necessary to map the text modality and image modality to feature subspaces with larger distribution distances to effectively capture unique emotional features.

[0031] Furthermore, multimodal sentiment analysis requires not only focusing on the unique expressions of each modality but also delving into the connections and complementarities between modalities. Therefore, a cross-attention mechanism is employed to capture interaction information between different modalities, enabling a particular modality to dynamically acquire highly relevant emotional cues from another modality during feature extraction, thereby enhancing the ability to extract multimodal features.

[0032] The modality unification features are extracted using a DUE (Domain-Unified Encoder, DUE) module constructed with MLP-Transformer. Specifically, it is expressed as follows:

[0033]

[0034] Where N represents the number of input samples, and D represents the dimension of the MLP-Transformer output. The parameters of the MLP-Transformer are represented by N = max{N t , N v}, D=max{D t D v1 D v2 D v}

[0035] The text-specific and visual-specific features are obtained by constructing a VSE (Visual-Specific Encoder, VSE) module and a TSE (Text-Specific Encoder, TSE) module using two MLP-Transformers, as shown below:

[0036]

[0037] in Indicates text-specific features, Indicates unique features of the image, The parameters of the MLP-Transformer represent the text-specific features. The parameters of the MLP-Transformer represent the image-specific features.

[0038] The modal interaction features are extracted using a Cross-Attention module to construct a Domain-Interactive Encoder (DIE). Specifically, it is expressed as follows:

[0039]

[0040] Where N represents the number of input samples, and D represents the dimension of the Cross-Attention output. This represents the parameters of Cross-Attention in modal interaction features.

[0041] Finally, to obtain modality-adaptive features, it is necessary not only to maximize the modality-specific features and modality-uniform features of the same modality to obtain their commonalities, but also to maximize the modality-specific features of different modalities to obtain their unique features. Multi-kernel maximum mean difference (MK-MMD) is used to measure the distance between features, as detailed below:

[0042]

[0043] in, It is a cross-modal adaptive loss function.

[0044] In step S03, feature fusion is performed using Dual Broad Learning (DBL) to better integrate the acquired features. DBL aims to extract interaction information between images and text through high-dimensional linear and non-linear mappings, thereby achieving efficient feature fusion and sentiment classification. The dual-width learning includes a first width learning and a second width learning.

[0045] In step S03, for the first width learning, the features are... and After concatenation, the input is used as the data, and then transformed through a linear mapping. Group of feature nodes, i-th group of feature nodes It is expressed as follows:

[0046]

[0047] in, Represents a linear activation function. This represents the number of nodes in each group of feature nodes. and Let represent the weight matrix and bias matrix randomly generated for the i-th group of feature nodes, respectively.

[0048] Will The feature nodes are concatenated to obtain a linear feature matrix. Specifically, it is expressed as follows:

[0049]

[0050] After nonlinear mapping transformation, it becomes Group of augmented nodes, where the j-th group of augmented nodes It is expressed as follows:

[0051]

[0052] in, Represents a non-linear activation function. This represents the number of nodes in each group of augmented nodes. and Let represent the weight matrix and bias matrix randomly generated for the j-th group of feature nodes, respectively.

[0053] Will The augmented nodes are concatenated to obtain the feature matrix of the augmented nodes. Specifically, it is expressed as follows:

[0054]

[0055] Will and By concatenating the features, the complete feature matrices of the text and image modalities are obtained. Specifically, it is expressed as follows:

[0056]

[0057] For the second width learning, the features and After concatenation, the input is used as the data, and then transformed through a linear mapping. Group of feature nodes, k-th group of feature nodes It is expressed as follows:

[0058]

[0059] in, Represents a linear activation function. This represents the number of nodes in each group of feature nodes. and Let represent the weight matrix and bias matrix randomly generated for the k-th group of feature nodes, respectively.

[0060] Will The feature nodes are concatenated to obtain a linear feature matrix. Specifically, it is expressed as follows:

[0061]

[0062] After nonlinear mapping transformation, it becomes Group of augmented nodes, where the l-th group of augmented nodes It is expressed as follows:

[0063]

[0064] in, Represents a non-linear activation function. This represents the number of nodes in each group of augmented nodes. and Let represent the weight matrix and bias matrix randomly generated for the l-th group of feature nodes, respectively.

[0065] Will The groups of augmented nodes are concatenated to obtain a matrix of augmented nodes. Specifically, it is expressed as follows:

[0066]

[0067] Will and By concatenating the features, the complete feature matrices of the text and image modalities are obtained. Specifically, it is expressed as follows:

[0068]

[0069] In order to effectively integrate features , , , The feature matrix needs to be , By splicing the components together, a fusion matrix is ​​obtained. Specifically, it is expressed as follows:

[0070]

[0071] Therefore, the approximate output of dual-width learning DBL As shown below:

[0072]

[0073] in, This indicates an approximate output. Represents the weight matrix. This indicates the number of emotion categories. Step S04 involves co-training for emotion prediction:

[0074] In step S04, to obtain the optimal weights of the output layer, ridge regression is used as the objective function, as shown below:

[0075]

[0076] in, , , This represents the regularization coefficient. , , Indicates the actual output;

[0077] Based on regularized least squares method , , It is expressed as follows:

[0078]

[0079] Where I represents the identity matrix, This represents the transpose of the matrix; summing the above outputs yields the actual output Y of the cross-modal and double-width learning CMDBL, as shown below:

[0080]

[0081] Finally, the cross-entropy loss is used as the classification loss, as shown below:

[0082]

[0083] in, Represents classification loss. Represents the comprehensive loss function. This represents the cross-modal adaptive loss function. The weights represent the classification loss. The weights represent the cross-modal adaptive loss.

[0084] The image-text multimodal emotion classification method provided by this invention, based on cross-modal and dual-width learning, brings the following significant benefits by systematically and collaboratively optimizing intramodal feature extraction, intermodal interaction, and multi-granularity feature fusion:

[0085] 1. At the feature extraction level: More refined and comprehensive multimodal representations are achieved, fundamentally improving the model's perceptual capabilities. Fine-grained single-modal feature extraction: By employing various dedicated pre-trained models such as BERT, PIXEL, DeepLabv3, and CLIP, targeted encoding is performed on text, emojis, and image foreground and background, enabling in-depth mining of fine-grained discriminative features related to emotion within each modality. In particular, the specialized processing of emojis and image backgrounds effectively captures implicit and contextual information that is easily overlooked by traditional methods but has a crucial impact on emotion judgment, providing a higher-quality and more robust single-modal representation foundation for subsequent fusion.

[0086] 2. At the cross-modal interaction level: Deep modeling and decoupling of complex relationships between modalities are achieved, breaking through the bottleneck of information fusion. Effective decoupling and alignment of features: By introducing domain adaptation, and utilizing MLP-Transformer and MK-MMD loss functions, multimodal features are innovatively decoupled into "modal unified features," "modal-specific features," and "modal interaction features." This decoupling mechanism can: capture potential commonalities across modalities (unified features), ensuring the model understands the consistency of emotional expression across different modalities; retain the unique expression methods of each modality (specific features), avoiding the information annihilation of weak modalities by strong modalities; explicitly model the dynamic complementary relationships between modalities (interaction features), enabling one modality to dynamically focus on highly relevant emotional cues in another modality through a cross-attention mechanism; and enhance the semantic understanding depth of the model: This design elevates the model from simple feature stacking to a structured and semantic understanding of multimodal information, laying a solid foundation for achieving "comprehensive understanding."

[0087] 3. At the feature fusion and decision-making level: Efficient, robust, and interpretable fusion and classification are achieved. The efficiency and performance advantages of dual-width learning: Employing a dual-width learning model for fusion offers advantages such as high computational efficiency, fast training speed, and low overfitting risk. By using two parallel width learning networks to process "unified and interactive features" and "specific features" respectively, multi-granularity and multi-perspective feature fusion are achieved, enabling more full utilization of decoupled feature information and thus significantly improving classification accuracy.

[0088] 4. Co-training enhances the model's generalization and robustness: Through co-training of the three output heads and ridge regression optimization, the model can integrate prediction results from different feature views. This mechanism effectively improves the model's generalization ability. When a certain modality signal is missing or contains noise (such as a blurry image or text containing typos), the model can adaptively compensate and correct through strong signals from other modalities and their interactions, thus maintaining stable and reliable performance in complex and ever-changing real-world application scenarios.

[0089] 5. The decision-making process is more targeted and interpretable: The final classification result is jointly determined by unified features, specific features and interaction features, so that the model's prediction is no longer a "black box" but can be traced back to the contributions of different feature sources, which enhances the transparency and interpretability of the decision-making process.

[0090] In summary, this invention effectively overcomes the core shortcomings of existing technologies, such as insufficient intramodal feature extraction, shallow intermodal relationship modeling, and inefficient fusion mechanisms, through a complete end-to-end solution. It achieves comprehensive improvement in multiple dimensions, including accuracy, robustness, efficiency, and interpretability, and possesses extremely high academic value and promising industrial application prospects. Attached Figure Description

[0091] Figure 1 This is a schematic diagram of a text-image multimodal emotion classification method based on cross-modal and dual-width learning provided in an embodiment of the present invention. Detailed Implementation

[0092] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, if the embodiments of the present invention involve descriptions such as "first" and "second", such descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one of such features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0093] Current emotional classification methods suffer from the following drawbacks: inadequate intra-modal feature extraction and inter-modal feature fusion, which limits the accuracy and robustness of emotional classification. To address these issues, this invention proposes a multimodal emotional classification method for images and text based on cross-modal and dual-width learning.

[0094] Example 1

[0095] like Figure 1 As shown, a multimodal emotion classification method based on cross-modal and dual-width learning for images and text includes the following steps:

[0096] S01. Single-modal coding: Responsible for extracting features from text modalities and image modalities; for text modalities, a BERT pre-trained model is used for encoding; the image modalities include emoticons in images and text, for emoticons, a PIXEL pre-trained model is used for encoding, for images, a Deeplabv3 pre-trained model is used to extract foreground image features and background image features, and finally a CLIP pre-trained model is used to uniformly encode the image modal features;

[0097] S02. Cross-modal interaction feature extraction: Extract modality-unified features, text-specific features, visual-specific features, and modal interaction features; use MLP-Transformer to extract modality-unified features, text-specific features, and visual-specific features, and use Cross-Attention to extract modal interaction features; use multi-kernel maximum mean difference to measure the distribution differences between modalities;

[0098] S03. Feature Fusion: Responsible for fusing multimodal features, using double-width learning to fuse these features;

[0099] S04. Emotion Prediction: Obtain the classification results of emotions through collaborative training.

[0100] The single-modal encoding in step S01 is to obtain image and text features for multimodal fusion, and different encoders are used to extract features from the image and text respectively.

[0101] In step S01, for the text modality, the BERT pre-trained model is used to extract text features H. t , means as follows:

[0102]

[0103] in, This represents the input text. express The parameter N t D represents the number of samples in the input text. t This indicates the dimension of the BERT output.

[0104] Since text often contains emojis, emojis are encoded using a pre-trained PIXEL model, specifically represented as follows:

[0105]

[0106] in, N represents the parameter of PIXEL. e D represents the number of emojis. v1 This indicates the dimension of the PIXEL output.

[0107] Then, since the background features of the image can affect emotion recognition, the DeepLabv3 model is used to analyze the foreground image features. and background image features Extraction is performed, specifically as follows:

[0108]

[0109] in, This represents the original input image. N represents the parameters of DeepLabv3. v D represents the number of input images. v2 This indicates the dimension of the DeepLabv3 output.

[0110] Finally, the CLIP pre-trained model is used to uniformly encode the emojis, foreground images, and background image features of the image modality. Specifically, it is expressed as follows:

[0111]

[0112] in, The original image is the input. for Extracted background image features for Extracted foreground image features, for The parameter N v =max{N t , N e , N v}, D v =max{D v1 D v2 D v}

[0113] The cross-modal interaction feature extraction in step S02 aims to effectively acquire interaction feature information in the text modality and the image modality. Based on the domain adaptation concept, the text modality and the image modality are regarded as two domains with different feature distributions, and their common features and unique features are extracted simultaneously.

[0114] In step S02, by mapping the text modality and image modality to feature subspaces with smaller distribution distances, the potential commonalities between images and text at the emotion and motivation levels are extracted to obtain domain-shared features. These shared features can characterize the inherent consistency of emotional expression among different modalities. Meanwhile, since each modality has different ways of expressing emotion—for example, images may convey emotion through color changes, while text relies on specific words or phrases—it is necessary to map the text modality and image modality to feature subspaces with larger distribution distances to effectively capture their unique emotional features.

[0115] Furthermore, multimodal sentiment analysis requires not only focusing on the unique expressions of each modality but also delving into the connections and complementarities between modalities. Therefore, a cross-attention mechanism is employed to capture interaction information between different modalities, enabling a particular modality to dynamically acquire highly relevant emotional cues from another modality during feature extraction, thereby enhancing the ability to extract multimodal features.

[0116] First, in order to obtain modal uniformity features The DUE (Domain-Unified Encoder) module is constructed using MLP-Transformer to extract modal unified features. Specifically, it is expressed as follows:

[0117]

[0118] Where N represents the number of input samples, and D represents the dimension of the MLP-Transformer output. The parameters of the MLP-Transformer are represented by N = max{N t , N v}, D=max{D t D v1 D v2 D v}

[0119] Secondly, two MLP-Transformers are used to construct VSE (Visual-Specific Encoder, VSE) and TSE (Text-Specific Encoder, TSE) respectively to obtain unique features, as shown below:

[0120]

[0121] in Indicates text-specific features, Indicates unique features of the image, The parameters of the MLP-Transformer represent the text-specific features. The parameters of the MLP-Transformer represent the image-specific features.

[0122] Then, in order to effectively extract modal interaction features Cross-Attention is used to construct the Domain-Interactive Encoder (DIE), as shown below:

[0123]

[0124] Where N represents the number of input samples, and D represents the dimension of the Cross-Attention output. The parameters represent Cross-Attention in domain interaction features.

[0125] Finally, to obtain modality-adaptive features, it is necessary not only to maximize the modality-specific features and modality-uniform features of the same modality to obtain their commonalities, but also to maximize the modality-specific features of different modalities to obtain their unique features. MK-MMD is used to measure the distance between these features, as shown below:

[0126]

[0127] in, It is a cross-modal adaptive loss function.

[0128] In step S03, feature fusion is performed using Dual Broad Learning (DBL) to better integrate the acquired features. DBL aims to extract interaction information between images and text through high-dimensional linear and non-linear mappings, thereby achieving efficient feature fusion and sentiment classification. The dual-width learning includes a first width learning and a second width learning.

[0129] In step S03, for the first width learning, the features are... and After being concatenated, it serves as its input, and is then transformed through a linear mapping. Group of feature nodes, i-th group of feature nodes It is expressed as follows:

[0130]

[0131] in, Represents a linear activation function. This represents the number of nodes in each group of feature nodes. and Let represent the weight matrix and bias matrix randomly generated for the i-th group of feature nodes, respectively.

[0132] Will The feature nodes are concatenated to obtain a linear feature matrix. Specifically, it is expressed as follows:

[0133]

[0134] After nonlinear mapping transformation, it becomes Group of augmented nodes, where the j-th group of augmented nodes It is expressed as follows:

[0135]

[0136] in, Represents a non-linear activation function. This represents the number of nodes in each group of augmented nodes. and Let represent the weight matrix and bias matrix randomly generated for the j-th group of feature nodes, respectively.

[0137] Will The augmented nodes are concatenated to obtain the feature matrix of the augmented nodes. Specifically, it is expressed as follows:

[0138]

[0139] Will and By concatenating the features, the complete feature matrices of the text and image modalities are obtained. Specifically, it is expressed as follows:

[0140]

[0141] For the second width learning, the features and After being concatenated, it serves as its input, and is then transformed through a linear mapping. Group of feature nodes, k-th group of feature nodes It is expressed as follows:

[0142]

[0143] in, Represents a linear activation function. This represents the number of nodes in each group of feature nodes. and Let represent the weight matrix and bias matrix randomly generated for the k-th group of feature nodes, respectively.

[0144] Will The feature nodes are concatenated to obtain a linear feature matrix. Specifically, it is expressed as follows:

[0145]

[0146] After nonlinear mapping transformation, it becomes Group of augmented nodes, where the l-th group of augmented nodes It is expressed as follows:

[0147]

[0148] in, Represents a non-linear activation function. This represents the number of nodes in each group of augmented nodes. and Let represent the weight matrix and bias matrix randomly generated for the l-th group of feature nodes, respectively.

[0149] Will The groups of augmented nodes are concatenated to obtain a matrix of augmented nodes. Specifically, it is expressed as follows:

[0150]

[0151] Concatenating F2 and E2 yields the complete feature matrix A2 for both text and image modalities, as shown below:

[0152]

[0153] In order to effectively integrate features , , , The feature matrix needs to be , By splicing the components together, a fusion matrix is ​​obtained. Specifically, it is expressed as follows:

[0154]

[0155] Therefore, the approximate output of DBL , , As shown below:

[0156]

[0157] in, , This indicates an approximate output. Represents the weight matrix. This indicates the number of emotion categories.

[0158] Step S04: Co-Training for Sentiment Prediction

[0159] In step S04, to obtain the optimal weights of the output layer, ridge regression is used as the objective function, as shown below:

[0160]

[0161] in, , , This represents the regularization coefficient. , , The fraction represents the actual output;

[0162] Based on regularized least squares method , , It is expressed as follows:

[0163]

[0164] Where I represents the identity matrix, This indicates transposing the matrix;

[0165] Adding the above outputs together, we can obtain the actual output Y of CMDBL, as follows:

[0166]

[0167] Finally, the cross-entropy loss is used as the classification loss, as shown below:

[0168]

[0169] in, Represents classification loss. Represents the comprehensive loss function. This represents the domain-adaptive loss function. The weights represent the classification loss. The weights represent the domain adaptive loss.

[0170] Emotion classification results were obtained through collaborative training; specific classification results include: Happiness, Anticipation, Sadness, Anger, Fear, Disgust, and Surprise.

[0171] The experimental setup and relevant detection data for the above-mentioned image-text multimodal emotion classification method based on cross-modal and dual-width learning are as follows:

[0172] (1) The dataset setup: Table 1 shows the corpus used for the text-image multimodal emotion classification method based on cross-modal and dual-width learning in this application.

[0173] Table 1 lists the corpora used in this application.

[0174]

[0175] (2) Model selection for comparison: In order to verify the performance of the image-text multimodal sentiment classification method based on cross-modal and dual-width learning in this application, it is compared with some mainstream and advanced baseline models of multimodal sentiment analysis. The baseline models of multimodal sentiment analysis include HSAN, MultiSentiNet, CoMN, MVAN, CNN-Multi and CTMWA. The specific sources of the above baseline models are as follows:

[0176] HSAN: [N. Xu. Analyzing multimodal public sentiment based on onhierarchical semantic attentional network. Proceedings of the 2017 IEEEInternational Conference on Intelligence and Security Informatics (ISI 2017), 2017, pp. 152-154.]

[0177] MultiSentiNet: [N. Xu and W. Mao. MultiSentiNet: A deep semantic network for multimodal sentiment analysis. Proceedings of the 2017 ACM onConference on Information and Knowledge Management (CIKM 2017), 2017, pp.2399- 2402.]

[0178] CoMN: [N. Xu, W. Mao, and G. Chen. A co-memory network for multimodalsentiment analysis. Proceedings of the 41st International ACM SIGIRConference on Research & Development in Information Retrieval (SIGIR 2018).2018, pp. 929-932.]

[0179] MVAN: [X. Yang, S. Feng, D. Wang and Y. Zhang. Image-Text MultimodalEmotion Classification via Multi-View Attentional Network. IEEE Transactionson Multimedia, vol. 23, pp. 4014-4026, 2021]

[0180] CNN-Multi: [Cai G, Xia B. Convolutional neural networks for multimedia sentiment analysis. Proceedings of the 4 th Natural Language Processing and Chinese Computing (NLPCC 2015), 2015, pp. 159-167.]

[0181] CTMWA: [Zhang B, Yuan Z, Xu H, et al. Crossmodal Translation BasedMeta Weight Adaption for Robust Image-Text Sentiment Analysis. IEEETransactions on Multimedia, vol. 26, pp. 9949-9961, 2024.]

[0182] (3) The experimental results are shown in Table 2;

[0183] Table 2 shows the test results of this application and the comparative model.

[0184]

[0185] The test results show that the image-text multimodal emotion classification method based on cross-modal and dual-width learning adopted in this application significantly outperforms existing comparative models in terms of detection results (Acc, 87.34, F1, 87.19). This addresses the problems of insufficient intra-modal fine-grained feature extraction and inefficient cross-modal interaction fusion mechanisms in existing emotion classification methods.

[0186] The image-text multimodal emotion classification method provided by this invention, based on cross-modal and dual-width learning, brings the following significant benefits by systematically and collaboratively optimizing intramodal feature extraction, intermodal interaction, and multi-granularity feature fusion:

[0187] 1. At the feature extraction level: More refined and comprehensive multimodal representations are achieved, fundamentally improving the model's perceptual capabilities. Fine-grained single-modal feature extraction: By employing various dedicated pre-trained models such as BERT, PIXEL, DeepLabv3, and CLIP, targeted encoding is performed on text, emojis, and image foreground and background, enabling in-depth mining of fine-grained discriminative features related to emotion within each modality. In particular, the specialized processing of emojis and image backgrounds effectively captures implicit and contextual information that is easily overlooked by traditional methods but has a crucial impact on emotion judgment, providing a higher-quality and more robust single-modal representation foundation for subsequent fusion.

[0188] 2. At the cross-modal interaction level: Deep modeling and decoupling of complex relationships between modalities are achieved, breaking through the bottleneck of information fusion. Effective decoupling and alignment of features: By introducing domain adaptation, and utilizing MLP-Transformer and MK-MMD loss functions, multimodal features are innovatively decoupled into "modal unified features," "modal-specific features," and "modal interaction features." This decoupling mechanism can: capture potential commonalities across modalities (unified features), ensuring the model understands the consistency of emotional expression across different modalities; retain the unique expression methods of each modality (specific features), avoiding the information annihilation of weak modalities by strong modalities; explicitly model the dynamic complementary relationships between modalities (interaction features), enabling one modality to dynamically focus on highly relevant emotional cues in another modality through a cross-attention mechanism; and enhance the semantic understanding depth of the model: This design elevates the model from simple feature stacking to a structured and semantic understanding of multimodal information, laying a solid foundation for achieving "comprehensive understanding."

[0189] 3. At the feature fusion and decision-making level: Efficient, robust, and interpretable fusion and classification are achieved. The efficiency and performance advantages of dual-width learning: Employing a dual-width learning model for fusion offers advantages such as high computational efficiency, fast training speed, and low overfitting risk. By using two parallel width learning networks to process "unified and interactive features" and "specific features" respectively, multi-granularity and multi-perspective feature fusion are achieved, enabling more full utilization of decoupled feature information and thus significantly improving classification accuracy.

[0190] 4. Co-training enhances the model's generalization and robustness: Through co-training of the three output heads and ridge regression optimization, the model can integrate prediction results from different feature views. This mechanism effectively improves the model's generalization ability. When a certain modality signal is missing or contains noise (such as a blurry image or text containing typos), the model can adaptively compensate and correct through strong signals from other modalities and their interactions, thus maintaining stable and reliable performance in complex and ever-changing real-world application scenarios.

[0191] 5. The decision-making process is more targeted and interpretable: The final classification result is jointly determined by unified features, specific features and interaction features, so that the model's prediction is no longer a "black box" but can be traced back to the contributions of different feature sources, which enhances the transparency and interpretability of the decision-making process.

[0192] In summary, this invention effectively overcomes the core shortcomings of existing technologies, such as insufficient intramodal feature extraction, shallow intermodal relationship modeling, and inefficient fusion mechanisms, through a complete end-to-end solution. It achieves comprehensive improvement in multiple dimensions, including accuracy, robustness, efficiency, and interpretability, and possesses extremely high academic value and promising industrial application prospects.

[0193] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal emotion classification method for images and text based on cross-modal and dual-width learning, characterized in that: Includes the following steps: S01. Single-modal coding: Responsible for extracting features from text modalities and image modalities; for text modalities, a BERT pre-trained model is used for encoding; the image modalities include emoticons in images and text, for emoticons, a PIXEL pre-trained model is used for encoding, for images, a Deeplabv3 pre-trained model is used to extract foreground image features and background image features, and finally a CLIP pre-trained model is used to uniformly encode the image modal features; S02. Cross-modal interaction feature extraction: Extract modality-unified features, text-specific features, visual-specific features, and modal interaction features. MLP-Transformer is used to extract modality-unified features, text-specific features, and visual-specific features, and a cross-attention mechanism is used to extract modal interaction features. Multi-kernel maximum mean difference (MK-MMD) is used to measure the distribution differences between modalities. S03. Feature Fusion: Responsible for fusing multimodal features, using double-width learning to fuse these features; S04. Sentiment Prediction: Sentiment classification is achieved through collaborative training and loss optimization based on fusion features.

2. The image-text multimodal emotion classification method based on cross-modal and dual-width learning according to claim 1, characterized in that: In step S01, for the text modality, the BERT pre-trained model is used to extract text features H. t ; For emojis, a PIXEL pre-trained model is used for encoding; Using the DeepLabv3 model to analyze foreground image features and background image features Extract; The CLIP pre-trained model is used to uniformly encode emoticons, foreground images, and background images of image modalities. .

3. The image-text multimodal emotion classification method based on cross-modal and dual-width learning according to claim 2, characterized in that: In step S02, by mapping the text modality and image modality to feature subspaces with smaller distribution distances, the potential commonalities between images and text at the emotion and motivation levels are extracted to obtain domain-shared features. These shared features can characterize the inherent consistency of emotional expression among different modalities. At the same time, since each modality has different emotional expression methods, it is necessary to map the text modality and image modality to feature subspaces with larger distribution distances to effectively capture unique emotional features. By employing a cross-attention mechanism to obtain interaction information between different modalities, a certain modality can dynamically acquire highly relevant emotional cues from another modality during feature extraction, thereby improving the ability to extract multimodal features.

4. The image-text multimodal emotion classification method based on cross-modal and dual-width learning according to claim 3, characterized in that: In step S02, the modal uniformity features are extracted using the DUE module constructed by MLP-Transformer. ; The text-specific features and visual-specific features are obtained by constructing a VSE module and a TSE module using two MLP-Transformers, respectively. and image-specific features ; The modal interaction features are extracted by constructing a modal interaction module using a cross-attention mechanism. ; Multi-kernel maximum mean difference (MK-MMD) is used to measure the distance between features, and a cross-modal adaptive loss function is constructed.

5. The image-text multimodal emotion classification method based on cross-modal and dual-width learning according to claim 4, characterized in that: In step S03, feature fusion employs dual-width learning (DBL) to fuse the acquired features. DBL aims to further extract the interaction information between images and text through high-dimensional linear and non-linear mappings, thereby achieving efficient feature fusion and emotion classification. The dual-width learning includes a first width learning and a second width learning.

6. The image-text multimodal emotion classification method based on cross-modal and dual-width learning according to claim 5, characterized in that: In step S03, for the first width learning, the features are... and After concatenation, the input is used as the data, and then transformed through a linear mapping. Group feature nodes; Will The feature nodes are concatenated to obtain a linear feature matrix. ; After nonlinear mapping transformation, it becomes Group enhancement nodes; Will The enhanced nodes are concatenated to obtain the feature matrix of the enhanced nodes. ; Will and By concatenating the features, the complete feature matrices of the text and image modalities are obtained. .

7. The image-text multimodal emotion classification method based on cross-modal and dual-width learning according to claim 6, characterized in that: In step S03, for the second width learning, the features are... and After concatenation, the input is used as the data, and then transformed through a linear mapping. Group feature nodes; Will The feature nodes are concatenated to obtain a linear feature matrix. ; After nonlinear mapping transformation, it becomes Group enhancement nodes; Will The groups of augmented nodes are concatenated to obtain a matrix of augmented nodes. ; Will and By concatenating the features, the complete feature matrices of the text and image modalities are obtained. 。 8. The image-text multimodal emotion classification method based on cross-modal and dual-width learning according to claim 7, characterized in that: In order to effectively integrate features , , , The feature matrix needs to be , By splicing the components together, a fusion matrix is ​​obtained. ; Approximate output of dual-width learning DBL , , As shown below: ； in, , , express , , express This indicates the number of emotion categories.

9. The image-text multimodal emotion classification method based on cross-modal and dual-width learning according to claim 8, characterized in that: In step S04, ridge regression is used as the objective optimization function to obtain the optimal weights of the output layer. The outputs are summed using regularized least squares to obtain the actual output of cross-modal and dual-width learning CMDBL. Finally, the cross-entropy loss is used as the classification loss, as shown below: ； in, Represents classification loss. Represents the comprehensive loss function. This represents the cross-modal adaptive loss function. The weights represent the classification loss. The weights represent the cross-modal adaptive loss.

Images