Multi-modal emotion recognition method, device, system and readable storage medium based on direction-amplitude decomposition

Abstract

The application relates to the technical field of deep learning, in particular to a multi-modal emotion recognition method, device and system based on direction-amplitude decomposition and a readable storage medium. Initial modal features of three modes of text, image and audio of to-be-detected video data are acquired; the multi-modal features are decomposed into direction components and amplitude components through a direction-amplitude polar coordinate decomposition mechanism, and cross-modal interaction is carried out through a double path; a mixed contrast learning mechanism is combined with intra-modal decoupling and cross-modal alignment, and a soft label strategy based on a Gaussian kernel is adopted to adapt to emotion continuity; emotion recognition processing is carried out according to the fusion features, and an emotion prediction result is output. The application aims to solve the problem of processing multi-modal sample features to adapt to the continuity of multi-modal emotion analysis.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a multimodal emotion recognition method, apparatus, system and readable storage medium based on direction-amplitude decomposition. Background Technology

[0002] Multimodal sentiment analysis aims to comprehensively utilize complementary information from multiple modalities, including text, visual, and audio, to achieve more accurate and robust recognition of human emotional states. Compared to traditional unimodal methods, multimodal analysis can capture the multidimensional features of emotional expression—text conveys semantic content, speech reflects rhythm and intonation, and facial expressions reveal nonverbal cues—thus achieving a more comprehensive understanding of complex emotional states.

[0003] Feature decomposition (VDE) offers a novel perspective for understanding multimodal representations. Existing research has proposed decomposing multimodal representations into modality-shared factors and modality-specific factors, achieving decoupling of cross-modal commonality and modality specificity through factorization modeling. However, existing decomposition methods are primarily based on latent space learning, lacking clear geometric interpretation, and the semantic meaning of the decomposed subspaces remains ambiguous. Furthermore, efficiently performing cross-modal interactions on the decomposed features remains an unresolved issue.

[0004] Contrastive learning, as a powerful representation learning paradigm, has been introduced into multimodal tasks. By bringing semantically similar samples closer together and distancing dissimilar samples, it can obtain discriminative cross-modal representations. However, traditional contrastive learning methods such as SimCLR and MoCo typically use binary / discrete positive / negative sample partitioning, which contradicts the continuous nature of sentiment. In sentiment analysis tasks, sentiment labels are usually continuous values. For example, the sentiment intensity range in the CMU-MOSEI dataset is [-3, 3]. Samples with labels 2.3 and 2.1 might be considered negative pairs in traditional contrastive learning methods, even though these two types of samples have high semantic similarity. This discrete partitioning hinders contrastive learning from fully utilizing the fine-grained information in sentiment labels.

[0005] In view of this, this application proposes a novel multimodal emotion recognition method, which aims to adapt to the continuous characteristics of emotion analysis through a contrastive learning mechanism based on continuous sample partitioning, thereby improving the accuracy and robustness of multimodal emotion recognition. Summary of the Invention

[0006] The main objective of this application is to provide a multimodal sentiment recognition method based on orientation-amplitude decomposition, which aims to solve the problem of processing multimodal sample features to adapt to the continuous characteristics of multimodal sentiment analysis.

[0007] To achieve the above objectives, this application provides a multimodal emotion recognition method based on direction-amplitude decomposition, the method comprising:

[0008] S10: Extract features from the input text modal data, image modal data, and audio modal data respectively, and map them to a feature space of a preset dimension to form an initial feature representation;

[0009] S20, the initial feature representation is decomposed into directional features representing cross-modal shared semantic direction and amplitude features representing modality-specific emotional intensity, wherein each modality data corresponds to one directional feature and one amplitude feature;

[0010] S30, stack the directional features and amplitude features of the three modal data along their respective modal dimensions to form a directional feature matrix and an amplitude feature matrix, and perform cross-modal information interaction processing on the directional feature matrix and the amplitude feature matrix through a directional feature mixer and an amplitude feature mixer respectively to obtain enhanced directional features and enhanced amplitude features after cross-modal interaction enhancement;

[0011] S40, the enhancement direction feature and the enhancement amplitude feature are weighted and recombined to obtain partial enhancement direction feature and partial enhancement amplitude feature. The enhancement direction feature is used as the dominant component and integrated with the partial enhancement amplitude feature information to generate similarity feature. The enhancement amplitude feature is used as the dominant component and integrated with the partial enhancement direction feature information to generate difference feature, resulting in six recombined multimodal fusion features.

[0012] S50: The six recombined multimodal fusion features are spliced ​​together, and the spliced ​​multimodal fusion features are input into the multimodal emotion model to obtain the emotion recognition result.

[0013] Optionally, S20 includes:

[0014] S21, invoke the orientation feature projector and amplitude feature projector corresponding to each of the three modal data, wherein the orientation feature projector and the amplitude feature projector each include a normalization layer, a linear transformation layer, a hyperbolic tangent activation layer and a random deactivation layer connected in sequence;

[0015] S22, extract the orientation features of each of the three modal data based on the orientation feature projector, and extract the amplitude features of each of the three modal data based on the amplitude feature projector.

[0016] Optionally, in step S30, both the direction feature mixer and the amplitude feature mixer are configured based on an MLP-Mixe architecture and include:

[0017] The token mixing operation is used to perform layer normalization on the input orientation feature matrix and amplitude feature matrix, then perform matrix transpose to make the modality dimension become the feature dimension, perform cross-modal information mixing through the first multilayer perceptron, and finally perform matrix transpose and perform residual connection with the original input.

[0018] The first multilayer perceptron includes a first linear layer from the modality number dimension to the hidden layer dimension, a Gaussian error linear unit activation function, and a second linear layer from the hidden layer dimension back to the modality number dimension.

[0019] The channel mixing operation is used to perform layer normalization on the direction feature matrix and amplitude feature matrix after token mixing, and then perform cross-feature dimension information mixing through a second multilayer perceptron. Finally, it is residually connected with the output of token mixing.

[0020] The second multilayer perceptron includes a third linear layer from the feature dimension to the hidden layer dimension, a Gaussian error linear unit activation function, and a fourth linear layer from the hidden layer dimension back to the feature dimension.

[0021] Optionally, S40 includes:

[0022] S41, according to the first preset recombination weighting coefficient, the enhanced amplitude feature after the mixer processing is used as an auxiliary component, and weighted and fused with the enhanced direction feature after the mixer processing as the dominant component to obtain the similarity feature;

[0023] S42, according to the second preset recombination weighting coefficient, the enhanced direction feature after the mixer processing is used as an auxiliary component, and weighted and fused with the enhanced amplitude feature after the mixer processing, which is the dominant component, to obtain the difference feature;

[0024] The weight coefficients of the first preset recombination and the second preset recombination are [0,1].

[0025] Optionally, after step S40, the method further includes:

[0026] S60, assign pseudo-labels with preset offsets to the similarity features and the difference features to map the similarity features and the difference features to different semantic subspaces;

[0027] S70, calculate the contrast loss values ​​between three pairs of modalities: text modal data and image modal data, text modal data and audio modal data, and image modal data and audio modal data, so as to align the similarity features of different modalities based on the contrast loss values;

[0028] S80, calculate the semantic similarity weight between sample pairs based on the continuous distance between sample sentiment labels, and divide the training samples based on the semantic similarity weight, wherein the magnitude of the semantic similarity weight is negatively correlated with the continuous distance.

[0029] Optionally, step S80 includes:

[0030] S81, select any two samples in the training samples as target sample pairs, calculate the difference between the sentiment label values ​​of the target sample pairs, divide the square of the difference by the square of twice the standard deviation of the Gaussian kernel, take the negative number of the result and perform exponential operation to obtain the unnormalized similarity weight of the target sample pairs.

[0031] S82, determine the unnormalized similarity weight corresponding to each sample pair in the training samples, and sum the unnormalized similarity weights to obtain the normalization factor;

[0032] S83, the ratio of the unnormalized similarity weight to the normalization factor is used as the semantic similarity weight of the target sample pair.

[0033] Optionally, in step S50, the training process of the multimodal emotion model includes the following steps:

[0034] S51, calculate the sentiment prediction value of the spliced ​​multimodal fusion features through the multimodal decoder in the multimodal sentiment model, and calculate the regression loss between the sentiment prediction value and the real sentiment label as the main task loss;

[0035] S52, input the six multimodal fusion features into the single-modal decoder respectively to obtain six single-modal prediction values, calculate the average regression loss between each single-modal prediction value and the real sentiment label, and use it as the single-modal auxiliary supervision loss;

[0036] S53, the intra-modal contrastive loss and cross-modal contrastive loss are weighted and combined according to preset weight coefficients to obtain the hybrid contrastive learning loss;

[0037] S54, the main task loss, the single-modal auxiliary supervision loss and the hybrid contrastive learning loss are weighted and summed according to preset weight coefficients to obtain the total loss;

[0038] S55, supervise the training process of the multimodal emotion model based on the total loss.

[0039] Furthermore, to achieve the above objectives, this application also provides a multimodal emotion recognition device based on direction-amplitude decomposition, the multimodal emotion recognition device based on direction-amplitude decomposition comprising:

[0040] The feature extraction module is used to extract features from the input text modal data, image modal data and audio modal data respectively and map them to a feature space of preset dimensions to form an initial feature representation;

[0041] The orientation-amplitude decomposition module is used to decompose the initial feature representation into orientation features that characterize the shared semantic orientation across modalities and amplitude features that characterize the intensity of modality-specific emotions, wherein each modality data corresponds to one orientation feature and one amplitude feature;

[0042] The cross-modal interaction module is used to stack the directional features and amplitude features of the three modal data along their respective modal dimensions to form directional feature matrices and amplitude feature matrices, and to perform cross-modal information interaction processing on the directional feature matrices and amplitude feature matrices through directional feature mixers and amplitude feature mixers, respectively, to obtain enhanced directional features and enhanced amplitude features enhanced by cross-modal interaction.

[0043] The recombination module is used to perform weighted recombination of the enhancement direction feature and the enhancement amplitude feature respectively to obtain partial enhancement direction feature and partial enhancement amplitude feature. The enhancement direction feature is used as the dominant component and integrated with the partial amplitude feature information to generate similarity feature. The enhancement amplitude feature is used as the dominant component and integrated with the partial direction feature information to generate difference feature, resulting in six recombined multimodal fusion features.

[0044] The sentiment prediction module is used to concatenate the six recombined multimodal fusion features, and then input the concatenated multimodal fusion features into the multimodal sentiment model to obtain the sentiment recognition result.

[0045] In addition, to achieve the above objectives, this application also provides a computer system comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the multimodal emotion recognition method based on direction-amplitude decomposition as described in any of the preceding claims.

[0046] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the multimodal emotion recognition method based on direction-amplitude decomposition as described in any of the preceding claims.

[0047] This application has at least the following beneficial effects:

[0048] 1. The proposed orientation-amplitude polar coordinate decomposition mechanism explicitly decomposes multimodal features into orientation components (representing semantic orientation shared across modalities) and amplitude components (representing modality-specific intensity / confidence), providing a clear geometric interpretation. Compared to existing modality-invariant / modality-specific decomposition methods, the decomposition in this application more naturally captures the directional features of sentiment semantics, enhancing the interpretability of the model;

[0049] 2. Employing a dual-path approach to process directional and amplitude features separately across modal interactions enables efficient cross-modal information exchange;

[0050] 3. A soft-label contrastive learning mechanism based on Gaussian kernels is introduced, which maps continuous sentiment labels to similarity weights between samples. This solves the discrete partitioning problem of traditional contrastive learning and makes full use of the fine-grained information in sentiment labels to improve the effectiveness of contrastive learning in sentiment analysis tasks.

[0051] 4. A hybrid contrastive learning framework was designed to enhance the discriminativeness between directional and amplitude features within a modality and to align directional features across modalities to promote cross-modal semantic consistency. This ensures both the discriminativeness of features within a modality and promotes semantic alignment across modalities. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating the multimodal emotion recognition method based on direction-amplitude decomposition involved in the embodiments of this application;

[0053] Figure 2 This is a schematic diagram of the architecture of the multimodal emotion recognition device based on direction-amplitude decomposition according to this application;

[0054] Figure 3 This is a schematic diagram of the hardware operating environment of the computer system involved in the embodiments of this application.

[0055] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0056] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.

[0057] First Embodiment

[0058] Reference Figure 1This embodiment provides a multimodal emotion recognition method based on direction-amplitude decomposition, the method comprising the following steps:

[0059] S10: Extract features from the input text modal data, image modal data, and audio modal data respectively, and map them to a feature space of a preset dimension to form an initial feature representation;

[0060] In this embodiment, for text modal data, the input text sequence is encoded by a pre-trained BERT model, and the hidden states corresponding to special classification labels are extracted as the initial feature representation of the text modality. The special classification labels are used to aggregate the semantic information of the entire text sequence.

[0061] For image modal data, feature extraction is performed through an image encoder containing multiple Transformer encoder layers. Each encoder layer contains a multi-head self-attention mechanism and a feedforward neural network. First, learnable positional encoding is added to the image feature sequence to capture temporal relationships. Then, feature transformation is performed through a multi-layer Transformer encoder. Average pooling is performed on the output features at all time steps to obtain the initial feature representation of the image modality.

[0062] For audio modal data, feature extraction is performed using an audio encoder containing multiple Transformer encoder layers. Its architecture is similar to that of an image encoder. After adding positional encoding to the audio feature sequence, it is transformed by multiple Transformer encoders. The output features at all time steps are averaged and pooled to obtain the initial feature representation of the audio modality.

[0063] The image encoder and audio encoder preferably have 3 Transformer layers, and the multi-head self-attention mechanism preferably has 8 attention heads, to ensure a balance between computational efficiency and feature representation capability.

[0064] Specifically, the video data to be tested includes three modalities: text modal data, image modal data, and audio modal data. For each modality, a dedicated encoder is used to extract discriminative features. Each encoder is designed to capture the unique features of its respective modality while outputting a unified representation for subsequent fusion.

[0065] For the text modality, a pre-trained BERT model (Bidirectional Encoder Representations from Transformers) is used to extract contextual representations. BERT is a pre-trained language model based on the Transformer architecture, which can capture rich semantic information through bidirectional training on a large-scale corpus.

[0066] Specifically, the text modality data is first segmented using a word vector dictionary, breaking it down into multiple words to facilitate subsequent text feature extraction. Then, the BERT model is used to embed the segmented words, yielding word-level text features. These word-level text features reflect the semantic relationships between words and can be used for subsequent tasks such as sentiment analysis.

[0067] In one possible implementation, given a sequence of text:

[0068]

[0069] in Let represent the i-th word, n represent the length of the text sequence, and the output of the text encoder is:

[0070]

[0071] Here, [CLS] represents a special classification marker used to aggregate the semantic information of the entire sequence. It should be noted that... BERT The model encodes the input text using a multi-layer bidirectional Transformer encoder. Each layer contains a multi-head self-attention mechanism and a feedforward neural network, enabling it to capture complex semantic relationships between words. In this embodiment, a pre-trained model is used, which contains a 12-layer Transformer encoder with a hidden layer dimension of 768, providing high-quality text representations for sentiment analysis.

[0072] Furthermore, for image modalities, visual feature extraction is a key component of this framework. Given visual features extracted from video frames, the visual encoder operates through three stages:

[0073] In one possible implementation, feature extraction is performed on the image modality data to obtain multiple vocabulary-level image features in the image modality data, including:

[0074] The start and end times of multiple words in the text modal data are marked in the image modal data to obtain the first time marking information of multiple words in the image modal data;

[0075] The image modal data is segmented based on the first time marker information and the preset acquisition frequency to obtain multiple segmented images corresponding to multiple words;

[0076] Face recognition is performed on multiple segmented images using a neural network model to obtain multiple face regions in the segmented images;

[0077] Image features of multiple face regions are extracted to obtain multiple image features from multiple segmented images;

[0078] The image features corresponding to multiple words are summed and averaged to obtain multiple word-level expression features.

[0079] Specifically, Phase 1: Transformer Encoding. First, positional encoding and Transformer layers are applied to capture temporal dependencies. :

[0080]

[0081] in, This represents the learnable positional embedding matrix. This represents the visual feature matrix obtained after preprocessing. This embodiment employs a 3-layer Transformer encoder, with each layer containing 8 attention heads. It should be noted that the introduction of positional encoding enables the model to understand the temporal relationships between different frames in an image sequence, which is crucial for capturing dynamic emotional expressions in video.

[0082] Phase 2: CBAM Attention Enhancement. To focus on discriminative visual features, a Convolutional Block Attention Module (CBAM) is integrated, which sequentially applies channel attention and spatial attention. Channel attention captures important feature channels through max pooling and average pooling:

[0083]

[0084]

[0085] Spatial attention identification of key spatial regions:

[0086]

[0087]

[0088] in, and Let represent the channel attention mask and the spatial attention mask, respectively. σ represents the sigmoid activation function, and ⨀ represents element-wise multiplication. It should be noted that channel attention is used to emphasize feature channels containing important emotional information (such as facial expression-related features), while spatial attention is used to locate key regions in an image (such as faces, gestures, etc.). Combining the two can effectively improve the discriminative power of visual features. This represents the visual features after channel attention processing, emphasizing the feature channels that contain important emotional information. These are visual features enhanced by attention.

[0089] Phase 3, temporal aggregation. The final visual representation is obtained through averaging:

[0090]

[0091] in, Indicates the length of the visual feature sequence. This represents the enhanced visual feature at time step t.

[0092] Temporal aggregation can integrate the visual information of the entire video segment into a unified representation vector.

[0093] The reason why CBAM was specifically introduced for visual features is:

[0094] (1) Video frames contain redundant information and require attention mechanisms to identify key frames;

[0095] (2) Different feature channels encode diverse visual patterns (facial expressions, gestures, etc.);

[0096] (3) The attention mechanism can adaptively emphasize discriminative channels and spatial regions. Experiments show that after introducing CBAM, the model's ability to recognize micro-expressions and subtle emotional changes is significantly improved.

[0097] Furthermore, for audio modalities, a Transformer-based encoder is used to process acoustic features.

[0098] In one possible implementation, feature extraction is performed on the audio modality data to obtain multiple word-level audio features from the audio modality data, including:

[0099] By labeling the start and end times of multiple words in the text modal data within the audio modal data, we obtain the first time labeling information for multiple words in the audio modal data:

[0100] The audio modal data is segmented based on the first time marker information to obtain multiple audio segment data corresponding to multiple words;

[0101] By using an automatic speech recognition model to extract features from multiple audio segments, multiple word-level audio features are obtained.

[0102] It should be noted that the start and end times refer to the specific times when each word in the audio modality data begins and ends. The first time stamp information identifies the start and end times of words in the text modality data and their corresponding audio segments aligned on the timeline in the audio modality data. The start and end times determine the boundaries of audio segments in the audio modality data; this first time stamp information allows the association between text and audio content; and the audio segment data consists of audio segments segmented from the audio modality data based on the time stamp information, corresponding to the words in the text modality data on the timeline.

[0103] Specifically, firstly, the start and end times of each word in the text modal data summary within the audio modal data are determined, i.e., the first time marker information. Then, based on the first time marker information, audio segment data corresponding to each word in the text modal data is segmented from the audio modal data. Finally, an automatic speech recognition model is used to extract features from the audio segment data, thereby obtaining word-level audio features corresponding to each word in the text modal data. The parameters of the automatic speech recognition model are adjusted to ensure that the dimensions of the word-level audio features and the word-level text features are the same. These word-level audio features provide information on speech content, emotional tone, etc., for analyzing emotional expression in speech. In this embodiment, the calculation formula for audio feature extraction is:

[0104]

[0105]

[0106] in, Indicates audio position encoding. Indicates the length of the audio feature sequence. Let represent the audio features at time step t. The audio encoder uses the same architecture as the visual encoder (3 layers, 8 attention heads), but does not include CBAM because audio features exhibit simpler temporal patterns. It should be noted that for audio modalities, due to their relatively simple temporal characteristics, a complex attention mechanism is not required as in visual modalities; a standard Transformer encoder can obtain good feature representations.

[0107] The encoder described above maps the original inputs of the three modalities to a unified 768-dimensional feature space. This facilitates subsequent multimodal fusion. This unified feature space design enables information from different modalities to be compared and fused in the same semantic space, laying the foundation for subsequent direction-amplitude decomposition.

[0108] S20, the initial feature representation is decomposed into directional features representing cross-modal shared semantic direction and amplitude features representing modality-specific emotional intensity, wherein each modality data corresponds to one directional feature and one amplitude feature;

[0109] After extracting uniform features, this embodiment proposes a novel polar coordinate-intensity decomposition mechanism for explicitly modeling cross-modal similarity and modality-specific differences. This mechanism is inspired by polar coordinate representation, where a vector can be decomposed into direction and magnitude.

[0110] Traditional multimodal fusion methods treat features as opaque vectors and directly concatenate them, failing to distinguish between shared semantic information and unique modal features. This embodiment decomposes multimodal features into two orthogonal components:

[0111] (1) Direction: Represents the semantic direction shared across modalities, capturing category information of sentiment (such as positive / negative).

[0112] (2) Magnitude: indicates the intensity of emotion specific to a modality, reflecting the confidence level of each modality in the expression of emotion.

[0113] It's important to note that directional features are used to capture the semantic direction shared across modalities, i.e., the consistent parts of emotional expression across different modalities, such as text, visual, and audio all expressing the emotion category of "happiness." Amplitude features, on the other hand, are used to capture modality-specific emotional intensity, i.e., the confidence and strength of each modality's emotional expression; for example, the visual modality might display a strong expression of happiness, while the text modality might only show a slight tendency towards happiness. This decomposition approach allows for a more granular characterization of the complexity of multimodal emotional expression.

[0114] For each mode m∈{ T,V,A}, using an independent projection network:

[0115]

[0116]

[0117] Each projector The dimension is reduced from 768 to 384, resulting in six feature vectors. Specifically, the projector first performs layer normalization on the input features ( LayerNorm To stabilize the training process, a linear transformation was then used to reduce the feature dimension to 384, reducing computational complexity. Then, [the following was used]... Tanh The activation function introduces non-linearity, restricting the feature values ​​to the range [-1, 1]; finally, it uses... DropoutRegularization is applied to prevent overfitting. This projection mechanism maps high-dimensional initial features to a more compact representation space.

[0118] Further and optionally, in some optional embodiments, the directional feature projector and amplitude feature projector corresponding to each of the three modal data are invoked, wherein the directional feature projector and the amplitude feature projector each include a normalization layer, a linear transformation layer, a hyperbolic tangent activation layer and a random deactivation layer connected in sequence; the directional features of each of the three modal data are extracted based on the directional feature projector, and the amplitude features of each of the three modal data are extracted based on the amplitude feature projector.

[0119] S30, stack the directional features and amplitude features of the three modal data along their respective modal dimensions to form a directional feature matrix and an amplitude feature matrix, and perform cross-modal information interaction processing on the directional feature matrix and the amplitude feature matrix through a directional feature mixer and an amplitude feature mixer respectively to obtain enhanced directional features and enhanced amplitude features after cross-modal interaction enhancement;

[0120] S40, the enhancement direction feature and the enhancement amplitude feature are weighted and recombined to obtain partial enhancement direction feature and partial enhancement amplitude feature. The enhancement direction feature is used as the dominant component and integrated with the partial enhancement amplitude feature information to generate similarity feature. The enhancement amplitude feature is used as the dominant component and integrated with the partial enhancement direction feature information to generate difference feature, resulting in six recombined multimodal fusion features.

[0121] In this embodiment, after obtaining the directional and amplitude features, optimization is performed on these two features in steps S30 and S40 respectively. To achieve intermodal information exchange, an MLP-Mixer architecture is adopted, which achieves efficient cross-modal fusion with linear complexity. Two independent Mixer modules are applied to the directional and amplitude features respectively. Unlike traditional attention-based fusion methods, MLP-Mixer achieves cross-modal interaction through a simple multilayer perceptron (MLP), with a computational complexity of O(N), significantly lower than the O(N²) complexity of Transformer, thus greatly improving computational efficiency while ensuring fusion performance.

[0122] It should be noted that similarity features mainly inherit the cross-modal sharing characteristics of directional features, supplemented by confidence information provided by amplitude features, to capture the consistency of different modalities in sentiment semantics; difference features mainly inherit the modality-specific characteristics of amplitude features, supplemented by semantic guidance information provided by directional features, to capture the unique contribution and discriminative features of each modality.

[0123] As an optional implementation method:

[0124] First, stack the orientation and amplitude features along the modal dimension:

[0125]

[0126]

[0127] in, D,M ∈R3×384 represents the stacked orientation and amplitude matrices. By stacking the orientation and amplitude features of different modalities separately, two independent feature matrices are formed, preparing for subsequent cross-modal interactions.

[0128] It should be noted that direction and magnitude features are processed by separate Mixer modules. This maintains the independence of the two features and prevents them from being merged prematurely and losing their individual characteristics. DirectionMixer focuses on extracting semantic direction information shared across modalities, while MagnitudeMixer focuses on extracting modality-specific intensity information.

[0129] After cross-modal interaction, the directional and amplitude features are enhanced and reconstructed using an adaptive gating mechanism. The core idea of ​​this embodiment is to adaptively fuse directional and amplitude information through a gating mechanism, so that the reconstructed directional features can be enhanced using amplitude information, and the reconstructed amplitude features can be adjusted using directional information.

[0130] Unlike fixed-weighted combinations, a learnable gating mechanism is employed, which adaptively determines the optimal fusion ratio for each sample and modality. Orientation features are reconstructed through adaptive gating:

[0131]

[0132]

[0133] in, The gating weights are represented by σ, and the sigmoid activation function is represented by σ. LN Representation layer normalization. Gating weights are derived from amplitude features via... MLP Generate and dynamically control the fusion ratio of directional and amplitude features.

[0134] Amplitude features are similarly generated and gated for reconstruction:

[0135]

[0136]

[0137] in, The gating weights represent the amplitude features, and σ represents the sigmoid activation function. This represents a multilayer perceptron used to generate the gating weights. Indicates the directional characteristics after mixing. Indicates the amplitude characteristics after mixing. This represents the amplitude characteristics after reconstruction. Tanh This represents the hyperbolic tangent activation function. LN The representation is normalized, and ⊙ represents element-wise multiplication. The gating weights are derived from directional features. MLP Generate and dynamically control the fusion ratio of amplitude and direction features.

[0138] It should be noted that the gating of directional features is generated by the amplitude features, while the gating of amplitude features is generated by the directional features. This cross-gating mechanism allows the two features to reinforce each other. The directional features can be adjusted according to the amplitude information, and the amplitude features can also be adjusted according to the directional information, thereby obtaining more robust and discriminative feature representations.

[0139] The final expression is:

[0140]

[0141] in, This represents the final vocabulary-level multimodal fusion feature vector. Represents the reconstruction direction features of the text modality. Represents the reconstruction direction features of the visual modality. This represents the reconstruction direction feature of the audio modality. Represents the reconstruction magnitude feature of the text modality. Represents the reconstruction amplitude features of the visual modality. These six feature vectors represent the reconstructed amplitude features of the audio modality. They are combined into a unified feature representation through a concatenation operation.

[0142] This feature is a vocabulary-level multimodal fusion feature, which includes reconstruction direction features and reconstruction magnitude features of the three modalities. Specifically, It is a 2304-dimensional vector (six 384-dimensional features concatenated together), which simultaneously contains semantic direction information shared across modalities and modality-specific intensity information, providing a rich and comprehensive multimodal representation for subsequent emotion recognition.

[0143] S50: The six recombined multimodal fusion features are spliced ​​together, and the spliced ​​multimodal fusion features are input into the multimodal emotion model to obtain the emotion recognition result.

[0144] The six recombined multimodal fusion features are concatenated into a unified multimodal representation vector. Sentiment prediction is then performed using the multimodal decoder within the multimodal fusion features. The multimodal decoder contains multiple linear layers and activation functions. After dimensionality reduction layer by layer, the final sentiment prediction value is output as the sentiment recognition result.

[0145] In the technical solution provided in this embodiment, initial modal features of text, image, and audio modalities are acquired from the video data to be tested. Multimodal features are decomposed into directional and amplitude components using a direction-amplitude polar coordinate decomposition mechanism, and cross-modal interaction is performed using a dual-path MLP-Mixer. A hybrid contrastive learning mechanism is used to combine intramodal decoupling and cross-modal alignment, and a Gaussian kernel-based soft labeling strategy is employed to adapt to sentiment continuity. Sentiment recognition processing is performed based on the fused features, and sentiment prediction results are output. This provides clear geometric interpretability and effectively distinguishes between cross-modal shared information and modality-specific information.

[0146] Second Embodiment

[0147] Based on the first embodiment, this embodiment provides a configuration architecture for a direction feature mixer and an amplitude feature mixer. Both the direction feature mixer and the amplitude feature mixer are configured based on an MLP-Mixe architecture and include:

[0148] The token mixing operation is used to perform layer normalization on the input orientation feature matrix and amplitude feature matrix, then perform matrix transpose to make the modality dimension become the feature dimension, perform cross-modal information mixing through the first multilayer perceptron, and finally perform matrix transpose and perform residual connection with the original input.

[0149] The first multilayer perceptron includes a first linear layer from the modality number dimension to the hidden layer dimension, a Gaussian error linear unit activation function, and a second linear layer from the hidden layer dimension back to the modality number dimension.

[0150] The channel mixing operation is used to perform layer normalization on the direction feature matrix and amplitude feature matrix after token mixing, and then perform cross-feature dimension information mixing through a second multilayer perceptron. Finally, it is residually connected with the output of token mixing.

[0151] The second multilayer perceptron includes a third linear layer from the feature dimension to the hidden layer dimension, a Gaussian error linear unit activation function, and a fourth linear layer from the hidden layer dimension back to the feature dimension.

[0152] In this embodiment, the token mixing operation is used to realize cross-modal dimension information interaction, enabling the features of the three modalities of text modality, image modality and audio modality to pay attention to each other and exchange information, promoting semantic alignment between different modalities;

[0153] Channel mixing operations are used to achieve information exchange within feature dimensions, enhancing the expressive power and feature correlation between different dimensions within the same feature vector.

[0154] Third Embodiment

[0155] Based on any embodiment, this embodiment provides a weighted recombination method for enhancement direction features and enhancement amplitude features, including:

[0156] S41, according to the first preset recombination weighting coefficient, the enhanced amplitude feature after the mixer processing is used as an auxiliary component, and weighted and fused with the enhanced direction feature after the mixer processing as the dominant component to obtain the similarity feature;

[0157] S42, according to the second preset recombination weighting coefficient, the enhanced direction feature after the mixer processing is used as an auxiliary component, and weighted and fused with the enhanced amplitude feature after the mixer processing, which is the dominant component, to obtain the difference feature;

[0158] The weight coefficients of the first preset recombination and the second preset recombination are [0,1].

[0159] In this embodiment, for each modality, the directional features processed by the mixer are used as the dominant component, and the amplitude features processed by the mixer are used as the auxiliary component according to the preset recombination weight coefficient to perform weighted fusion to obtain the similarity features of the modality. The similarity features mainly represent the semantic information shared across modalities.

[0160] For each mode, the amplitude feature processed by the mixer is taken as the dominant component, and the directional feature processed by the mixer is taken as the auxiliary component according to the preset recombination weighting coefficient, and the difference features of the mode are weighted and fused to obtain the difference features of the mode. The difference features mainly characterize the unique information specific to the mode.

[0161] The preset recombination weight coefficient is used to control the fusion ratio between the dominant component and the auxiliary component. Its value ranges from 0 to 1, with a preferred value of 0.5, indicating that the auxiliary component contributes half the weight to the recombinated feature.

[0162] Fourth embodiment

[0163] Based on any embodiment, this embodiment also sets up a hybrid contrastive learning mechanism in the model to optimize the discriminativeness and cross-modal alignment of feature representations during model training. This mechanism includes:

[0164] S60, assign pseudo-labels with preset offsets to the similarity features and the difference features to map the similarity features and the difference features to different semantic subspaces;

[0165] In this embodiment, S60 is intra-modal contrastive learning, which is used to enhance the discriminativeness between similar and dissimilar features within the same modality. By assigning pseudo-labels with preset offsets to similar and dissimilar features, the model is forced to map similar and dissimilar features to different semantic subspaces, wherein the preset offset is preferably 10.

[0166] S70, calculate the contrast loss values ​​between three pairs of modalities: text modal data and image modal data, text modal data and audio modal data, and image modal data and audio modal data, so as to align the similarity features of different modalities based on the contrast loss values;

[0167] In this embodiment, S70 is cross-modal contrastive learning, which is used to explicitly align the similarity features of different modalities. By calculating the contrast loss between three pairs of modalities—text-image, text-audio, and image-audio—it promotes consistent expression of different modalities in the semantic direction.

[0168] S80, calculate the semantic similarity weight between sample pairs based on the continuous distance between sample sentiment labels, and divide the training samples based on the semantic similarity weight, wherein the magnitude of the semantic similarity weight is negatively correlated with the continuous distance.

[0169] In this embodiment, S80 is the soft label similarity calculation. Unlike traditional contrastive learning which uses a binary positive and negative sample division, this method uses a soft label strategy based on the Gaussian kernel function. It calculates the semantic similarity weight between sample pairs based on the continuous distance between the sentiment labels of the samples, so that samples with close labels get higher similarity weights and samples with far labels get lower similarity weights, thus better adapting to the continuous nature of sentiment analysis tasks.

[0170] Further, and optionally, the steps for calculating semantic similarity weights include:

[0171] S81, select any two samples in the training samples as target sample pairs, calculate the difference between the sentiment label values ​​of the target sample pairs, divide the square of the difference by the square of twice the standard deviation of the Gaussian kernel, take the negative number of the result and perform exponential operation to obtain the unnormalized similarity weight of the target sample pairs.

[0172] S82, determine the unnormalized similarity weight corresponding to each sample pair in the training samples, and sum the unnormalized similarity weights to obtain the normalization factor;

[0173] S83, the ratio of the unnormalized similarity weight to the normalization factor is used as the semantic similarity weight of the target sample pair.

[0174] In addition, as some optional implementation methods, this embodiment proposes to use a Gaussian kernel to calculate the soft similarity weights:

[0175]

[0176] in, This represents the semantic similarity between sample i and sample j based on label proximity. and Let i and j represent the sentiment labels, respectively. This is the Gaussian kernel bandwidth parameter.

[0177] It should be noted that, unlike traditional contrastive learning which simply divides samples into positive and negative samples, this embodiment uses a Gaussian kernel function to calculate a continuous similarity weight for each pair of samples. Samples with closer labels receive higher weights, thus better adapting to the continuous nature of sentiment analysis.

[0178] To enhance the discriminative power between directional and amplitude features, pseudo-labels with an offset of δ=10 are assigned:

[0179]

[0180] This allows directional and amplitude features to form different semantic clusters during contrastive learning. By assigning different pseudo-labels to directional and amplitude features, the model is forced to map them to different representation spaces, thereby enhancing the discriminative power of the two features. The intra-modal loss with soft labels is:

[0181]

[0182] in, For cosine similarity, and Let be the feature vectors of the i-th and j-th samples in the training batch, respectively. Here, N represents the temperature parameter, and N represents the batch size. Controlling the smoothness of the similarity distribution, a smaller This makes the model pay more attention to hard negative samples, which are larger. This makes the distribution smoother. In this embodiment, Set to 0.07.

[0183] To align cross-modal orientation features (since orientation features represent shared semantic orientations across modalities), pairwise cross-modal contrastive learning losses are computed. :

[0184]

[0185] in, It is a cross-modal contrast loss. L TV It is a cross-modal contrast loss between text and vision. L TA It is a cross-modal contrast loss between text and audio. L VA It is a cross-modal contrast loss between visual and audio.

[0186] It should be noted that cross-modal contrastive learning is only applied to orientation features, because orientation features represent semantic orientation shared across modalities and need to be aligned across different modalities; while amplitude features represent modality-specific intensity information and do not require cross-modal alignment.

[0187] Fifth embodiment

[0188] Based on any embodiment, this embodiment provides a constraint on the training process of a multimodal emotion model, including the following steps:

[0189] S51, calculate the sentiment prediction value of the spliced ​​multimodal fusion features through the multimodal decoder in the multimodal sentiment model, and calculate the regression loss between the sentiment prediction value and the real sentiment label as the main task loss;

[0190] S52, input the six multimodal fusion features into the single-modal decoder respectively to obtain six single-modal prediction values, calculate the average regression loss between each single-modal prediction value and the real sentiment label, and use it as the single-modal auxiliary supervision loss;

[0191] S53, the intra-modal contrastive loss and cross-modal contrastive loss are weighted and combined according to preset weight coefficients to obtain the hybrid contrastive learning loss;

[0192] S54, the main task loss, the single-modal auxiliary supervision loss and the hybrid contrastive learning loss are weighted and summed according to preset weight coefficients to obtain the total loss;

[0193] S55, supervise the training process of the multimodal emotion model based on the total loss.

[0194] In this embodiment, hybrid contrastive learning loss The expression is:

[0195]

[0196] In the formula, For intramodal contrast loss, For cross-modal contrast loss, α and β are the weights of intra-modal contrast loss and cross-modal contrast loss, respectively.

[0197] Main task loss The calculation expression is:

[0198]

[0199] Single-modal auxiliary supervision loss The calculation expression is:

[0200]

[0201] The formula for calculating the total loss is:

[0202]

[0203] In the formula, λ1 and λ2 are the loss weight coefficients.

[0204] In addition, as an implementation scheme, refer to Figure 2 This embodiment also proposes a multimodal emotion recognition device based on direction-amplitude decomposition, the multimodal emotion recognition device based on direction-amplitude decomposition includes:

[0205] The feature extraction module 100 is used to extract features from the input text modal data, image modal data and audio modal data respectively and map them to a feature space of a preset dimension to form an initial feature representation;

[0206] The orientation-amplitude decomposition module 200 is used to decompose the initial feature representation into orientation features representing cross-modal shared semantic orientation and amplitude features representing modality-specific emotional intensity, wherein each modality data corresponds to one orientation feature and one amplitude feature;

[0207] The cross-modal interaction module 300 is used to stack the directional features and amplitude features of the three modal data along their respective modal dimensions to form a directional feature matrix and an amplitude feature matrix, and to perform cross-modal information interaction processing on the directional feature matrix and the amplitude feature matrix through a directional feature mixer and an amplitude feature mixer, respectively, to obtain enhanced directional features and enhanced amplitude features enhanced by cross-modal interaction.

[0208] The recombination module 400 is used to perform weighted recombination on the enhancement direction feature and the enhancement amplitude feature respectively to obtain partial enhancement direction feature and partial enhancement amplitude feature. The enhancement direction feature is used as the dominant component and integrated with the partial amplitude feature information to generate similarity feature. The enhancement amplitude feature is used as the dominant component and integrated with the partial direction feature information to generate difference feature, resulting in six recombined multimodal fusion features.

[0209] The sentiment prediction module 500 is used to concatenate the six recombined multimodal fusion features and input the concatenated multimodal fusion features into the multimodal sentiment model to obtain the sentiment recognition result.

[0210] Furthermore, as an implementation scheme, Figure 3 This is a schematic diagram of the hardware operating environment of the computer system involved in the embodiments of this application.

[0211] like Figure 3 As shown, the computer system may include: a processor 1001, such as a CPU; a memory 1005; a user interface 1003; a network interface 1004; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0212] Those skilled in the art will understand that Figure 3 The computer system architecture shown does not constitute a limitation on the computer system and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0213] like Figure 3 As shown, the memory 1005, as a storage medium, may include an operating system, a network communication module, a user interface module, and computer programs. The operating system is a program that manages and controls the hardware and software resources of the computer system, as well as the operation of the computer programs and other software or programs.

[0214] exist Figure 3 In the computer system shown, the user interface 1003 is mainly used to connect to the terminal and communicate with the terminal; the network interface 1004 is mainly used to communicate with the backend server; and the processor 1001 can be used to call the computer program stored in the memory 1005.

[0215] In this embodiment, the computer system includes: a memory 1005, a processor 1001, and a computer program stored in the memory and executable on the processor, wherein:

[0216] When processor 1001 calls a computer program stored in memory 1005, it performs the following operations:

[0217] S10: Extract features from the input text modal data, image modal data, and audio modal data respectively, and map them to a feature space of a preset dimension to form an initial feature representation;

[0218] S20, the initial feature representation is decomposed into directional features representing cross-modal shared semantic direction and amplitude features representing modality-specific emotional intensity, wherein each modality data corresponds to one directional feature and one amplitude feature;

[0219] S30, stack the directional features and amplitude features of the three modal data along their respective modal dimensions to form a directional feature matrix and an amplitude feature matrix, and perform cross-modal information interaction processing on the directional feature matrix and the amplitude feature matrix through a directional feature mixer and an amplitude feature mixer respectively to obtain enhanced directional features and enhanced amplitude features after cross-modal interaction enhancement;

[0220] S40, the enhancement direction feature and the enhancement amplitude feature are weighted and recombined to obtain partial enhancement direction feature and partial enhancement amplitude feature. The enhancement direction feature is used as the dominant component and integrated with the partial enhancement amplitude feature information to generate similarity feature. The enhancement amplitude feature is used as the dominant component and integrated with the partial enhancement direction feature information to generate difference feature, resulting in six recombined multimodal fusion features.

[0221] S50: The six recombined multimodal fusion features are spliced ​​together, and the spliced ​​multimodal fusion features are input into the multimodal emotion model to obtain the emotion recognition result.

[0222] When processor 1001 calls a computer program stored in memory 1005, it performs the following operations:

[0223] S21, invoke the orientation feature projector and amplitude feature projector corresponding to each of the three modal data, wherein the orientation feature projector and the amplitude feature projector each include a normalization layer, a linear transformation layer, a hyperbolic tangent activation layer and a random deactivation layer connected in sequence;

[0224] S22, extract the orientation features of each of the three modal data based on the orientation feature projector, and extract the amplitude features of each of the three modal data based on the amplitude feature projector.

[0225] When processor 1001 calls a computer program stored in memory 1005, it performs the following operations:

[0226] The token mixing operation is used to perform layer normalization on the input orientation feature matrix and amplitude feature matrix, then perform matrix transpose to make the modality dimension become the feature dimension, perform cross-modal information mixing through the first multilayer perceptron, and finally perform matrix transpose and perform residual connection with the original input.

[0227] The first multilayer perceptron includes a first linear layer from the modality number dimension to the hidden layer dimension, a Gaussian error linear unit activation function, and a second linear layer from the hidden layer dimension back to the modality number dimension.

[0228] The channel mixing operation is used to perform layer normalization on the direction feature matrix and amplitude feature matrix after token mixing, and then perform cross-feature dimension information mixing through a second multilayer perceptron. Finally, it is residually connected with the output of token mixing.

[0229] The second multilayer perceptron includes a third linear layer from the feature dimension to the hidden layer dimension, a Gaussian error linear unit activation function, and a fourth linear layer from the hidden layer dimension back to the feature dimension.

[0230] When processor 1001 calls a computer program stored in memory 1005, it performs the following operations:

[0231] S41, according to the first preset recombination weighting coefficient, the enhanced amplitude feature after the mixer processing is used as an auxiliary component, and weighted and fused with the enhanced direction feature after the mixer processing as the dominant component to obtain the similarity feature;

[0232] S42, according to the second preset recombination weighting coefficient, the enhanced direction feature after the mixer processing is used as an auxiliary component, and weighted and fused with the enhanced amplitude feature after the mixer processing, which is the dominant component, to obtain the difference feature;

[0233] The weight coefficients of the first preset recombination and the second preset recombination are [0,1].

[0234] When processor 1001 calls a computer program stored in memory 1005, it performs the following operations:

[0235] S60, assign pseudo-labels with preset offsets to the similarity features and the difference features to map the similarity features and the difference features to different semantic subspaces;

[0236] S70, calculate the contrast loss values ​​between three pairs of modalities: text modal data and image modal data, text modal data and audio modal data, and image modal data and audio modal data, so as to align the similarity features of different modalities based on the contrast loss values;

[0237] S80, calculate the semantic similarity weight between sample pairs based on the continuous distance between sample sentiment labels, and divide the training samples based on the semantic similarity weight, wherein the magnitude of the semantic similarity weight is negatively correlated with the continuous distance.

[0238] When processor 1001 calls a computer program stored in memory 1005, it performs the following operations:

[0239] S81, select any two samples in the training samples as target sample pairs, calculate the difference between the sentiment label values ​​of the target sample pairs, divide the square of the difference by the square of twice the standard deviation of the Gaussian kernel, take the negative number of the result and perform exponential operation to obtain the unnormalized similarity weight of the target sample pairs.

[0240] S82, determine the unnormalized similarity weight corresponding to each sample pair in the training samples, and sum the unnormalized similarity weights to obtain the normalization factor;

[0241] S83, the ratio of the unnormalized similarity weight to the normalization factor is used as the semantic similarity weight of the target sample pair.

[0242] When processor 1001 calls a computer program stored in memory 1005, it performs the following operations:

[0243] S51, calculate the sentiment prediction value of the spliced ​​multimodal fusion features through the multimodal decoder in the multimodal sentiment model, and calculate the regression loss between the sentiment prediction value and the real sentiment label as the main task loss;

[0244] S52, input the six multimodal fusion features into the single-modal decoder respectively to obtain six single-modal prediction values, calculate the average regression loss between each single-modal prediction value and the real sentiment label, and use it as the single-modal auxiliary supervision loss;

[0245] S53, the intra-modal contrastive loss and cross-modal contrastive loss are weighted and combined according to preset weight coefficients to obtain the hybrid contrastive learning loss;

[0246] S54, the main task loss, the single-modal auxiliary supervision loss and the hybrid contrastive learning loss are weighted and summed according to preset weight coefficients to obtain the total loss;

[0247] S55, supervise the training process of the multimodal emotion model based on the total loss.

[0248] Furthermore, those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in a computer system to implement the process steps of the embodiments of the above methods.

[0249] Therefore, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the various steps of the multimodal emotion recognition method based on direction-amplitude decomposition as described in the above embodiments.

[0250] The computer-readable storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0251] It should be noted that, since the storage medium provided in the embodiments of this application is the storage medium used to implement the methods of the embodiments of this application, those skilled in the art can understand the specific structure and variations of the storage medium based on the methods described in the embodiments of this application, and therefore will not be repeated here. All storage media used in the methods of the embodiments of this application fall within the scope of protection of this application.

[0252] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0253] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0254] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0255] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0256] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. This application can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0257] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0258] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A multimodal emotion recognition method based on direction-amplitude decomposition, characterized in that, The method includes the following steps: S10: Extract features from the input text modal data, image modal data, and audio modal data respectively, and map them to a feature space of a preset dimension to form an initial feature representation; S20, the initial feature representation is decomposed into directional features representing cross-modal shared semantic direction and amplitude features representing modality-specific emotional intensity, wherein each modality data corresponds to one directional feature and one amplitude feature; S30, stack the directional features and amplitude features of the three modal data along their respective modal dimensions to form a directional feature matrix and an amplitude feature matrix, and perform cross-modal information interaction processing on the directional feature matrix and the amplitude feature matrix through a directional feature mixer and an amplitude feature mixer respectively to obtain enhanced directional features and enhanced amplitude features after cross-modal interaction enhancement; S40, the enhancement direction feature and the enhancement amplitude feature are weighted and recombined to obtain partial enhancement direction feature and partial enhancement amplitude feature. The enhancement direction feature is used as the dominant component and integrated with the partial enhancement amplitude feature information to generate similarity feature. The enhancement amplitude feature is used as the dominant component and integrated with the partial enhancement direction feature information to generate difference feature, resulting in six recombined multimodal fusion features. S50: The six recombined multimodal fusion features are spliced ​​together, and the spliced ​​multimodal fusion features are input into the multimodal emotion model to obtain the emotion recognition result; S20 includes: S21, invoke the orientation feature projector and amplitude feature projector corresponding to each of the three modal data, wherein the orientation feature projector and the amplitude feature projector each include a normalization layer, a linear transformation layer, a hyperbolic tangent activation layer and a random deactivation layer connected in sequence; S22, extract the orientation features of each of the three modal data based on the orientation feature projector, and extract the amplitude features of each of the three modal data based on the amplitude feature projector; After step S40, the method further includes: S60, assign pseudo-labels with preset offsets to the similarity features and the difference features to map the similarity features and the difference features to different semantic subspaces; S70, calculate the contrast loss values ​​between three pairs of modalities: text modal data and image modal data, text modal data and audio modal data, and image modal data and audio modal data, so as to align the similarity features of different modalities based on the contrast loss values; S80, calculate the semantic similarity weight between sample pairs based on the continuous distance between sample sentiment labels, and divide training samples based on the semantic similarity weight, wherein the magnitude of the semantic similarity weight is negatively correlated with the continuous distance; Step S80 includes: S81, select any two samples in the training samples as target sample pairs, calculate the difference between the sentiment label values ​​of the target sample pairs, divide the square of the difference by the square of twice the standard deviation of the Gaussian kernel, take the negative number of the result and perform exponential operation to obtain the unnormalized similarity weight of the target sample pairs. S82, determine the unnormalized similarity weight corresponding to each sample pair in the training samples, and sum the unnormalized similarity weights to obtain the normalization factor; S83, the ratio of the unnormalized similarity weight to the normalization factor is used as the semantic similarity weight of the target sample pair.

2. The multimodal emotion recognition method based on direction-amplitude decomposition as described in claim 1, characterized in that, In step S30, both the direction feature mixer and the amplitude feature mixer are configured based on the MLP-Mixe architecture and include: The token mixing operation is used to perform layer normalization on the input orientation feature matrix and amplitude feature matrix, then perform matrix transpose to make the modality dimension become the feature dimension, perform cross-modal information mixing through the first multilayer perceptron, and finally perform matrix transpose and perform residual connection with the original input. The first multilayer perceptron includes a first linear layer from the modality number dimension to the hidden layer dimension, a Gaussian error linear unit activation function, and a second linear layer from the hidden layer dimension back to the modality number dimension. The channel mixing operation is used to perform layer normalization on the direction feature matrix and amplitude feature matrix after token mixing, and then perform cross-feature dimension information mixing through a second multilayer perceptron. Finally, it is residually connected with the output of token mixing. The second multilayer perceptron includes a third linear layer from the feature dimension to the hidden layer dimension, a Gaussian error linear unit activation function, and a fourth linear layer from the hidden layer dimension back to the feature dimension.

3. The multimodal emotion recognition method based on direction-amplitude decomposition as described in claim 1, characterized in that, S40 includes: S41, according to the first preset recombination weighting coefficient, the enhanced amplitude feature after the mixer processing is used as an auxiliary component, and weighted and fused with the enhanced direction feature after the mixer processing as the dominant component to obtain the similarity feature; S42, according to the second preset recombination weighting coefficient, the enhanced direction feature after the mixer processing is used as an auxiliary component, and weighted and fused with the enhanced amplitude feature after the mixer processing, which is the dominant component, to obtain the difference feature; The weight coefficients of the first preset recombination and the second preset recombination are [0,1].

4. The multimodal emotion recognition method based on direction-amplitude decomposition as described in claim 1, characterized in that, In step S50, the training process of the multimodal emotion model includes the following steps: S51, calculate the sentiment prediction value of the spliced ​​multimodal fusion features through the multimodal decoder in the multimodal sentiment model, and calculate the regression loss between the sentiment prediction value and the real sentiment label as the main task loss; S52, input the six multimodal fusion features into the single-modal decoder respectively to obtain six single-modal prediction values, calculate the average regression loss between each single-modal prediction value and the real sentiment label, and use it as the single-modal auxiliary supervision loss; S53, the intra-modal contrastive loss and cross-modal contrastive loss are weighted and combined according to preset weight coefficients to obtain the hybrid contrastive learning loss; S54, the main task loss, the single-modal auxiliary supervision loss and the hybrid contrastive learning loss are weighted and summed according to preset weight coefficients to obtain the total loss; S55, supervise the training process of the multimodal emotion model based on the total loss.

5. An apparatus for implementing the multimodal emotion recognition method based on direction-amplitude decomposition as described in claim 1, characterized in that, The device includes: The feature extraction module is used to extract features from the input text modal data, image modal data and audio modal data respectively and map them to a feature space of preset dimensions to form an initial feature representation; The orientation-amplitude decomposition module is used to decompose the initial feature representation into orientation features that characterize the shared semantic orientation across modalities and amplitude features that characterize the intensity of modality-specific emotions, wherein each modality data corresponds to one orientation feature and one amplitude feature; The cross-modal interaction module is used to stack the directional features and amplitude features of the three modal data along their respective modal dimensions to form directional feature matrices and amplitude feature matrices, and to perform cross-modal information interaction processing on the directional feature matrices and amplitude feature matrices through directional feature mixers and amplitude feature mixers, respectively, to obtain enhanced directional features and enhanced amplitude features enhanced by cross-modal interaction. The recombination module is used to perform weighted recombination on the enhancement direction feature and the enhancement amplitude feature respectively to obtain partial enhancement direction feature and partial enhancement amplitude feature. The enhancement direction feature is used as the dominant component and integrated with the partial enhancement amplitude feature information to generate similarity feature. The enhancement amplitude feature is used as the dominant component and integrated with the partial enhancement direction feature information to generate difference feature, resulting in six recombined multimodal fusion features. The sentiment prediction module is used to concatenate the six recombined multimodal fusion features, and then input the concatenated multimodal fusion features into the multimodal sentiment model to obtain the sentiment recognition result.

6. A computer system, characterized in that, The computer system includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the multimodal emotion recognition method based on direction-amplitude decomposition as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the multimodal emotion recognition method based on direction-amplitude decomposition as described in any one of claims 1 to 4.

Images