Cross-modal emotion recognition method and system considering interactive context

Abstract

The invention discloses a cross-modal emotion recognition method and system considering an interactive context, and relates to the technical field of emotion recognition, images, audios and texts are respectively input into a multi-modal emotion recognition model for processing, and an emotion analysis result is obtained; the multi-modal emotion recognition model comprises a multi-modal feature extraction module, a joint cross-modal attention module and a multi-layer perceptron which are connected in sequence; images, audios and texts are input into a multi-modal feature extraction module, time sequence enhanced spatial features, time sequence audio features and time sequence text features are obtained, the time sequence enhanced spatial features, the time sequence audio features and the time sequence text features are input into a combined cross-modal attention module together to obtain an emotion analysis result, and multi-modal information fusion is integrated to obtain an emotion analysis result. Compared with the prior art, the method has the advantages that the precision and robustness of sentiment analysis are remarkably improved, particularly, the modeling of deep interaction and context association among modals is developed, the dynamic change in a complex sentiment scene can be processed more accurately, and the context integration capability among the modals is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion recognition, and particularly to a cross-modal emotion recognition method and system considering interaction context. Background Art

[0002] The statements in this part only provide background technical information related to the present disclosure and do not necessarily constitute prior art.

[0003] With the rapid development of deep learning technology, emotion recognition has become an important application in multiple fields such as human-computer interaction, intelligent customer service, and mental health monitoring. Most traditional emotion recognition methods rely on single-modal data, such as audio signals, facial expression images, or text. However, emotional expressions are usually multi-dimensional, and single-modal data cannot fully capture the complexity and subtle changes of emotions. Therefore, more and more research has begun to turn to multi-modal emotion analysis, aiming to improve the effect of emotion recognition by integrating information from multiple modalities, especially in enhancing the interaction context between different modalities.

[0004] In multi-modal emotion analysis, the fusion of the three modalities of audio, vision, and text is crucial. The audio modality can reflect the acoustic features of emotions by extracting features such as Mel spectrogram (MFCC). The visual modality can capture the subtle changes in facial emotions through facial expression recognition technology, such as using a convolutional neural network (CNN) for feature extraction. The text modality usually relies on natural language processing (NLP) technology to analyze text data and identify the emotional information therein. However, single-modal data cannot fully capture the context relationship between modalities, so it is particularly important to effectively fuse the context information of these modalities.

[0005] Although the prior art has made certain progress in multi-modal information fusion, feature extraction, and emotion target recognition, it still faces problems such as environmental changes, background noise, and insufficient capture of the deep relationship between modalities. At the same time, existing emotion analysis methods face great challenges in capturing the context interaction between multi-modal data and are difficult to accurately capture the dynamic changes of emotional expressions.

[0006] In this context, in recent years, with the development of the attention mechanism, more and more research has begun to explore how to enhance the interaction and information fusion between different modalities through the attention mechanism. Recursive fusion has thus been mentioned, which can improve the information fusion between modalities to a certain extent and optimize the accuracy of emotion analysis. However, when dealing with long-term sequential data, these methods still face the problem of insufficient sequential modeling ability and are difficult to capture the complex context relationship in emotional changes.

[0007] In summary, the existing multi-modal sentiment analysis methods still have certain deficiencies in context association, information fusion, and dealing with complex environmental changes. Although the existing methods have improved the information interaction between modalities through cross-modal attention mechanisms, when dealing with long-term data, the model's ability to capture the dynamic changes of sentiment is still limited, especially in the performance of cross-modal interaction context association. At the same time, the extraction of visual features is often affected by various factors, which makes the visual modality fail to fully exert its potential in sentiment analysis, especially lacking effective modeling when capturing context interactions. In addition, the existing technologies do not utilize text information deeply enough, making it difficult to accurately identify the sentiment targets in text fragments, resulting in insufficient exploration of context association and further affecting the accuracy of sentiment analysis. Summary of the Invention

[0008] To overcome the deficiencies of the above-mentioned prior art, the present invention provides a cross-modal sentiment recognition method and system considering interactive context, which has achieved significant improvements in the accuracy and robustness of sentiment analysis through integrating multi-modal information fusion, especially making progress in the modeling of deep interaction and context association between modalities, being able to more accurately handle the dynamic changes in complex sentiment scenarios and enhancing the context integration ability between modalities.

[0009] To achieve the above object, one or more embodiments of the present invention provide the following technical solutions: In the first aspect, the present invention provides a cross-modal sentiment recognition method considering interactive context, including: Obtain the image, audio, and text to be recognized; Input the image, audio, and text into a multi-modal sentiment recognition model for processing respectively to obtain sentiment analysis results; The multi-modal emotion recognition model includes a multi-modal feature extraction module, a joint cross-modal attention module, and a multi-layer perceptron connected in sequence; the image, audio, and text are respectively input into the multi-modal feature extraction module to obtain temporal enhanced spatial features, temporal audio features, and temporal text features, and are input into the joint cross-modal attention module together; the joint cross-modal attention module fuses the temporal enhanced spatial features, temporal audio features, and temporal text features to obtain a joint feature representation, then calculates the cross-correlations between the temporal enhanced spatial features, temporal audio features, temporal text features and the joint feature representation respectively, generates cross-correlation matrices, generates attention maps based on the cross-correlation matrices, then performs weighted summation of each attention map with the corresponding temporal enhanced spatial features, temporal audio features, and temporal text features respectively to obtain weighted features, re-inputs the weighted features into the joint cross-modal attention model for recursive optimization to obtain the final weighted features, splices the final weighted features to obtain a multi-modal feature representation and inputs it into the multi-layer perceptron to obtain the emotion analysis result.

[0010] Further technical solution, the multi-modal feature extraction module includes a spatial feature extraction module, an audio feature extraction module, and a text feature extraction module arranged in parallel.

[0011] Further technical solution, the spatial feature extraction module includes a visual encoder, a spatial-channel attention network, and a temporal convolutional network connected in sequence; the image is input into the visual encoder for feature extraction to obtain spatial features, and the spatial-channel attention network strengthens the spatial features through a joint attention mechanism in the spatial dimension and the channel dimension to obtain enhanced spatial features and inputs them into the temporal convolutional network for temporal modeling to obtain temporal enhanced spatial features; the visual encoder uses a pre-trained ResNet-50 network.

[0012] Further technical solution, the spatial-channel attention network first performs convolution on the spatial features to obtain attention weights, and weights the spatial features based on the attention weights to obtain attention-weighted features.

[0013] Further technical solution, the audio feature extraction module includes an audio encoder and a temporal convolutional network connected in sequence; the audio is converted into a Mel spectrogram and input into the audio encoder for feature extraction to obtain audio features and input into the temporal convolutional network for temporal modeling to obtain temporal audio features; the audio encoder uses a convolutional neural network.

[0014] Further technical solution: The text feature extraction module includes a text encoder, a pairwise expert module, and a temporal convolutional network connected in sequence; the text is input into the text encoder for feature extraction to obtain text features and input into the pairwise expert module. The pairwise expert module optimizes the boundary matching of the text feature span through an energy function to obtain the pairing energy between text label pairs, multiplies it with the text features as a weighting coefficient and inputs it into the temporal convolutional network for temporal modeling to obtain temporal text features; the pairwise expert module calculates the pairing energy between text label pairs by calculating the system energy.

[0015] Further technical solution: The temporal enhanced spatial features, temporal audio features, and temporal text features are fused through a fully connected layer to obtain a joint feature representation.

[0016] In a second aspect, the present invention provides a cross-modal sentiment recognition system considering interaction context, including: A data acquisition module, configured to: acquire an image, audio, and text to be recognized; A model recognition module, configured to: input the image, audio, and text into a multi-modal sentiment recognition model for processing respectively to obtain sentiment analysis results; The multi-modal sentiment recognition model includes a multi-modal feature extraction module, a joint cross-modal attention module, and a multi-layer perceptron connected in sequence; the image, audio, and text are input into the multi-modal feature extraction module respectively to obtain temporal enhanced spatial features, temporal audio features, and temporal text features, and are input into the joint cross-modal attention module together; the joint cross-modal attention module fuses the temporal enhanced spatial features, temporal audio features, and temporal text features to obtain a joint feature representation, then calculates the cross-correlations between the temporal enhanced spatial features, temporal audio features, temporal text features and the joint feature representation respectively to generate cross-correlation matrices, generates attention maps based on the cross-correlation matrices, then performs weighted summation of each attention map with the corresponding temporal enhanced spatial features, temporal audio features, and temporal text features respectively to obtain weighted features, re-inputs the weighted features into the joint cross-modal attention model for recursive optimization to obtain the final weighted features, splices the final weighted features to obtain a multi-modal feature representation and inputs it into the multi-layer perceptron to obtain sentiment analysis results.

[0017] In a third aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the steps in a cross-modal sentiment recognition method considering interaction context as described in the first aspect.

[0018] In a fourth aspect, the present invention provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps in a cross-modal emotion recognition method considering interaction context as described in the first aspect are implemented.

[0019] The above one or more technical solutions have the following beneficial effects: The present invention innovatively designs a multi-modal emotion analysis framework, namely a multi-modal emotion recognition model, including a multi-modal feature extraction module, a joint cross-modal attention module, and a multi-layer perceptron connected in sequence. The modal features obtained by the multi-modal feature extraction module are fused through the recursive joint cross-modal attention module. Through recursive optimization, the model continuously strengthens the feature interaction between different modalities, ensures the effective cooperation between different information sources, and enables the model to better capture the deep association between the emotion target and the context. The recursive joint cross-attention mechanism enhances the information flow and cooperation between modalities, and improves the accuracy and robustness of emotion analysis.

[0020] The present invention combines a variety of advanced technical means to overcome the deficiencies of the prior art in context information processing and emotion target recognition. The model framework of the present invention mainly introduces a spatial-channel attention network, an energy-based pairwise expert module, a temporal convolutional network, and a recursive joint cross-modal attention mechanism, effectively improving the performance of the multi-modal emotion recognition model, especially achieving remarkable breakthroughs in information interaction between modalities and context collaboration. First, the spatial-channel attention network module enhances the expressiveness of the visual modality through an adaptive spatial and channel attention mechanism, especially performing well in capturing context information in the visual modality, which is one of the core innovations of the model. This module can effectively ensure that the extraction of spatial features is consistent with the context interaction of other modalities, thereby improving the accuracy and robustness of emotion analysis. Second, the energy-based pairwise expert module is introduced into the output part of the text encoder, and the recognition ability of emotion targets is optimized by calculating the pairing stability of text segment boundaries. This module has significant advantages in modeling text context, ensuring the accurate extraction of text emotion targets, strengthening context association, and improving the overall emotion analysis effect. In addition, the recursive joint cross-modal attention mechanism continuously optimizes the feature interaction between audio, visual, and text modalities in a recursive manner. This mechanism particularly focuses on the recursive optimization of context relationships between modalities, making the fusion of modal information closer, effectively capturing the dynamic context in cross-modal emotion changes, significantly enhancing the correlation between different modalities, and providing stronger robustness and accuracy for emotion analysis tasks.

[0021] The present invention significantly improves the overall performance of the multi-modal emotion recognition model, especially in the processing of temporal information, the deep interaction of information fusion between modalities, and the accurate recognition of emotion targets, providing a more efficient and stable emotion analysis framework. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings forming a part of this specification are used to provide a further understanding of the present invention. The schematic embodiments and descriptions thereof of the present invention are used to explain the present invention and do not constitute an improper limitation of the present invention.

[0023] Figure 1 is a framework diagram of the multi-modal emotion recognition model RJCMA-IC according to an embodiment of the present invention; Figure 2 is a flowchart of the spatial-channel attention network according to an embodiment of the present invention; Figure 3 is a flowchart of the paired expert module according to an embodiment of the present invention; Figure 4 is a flowchart of the joint cross-modal attention module according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0025] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0026] In the case of no conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0027] Embodiment 1 As Figure 1 shown, this embodiment discloses a cross-modal emotion recognition method considering interactive context, and the method includes the following steps: S1: Obtain the images, audio, and text to be recognized; In this embodiment, the image, audio, and text data are acquired from the same scene. The image data acquires the facial information of the target object, i.e., the face image, through a video device. To ensure the consistency of the input data, the image undergoes preprocessing operations such as cropping, alignment, and size adjustment. The audio data extracts the speech signal from the corresponding video, uses the Vggish library to extract the Mel spectrogram features, and inputs these features into the audio encoder (VGG-Net) for processing. The text data is extracted from the audio signal through a speech recognition system (Vosk toolkit), the speech is transcribed, and the extracted text is processed by a text encoder (BERT) to extract the semantic features related to emotions.

[0028] S2: Input the image, audio, and text into the emotion recognition model for processing respectively to obtain the emotion analysis result.

[0029] The emotion recognition model includes a multi-modal feature extraction module, a joint cross-modal attention module, and a multi-layer perceptron connected in sequence. Input the image, audio, and text into the multi-modal feature extraction module respectively to obtain the temporal enhanced spatial feature, the temporal audio feature, and the temporal text feature, and input them into the joint cross-modal attention module together. The joint cross-modal attention module fuses the temporal enhanced spatial feature, the temporal audio feature, and the temporal text feature to obtain a joint feature representation, then calculates the cross-correlations between the temporal enhanced spatial feature, the temporal audio feature, the temporal text feature and the joint feature representation respectively to generate each cross-correlation matrix, generates each attention map based on each cross-correlation matrix, then performs weighted summation of each attention map with the corresponding temporal enhanced spatial feature, temporal audio feature, and temporal text feature respectively to obtain each weighted feature, re-input each weighted feature into the joint cross-modal attention model for recursive optimization to obtain the final each weighted feature, splice the final each weighted feature to obtain the multi-modal feature representation and input it into the multi-layer perceptron to obtain the emotion analysis result. The multi-modal feature extraction module includes a spatial feature extraction module, an audio feature extraction module, and a text feature extraction module arranged in parallel.

[0030] In this embodiment, the spatial feature extraction module includes a visual encoder, a spatial-channel attention network, and a temporal convolutional network connected in sequence. The spatial feature extraction module uses a pre-trained ResNet-50 network to extract the spatial features of the face image, and further performs adaptive spatial and channel attention calculations through the spatial-channel attention network. The spatial-channel attention mechanism can dynamically focus on the key regions in the image. Especially in complex scenarios, it can effectively handle interferences such as illumination changes and pose changes, thereby enhancing the model's perception ability of emotion-related details. In this process, the visual modality can not only process image information independently, but also enhance the information association with other modalities in a multi-modal environment, improving the effect of cross-modal fusion. Then, temporal modeling is performed through the temporal convolutional network to strengthen the temporal context perception of emotional changes.

[0031] The visual encoder is implemented using a pre-trained ResNet-50 network. The input image, that is, the face image (the face image containing facial expressions), is input into the visual encoder for feature extraction, and the spatial features of the facial expressions are output.

[0032] The spatial-channel attention network is a deep learning method that improves the ability to represent image features. It mainly strengthens the effect of feature extraction through a joint attention mechanism in the spatial dimension and the channel dimension. Visual data refers to continuous-frame face images containing facial expressions, that is, each video sequence contains multiple frames, and each frame is a face image, which is used to extract the spatial features of facial expressions. When processing visual data, the spatial-channel attention network ensures that the multi-modal emotion recognition model RJCMA-IC can automatically focus on the most meaningful parts of the image through two aspects of attention mechanisms (spatial attention and channel attention), especially when capturing the context information in the image. By effectively extracting the spatial information and channel information in the image, the spatial-channel attention network can enhance the context interaction between different modalities, thereby enhancing the expression ability of spatial features and improving the accuracy of emotion recognition.

[0033] The specific steps for the spatial-channel attention network to process are as follows: First, as Figure 2 shown, the input of the spatial-channel attention network is a dimensional spatial feature map that is, the spatial feature, where represents the number of channels, and respectively represent the height and width of the spatial feature map. The spatial-channel attention network extracts the corresponding spatial and channel features from the spatial feature map through convolution operations and generates a set of attention weights , with the aim of adaptively learning important features in both the spatial dimension and the channel dimension. The spatial-channel attention network first performs a convolution operation on the input feature map to obtain the feature weights for each spatial position and channel. The formula for this process can be expressed as:

[0034] where, represents the convolution operation, represents batch normalization, is the activation function with parameters, is the Sigmoid activation function. This operation transforms the spatial feature map into an -dimensional attention weight , which is used for subsequent weighted processing of the spatial feature map. Through these attention weights, the spatial-channel attention network can adaptively focus on the key regions and important features in the image in both the spatial and channel dimensions.

[0035] Next, the spatial-channel attention network uses these calculated attention weights to perform a weighting operation on the original input feature map, i.e., the spatial feature map , so as to obtain the enhanced output feature map , i.e., the attention-weighted feature map. This process is completed through an element-wise multiplication operation, expressed as:

[0036] where, represents the element-wise multiplication operator. Through this weighting process, the spatial-channel attention network enables the multi-modal emotion recognition model to focus on the important regions and features relevant to the task in both the spatial and channel dimensions, and ignore those irrelevant or noisy parts.

[0037] This series of operations constitutes the spatial-channel attention network mechanism, which adaptively enhances the most emotion-relevant parts in the image by learning the local and global context information of the image. Through such detailed feature weighting, the spatial-channel attention network can effectively improve the representation ability of visual information, thereby providing higher-quality feature inputs for subsequent tasks such as emotion analysis or object detection.

[0038] The attention-weighted feature map output by the spatial-channel attention network is input into the temporal convolutional network for temporal modeling, and the spatial features with context information in the time dimension, i.e., the temporally enhanced spatial features, are output. The temporally enhanced spatial features represent the features of each time step in the face image sequence.

[0039] In this embodiment, the audio feature extraction module includes an audio encoder and a temporal convolutional network connected in sequence. The audio encoder is implemented using the convolutional neural network VGG-Net. First, the acquired audio (i.e., speech signal) is converted into a Mel spectrogram, and the Mel spectrogram is input into the audio encoder for feature extraction, outputting audio features, i.e., the frequency features of the audio signal. The audio features are input into the temporal convolutional network for temporal modeling, outputting enhanced features with temporal context information in the time dimension, i.e., temporal audio features. The temporal audio features are a temporal feature matrix.

[0040] The audio encoding part uses the VGG network to extract features from the Mel spectrogram. The VGG network can efficiently capture the frequency features in the audio signal and extract emotional information. Then, the audio features are temporally modeled through the temporal convolutional network, avoiding the problem of gradient disappearance that easily occurs in traditional RNN models for long temporal data and significantly enhancing the response ability of the audio modality to dynamic emotional changes.

[0041] In this embodiment, the text feature extraction module includes a text encoder, a pairwise expert module, and a temporal convolutional network connected in sequence. The text feature extraction module uses the BERT model to extract text features. To further enhance the expression of text information, an energy-based pairwise expert module is introduced, which optimizes the recognition ability of emotional targets by calculating the pairing stability of text segment boundaries. The energy-based pairwise expert module can further refine the emotional details of text information and strengthen the context docking of the text modality, thereby improving the accuracy of overall emotion recognition. After that, the text features are also temporally modeled through the temporal convolutional network to capture the dynamic changes of emotions.

[0042] The text encoder uses the BERT model. The acquired text is input into the text encoder for feature extraction, outputting text features. BERT can improve the accuracy of sentiment analysis through bidirectional encoding.

[0043] The pairwise expert module is specifically an energy-based pairwise expert module. Its core goal is to optimize the recognition of target spans in sentiment analysis, especially having significant advantages when dealing with context relationships in text. It evaluates the pairing stability between the start and end boundaries of text segments through an energy-based model, thereby improving the accuracy of emotion recognition. This module calculates the pairing energy for each pair of boundaries and assigns a score to them based on their consistency in the context. A lower energy value indicates that the boundary pair is more stable and has a stronger correlation in the emotional context, and is more likely to be an emotional target; while a higher energy value indicates that their pairing is not semantically tight enough and lacks sufficient context support. In this way, this module can accurately identify emotional targets in text and enhance the context interaction ability in sentiment analysis.

[0044] As Figure 3 shown, the process of the energy-based pairwise expert module involves defining an energy function to evaluate the pairing stability of each pair of text segment boundaries. Let the feature sequence of the text, i.e., the text feature, be , where is the length of the sequence, and the start and end boundaries of each segment are represented by and respectively. The energy-based pairwise expert module first calculates the energy value for each pair of boundaries, and the formula for this process is as follows:

[0045] where and are learnable parameter matrices representing the weights used to calculate the energy; and are the representations of the start and end boundaries, is the transpose of , and is the transpose of . This energy function calculates the mutual influence between the start and end positions of the text segments. A lower energy value indicates a stronger association between these two positions, while a higher energy value means a weaker association.

[0046] Each point in the figure represents a pair of sentiment span boundaries, which are distributed in terms of sentiment categories according to the energy values. Positive, neutral, and negative are sentiment classification labels indicating the sentiment tendencies of different sentiment spans. The stability of the pairing of each pair of sentiment span boundaries and the correlation with the sentiment category are shown through the energy distribution diagram.

[0047] Next, a system-level energy function is used to calculate the energy of all boundary pairs in the entire text sequence. Assume is the target label matrix, where indicates whether and belong to valid target segment boundaries (e.g., whether they correspond to "sentiment words" in sentiment analysis). The energy-based pairwise expert calculates the system-level energy through the following formula:

[0048] where represents the system energy, represents the system state, and , represent the row and column indices in the target label matrix respectively.

[0049] This system-level energy function will help evaluate the overall matching degree between multiple boundary pairs and optimize the performance of the model during the inference phase. By optimizing the system energy, select those label pairs with strong pairing ability (i.e., low energy). The system energy is calculated by computing the pairing energy for each pair of possible boundaries and summing them up, which can provide a quantitative assessment of the pairing ability of text label pairs. Minimizing the system energy value means that the model selects those most stable and relevant boundary pairings (i.e., text label pairs). Finally, the model is trained using the cross-entropy loss function to minimize the gap between the energy value and the actual labels. The form of the loss function is as follows:

[0050] where is the predicted value of the boundary pair calculated by the energy function, is the Sigmoid function. By minimizing this loss function, the energy-based pairwise expert module can optimize the identification of boundary pairs and improve the accuracy of the target span.

[0051] The energy-based pairwise expert module optimizes the boundary matching of the target span through the energy function and uses the cross-entropy loss to train the model, thereby significantly improving the identification effect of sentiment targets in sentiment analysis tasks. Through this method, the pairing relationship between different text segments can be effectively learned, providing strong support for multi-modal sentiment analysis. The target span refers to the start and end boundaries of the label pair in the sentiment analysis task.

[0052] The temporal convolutional network receives the pairing energy between the text label pairs output by the energy-based pairwise expert module, multiplies it element-wise with the text features as a weighting coefficient, and performs temporal modeling on it to obtain temporal text features.

[0053] In this embodiment, the joint cross-modal attention module introduces a recursive joint cross-modal attention mechanism, which continuously optimizes the feature interaction of temporal audio features (audio modality), temporal enhanced spatial features (visual modality), and temporal text features (text modality) in a recursive manner, especially playing a key role in modeling the context relationship between modalities. By gradually enhancing the connection between each modality feature and other modality features, the recursive joint cross-modal attention mechanism can effectively capture the context correlation in multi-modal information, thereby enhancing the synergy between different modalities. Finally, this recursive optimization process improves the accuracy and robustness of multi-modal information fusion, ensuring that the model can accurately identify sentiment changes and generate more accurate sentiment analysis results.

[0054] As Figure 4 shown, at the initial stage of the joint cross-modal attention module, the temporal audio features, temporal enhanced spatial features, and temporal text features are respectively represented as , and , these features are fused through a fully connected layer (FC) to obtain a joint feature representation . Here, , and , where , , are the dimensions of the temporal audio feature, the temporal enhanced spatial feature, and the temporal text feature respectively, is the temporal length (i.e., the time step of the input data). The fused joint feature representation has a dimension of , and .

[0055] Next, the joint cross-modal attention module calculates the cross-correlation between each modal feature and the joint feature representation to generate a cross-correlation matrix for each modality , and , denoted as:

[0056]

[0057]

[0058] where, represents the audio cross-correlation matrix, represents the visual cross-correlation matrix, represents the text cross-correlation matrix, , , are trainable weight matrices, is used for normalization. Through these matrices, the model calculates the cross-correlation between each modality and the joint modal feature (i.e., the joint feature representation) to obtain the attention weights for each modality.

[0059] Based on the cross-correlation matrix for each modality , and are further used to generate an attention map for each modality , , , denoted as:

[0060]

[0061]

[0062] Among them, represents the audio attention map, represents the visual attention map, represents the text attention map, , , are trainable weight matrices; is an activation function that turns the negative value of each element into 0 and retains the positive value. Through the calculated attention maps, the model can focus on the important features in each modality.

[0063] Next, the attention maps of each modality , , are respectively weighted and summed with the corresponding temporal audio features , temporal enhanced spatial features , and temporal text features to obtain the weighted feature representations , , , expressed as:

[0064]

[0065]

[0066] Among them, represents the weighted audio feature, represents the weighted enhanced spatial feature, represents the weighted text feature, , , are learned weight matrices. This process updates the features through weighting, enabling each modality to be further optimized in the subsequent recursive iterations.

[0067] In the recursive stage, the weighted features, namely the weighted audio feature , weighted enhanced spatial feature , and weighted text feature will be input into the joint cross-modal attention module again for optimization. That is to say, the weighted audio feature serves as the of the current round, the weighted enhanced spatial feature serves as the of the current round, and the weighted text feature serves as the of the current round for optimization. This recursive process is updated through the following formula:

[0068]

[0069]

[0070] Among them, represents the recursive round, and the weighted features of each round are used as the input for the next round of recursion. As the recursion progresses, the information interaction between modalities is gradually enhanced, and the representation of features becomes more accurate.

[0071] After the recursive optimization is completed, the weighted features of the final round , , will be concatenated together to form the final multi-modal feature representation:

[0072] Among them, represents the multi-modal feature representation.

[0073] The stopping condition of the recursion is usually based on the evaluation of the model performance. In this embodiment, the criterion for the completion of the recursion is that the model performance is the best when the number of recursions is 3 times. Increasing the number of recursive steps may lead to a decrease in performance, so it can be judged that the recursion is completed.

[0074] By recursively iterating this process, the multi-modal sentiment recognition model RJCMA-IC can continuously strengthen the fusion between modalities in multiple rounds of interaction, and finally obtain accurate sentiment analysis results. This recursive mechanism enables RJCMA-IC to effectively process multi-modal data, especially in complex sentiment scenarios, showing stronger adaptability and robustness.

[0075] In this embodiment, the multi-modal feature representation is input into a multi-layer perceptron (MLP) for sentiment prediction, and the regression prediction of the sentiment dimension is output, so as to obtain the result of sentiment analysis. The regression prediction includes sentiment value and arousal.

[0076] The present invention innovatively designs a multi-modal emotion recognition model, which mainly combines a spatial-channel attention network, energy-based pairwise experts, a temporal convolutional network, and a recursive joint cross-attention mechanism. The aim is to effectively fuse the features of audio, visual, and text modalities, improving the accuracy and robustness of emotion analysis. The spatial-channel attention network can effectively handle the interference caused by factors such as occlusion and pose changes in the extraction of spatial features of facial expressions through an adaptive spatial-channel attention mechanism. It not only optimizes the extraction of visual information but also enhances the context awareness in visual emotion analysis. The energy-based pairwise experts module optimizes the recognition of emotion targets by calculating the pairing stability of text segments, optimizing the interaction context of text emotion information between different modalities and ensuring the accurate modeling of emotion targets within the text. The temporal convolutional network enhances the ability to model temporal data, ensuring that the model can accurately capture emotion changes in the temporal context, making emotion analysis more accurate and stable. The recursive joint cross-attention mechanism enables the model to deeply explore the emotional connections between different modalities through recursive optimization, ensuring the deep interaction and integration of emotion targets across multiple modalities, thus enhancing the context correlation in multi-modal emotion analysis.

[0077] Based on the existing multi-modal emotion analysis technology, the present invention combines a variety of advanced technical means, overcomes the deficiencies in the existing methods, and proposes a new multi-modal emotion analysis framework. In particular, significant progress has been made in the modeling of interaction context, thus providing a more efficient and accurate technical solution for the field of emotion computing.

[0078] The present invention significantly improves the accuracy and robustness of emotion analysis, especially making remarkable progress in the modeling of context interaction. Its advantages are reflected in many aspects: First, in terms of visual processing, the spatial-channel attention network can effectively focus on emotion-related visual regions by adaptively calculating the spatial and channel attention of images, overcoming the influence of factors such as illumination changes, occlusion, and pose changes on the extraction of spatial features of facial expressions. Through the spatial and channel context interaction, this mechanism helps the model to more accurately understand the emotion information in the image, ensuring the deep association between the spatial features of facial expressions and other modalities, and enhancing the accuracy and robustness of emotion analysis.

[0079] Secondly, in terms of text processing, the present invention adopts an energy-based pairwise expert module, which optimizes the recognition of sentiment targets by calculating the pairing stability of text segment boundaries. This module can effectively improve the expressiveness of text features, especially in the sentiment analysis of long texts, and can more accurately identify sentiment targets. By optimizing the context connection in text information, the energy-based pairwise expert module not only enhances the model's perception ability of sentiment details, but also enables the model to more efficiently and accurately process complex contexts in text in cross-modal sentiment analysis.

[0080] For the processing of time series data, the present invention adopts a common time series convolutional network, which can more efficiently process long time series data and avoid the problem of gradient disappearance. In sentiment analysis tasks, especially when processing time series data such as audio and video, the time series convolutional network can significantly improve the ability to capture sentiment changes, ensure that the model can more accurately track the dynamic context of sentiment, and make sentiment analysis more stable and accurate.

[0081] By jointly using the features of audio, visual, and text modalities, the present invention performs information fusion through a recursive joint cross-modal attention mechanism. The recursive joint cross-modal attention mechanism continuously optimizes the interaction of features in different modalities in a recursive manner, making the association between different modalities closer. Through this recursive optimization of the interaction of context information, the model can strengthen the context fusion between modalities in multiple iterative steps, ensure the deep integration of information in each modality in sentiment analysis, and thus improve the accuracy and robustness of overall sentiment analysis.

[0082] By combining multiple technical means, the present invention significantly improves the multi-modal fusion ability, time series modeling ability, and sentiment target recognition accuracy of the sentiment analysis model, especially making remarkable progress in the optimization of context interaction. In the face of complex sentiment scenarios and dynamic sentiment changes, the present invention demonstrates better performance than the prior art.

[0083] In summary, in the visual modality part of the present invention, the pre-trained ResNet-50 is used to extract the spatial features of facial expressions, and then the spatial features are processed by a spatial-channel attention network. The spatial-channel attention network captures local and global important information in the input image by performing adaptive attention calculations in the spatial and channel dimensions, especially being able to effectively handle the interference caused by factors such as occlusion, pose changes, and lighting changes in complex environments. The local and global important information in the image is captured through an adaptive spatial-channel attention mechanism, which enables the model to focus on the visual context related to emotions rather than simply relying on all regions in the image, thus enhancing the model's perception ability of emotion-related details. At the same time, in the text modality part, the BERT model is used to encode text data. To further improve the expressiveness of text features, an energy-based pairwise expert module is introduced into the model. This module optimizes the recognition of emotion targets by calculating the pairing stability of the boundaries of text segments, ensuring that the model efficiently and accurately models the text structure in emotion analysis. The temporal convolutional network performs temporal modeling on the features, further enhancing the model's ability to capture dynamic emotion changes. After combining audio, visual, and text features, the RJCMA-IC model performs cross-modal information fusion through a recursive joint cross-modal attention mechanism. This mechanism continuously optimizes the interaction of features in each modality in a recursive manner, making the association between different modalities closer. In short, the RJCMA-IC model enables the model to more accurately capture emotion information in multi-modal emotion analysis through multiple levels of context mechanisms, including visual, text, and cross-modal contexts. At the same time, through the above design, audio, visual, and text information can be effectively combined in the emotion recognition task, and by strengthening the correlation and temporal features between modalities, the accuracy and robustness of emotion recognition are significantly improved.

[0084] Embodiment 2 This embodiment discloses a cross-modal emotion recognition system considering interaction context, including: A data acquisition module, which is configured to: acquire images, audio, and text to be recognized; A model recognition module, which is configured to: respectively input the images, audio, and text into an emotion recognition model for processing to obtain emotion analysis results; The emotion recognition model includes a multi-modal feature extraction module, a joint cross-modal attention module, and a multi-layer perceptron that are sequentially connected in order; the image, audio, and text are respectively input into the multi-modal feature extraction module to obtain a temporal enhanced spatial feature, a temporal audio feature, and a temporal text feature, and are input into the joint cross-modal attention module together; the joint cross-modal attention module fuses the temporal enhanced spatial feature, the temporal audio feature, and the temporal text feature to obtain a joint feature representation, and then calculates the cross-correlations between the temporal enhanced spatial feature, the temporal audio feature, the temporal text feature and the joint feature representation respectively, generates each cross-correlation matrix, generates each attention mapping based on each cross-correlation matrix, then performs weighted summation of each attention mapping with the corresponding temporal enhanced spatial feature, temporal audio feature, and temporal text feature respectively to obtain each weighted feature, re-inputs each weighted feature into the joint cross-modal attention model for recursive optimization to obtain the final weighted features, splices the final weighted features to obtain a multi-modal feature representation and inputs it into the multi-layer perceptron to obtain the emotion analysis result.

[0085] Embodiment III The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the method in Embodiment I are implemented.

[0086] Embodiment IV The purpose of this embodiment is to provide a computer-readable storage medium. A computer-readable storage medium stores a computer program, and when the program is executed by a processor, the steps of the method in Embodiment I are executed.

[0087] The steps involved in the devices in the above Embodiments III and IV correspond to those in Method Embodiment I, and the specific implementation manners can be referred to the relevant description part of Embodiment I. The term "computer-readable storage medium" should be understood to include a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode, or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0088] Those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computer device. Optionally, they can be implemented by program codes executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be separately made into individual integrated circuit modules, or multiple modules or steps among them can be made into a single integrated circuit module to implement. The present invention is not limited to any specific combination of hardware and software.

[0089] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

[0090] Although the specific implementation manners of the present invention are described above in conjunction with the accompanying drawings, it is not a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications or deformations that can be made without creative efforts on the basis of the technical solution of the present invention are still within the protection scope of the present invention.

Claims

1. A cross-modal emotion recognition method considering interaction context, characterized in that Including: Obtaining an image, audio, and text to be recognized; Respectively inputting the image, audio, and text into a multi-modal sentiment recognition model for processing to obtain a sentiment analysis result; The multi-modal sentiment recognition model includes a multi-modal feature extraction module, a joint cross-modal attention module, and a multi-layer perceptron connected in sequence; the image, audio, and text are respectively input into the multi-modal feature extraction module to obtain a temporal enhanced spatial feature, a temporal audio feature, and a temporal text feature, and are jointly input into the joint cross-modal attention module; the joint cross-modal attention module fuses the temporal enhanced spatial feature, the temporal audio feature, and the temporal text feature to obtain a joint feature representation, then calculates the cross-correlations between the temporal enhanced spatial feature, the temporal audio feature, the temporal text feature, and the joint feature representation respectively, generates each cross-correlation matrix, generates each attention map based on each cross-correlation matrix, then respectively performs weighted summation of each attention map with the corresponding temporal enhanced spatial feature, temporal audio feature, and temporal text feature to obtain each weighted feature, re-inputs each weighted feature into the joint cross-modal attention model for recursive optimization to obtain the final each weighted feature, splices the final each weighted feature to obtain a multi-modal feature representation and inputs it into the multi-layer perceptron to obtain a sentiment analysis result.

2. The cross-modal sentiment recognition method considering interaction context according to claim 1, characterized in that The multi-modal feature extraction module includes a spatial feature extraction module, an audio feature extraction module, and a text feature extraction module in parallel.

3. The cross-modal emotion recognition method considering interaction context according to claim 2, wherein The spatial feature extraction module includes a visual encoder, a spatial-channel attention network, and a temporal convolutional network connected in sequence; the image is input into the visual encoder for feature extraction to obtain a spatial feature, the spatial-channel attention network strengthens the spatial feature through a joint attention mechanism in the spatial dimension and the channel dimension to obtain an enhanced spatial feature and inputs it into the temporal convolutional network for temporal modeling to obtain a temporal enhanced spatial feature; The visual encoder uses a pre-trained ResNet-50 network.

4. The cross-modal sentiment recognition method considering interaction context according to claim 3, characterized in that The spatial-channel attention network first performs convolution on the spatial feature to obtain an attention weight, and weights the spatial feature based on the attention weight to obtain an attention weighted feature.

5. The cross-modal emotion recognition method considering interaction context according to claim 2, wherein The audio feature extraction module includes an audio encoder and a temporal convolutional network connected in sequence; the audio is converted into a Mel spectrogram and input into the audio encoder for feature extraction to obtain an audio feature and input into the temporal convolutional network for temporal modeling to obtain a temporal audio feature; the audio encoder uses a convolutional neural network.

6. The cross-modal sentiment recognition method considering interaction context according to claim 2, characterized in that, The text feature extraction module includes a text encoder, a pairwise expert module, and a temporal convolutional network connected in sequence; the text is input into the text encoder for feature extraction to obtain text features, which are then input into the pairwise expert module. The pairwise expert module optimizes the boundary matching of the text feature span through an energy function to obtain the pairing energy between text label pairs, multiplies it with the text features as a weighting coefficient, and inputs the result into the temporal convolutional network for temporal modeling to obtain temporal text features; the pairwise expert module calculates the system energy to obtain the pairing energy between text label pairs.

7. A cross-modal sentiment recognition method considering interaction context according to claim 1, characterized in that The temporal enhanced spatial features, temporal audio features, and temporal text features are fused through a fully connected layer to obtain a joint feature representation.

8. A cross-modal sentiment recognition system considering interaction context, characterized in that, It includes: A data acquisition module configured to: acquire images, audio, and text to be recognized; A model recognition module configured to: input the images, audio, and text into a multi-modal sentiment recognition model for processing respectively to obtain sentiment analysis results; The multi-modal sentiment recognition model includes a multi-modal feature extraction module, a joint cross-modal attention module, and a multi-layer perceptron connected in sequence; the images, audio, and text are input into the multi-modal feature extraction module respectively to obtain temporal enhanced spatial features, temporal audio features, and temporal text features, and are then input into the joint cross-modal attention module together; the joint cross-modal attention module fuses the temporal enhanced spatial features, temporal audio features, and temporal text features to obtain a joint feature representation, then calculates the cross-correlations between the temporal enhanced spatial features, temporal audio features, temporal text features and the joint feature representation respectively to generate cross-correlation matrices, generates attention maps based on the cross-correlation matrices, then performs weighted summation of each attention map with the corresponding temporal enhanced spatial features, temporal audio features, and temporal text features respectively to obtain weighted features, re-inputs the weighted features into the joint cross-modal attention model for recursive optimization to obtain the final weighted features, splices the final weighted features to obtain a multi-modal feature representation and inputs it into the multi-layer perceptron to obtain sentiment analysis results.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps in a cross-modal sentiment recognition method considering interaction context as described in any one of claims 1-7.

10. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in a cross-modal sentiment recognition method considering interaction context as described in any one of claims 1-7.

Images