A multi-modal sentiment analysis method, device, equipment and storage medium

Abstract

The application discloses a kind of multimodal sentiment analysis method, device, equipment and storage medium.The method includes: constructing the sentiment analysis dataset containing text, audio and visual sequence and inputting pre-training model, extracting each modality initial feature using feature coding module, comparing learning and dynamic fusion are carried out through modal common information processing module to obtain cross-modal common feature, with the aid of modal specificity information processing module, feature enhancement and pooling are implemented to generate global enhanced features, finally, each kind of feature is spliced and regression prediction is executed to output sentiment intensity prediction value by fusion prediction module;Since the application cooperates modeling modal common and specificity information, the modal heterogeneity is considered and the average processing bottleneck is broken, different dimension features can be dynamically distinguished, the modal conflict can be effectively relieved, and the prediction accuracy and robustness of sentiment analysis can be significantly improved.

Description

Technical Field

[0001] This invention relates to the field of multimodal data analysis, and more particularly to a multimodal sentiment analysis method, apparatus, device, and storage medium. Background Technology

[0002] Currently, research on multimodal sentiment analysis mainly relies on the joint modeling and analysis of heterogeneous data from multiple sources, such as text, audio, and video. Learning effective joint representations is fundamental to multimodal sentiment analysis and is crucial for improving machines' ability to understand complex human emotional interactions. Existing research methods typically use complex networks to directly construct joint multimodal representations. While this can fuse multiple types of information to some extent, it has significant limitations in handling the inherent heterogeneity of different modalities and preserving modality-specific information. With the rapid development of deep learning technology, fusion models based on attention mechanisms and graph neural networks have gradually become an important research direction in this field, demonstrating a more comprehensive understanding potential than traditional single-modal methods. However, in practical applications, the data distribution characteristics of different modalities vary greatly, and their contributions to sentiment recognition differ. Existing methods tend to treat all modalities equally, failing to fully consider the complementary and interfering relationships between modalities. This one-size-fits-all modeling approach often leads to models failing to fully explore the rich, fine-grained emotional cues in the text modality and also makes it difficult to effectively decouple redundant information between modalities during the fusion process. Therefore, how to construct a unified joint representation while taking into account the heterogeneity between modalities, breaking the averaging process for all modalities, and deeply mining the deep semantics in specific modalities (such as text) remains a key problem that needs to be solved in the field of multimodal sentiment analysis. Summary of the Invention

[0003] The main objective of this invention is to provide a multimodal sentiment analysis method, apparatus, device, and storage medium, aiming to solve the technical problems of existing technologies that are difficult to construct a unified joint representation while taking into account the heterogeneity between modalities, breaking the averaging process of all modalities, and deeply mining the deep semantics in specific modalities.

[0004] To achieve the above objectives, the present invention provides a multimodal sentiment analysis method, the method comprising the following steps: A multimodal sentiment analysis dataset is constructed, which includes text modality sequences, audio modality sequences, and visual modality sequences; The multimodal sentiment analysis dataset is input into a pre-built multimodal sentiment analysis prediction model to obtain the predicted sentiment intensity value; The multimodal sentiment analysis and prediction model includes a feature encoding module, a modality common information processing module, a modality specific information processing module, and a fusion prediction module. The feature encoding module is configured to extract features from the text modality sequence, the audio modality sequence, and the visual modality sequence to obtain initial text features, initial audio features, and initial visual features; The modal shared information processing module is configured to perform comparative learning and dynamic fusion of the initial text features, the initial audio features, and the initial visual features to obtain cross-modal shared features of text and audio and cross-modal shared features of text and vision. The modality-specific information processing module is configured to perform feature enhancement and pooling processing on the initial audio features and the initial visual features to obtain global audio enhancement features and global visual enhancement features. The fusion prediction module is configured to concatenate and regress the cross-modal shared features of the text audio, the cross-modal shared features of the text visual, the global audio enhancement features, and the global visual enhancement features, and output a predicted value of sentiment intensity.

[0005] Furthermore, to achieve the above objectives, the present invention also proposes a multimodal sentiment analysis device that applies the multimodal sentiment analysis method described above, the multimodal sentiment analysis device comprising: A dataset construction module is used to construct a multimodal sentiment analysis dataset, which includes text modality sequences, audio modality sequences, and visual modality sequences. The sentiment analysis module is used to input the multimodal sentiment analysis dataset into a pre-built multimodal sentiment analysis prediction model to obtain a predicted sentiment intensity value; The multimodal sentiment analysis and prediction model includes a feature encoding module, a modality common information processing module, a modality specific information processing module, and a fusion prediction module. The feature encoding module is configured to extract features from the text modality sequence, the audio modality sequence, and the visual modality sequence to obtain initial text features, initial audio features, and initial visual features; The modal shared information processing module is configured to perform comparative learning and dynamic fusion of the initial text features, the initial audio features, and the initial visual features to obtain cross-modal shared features of text and audio and cross-modal shared features of text and vision. The modality-specific information processing module is configured to perform feature enhancement and pooling processing on the initial audio features and the initial visual features to obtain global audio enhancement features and global visual enhancement features. The fusion prediction module is configured to concatenate and regress the cross-modal shared features of the text audio, the cross-modal shared features of the text visual, the global audio enhancement features, and the global visual enhancement features, and output a predicted value of sentiment intensity.

[0006] Furthermore, to achieve the above objectives, this application also proposes a multimodal sentiment analysis device, the device comprising: a memory, a processor, and a multimodal sentiment analysis program stored in the memory, the processor being configured to run the multimodal sentiment analysis program, the computer program being configured to implement the steps of the multimodal sentiment analysis method as described above.

[0007] In addition, to achieve the above objectives, this application also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the multimodal sentiment analysis method described above.

[0008] This invention addresses the challenges of eliminating intermodal heterogeneity in multimodal sentiment analysis, the tendency to ignore modality specificity during information fusion, and information loss due to averaging. It proposes a multimodal sentiment analysis prediction model that, by designing both a modality-shared information processing module and a modality-specific information processing module, achieves collaborative modeling of cross-modal shared features and single-modal specific features, overcoming the bottleneck of traditional models' averaging across all modalities. Regarding shared information mining, the model introduces a two-stage contrastive learning and text-guided dynamic fusion network. A hard negative sample mining algorithm enhances the model's ability to acquire deep shared similarity information and strengthens the dominant role of the text modality in the fusion process, effectively mitigating modality heterogeneity. For specific information enhancement, an autoencoder is used to reconstruct features from the visual and audio modalities, coupled with joint loss function constraints, ensuring that the enhanced features maintain consistency with the task label distribution while preserving the attributes of the original feature subspace. In the prediction and decision-making stage, the model innovatively uses a delimiter vector to concatenate cross-modal shared features and global enhancement features, enabling it to dynamically distinguish and utilize feature information from different dimensions based on the consistency or conflict between modalities. This achieves superior sentiment prediction accuracy in various complex situations, effectively improving the accuracy and robustness of multimodal sentiment analysis. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a schematic diagram of the structure of a multimodal sentiment analysis device in the hardware operating environment involved in the embodiments of the present invention; Figure 2 This is a flowchart illustrating the first embodiment of the multimodal sentiment analysis method of the present invention; Figure 3 This is a flowchart illustrating the second embodiment of the multimodal sentiment analysis method of the present invention; Figure 4 This is a flowchart illustrating the third embodiment of the multimodal sentiment analysis method of the present invention; Figure 5 This is a structural block diagram of the first embodiment of the multimodal emotion analysis device of the present invention.

[0011] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0012] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0013] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a multimodal sentiment analysis device in the hardware operating environment involved in the embodiments of the present invention.

[0014] like Figure 1 As shown, the multimodal sentiment analysis device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; the user interface 1003 may also include standard wired and wireless interfaces. The network interface 1004 may optionally include standard wired and wireless interfaces (such as Wireless-Fidelity (Wi-Fi) interfaces). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0015] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on a multimodal sentiment analysis device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0016] like Figure 1 As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a multimodal sentiment analysis program.

[0017] exist Figure 1 In the multimodal sentiment analysis device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the multimodal sentiment analysis device of the present invention can be set in the multimodal sentiment analysis device, and the multimodal sentiment analysis device calls the multimodal sentiment analysis program stored in the memory 1005 through the processor 1001 and executes the multimodal sentiment analysis method provided in the embodiment of the present invention.

[0018] This invention provides a multimodal sentiment analysis method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the multimodal sentiment analysis method of the present invention.

[0019] In this embodiment, the multimodal sentiment analysis method includes the following steps: Step S10: Construct a multimodal sentiment analysis dataset.

[0020] It should be noted that this embodiment can be applied to scenarios such as intelligent customer service, digital human interaction, and online psychological consultation.

[0021] It should be understood that the executing entity of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or a terminal electronic device capable of performing the above functions. The following description uses a multimodal sentiment analysis device (hereinafter referred to as the analysis device) as an example to illustrate this embodiment and the following embodiments.

[0022] It should be noted that a multimodal sentiment analysis dataset refers to a collection of data that includes information from multiple sensory dimensions such as text, audio, and visual data, and is labeled with sentiment tags. In the context of intelligent customer service, this dataset can be a combination of voice call recordings of customer complaints, text content transcribed from speech, and videos of users' anxious facial expressions captured by a camera.

[0023] Multimodal sentiment analysis datasets include text modal sequences, audio modal sequences, and visual modal sequences. A text modal sequence refers to a stream of characters arranged chronologically after speech signals have been converted into text using recognition technology. For example, in digital human interaction, this could be the text character sequence corresponding to a user saying "I'm in a good mood today."

[0024] An audio modal sequence can be a set of speech acoustic features (such as pitch, volume, and frequency) arranged in chronological order. For example, in intelligent quality inspection, the increased volume and rapid speech rate when a customer is angry are characteristics of this.

[0025] A visual modal sequence refers to a series of consecutive image frames arranged in chronological order. For example, in virtual anchor interaction, it refers to video clips sent by viewers in the live broadcast room that include facial expressions such as smiling, frowning, or nodding.

[0026] In the specific implementation, video data with complete facial and voice signals is acquired, such as customer satisfaction feedback videos recorded in intelligent customer service systems. The video is segmented into short clips of 2-5 seconds. For each clip, synchronous text is generated using speech recognition technology, and the video footage, audio, and text signals are time-aligned to form trimodal samples. Finally, the samples are labeled with sentiment intensity, and the labeling results are used for model training.

[0027] Furthermore, to enhance data quality and thus improve the accuracy of subsequent sentiment analysis and prediction, step S10 may include: Step S101: Obtain the original video of the person commenting, and divide the original video of the person commenting into multiple video segments according to a preset time interval.

[0028] It should be noted that the original personal commentary video can be multimedia material such as film reviews, confessions, interviews, or short videos on social media that include facial expressions, gestures, and clear audio signals.

[0029] The preset time interval refers to the fixed step size or duration of video cutting that is pre-set within the system in order to adapt to the length limitations of the input sequence of the deep learning model and to capture subtle changes in emotion.

[0030] Step S102: Extract speech signals from the multiple video segments, and perform speech-to-text processing on the extracted speech signals to generate text signals synchronized with the speech signals.

[0031] It should be noted that a text signal refers to a collection of electronic text data generated after speech recognition, which includes word content information and start and end timestamps for each word.

[0032] Step S103: Perform timeline alignment on the video clip, the audio signal, and the text signal to obtain a three-modal alignment sample, which includes a text modal sample, an audio modal sample, and a visual modal sample.

[0033] In some embodiments, the analysis device can use a built-in temporal alignment algorithm to register the three—the frame sequence of the video clip as a global time reference, the text time stamps generated in the previous steps, and the audio sampling point index—at the millisecond level, eliminating any asynchronous parts caused by network packet loss or recognition delay. This ensures that the micro-expressions in the visual image, the volume fluctuations in the audio, and the strong words in the text all mathematically point to the same emotional moment. For example, in a sample showing the emotion of "anger," it ensures that the frame of the user frowning, the audio track of raising their voice, and the text vector of the word "bad review" are tightly locked in the same trimodal aligned sample container.

[0034] Step S104: Label the three-modal aligned samples with sentiment intensity labels, and summarize the labeled three-modal aligned samples to construct a multimodal sentiment analysis dataset.

[0035] It should be noted that the emotional intensity label is a numerical identifier used to quantify the direction and intensity of the emotional polarity of a sample, usually expressed as a continuous numerical value or a discrete level.

[0036] In practice, the analysis device acquires multimodal sentiment analysis data. Optional data sources include videos of film reviews featuring complete facial and voice signals, or videos of personal monologues with complete facial and voice signals.

[0037] In constructing the multimodal sentiment analysis dataset, videos need to be segmented into video clips ranging from 2 to 5 seconds. For each video clip, speech recognition is performed to generate a text signal synchronized with the speech. The video, speech, and text signals are then time-aligned to obtain samples containing text, audio, and visual modalities. For each sample, sentiment intensity is labeled (ranging from -3 to 3, where positive and negative represent positive and negative sentiment polarities, respectively, and the absolute value represents sentiment intensity). The labeled results serve as sample labels for training the multimodal sentiment analysis model.

[0038] Step S20: Input the multimodal sentiment analysis dataset into the pre-built multimodal sentiment analysis prediction model to obtain the predicted sentiment intensity value.

[0039] It should be noted that the multimodal sentiment analysis prediction model includes a feature encoding module, a modality common information processing module, a modality specific information processing module, and a fusion prediction module; In the specific implementation, during the model building phase, the analysis device initializes the parameters of each network layer according to the aforementioned multimodal sentiment analysis prediction model topology. The weights of the pre-trained language model are kept frozen or fine-tuned with a low learning rate, while the remaining components use a standard random initialization strategy. During the training phase, the pre-defined training batches of data are input into the model for forward propagation, sequentially calculating the task loss, the first-stage contrastive loss, the second-stage contrastive loss, and the joint loss. The task loss is measured using the mean absolute error between the predicted value and the true sentiment intensity label. The optimizer uses an adaptive moment estimation algorithm to weight and sum the four losses according to preset hyperparameter weights to obtain the total loss function. The gradient of the total loss with respect to each trainable parameter is calculated using the backpropagation algorithm, and parameter updates are performed. During training, a learning rate warm-up and cosine decay strategy are introduced, and an early stopping mechanism is set to monitor the validation set performance metrics. Training is terminated when the validation set error fails to decrease for several consecutive cycles, and the optimal weight file is saved as the final multimodal sentiment analysis prediction model.

[0040] In this embodiment, each module in the multimodal sentiment analysis and prediction model is configured to perform the following steps: Step S21: The feature encoding module extracts features from the text modality sequence, the audio modality sequence, and the visual modality sequence to obtain initial text features, initial audio features, and initial visual features.

[0041] It should be noted that the feature encoding module is a network component used to convert raw unstructured sequences (such as sound and images) into high-dimensional continuous vectors. For example, in intelligent tour guide interaction, it acts as the "sensory nerve" of the intelligent tour guide, digitizing the received external signals.

[0042] It should be noted that initial text features, initial audio features, and initial visual features refer to vectors containing basic semantic or physical features extracted from the original data after preliminary processing. For example, in video comment analysis, this could be a text embedding vector containing the user's sarcastic meaning, or an acoustic feature vector containing their sharp voice.

[0043] In some embodiments, for text, the head embeddings of the last layer are extracted as features using a BERT model. For visual and audio input, a Long Short-Term Memory (LSTM) network is used to capture temporal dynamic changes, thereby obtaining initial feature representations for the visual and audio input. For example, in a virtual tour guide application, this module simultaneously encodes the tourist's question text and the urgent tone of their inquiry into corresponding mathematical vectors.

[0044] Furthermore, to accurately extract multimodal features, in one embodiment, the feature encoding module includes a BERT model, a first long short-term memory network, and a second long short-term memory network. Step S21 above may include: Step S211: Encode the text modal sequence using the BERT model, extract the head embedding vector from the output of the last layer of the BERT model, and obtain the text encoding features; Step S212: Input the audio modal sequence into the first long short-term memory network to obtain audio coding features. The first long short-term memory network is configured to perform audio modal coding operations. Step S213: Input the visual modality sequence into the second long short-term memory network to obtain visual encoding features. The second long short-term memory network is configured to perform visual modality encoding operations. Step S214: Input the text encoding features, the audio encoding features, and the visual encoding features into the fully connected layer to obtain the initial text features, the initial audio features, and the initial visual features. The fully connected layer is configured to perform dimension alignment operations.

[0045] In its implementation, the feature encoding module is used to extract features from text. Audio and vision Initial text, audio, and visual features are extracted from each modality. In the feature extraction module, for the text input, the BERT model is used as the feature extractor to encode the text information, and the head embedding from the output of the last layer of the model is selected as the feature representation of the input text. in, These are model parameters. It is a representation of text features.

[0046] For visual and audio modal inputs, LSTM is used to encode the visual and audio modalities: in, and These are the model parameters for visual modality and audio modality coding, respectively. and These are audio and visual feature representations, respectively.

[0047] The obtained text, visual, and audio features are then fed into the modality common information processing module to mine common information between modalities. In particular, visual and audio features are separately fed into the modality specific information processing module to mine modality specific information.

[0048] Step S22: The modal shared information processing module performs comparative learning and dynamic fusion on the initial text features, the initial audio features, and the initial visual features to obtain cross-modal shared features of text and audio and cross-modal shared features of text and vision.

[0049] It should be noted that the modal shared information processing module refers to a computing unit used to discover similarities between different modalities and eliminate expression barriers between modalities. For example, in intelligent tour guide interaction, it is used to identify the positive emotions jointly conveyed by tourists' "excitement in their speech" and "joy in their facial expressions".

[0050] It should be noted that cross-modal shared features refer to the extracted unified emotional features that are reflected in multiple modalities.

[0051] In its implementation, a two-stage contrastive learning network treats different modalities from the same sample as positive samples and introduces a hard negative sample mining mechanism to force the model to learn deep associations between different modalities. Subsequently, a text-guided dynamic fusion network is used, with text as the core query vector. Through an attention mechanism and audio / video feature transformation, key and value vectors are obtained and interactively fused. For example, in automated e-commerce customer service replies, the model combines the customer's textual "thank you" and the "lighthearted" tone in their voice to extract the shared feature of "high satisfaction."

[0052] Understandably, this embodiment effectively alleviates the heterogeneity problem between modalities by using the modal common information processing module. By strengthening the dominant role of text, it ensures that the model can grasp the most important emotional cues when processing tasks such as daily dialogues, thereby improving the accuracy of emotion recognition.

[0053] Step S23: The modality-specific information processing module performs feature enhancement and pooling processing on the initial audio features and the initial visual features to obtain global audio enhancement features and global visual enhancement features.

[0054] It should be noted that the modality-specific information processing module is a dedicated preservation mechanism designed to prevent the loss of unique non-verbal (audio and video) information during fusion. In the context of ironic sentiment analysis, this module is specifically responsible for capturing subtle differences that cannot be reflected in the text, such as "although it was said nicely, the tone was perfunctory."

[0055] It should be noted that global audio / visual enhancement features refer to the aggregated vector that represents the global core emotional characteristics of the modality after processing by an autoencoder.

[0056] In the specific implementation, the analysis device uses an autoencoder to learn the initial audio and visual features separately to obtain enhanced features, and then obtains the global feature vector through average pooling. During training, a joint loss function is applied to ensure that the enhanced features do not deviate from the original subspace and maintain a consistent distribution with the final task labels (such as positive / negative).

[0057] Step S24: The fusion prediction module concatenates and regresses the text audio cross-modal shared features, the text visual cross-modal shared features, the global audio enhancement features, and the global visual enhancement features to output the predicted sentiment intensity value.

[0058] It should be noted that the fusion prediction module refers to the decision layer that integrates all processed features and calculates the final prediction result. For example, in the context of intelligent recruitment in enterprises, it is used to comprehensively consider the candidate's written answers, tone of voice, confidence, and facial expressions during the interview to give a final emotional state score.

[0059] In the specific implementation, cross-modal shared features and global unimodal enhanced features are arranged in a preset order, and delimiter vectors are inserted between them for concatenation. Finally, the concatenated vector is input into a multilayer perceptron for regression calculation, outputting sentiment polarity. Here, the delimiter vector refers to the special delimiter used when concatenating different features. It is like a "punctuation mark" in a sentence, helping the model to clarify the boundaries between different features.

[0060] For example, in intelligent voice assistants, this module can distinguish between "common features of heard commands" and "user-specific features in environmental noise" to determine whether the user is in a state of anxiety.

[0061] Furthermore, in order to accurately predict the intensity of emotion, step S24 above may include: Step S241: Generate a separator vector and initialize the separator vector with learnable parameters; Step S242: Concatenate the text audio cross-modal shared features, the text visual cross-modal shared features, the global audio enhancement features, and the global visual enhancement features along the feature dimension using multiple of the aforementioned delimiter vectors to obtain multimodal concatenated features; Step S243: Input the multimodal splicing features into a multilayer perceptron for linear transformation to obtain the hidden layer representation; Step S244: The hidden layer represents the input and the regression output layer, and the predicted value of the sentiment intensity is output.

[0062] It is understandable that, after processing the modal common information module and the modal specific information module, cross-modal common features are obtained. and and global single-modal enhancement features and .

[0063] To enable the prediction module to effectively integrate cross-modal shared features and global unimodal augmentation features, and to dynamically distinguish and utilize them when there are conflicts between shared and augmentation features, a separator label vector is used along the feature dimension to concatenate general and specific features: in, This represents the concatenation function. Represents the separator vector. This indicates the features after splicing. This serves as a boundary to distinguish different types of modes.

[0064] Finally, The input is fed into a multilayer perceptron, which outputs a predicted value of emotional intensity. : The loss function for the final training of the model consists of four parts: task loss, first-stage contrastive loss, second-stage contrastive loss, and joint loss. Task loss The definition is as follows: in, This represents the total number of training samples. These are the model's predicted values. It is the actual value.

[0065] The total loss function of the multimodal sentiment analysis prediction model consists of the task loss, the first-stage contrastive loss, the second-stage contrastive loss, and the joint loss. in, , and These represent the weights of the first-stage contrast loss, the second-stage contrast loss, and the joint loss, respectively.

[0066] This embodiment achieves efficient mining and deep fusion of multi-source heterogeneous data by constructing a collaborative processing mechanism for shared and specific information of multiple modalities. It significantly improves the accuracy of emotion perception in complex and ever-changing environments and effectively solves the pain points of mutual interference of modal information or the averaging and masking of key clues in traditional methods.

[0067] refer to Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the multimodal sentiment analysis method of the present invention.

[0068] Based on the first embodiment described above, in this embodiment, step S22 further includes: Step S221: Pair the initial text features, the initial audio features, and the initial visual features to construct positive sample pairs and negative sample pairs.

[0069] It should be noted that positive sample pairs refer to correct combinations of different modal feature representations from the same video segment. Negative sample pairs can be random cross-combinations of different modal feature representations from different video segments within the same computational batch.

[0070] In one embodiment, the analysis device can read pre-extracted initial text, audio, and visual features through a memory scheduling module. Within the current processing batch, it combines different sensory features from the same movie review video clip into correct matching combinations of text and visual, text and audio, and visual and audio according to strict homology logic to form positive sample pairs. At the same time, it randomly shuffles the sample index and cross-pairs the text features of a certain video clip with the audio and video features of other completely unrelated video clips to generate a large number of negative sample pairs. For example, when processing a batch of movie review videos, the analysis device will pair the text feature of the narrator saying "wonderful" with his excited voice feature at the time as a positive reference, and pair this text feature with the sad background music feature in another video as a negative reference.

[0071] Step S222: Calculate feature similarity based on the positive sample pairs and the negative sample pairs, generate a similarity matrix, and filter difficult negative samples from the negative sample pairs based on the difficult negative sample mining mechanism.

[0072] It should be noted that the similarity matrix refers to a numerical calculation result matrix where the diagonal elements represent the similarity of positive sample pairs, while the non-diagonal elements represent the similarity of negative sample pairs. Difficult negative samples can be highly challenging feature combinations that have extremely high similarity scores but actually belong to mismatched different samples.

[0073] Understandably, this implementation cleverly quantifies the distance relationships between cross-modal features and forces the model to step out of its comfort zone of processing simple samples through an adversarial mining strategy, directly targeting the boundary-ambiguous data that is most likely to lead to sentiment misjudgment, which greatly enhances the feature recognition lower limit and robustness of the model when facing complex or even fake emotions.

[0074] Step S223: Calculate the first-stage contrastive learning loss based on the similarity matrix and the difficult negative samples, update the feature mapping parameters based on the first-stage contrastive learning loss, and obtain the first-stage enhanced features.

[0075] It should be noted that the first-stage contrastive learning loss refers to the overall network optimization error calculated by combining the cross-entropy loss and the hard negative sample loss. The feature mapping parameters can control the weight values ​​of all network nodes in the process of projecting different modal features into the same dimensional space.

[0076] Understandably, this embodiment uses composite error metrics to drive the evolution of the underlying network, successfully breaking down the naturally existing heterogeneous distribution barriers between different sensory data. It not only completes basic cross-modal alignment, but also specifically improves the ability to resist strongly interfering data, providing high-quality, low-noise basic enhancement features for subsequent high-order semantic fusion.

[0077] Step S224: Input the first-stage enhanced features into a multilayer perceptron for dimension mapping to obtain the second-stage mapped features.

[0078] It should be noted that, in this embodiment, the multilayer perceptron refers to a high-level nonlinear network model used to uniformly map visual features, acoustic features, and text features to the same dimension. The second-stage mapped features can be high-purity, same-dimensional deep feature vectors generated after eliminating residual heterogeneity.

[0079] It is understandable that this embodiment utilizes the high-dimensional fitting capability of the deep perceptron for secondary mapping, effectively compensating for the computational limitations of shallow alignment when dealing with extremely heterogeneous data, completely eliminating the residual microscopic distribution inconsistencies between modalities, and constructing a perfectly aligned high-order semantic interaction pool.

[0080] Step S225: Reconstruct positive and negative sample pairs based on the second-stage mapping features, and apply the hard negative sample mining mechanism again to calculate the second-stage contrastive learning loss. Perform gradient backpropagation and parameter iterative optimization based on the second-stage contrastive learning loss to obtain the second-stage enhanced features. Step S226: Input the enhanced features of the second stage into the text-guided dynamic fusion network to perform attention interaction calculation, and obtain the text-audio cross-modal shared features and the text-visual cross-modal shared features.

[0081] It is understandable that multimodal information inherently exhibits heterogeneity, and there are often distributional differences between modalities. To address these issues, better uncover similarities between modalities, and reduce heterogeneity, this embodiment designs a two-stage contrastive learning network within the modal general information processing module. The effectiveness of contrastive learning hinges on the construction of positive and negative sample pairs.

[0082] In the positive and negative sample construction process of this embodiment, different modal representations from the same sample are considered positive samples, while different modal representations from different samples in the same batch are considered negative samples. Furthermore, to prevent the model from over-relying on easy negative samples and getting trapped in local optima, a difficult negative sample mining mechanism is used in the positive and negative sample construction process. This mechanism selects the most challenging negative samples from each batch of samples. The specific implementation of the two-stage contrastive learning network is as follows.

[0083] In the first stage of contrastive learning, feature representations of visual and audio modalities were developed. , and text modality Pairing is performed in pairs: text-visual pairing, text-audio pairing, and visual-text pairing. Considering the different dimensions of the various modalities after feature extraction, fully connected layers are used to map the features of different modalities to the same dimensional space. Subsequently, the similarity between features is calculated, and a similarity matrix is ​​generated. The diagonal elements of the similarity matrix represent the similarity of positive sample pairs, while the off-diagonal elements represent the similarity of negative sample pairs. Furthermore, the cross-entropy loss function is used to calculate the first-stage contrastive learning loss. in, Indicates batch size. It is a similarity matrix Temperature scaling factor, Indicates from modality To mode (e.g., text-to-visual) computation, and vice versa for That's also true.

[0084] Building upon the cross-entropy loss, a hard negative sample loss is introduced, which uses a mask matrix to mask positive samples in the similarity matrix to identify the most challenging negative samples: in, It is an identity matrix that indicates the positions of positive samples. Indicates from modality To mode (e.g., text-to-visual) computation, and vice versa for That's also true. This is the negative sample similarity matrix obtained through masking operations. Then, for each positive sample, the similarity to the hardest negative sample is identified, and the hard negative sample loss is calculated: in, It is the boundary parameter that controls the separation between positive samples and the most difficult negative samples in the feature space. Indicates from modality To mode (e.g., text-to-visual) computation, and vice versa for That's also true. It is the first The hard negative sample score of each sample. It is the first The similarity score of each positive sample pair.

[0085] Finally, the total loss of contrastive learning is expressed as : Among them, parameters It is a weighted parameter.

[0086] Overall losses in the first phase The sum of losses obtained from contrastive learning between different modalities: in, This is the first stage of text visual contrast loss. This is the text-to-audio contrast loss in the first stage. This is the visual-audio contrast loss in the first stage.

[0087] Because different modalities have inherent heterogeneity in their original features, the first aligned representation may still have some inconsistencies. Therefore, a second stage of contrastive learning is performed on this basis.

[0088] The second stage of comparative learning uses a multilayer perceptron. visual features Acoustic characteristics and text features Mapping to the same dimension : in, These are the mapped features. .

[0089] Then, following the processing procedure of the first stage, the features mapped from the three modalities are paired, and the same hard negative sample mining mechanism is applied to calculate the total loss of the second stage of contrastive learning: in, This is the second stage of textual visual contrast loss. This is the second stage of text-to-audio contrast loss. This is the visual-audio contrast loss in the second stage.

[0090] Furthermore, in order to fully utilize the inherent advantages of text modality in emotional expression and enhance emotional interaction between different modalities, step S226 above may include: Step S2261: Map the text modal features in the second-stage enhancement features to query vectors, and the acoustic modal features and visual modal features to key vectors and value vectors; Step S2262: Perform scaling dot product calculation based on the query vector and the key vector to obtain the attention score matrix, and calculate the attention weight matrix based on the attention score matrix; Step S2263: Perform a weighted summation operation on the attention weight matrix and the value vector to obtain single-head attention output features; perform a concatenation operation on multiple single-head attention output features along the feature dimension to obtain concatenated attention features; and perform a linear transformation operation on the concatenated attention features to obtain multi-head attention output features. Step S2264: Perform residual connection between the multi-head attention output features and the text modal features in the second-stage enhancement features to obtain residual connection features. Perform layer normalization processing on the residual connection features to obtain text-audio cross-modal common features and text-visual cross-modal common features.

[0091] It should be noted that after completing the contrastive learning module and obtaining enhanced features... (in Based on this, a text-guided dynamic fusion module is designed to fully leverage the inherent advantages of text modality in emotional expression and enhance emotional interaction between different modalities, as detailed below: First, assign text features to To preserve the original information of the text, and to adapt to the input criteria of the attention mechanism, timestamp information is first added to modal features lacking a time dimension. Taking the fusion process of visual and text modalities as an example (the same applies to audio and text modalities), a learnable projection matrix is ​​used. , and The features with added timestamp information are mapped to the corresponding space to construct a query vector with the text modality. Acoustic modes as key vectors Sum value vector The interactive relationship.

[0092] To enhance the model's representational power, a multi-head attention mechanism is employed. During the multi-head attention computation, scaled dot product attention is used on the query and key. First, matrix multiplication is used to compute the dot product between the query and key. This process produces an attention score matrix, representing the similarity between each query and key.

[0093] To stabilize the gradient, this embodiment divides the attention score matrix by a scaling factor. Then, a Softmax operation is applied to the scaled attention score matrix to transform it into a probability distribution. Finally, the computed attention weights are applied to the values ​​to obtain the output of an attention head. Each output vector is a weighted sum of the value vectors, with the weights determined by the attention weight matrix. in, It is the first The query vector is a projection of the attention head text features. It is the first The key vector of the audio feature projection of the attention head. It is the first The value vector of the audio feature projection of the attention head. This indicates a connection along the feature dimension. It is the output projection matrix. It refers to the number of heads to focus on.

[0094] The outputs of each head are then concatenated and mapped back to the original feature space through a linear layer. To preserve the original text information, skip connections are used to transfer the text features. Add to the attention output and apply layer normalization to the result to obtain text-audio cross-modal fusion features. : in, It is the text-audio cross-modal shared feature output from the modal shared information module. Representation layer normalization, and text-visual cross-modal shared features are calculated in the same way as described above. .

[0095] This embodiment establishes the guiding role of the text modality in the complex multi-head attention mechanism, so that multimodal fusion is no longer a blind superposition of underlying information, but strengthens the dominant role of the text modality in the fusion process. This dynamic attention allocation and residual jump fusion mechanism not only perfectly preserves the core emotional intent in the user's text, but also effectively absorbs the auxiliary emotional temporal clues in the audio and video, obtains the common similar information between different modalities, and effectively improves the predictive performance of the model.

[0096] refer to Figure 4 , Figure 4 This is a flowchart illustrating the third embodiment of the multimodal sentiment analysis method of the present invention.

[0097] Based on the above embodiments, in this embodiment, the modality-specific information processing module includes a first autoencoder and a second autoencoder, and step S23 further includes: Step S231: Input the initial audio features into the first autoencoder for feature reconstruction to obtain audio single-modal enhanced features.

[0098] It should be noted that the first autoencoder refers to an unsupervised neural network model specifically designed for processing audio features, which extracts core audio information through encoder dimensionality reduction compression and decoder reconstruction and restoration mechanisms.

[0099] Audio single-modal enhancement features can be high-dimensional feature vectors that, after being purified and denoised by an autoencoder, filter out background noise and highlight unique acoustic emotional cues in the audio.

[0100] Understandably, this embodiment utilizes an autoencoder for audio feature reconstruction, effectively eliminating environmental noise and irrelevant redundant information in the original acoustic signal, highly refining the emotional expression cues specific to the audio modality, and greatly improving the model's sensitivity to changes in speech emotion and its anti-interference ability in noisy environments.

[0101] Step S232: Input the initial visual features into the second autoencoder for feature reconstruction to obtain visual single-modal enhanced features.

[0102] It should be noted that the second autoencoder refers to a deep learning network that specifically processes visual frame sequence features, uses deep networks for feature encoding, compression, decoding and reconstruction, in order to extract core emotional cues from the face or body.

[0103] Visual monomodal augmentation features can be visual feature representations that, after reconstruction learning, remove background lighting interference and highly focus on key cues such as facial expressions or micro-movements.

[0104] Understandably, this embodiment, through targeted reconstruction and enhancement of visual features, successfully eliminates interference from non-emotional physical factors such as background and drastic changes in ambient lighting, making the representation of visual modalities more purely focused on the real facial and body language of the person, and significantly enhancing the ability to capture and amplify subtle visual emotional signals.

[0105] Step S233: Calculate the first distribution difference between the audio monomodal enhancement feature and the audio initial feature, and calculate the second distribution difference between the visual monomodal enhancement feature and the visual initial feature.

[0106] It should be noted that the first distribution difference refers to the degree of deviation between the reconstructed audio single-modal enhanced features and the original audio features before input in the mathematical feature space.

[0107] The second distributional difference can be the distance quantification value of the visual monomodal augmentation feature compared to the original visual feature in the multidimensional spatial distribution, calculated using mathematical measurement methods.

[0108] Step S234: Calculate the third distribution difference between the audio monomodal enhancement features and the task label, and calculate the fourth distribution difference between the visual monomodal enhancement features and the task label.

[0109] It should be noted that task labels refer to standard answer values ​​that are manually labeled and set during the construction of the dataset, representing the true sentiment polarity and specific sentiment intensity of the samples.

[0110] The third and fourth distribution differences can be used as evaluation metrics to measure the magnitude of the error between the predicted probability tendency of audio-enhanced features or visual-enhanced features and the distribution of real human emotion labels, respectively.

[0111] It is understandable that this embodiment introduces the calculation of the distribution difference between enhanced features and target task labels, establishing a bridge that directly connects the single-modal low-level enhanced features with the final high-level semantic prediction target. This forces the enhancement direction of visual and audio features to closely align with the final business objective of emotion recognition, greatly improving the effective decision contribution rate of specific anti-conflict features in actual multimodal prediction.

[0112] Step S235: Construct a joint loss function based on the first distribution difference, the second distribution difference, the third distribution difference, and the fourth distribution difference, and assign weight hyperparameters to the joint loss function; Step S236: Based on the joint loss function, constrain the parameter updates of the first autoencoder and the second autoencoder, and output global audio enhancement features and global visual enhancement features that conform to the feature subspace constraints.

[0113] Understandably, after two stages of contrastive learning and text-guided dynamic semantic fusion, cross-modal shared features of text-audio and text-visual were obtained respectively. and However, there is a problem in the text-guided feature fusion process: the unique specific information of visual and acoustic modalities is often forcibly mapped into the semantic space of the text, resulting in a weakening of non-verbal representations. This can have a serious impact on some tasks in multimodal sentiment analysis, especially in complex sentiment analysis tasks such as irony, where visual and acoustic modalities often carry key cues that contradict the semantics of the text surface.

[0114] Therefore, this embodiment constructs a modality-specific information processing module to enhance the unique information of visual and audio modalities, thereby solving the problem of modality specificity loss. This module ensures that the enhanced features reside in the same subspace as the original modality features by constraining the distribution difference between the enhanced modality and the original modality, thus reducing modality heterogeneity. Secondly, it ensures that the enhanced features are as consistent as possible with the task label distribution by minimizing the difference between the enhanced modality and the task label distribution. The specific implementation is as follows.

[0115] First, use an autoencoder. Learning audio single-modal augmentation features : in, It is the initial audio feature vector output from the audio encoder.

[0116] And enhance audio single-modal features Perform average pooling to obtain the global feature vector. : in, This indicates the average pooling operation.

[0117] Similarly, using a self-encoder Learning visual unimodal augmentation features : in, It is the initial visual feature vector output from the visual encoder.

[0118] And enhance visual single-modal features Perform average pooling to obtain the global feature vector. : in, This indicates the average pooling operation.

[0119] To ensure that the enhanced modal remains within the same feature subspace as the original modal, reduce the specificity of the modal enhancement process, and guarantee that the distribution of the enhanced modal is consistent with that of the task labels, a joint loss function is constructed for the single-modal enhancement process under the constraint of consistency between the feature subspace and the task labels. : in, and Representing the speech modality and text modality respectively The initial feature vector of each sample, and These represent the enhanced feature vectors. It's the batch size. This represents the total number of all label categories. Indicates the middle Each sample in category The real labels in This represents the predicted probability of samples belonging to the same category. It is a weight hyperparameter.

[0120] This embodiment reconstructs the initial features of audio and video using an autoencoder to extract single-modal enhanced features, and employs a joint loss function to strictly constrain the feature distribution. This effectively solves the problem of vanishing specificity when unique audio and video information is forcibly mapped to text semantics. While ensuring that the enhanced features retain the original physical attribute subspace to prevent modality shift, it forces them to maintain a high degree of consistency with the label distribution of the sentiment prediction task. This multi-objective collaborative optimization closed loop endows the model with the crucial non-verbal cue correction ability when facing modality surface semantic conflict situations such as irony, thereby significantly improving the overall prediction accuracy and environmental robustness of the multimodal sentiment analysis system in complex real-world scenarios.

[0121] To verify the effectiveness of the multimodal sentiment analysis prediction model of this invention, comparative experiments were conducted on two widely used benchmark datasets in the field of multimodal sentiment analysis (CMU-MOSI and CMU-MOSEI datasets). Specifically, the performance of the multimodal sentiment analysis prediction model of this invention was compared with commonly used baseline models and state-of-the-art models. The model performance evaluation metrics used included MAE (Mean Absolute Error: the average of the absolute values ​​of the differences between predicted and true values; the smaller the value, the better the model's predictive ability), Corr (Pearson correlation coefficient: measures the linear correlation between the model's predicted results and the true labels; the closer the value is to 1, the better the model's performance), Acc-7 / 5 / 2 (sentiment level intensity classification accuracy; the higher the value, the better the model's predictive performance), and F1 (F1 score: the harmonic mean of precision and recall; the higher the value, the better the model's performance).

[0122] The multimodal sentiment analysis prediction model proposed in this invention has achieved better results than other models on the CMU-MOSI dataset. The comparison of experimental results is shown in Table 1 below.

[0123] Table 1. Comparison of experimental results on the CMU-MOSI dataset The model proposed in this invention also achieved better results than other models on the CMU-MOSEI dataset. The comparison of experimental results is shown in Table 2 below.

[0124] Table 2. Comparison of experimental results on the CMU-MOSEI dataset. Furthermore, embodiments of the present invention also propose a computer-readable storage medium storing a multimodal sentiment analysis program, wherein the multimodal sentiment analysis program, when executed by a processor, implements the steps of the multimodal sentiment analysis method as described above.

[0125] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0126] The aforementioned computer-readable storage medium may be included in the multimodal sentiment analysis device; or it may exist independently and not assembled into the multimodal sentiment analysis device.

[0127] Furthermore, this invention also proposes a computer program product, including a multimodal sentiment analysis program, which, when executed by a processor, implements the steps of the multimodal sentiment analysis method described above.

[0128] The specific implementation of the computer program product of the present invention is basically the same as the embodiments of the above-described multimodal sentiment analysis method, and will not be repeated here.

[0129] Reference Figure 5 , Figure 5 This is a structural block diagram of the first embodiment of the multimodal emotion analysis device of the present invention.

[0130] like Figure 5 As shown, the multimodal emotion analysis device proposed in this embodiment of the invention includes: The dataset construction module 10 is used to construct a multimodal sentiment analysis dataset, which includes text modality sequences, audio modality sequences, and visual modality sequences. The sentiment analysis module 20 is used to input the multimodal sentiment analysis dataset into a pre-built multimodal sentiment analysis prediction model to obtain a predicted sentiment intensity value. The multimodal sentiment analysis and prediction model includes a feature encoding module, a modality common information processing module, a modality specific information processing module, and a fusion prediction module. The feature encoding module is configured to extract features from the text modality sequence, the audio modality sequence, and the visual modality sequence to obtain initial text features, initial audio features, and initial visual features; The modal shared information processing module is configured to perform comparative learning and dynamic fusion of the initial text features, the initial audio features, and the initial visual features to obtain cross-modal shared features of text and audio and cross-modal shared features of text and vision. The modality-specific information processing module is configured to perform feature enhancement and pooling processing on the initial audio features and the initial visual features to obtain global audio enhancement features and global visual enhancement features. The fusion prediction module is configured to concatenate and regress the cross-modal shared features of the text audio, the cross-modal shared features of the text visual, the global audio enhancement features, and the global visual enhancement features, and output a predicted value of sentiment intensity.

[0131] This embodiment achieves efficient mining and deep fusion of multi-source heterogeneous data by constructing a collaborative processing mechanism for shared and specific information of multiple modalities. It significantly improves the accuracy of emotion perception in complex and ever-changing environments and effectively solves the pain points of mutual interference of modal information or the averaging and masking of key clues in traditional methods.

[0132] The multimodal sentiment analysis device provided in this application, employing the multimodal sentiment analysis method described in the above embodiments, can solve the technical problems of multimodal sentiment analysis. Compared with the prior art, the beneficial effects of the multimodal sentiment analysis device provided in this application are the same as those of the multimodal sentiment analysis method described in the above embodiments, and other technical features in the multimodal sentiment analysis device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0133] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.

[0134] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0135] In addition, for technical details not described in detail in this embodiment, please refer to the multimodal sentiment analysis method provided in any embodiment of the present invention, which will not be repeated here.

[0136] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0137] It should be noted that the user information (including but not limited to user device information, user personal information, user location information, user behavior information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0138] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0139] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0140] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A multimodal sentiment analysis method, characterized in that, The multimodal sentiment analysis method includes: A multimodal sentiment analysis dataset is constructed, which includes text modality sequences, audio modality sequences, and visual modality sequences; The multimodal sentiment analysis dataset is input into a pre-built multimodal sentiment analysis prediction model to obtain the predicted sentiment intensity value; The multimodal sentiment analysis and prediction model includes a feature encoding module, a modality common information processing module, a modality specific information processing module, and a fusion prediction module. The feature encoding module is configured to extract features from the text modality sequence, the audio modality sequence, and the visual modality sequence to obtain initial text features, initial audio features, and initial visual features; The modal shared information processing module is configured to perform comparative learning and dynamic fusion of the initial text features, the initial audio features, and the initial visual features to obtain cross-modal shared features of text and audio and cross-modal shared features of text and vision. The modality-specific information processing module is configured to perform feature enhancement and pooling processing on the initial audio features and the initial visual features to obtain global audio enhancement features and global visual enhancement features. The fusion prediction module is configured to concatenate and regress the text-audio cross-modal shared features, the text-visual cross-modal shared features, the global audio enhancement features, and the global visual enhancement features to output a predicted sentiment intensity value. The step of comparing and dynamically fusing the initial text features, the initial audio features, and the initial visual features to obtain cross-modal shared features between text and audio and cross-modal shared features between text and vision includes: The initial text features, the initial audio features, and the initial visual features are paired up to construct positive sample pairs and negative sample pairs; Based on the positive sample pairs and the negative sample pairs, feature similarity is calculated to generate a similarity matrix, and difficult negative samples are screened from the negative sample pairs based on the difficult negative sample mining mechanism; The first-stage contrastive learning loss is calculated based on the similarity matrix and the difficult negative samples. The feature mapping parameters are then updated based on the first-stage contrastive learning loss to obtain the first-stage enhanced features. The first-stage contrastive learning loss is calculated using the following formula: in, This indicates the learning loss in the first stage of comparison. This indicates the loss of visual contrast in the text during the first stage. This represents the text-to-audio contrast loss in the first stage. This indicates the visual-audio contrast loss in the first stage; The first-stage enhanced features are input into a multilayer perceptron for dimension mapping to obtain the second-stage mapped features. Based on the second-stage mapping features, positive and negative sample pairs are reconstructed, and the hard negative sample mining mechanism is applied again to calculate the second-stage contrastive learning loss. Based on the second-stage contrastive learning loss, gradient backpropagation and parameter iterative optimization are performed to obtain the second-stage enhanced features. The second-stage contrastive learning loss is calculated using the following formula: in, This indicates the learning loss in the second phase of comparison. This indicates the loss of visual contrast in the text during the second stage. This indicates the text-to-audio contrast loss in the second stage. This indicates the visual-audio contrast loss in the second stage; The enhanced features from the second stage are input into a text-guided dynamic fusion network to perform attention-based interaction computation, resulting in cross-modal shared features between text and audio, and cross-modal shared features between text and vision.

2. The multimodal sentiment analysis method as described in claim 1, characterized in that, The feature encoding module includes a BERT model, a first long short-term memory network, and a second long short-term memory network; the feature extraction of the text modality sequence, the audio modality sequence, and the visual modality sequence to obtain initial text features, initial audio features, and initial visual features includes: The text modal sequence is encoded using the BERT model, and the head embedding vector in the output of the last layer of the BERT model is extracted to obtain the text encoding features. The audio modal sequence is input into a first long short-term memory network to obtain audio coding features. The first long short-term memory network is configured to perform audio modal coding operations. The visual modality sequence is input into a second long short-term memory network to obtain visual encoding features. The second long short-term memory network is configured to perform visual modality encoding operations. The text encoding features, the audio encoding features, and the visual encoding features are input into a fully connected layer to obtain initial text features, initial audio features, and initial visual features. The fully connected layer is configured to perform dimension alignment operations.

3. The multimodal sentiment analysis method as described in claim 1, characterized in that, The second-stage enhanced features are input into the text-guided dynamic fusion network to perform attention interaction calculations, resulting in cross-modal shared features between text and audio, and cross-modal shared features between text and vision, including: The text modal features in the second-stage enhancement features are mapped to query vectors, and the acoustic modal features and visual modal features are mapped to key vectors and value vectors; A scaling dot product is performed based on the query vector and the key vector to obtain an attention score matrix, and an attention weight matrix is ​​calculated based on the attention score matrix. The attention weight matrix and the value vector are weighted and summed to obtain single-head attention output features. Multiple single-head attention output features are concatenated along the feature dimension to obtain concatenated attention features. The concatenated attention features are then linearly transformed to obtain multi-head attention output features. The multi-head attention output features are residually connected with the text modal features in the second-stage enhancement features to obtain residual connection features. The residual connection features are then subjected to layer normalization to obtain text-audio cross-modal shared features and text-visual cross-modal shared features.

4. The multimodal sentiment analysis method as described in claim 1, characterized in that, The modality-specific information processing module includes a first autoencoder and a second autoencoder; the feature enhancement and pooling processing of the initial audio features and the initial visual features to obtain global audio enhancement features and global visual enhancement features includes: The initial audio features are input into a first autoencoder for feature reconstruction to obtain audio single-modal enhanced features. The initial visual features are input into a second autoencoder for feature reconstruction to obtain visual single-modal enhanced features. Calculate the first distribution difference between the audio unimodal enhancement features and the initial audio features, and calculate the second distribution difference between the visual unimodal enhancement features and the initial visual features; Calculate the third distribution difference between the audio unimodal enhancement features and the task label, and calculate the fourth distribution difference between the visual unimodal enhancement features and the task label; A joint loss function is constructed based on the first distribution difference, the second distribution difference, the third distribution difference, and the fourth distribution difference, and weight hyperparameters are assigned to the joint loss function. The joint loss function is defined by the following formula: in, Indicates joint loss, This represents the total number of training samples. and They represent the first Initial audio and visual features of each sample. and These represent audio unimodal enhancement features and visual unimodal enhancement features, respectively. This represents the total number of all label categories. Indicates the first Each sample in category The real labels in Indicates the index number of the category. This represents the predicted probability of samples belonging to the same category. It is a weight hyperparameter; Based on the joint loss function constraining the parameter updates of the first and second autoencoders, global audio enhancement features and global visual enhancement features that conform to the feature subspace constraints are output.

5. The multimodal sentiment analysis method as described in claim 1, characterized in that, The process of concatenating and regressing the shared features of the text-audio cross-modal data, the shared features of the text-visual cross-modal data, the global audio enhancement features, and the global visual enhancement features to output a predicted sentiment intensity value includes: Generate a separator vector and initialize the separator vector with learnable parameters; The text audio cross-modal shared features, the text visual cross-modal shared features, the global audio enhancement features, and the global visual enhancement features are concatenated along the feature dimension using multiple of the aforementioned delimiter vectors to obtain multimodal concatenated features; The multimodal spliced ​​features are input into a multilayer perceptron and linearly transformed to obtain the hidden layer representation; The hidden layer is used to represent the input and the regression output layer, and the predicted value of the sentiment intensity is output.

6. The multimodal sentiment analysis method as described in claim 1, characterized in that, The construction of the multimodal sentiment analysis dataset includes: Obtain the original video of the person commenting, and divide the original video of the person commenting according to a preset time interval to obtain multiple video segments; Speech signals are extracted from the multiple video segments, and the extracted speech signals are processed into text to generate text signals synchronized with the speech signals. The video clip, the audio signal, and the text signal are time-axis aligned to obtain a three-modal alignment sample, which includes a text modal sample, an audio modal sample, and a visual modal sample. The three-modal aligned samples are labeled with sentiment intensity tags, and the labeled three-modal aligned samples are summarized to construct a multimodal sentiment analysis dataset.

7. A multimodal sentiment analysis device applying the multimodal sentiment analysis method according to any one of claims 1 to 6, characterized in that, The device includes: A dataset construction module is used to construct a multimodal sentiment analysis dataset, which includes text modality sequences, audio modality sequences, and visual modality sequences. The sentiment analysis module is used to input the multimodal sentiment analysis dataset into a pre-built multimodal sentiment analysis prediction model to obtain a predicted sentiment intensity value; The multimodal sentiment analysis and prediction model includes a feature encoding module, a modality common information processing module, a modality specific information processing module, and a fusion prediction module. The feature encoding module is configured to extract features from the text modality sequence, the audio modality sequence, and the visual modality sequence to obtain initial text features, initial audio features, and initial visual features; The modal shared information processing module is configured to perform comparative learning and dynamic fusion of the initial text features, the initial audio features, and the initial visual features to obtain cross-modal shared features of text and audio and cross-modal shared features of text and vision. The modality-specific information processing module is configured to perform feature enhancement and pooling processing on the initial audio features and the initial visual features to obtain global audio enhancement features and global visual enhancement features. The fusion prediction module is configured to concatenate and regress the cross-modal shared features of the text audio, the cross-modal shared features of the text visual, the global audio enhancement features, and the global visual enhancement features, and output a predicted value of sentiment intensity.

8. A multimodal sentiment analysis device, characterized in that, The multimodal sentiment analysis device includes: a memory, a processor, and a multimodal sentiment analysis program stored in the memory, the processor being used to run the multimodal sentiment analysis program, the multimodal sentiment analysis program being configured to implement the multimodal sentiment analysis method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a multimodal sentiment analysis program, which, when executed by a processor, implements the multimodal sentiment analysis method as described in any one of claims 1 to 6.

Images