A multi-modal sentiment analysis method and system based on cross-modal redundancy resolution

Abstract

The application belongs to the technical field of artificial intelligence, and provides a multi-modal sentiment analysis method and system based on cross-modal redundancy resolution, which comprises the following steps: acquiring video data; extracting multi-modal features and global sequence aggregation vectors of the video data; extracting cross-modal shared semantic consensus based on the global sequence aggregation vectors, eliminating the shared semantic consensus from the global sequence aggregation vectors, separating out discriminative residual features of specific modalities, and constructing a supervised contrast loss based on the discriminative residual features; processing the multi-modal features and the global sequence aggregation vectors based on a hierarchical cross-modal information bottleneck to obtain pure fusion features and an information bottleneck loss function; performing reverse back propagation calibration on the pure fusion features, and iteratively processing a preset maximum cross-modal fusion layer number; extracting single-modal global sequence aggregation vectors and bottleneck pure fusion features at the end of the cycle to obtain joint classification representation; performing sentiment prediction based on the joint classification representation to obtain a sentiment prediction result, and completing multi-modal sentiment analysis.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a multimodal sentiment analysis method and system based on cross-modal redundancy resolution. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the development of artificial intelligence and affective computing technologies, sentiment analysis methods based on multimodal data have gradually become an important research direction in the field of emotion recognition. Multimodal sentiment analysis typically utilizes information from multiple modalities such as text, speech, and vision to characterize the multidimensional features of emotional states, thereby improving the accuracy and robustness of emotion recognition.

[0004] Currently, multimodal sentiment analysis methods can extract features from data of different modalities using deep neural networks, and combine cross-modal attention, feature alignment, or fusion mechanisms to achieve information interaction between modalities. Features from different modalities are aligned in the temporal or semantic dimensions and then fused to form a unified sentiment representation. However, this approach usually focuses on enhancing the information aggregation capability between modalities, assuming that the information after cross-modal fusion is all valid information, and lacks explicit suppression mechanisms for redundant information across modalities.

[0005] Different modalities often express the same or similar emotional cues simultaneously, such as the emotional consistency between text semantics and speech intonation. When the above information is repeatedly introduced in the cross-modal fusion process, redundant information is easily reinforced multiple times in the fused representation, which not only increases the redundancy of feature representation, but may also cover up the effective information that plays a key role in emotion discrimination, thereby affecting the discrimination performance of the model.

[0006] As multimodal sentiment analysis models become increasingly deeper, cross-modal fusion operations are often repeatedly performed across multiple network layers. Current techniques typically constrain or regularize fusion features only at a single layer or in the final output stage, failing to effectively control the propagation of cross-modal information across multiple network layers. Redundant information generated during the cross-modal fusion stage easily accumulates layer by layer, creating a cumulative effect of information redundancy, thereby amplifying the interference of noisy modalities on the overall sentiment judgment.

[0007] To address the aforementioned issues, some existing technologies attempt to introduce information constraints or feature compression mechanisms to reduce interference from irrelevant information. However, applying information constraints to single-modal features or the final fused representation is difficult to directly apply to the crucial stage of cross-modal fusion, and it also struggles to adapt to the semantic evolution characteristics of different levels in multi-layered cross-modal interaction structures. Therefore, existing multimodal sentiment analysis methods still lack a technical solution capable of hierarchically controlling and effectively suppressing redundant information during cross-modal fusion. Summary of the Invention

[0008] To address the aforementioned issues, this invention proposes a multimodal sentiment analysis method and system based on cross-modal redundancy resolution. This method fully utilizes shared semantics and modality-specific discriminative features in multimodal signals and effectively filters out temporal redundancy noise, thereby improving the accuracy and robustness of multimodal sentiment recognition.

[0009] According to some embodiments, the first aspect of the present invention provides a multimodal sentiment analysis method based on cross-modal redundancy resolution, employing the following technical solution: A multimodal sentiment analysis method based on cross-modal redundancy resolution includes: Obtain the video data to be identified; Extract the multimodal features and their global sequence aggregation vectors from the acquired video data; Cross-modal shared semantic consensus is extracted based on global sequence aggregation vectors. The shared semantic consensus is removed from the global sequence aggregation vectors to separate discriminative residual features of specific modalities. A supervised contrastive loss function is constructed based on the discriminative residual features. Based on the multimodal features extracted by hierarchical cross-modal information bottleneck processing and their global sequence aggregation vector, pure fusion features and information bottleneck loss function are obtained; The obtained pure fusion features are backpropagated for calibration, and the preset maximum number of cross-modal fusion layers is iterated in a loop. Extract the single-modal global sequence aggregation vector and bottleneck pure fusion features at the end of the loop to obtain a joint classification representation; Sentiment prediction is performed based on the obtained joint classification representation, and the sentiment prediction results are obtained, thus completing the multimodal sentiment analysis based on cross-modal redundancy resolution.

[0010] As a further technical limitation, the process of constructing the supervised contrastive loss function is as follows: Feature fusion is performed on the global sequence aggregation vectors of each modality to extract the shared semantic consensus among the multimodalities; By fixing the shared semantic consensus using the stopping gradient operation, the global sequence aggregation vector of each modality is subtracted from the shared semantic consensus to separate the discriminative residual features of a specific modality that do not contain cross-modal global redundancy; Based on the sentiment label, a distance threshold is set to divide positive and negative sample pairs. The contrast loss is calculated using the discriminative residual features to obtain the supervised contrast loss.

[0011] As a further technical limitation, the calculation process of the information bottleneck loss function is as follows: Based on the sequence features of each modality and the bottleneck query vector of the current layer, a multimodal cross-attention mechanism is used to perform cross-modal interaction and aggregate to obtain intermediate fusion features; The intermediate fused features are modeled by probability distribution through random variational approximation and reparameterization, and constrained pure fused features are generated by sampling. The information bottleneck loss function representing the amount of information retained is calculated.

[0012] As a further technical limitation, during the reverse backhaul calibration of the obtained pure fusion features, a cross-transform encoder is used to construct a bidirectional feedback loop, broadcasting the pure fusion features generated in the current layer back to each single-modal branch; using the pure fusion features as the key matrix and value matrix, and using the single-modal sequence features output by the upper layer as the query matrix, cross-interactive calculation is performed to filter out the temporal redundancy of the single-modal sequence at the bottom layer; the updated and denoised single-modal features are used as the input of the next layer network module, and the above feature purification and bidirectional calibration process is executed cyclically until the set maximum number of cross-modal fusion layers is reached.

[0013] As a further technical limitation, in the sentiment prediction process, a joint classification representation is constructed based on the final updated global sequence aggregation vectors of each modality and the pure fusion features of the last layer. This representation is then input into the prediction network to output the sentiment prediction result, i.e.: Extract the final updated single-modal global sequence aggregation vector at the end of the network loop, and concatenate it with the flattened last layer of pure fusion features in the feature dimension to form the final joint classification representation; The joint classification representation is fed into the regression prediction network to output a continuous sentiment prediction score, and the mean squared error is calculated based on the real sentiment label and the prediction score as the task prediction loss. The overall loss function is obtained by weighting and summing the task prediction loss, the supervised comparison loss multiplied by the first weight coefficient, and the information bottleneck loss of all levels multiplied by the second weight coefficient.

[0014] As a further technical limitation, the process of extracting the global sequence aggregation vector corresponding to each modality is as follows: Pre-trained language models are used to extract initial text sequence features, and Transformer encoders with specific classification labels and positional encodings are used to extract initial speech and visual sequence features. The classification label or equivalent feature representation after pooling operation is extracted from the initial sequence feature header of each modality and used as the global sequence aggregation vector of each modality to characterize the overall contextual semantics of a single modality.

[0015] According to some embodiments, the second aspect of the present invention provides a multimodal sentiment analysis system based on cross-modal redundancy resolution, employing the following technical solution: A multimodal sentiment analysis system based on cross-modal redundancy resolution includes: The acquisition module is configured to acquire the video data to be identified; The extraction module is configured to extract multimodal features and their global sequence aggregation vectors from the acquired video data; The construction module is configured to extract cross-modal shared semantic consensus based on global sequence aggregation vectors, remove the shared semantic consensus from the global sequence aggregation vectors, separate discriminative residual features of specific modalities, and construct a supervised contrastive loss function based on the discriminative residual features; and obtain pure fusion features and information bottleneck loss function based on the multimodal features extracted by hierarchical cross-modal information bottleneck processing and their global sequence aggregation vectors. The iterative module is configured to backpropagate and calibrate the obtained pure fusion features, iterating cyclically to the preset maximum number of cross-modal fusion layers; extracting the single-modal global sequence aggregation vector and bottleneck pure fusion features at the end of the loop to obtain the joint classification representation; The analysis module is configured to perform sentiment prediction based on the obtained joint classification representation, obtain the sentiment prediction results, and complete multimodal sentiment analysis based on cross-modal redundancy resolution.

[0016] According to some embodiments, a third aspect of the present invention provides a computer-readable storage medium, employing the following technical solution: A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the multimodal sentiment analysis method based on cross-modal redundancy resolution as described in the first aspect of the present invention.

[0017] According to some embodiments, the fourth aspect of the present invention provides an electronic device, which adopts the following technical solution: An electronic device includes a memory, a processor, and a program stored in the memory and running on the processor, wherein the processor executes the program to implement the steps in the multimodal sentiment analysis method based on cross-modal redundancy resolution as described in the first aspect of the present invention.

[0018] According to some embodiments, the fifth aspect of the present invention provides a computer program product, which adopts the following technical solution: A computer program product includes software code, wherein the program in the software code performs the steps of the multimodal sentiment analysis method based on cross-modal redundancy resolution as described in the first aspect of the present invention.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: In terms of overall architecture design, this invention adopts a collaborative computing paradigm that eliminates redundancy in both the representation layer and the fusion layer. Addressing the problem that existing multimodal sentiment analysis methods often focus only on coarse feature splicing and alignment and easily ignore the cumulative effect of redundant information, this invention processes geometric redundancy in the semantic space at the representation layer and physical redundancy in the time series at the cross-modal fusion layer. This effectively alleviates the problem of task-irrelevant noise being amplified step by step in deep networks, thereby obtaining purer and more robust sentiment features.

[0020] In terms of semantic space optimization, this invention effectively overcomes the limitations of traditional multimodal contrastive learning, which blindly performs "overall feature alignment," by employing Redundant Perceptual Contrastive Learning (RACL). By cleverly utilizing the "Stop-Gradient" mechanism, "cross-modal shared consensus" and "modality-specific residuals" are effectively decoupled, so that supervised contrastive constraints only act on discriminative residuals. While achieving a compact semantic space, the diversity of features is protected, achieving a manifold state where feature alignment and modality individuality coexist.

[0021] In terms of cross-modal fusion, this invention adopts the Hierarchical Cross-Modal Information Bottleneck (HCIB) to improve the conventional fusion paradigm of direct cross-attention in dense sequences. It transforms cross-modal fusion into a "variable compression optimization problem" by designing a "dual filtering mechanism"—the first layer uses learnable bottleneck tokens that are halved layer by layer to construct a structured funnel to truncate temporal redundancy; the second layer introduces a stochastic variational approximation and uses KL divergence as a probability upper bound constraint for information retention. The dimensionality reduction method combining structured and probabilistic approaches significantly improves the model's noise resistance and generalization ability.

[0022] Regarding the feature interaction mechanism, this invention constructs a "top-down" dynamic calibration closed loop; by introducing a cross-transform encoder (CTE) feedback mechanism, it allows the advanced fusion features purified by HCIB to guide and update the underlying single-modal branch sequence in reverse; the bidirectional collaborative feature calibration mechanism in this invention further enhances the model's recognition performance and system robustness in complex real-world scenarios such as high noise and missing data. Attached Figure Description

[0023] The accompanying drawings, which form part of this embodiment, are used to provide a further understanding of this embodiment. The illustrative embodiments and their descriptions are used to explain this embodiment and do not constitute an improper limitation of this embodiment.

[0024] Figure 1 This is a flowchart of the multimodal sentiment analysis method based on cross-modal redundancy resolution in Embodiment 1 of the present invention; Figure 2 This is a flowchart of the multimodal sentiment analysis method based on cross-modal redundancy resolution in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the overall structure of the Hierarchical Cross-Modal Information Bottleneck (HCIB) module in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the internal structure of the cross-modal fusion and variational sampling unit in Embodiment 1 of the present invention; Figure 5 This is a schematic diagram of the internal structure of the feature bidirectional calibration and single-mode sequence update in Embodiment 1 of the present invention; Figure 6 This is a schematic diagram of the internal structure of the redundant perception contrast learning unit in Embodiment 1 of the present invention; Figure 7 This is a structural block diagram of the multimodal sentiment analysis system based on cross-modal redundancy resolution in Embodiment 2 of the present invention. Detailed Implementation

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0026] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0027] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0028] In this invention, terms such as "upper," "lower," "left," "right," "front," "back," "vertical," "horizontal," "side," and "bottom" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used only to facilitate the description of the structural relationships of the various components or elements of this invention and do not specifically refer to any component or element in this invention. They should not be construed as limiting the invention.

[0029] In this invention, terms such as "fixed connection," "connected," and "linked" should be interpreted broadly, indicating a fixed connection, an integral connection, or a detachable connection; a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can determine the specific meaning of these terms in this invention based on the specific circumstances, and they should not be construed as limitations on the invention.

[0030] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0031] Example 1 Embodiment 1 of this invention introduces a multimodal sentiment analysis method based on cross-modal redundancy resolution.

[0032] like Figure 1 and Figure 2 The multimodal sentiment analysis method shown includes: Obtain the video data to be identified; Extract the multimodal features and their global sequence aggregation vectors from the acquired video data; Cross-modal shared semantic consensus is extracted based on global sequence aggregation vectors. The shared semantic consensus is removed from the global sequence aggregation vectors to separate discriminative residual features of specific modalities. A supervised contrastive loss function is constructed based on the discriminative residual features. Based on the multimodal features extracted by hierarchical cross-modal information bottleneck processing and their global sequence aggregation vector, pure fusion features and information bottleneck loss function are obtained; The obtained pure fusion features are backpropagated for calibration, and the preset maximum number of cross-modal fusion layers is iterated in a loop. Extract the single-modal global sequence aggregation vector and bottleneck pure fusion features at the end of the loop to obtain a joint classification representation; Sentiment prediction is performed based on the obtained joint classification representation, and the sentiment prediction results are obtained, thus completing the multimodal sentiment analysis based on cross-modal redundancy resolution.

[0033] As one or more implementation methods, this embodiment considers preprocessing the multimodal video data to be identified, extracting initial sequence features for text, speech, and visual modalities respectively, and extracting global sequence aggregation vectors corresponding to each modality, specifically including: (1) Obtain the multimodal input data of the object to be analyzed and perform modal decoupling.

[0034] The video data segments to be identified are acquired through a multimodal acquisition device and decoupled into text. ),voice( ) and visual ( Three types of heterogeneous modal data. The multimodal feature extraction module is used to acquire and process this data. To ensure seamless logical alignment with the input of the subsequent hierarchical cross-modal fusion structure, the extracted initial sequence feature representation set is denoted as: For the sake of consistency in formula expression, the above initial single-mode features can be collectively referred to as... ,in .

[0035] (2) Encode the sequence features of the text modality.

[0036] Text transcription is tokenized and encoded using a pre-trained language model (such as BERT). This tokenization is then mapped to the target hidden dimension using a linear projection layer. To obtain the initial sequence features of the text ,in, Indicates the time step of the text sequence.

[0037] (3) Sequence feature encoding of speech and visual modalities.

[0038] For speech modalities, low-level acoustic features (such as MFCCs) are extracted using feature extraction tools (such as COVAREP), and special classification markers (`[CLS]`) are appended to the beginning of the sequence and positional encoding is incorporated. Subsequently, the sequence is processed by a multi-layer Transformer encoder to obtain the initial speech sequence features. .

[0039] For visual modalities, facial visual features are extracted using visual extraction tools (such as OpenFace 2.0) and projected onto... Similarly, a visual Transformer encoder is used to process the data to obtain the initial visual sequence features. .

[0040] (4) Extract the global sequence aggregation vector for each modality.

[0041] Extract the `[CLS]` marker from the beginning of each sequence feature (for text modalities, use the pooled equivalent representation) as the global sequence aggregation vector for each modality, denoted as... :

[0042] Here, pool represents the pooling operation, and this vector is used to characterize the overall contextual semantics of a single modality within the global time window.

[0043] As one or more implementation methods, such as Figure 6 As shown, in this embodiment, at the representation level, based on the global sequence aggregation vector of each modality, the discriminative residual features of a specific modality are separated using the Redundant Aware Contrast Learning (RACL) module, and a supervised contrastive loss is constructed accordingly to optimize the multimodal semantic space.

[0044] Global sequence aggregation vectors for three modalities A mean-pooling operation is performed. Since redundant sentiment signals are usually consistent across different modalities, the mean can serve as a robust first-order approximation of shared cross-modal semantics, extracting the shared semantic consensus. ,Right now .

[0045] To prevent harmful gradient coupling between modal encoders and the mean during optimization (modal collapse), the extracted consensus is... Apply a stopping gradient (i.e., The operator is then used. Subsequently, the frozen consensus is subtracted from the global vectors of each modality to extract the discriminative residual features specific to each modality, free from cross-modal redundancy. ,Right now .

[0046] This operation isolates non-redundant, modality-specific sentiment metrics.

[0047] Utilizing continuous sentiment labels Set distance threshold For samples within a batch If the sample satisfy Then it is considered as a set of positive sample pairs. The rest are considered as negative samples. For pure residuals Apply the normalized temperature-scaled cross-entropy loss (NT-Xent), calculated as follows: ; in, The cosine similarity function is used. is the temperature scaling factor. This loss function ensures that similar residual features cluster tightly in the latent space, while perfectly preserving the discriminative cues unique to specific modalities (such as vocal prosody and visual micro-expressions).

[0048] like Figure 3As shown, at the cross-modal fusion level, based on the initial sequence features of each modality, the hierarchical cross-modal information bottleneck (HCIB) module is used to perform layer-by-layer interaction and structured compression processing, sample and generate constrained pure fusion features, and calculate the information bottleneck loss.

[0049] The layered cross-modal fusion module is used to achieve layer-by-layer interaction and aggregation of multimodal emotional information. Let the number of fusion layers be . (in For any current fusion level ( Within each layer, the shared fusion representation serves not only as a query carrier for cross-modal interactions but also as a unified representation space for multimodal information filtering and aggregation, performing the following operations: (1) Initialization of shared fusion representation and structured physical compression.

[0050] In the During the layer fusion process, a set of learnable bottleneck query vectors (Bottleneck Tokens) is introduced. As a shared and fused representation. To achieve physical dimensionality reduction and temporal redundancy removal of information structurally, the number of bottleneck query vectors... With fusion level The compression rate decreases progressively with each layer, employing a halving compression strategy: ; This structured compression mechanism acts as a rigid information funnel, forcing the network to retain only the most salient emotional cues as it progresses through layers.

[0051] (2) Cross-modal fusion computation and intermediate fusion representation extraction.

[0052] like Figure 4 As shown on the left, selective absorption of cross-modal information is achieved through interactive computation between shared fusion representation and multimodal features. A multimodal cross-attention mechanism (Multi-CA) is employed to address the bottleneck query vector of the current layer. As a query term, the modal sequence features output from the previous layer are respectively... As keys and values, it aggregates supplementary information from text, speech, and visual sequences. Its specific calculation formula can be expressed as: ; in, Presentation layer normalization operation, This is the intermediate representation after fusion. It is important to note that due to the logical closed-loop nature of the formula features: when in the first layer of fusion (i.e....) When ), the upper-level feature in the formula That is, corresponding to This refers to the initial single-modal features output by the feature extraction module. This design allows the initially extracted features to be seamlessly integrated and incorporated into the cross-modal fusion process.

[0053] (3) Stochastic variational modeling of fusion representation like Figure 4 As shown on the right, for the first Intermediate representation obtained by cross-modal fusion Construct the corresponding random variables This is used to represent the fused representation after redundancy elimination. A conditional probability distribution is constructed based on the intermediate representation. .

[0054] To deeply mine redundant patterns in features, the mapping function employs a parallel multilayer perceptron (MLP) structure to predict the mean vector of a multivariate Gaussian distribution. Sum of logarithmic variance vector Each prediction branch contains, in sequence, a linear mapping layer, a layer normalization layer, and a GELU activation function.

[0055] (4) Reparameterized generation of constrained pure fusion representations.

[0056] Based on the above probability distribution, we introduce the standard normal distribution. Independently sampled random disturbance variables Constrained pure fusion features are generated through reparameterization techniques: ; in, This indicates element-wise multiplication.

[0057] (5) Calculation of the redundancy elimination constraint objective (information bottleneck loss).

[0058] The hierarchical cross-modal redundancy resolution module is not a post-processing unit independent of the fusion process, but rather it is tightly coupled with the hierarchical cross-modal fusion module in terms of structure and process, so that cross-modal information is always in an alternating state of "fusion-constraint-refusion".

[0059] To limit the amount of information contained in the fused representation, a redundancy resolution constraint is introduced. The Kullback-Leibler (KL) divergence between the predicted Gaussian distribution and the preset reference distribution (standard normal distribution) is calculated: ; The KL divergence constraint term serves as a variational upper bound for the information bottleneck, forcing the model to filter out task-irrelevant noise redundancy.

[0060] As one or more implementation methods, this embodiment performs backpropagation calibration of single-modal sequence features using pure fusion features, and iterates cyclically until the maximum number of network layers is reached. Specifically, this includes: (1) Hierarchical transfer and bidirectional calibration of fusion representation.

[0061] like Figure 5 As shown, the pure fusion representation after redundancy elimination processing Not only as the first The input to the layer module is also broadcast back to each single-modal branch via a Cross-Transformer Encoder (CTE) for calibrating the single-modal sequences at the bottom layer. As a key-value matrix, the features of the previous layer's single-modal sequence are... Interactive calculations are performed as a query matrix: ; Through this feedback mechanism, the first The single-modal features of layer 1 are updated to the first layer. Features of the layer And just as the next level (the first) The single-modal input features (layer) are extracted. Thus far, the initial feature extraction... The input, after fusion, resolution, and feedback update, forms a complete closed loop of feature flow logic, realizing collaborative noise reduction between the fusion process and the single-modal representation.

[0062] (2) Execute repeatedly until the set depth is reached.

[0063] The updated features of each modality sequence are input into the next level for iterative processing until the preset maximum number of cross-modal fusion layers is reached. .

[0064] As one or more implementation methods, this embodiment performs classification prediction based on joint classification representation and combines it with a joint loss function for end-to-end optimization, specifically including: After going through the complete After layer processing, extract the latest updated single-modal global aggregation vector at the end of the loop (i.e. The first `[CLS]` mark), and the final layer of bottleneck pure fusion features after flattening. The features are concatenated along the feature dimension to form the final joint classification representation. .

[0065] Will The data is fed into a regression prediction network composed of a multilayer perceptron (MLP) to output continuous sentiment prediction values. In practical applications, continuous values ​​can be directly output to represent the precise intensity of sentiment, or sentiment classification results can be output based on spatial discretization of the decision threshold.

[0066] During the model training phase, the sentiment analysis task objective and the two-level redundancy resolution constraint objective are jointly optimized. This is for batch processing. Calculate the true sentiment label from a sample. Compared with the predicted value The mean squared error (MSE) between the two is used as the task prediction loss. : ; The overall information bottleneck loss is obtained by summing the redundancy resolution constraints at each fusion level. Introducing contrastive learning weight hyperparameters. Information bottleneck weight hyperparameter Construct the overall loss function: ; Relying on total loss Backpropagation algorithms (such as the AdamW optimizer) enable end-to-end joint optimization of all learnable parameters in the multimodal feature extraction module, redundant perception comparison module, hierarchical information bottleneck module, and sentiment prediction module, thereby forming a rigorous computational and logical closed loop throughout the entire process of "data input - feature purification - sentiment prediction".

[0067] In terms of overall architecture design, this embodiment proposes a collaborative computing paradigm of "dual redundancy resolution in the representation layer and fusion layer". Existing multimodal sentiment analysis methods often only focus on coarse feature splicing and alignment, easily ignoring the cumulative effect of redundant information. This embodiment processes geometric redundancy in the semantic space at the representation layer and physical redundancy in the time series at the cross-modal fusion layer, effectively mitigating the problem of task-irrelevant noise being amplified step by step in deep networks, thereby obtaining purer and more robust sentiment features.

[0068] In terms of semantic space optimization, the Redundant Perceptual Contrastive Learning (RACL) module designed in this embodiment effectively overcomes the limitation of traditional multimodal contrastive learning that blindly performs "overall feature alignment." Existing contrastive learning techniques often lead to "modal collapse" when bridging heterogeneous features, meaning they easily lose the discriminative advantages unique to specific modalities such as speech prosody or visual micro-expressions. This embodiment cleverly utilizes a "Stop-Gradient" mechanism to effectively decouple "cross-modal shared consensus" from "modal-specific residuals," ensuring that supervised contrastive constraints only apply to discriminative residuals. This design achieves both compact bridging of the semantic space and better protects feature diversity, reaching a manifold state where feature alignment and preservation of modal individuality coexist.

[0069] In cross-modal fusion, the Hierarchical Cross-Modal Information Bottleneck (HCIB) module designed in this embodiment improves upon the conventional fusion paradigm of direct cross-attention in dense sequences. Existing attention mechanisms tend to absorb background noise when faced with lengthy video features (such as long periods of silence or expressionless scenes). This embodiment transforms cross-modal fusion into a "variable compression optimization problem" by designing a "dual filtering mechanism"—the first layer utilizes learnable bottleneck tokens that are halved layer by layer to construct a structured funnel to truncate temporal redundancy; the second layer introduces a stochastic variational approximation, using KL divergence as a probability upper bound constraint for information retention. This dimensionality reduction approach, combining structured and probabilistic methods, significantly improves the model's robustness and generalization ability.

[0070] Regarding the feature interaction mechanism, this embodiment constructs a "top-down" dynamic calibration closed loop. Unlike the unidirectional feature feedforward in traditional methods (i.e., the bottom-level features flow unidirectionally to the fusion layer), the cross-transform encoder (CTE) feedback mechanism introduced in this embodiment allows the high-level fusion features purified by HCIB to guide and update the bottom-level single-modal branch sequence in reverse. The bidirectional collaborative feature calibration mechanism further enhances the model's recognition performance and system robustness in complex real-world scenarios such as high noise and missing data.

[0071] Example 2 Embodiment 2 of this invention introduces a multimodal sentiment analysis system based on cross-modal redundancy resolution.

[0072] like Figure 7 The multimodal sentiment analysis system shown includes: The acquisition module is configured to acquire the video data to be identified; The extraction module is configured to extract multimodal features and their global sequence aggregation vectors from the acquired video data; The construction module is configured to extract cross-modal shared semantic consensus based on global sequence aggregation vectors, remove the shared semantic consensus from the global sequence aggregation vectors, separate discriminative residual features of specific modalities, and construct a supervised contrastive loss function based on the discriminative residual features; and obtain pure fusion features and information bottleneck loss function based on the multimodal features extracted by hierarchical cross-modal information bottleneck processing and their global sequence aggregation vectors. The iterative module is configured to backpropagate and calibrate the obtained pure fusion features, iterating cyclically to the preset maximum number of cross-modal fusion layers; extracting the single-modal global sequence aggregation vector and bottleneck pure fusion features at the end of the loop to obtain the joint classification representation; The analysis module is configured to perform sentiment prediction based on the obtained joint classification representation, obtain the sentiment prediction results, and complete multimodal sentiment analysis based on cross-modal redundancy resolution.

[0073] The detailed steps are the same as those of the multimodal sentiment analysis method based on cross-modal redundancy resolution provided in Example 1, and will not be repeated here.

[0074] Example 3 Embodiment 3 of the present invention provides a computer-readable storage medium.

[0075] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the multimodal sentiment analysis method based on cross-modal redundancy resolution as described in Embodiment 1 of the present invention.

[0076] The detailed steps are the same as those of the multimodal sentiment analysis method based on cross-modal redundancy resolution provided in Example 1, and will not be repeated here.

[0077] Example 4 Embodiment 4 of the present invention provides an electronic device.

[0078] An electronic device includes a memory, a processor, and a program stored in the memory and running on the processor. When the processor executes the program, it implements the steps in the multimodal sentiment analysis method based on cross-modal redundancy resolution as described in Embodiment 1 of the present invention.

[0079] The detailed steps are the same as those of the multimodal sentiment analysis method based on cross-modal redundancy resolution provided in Example 1, and will not be repeated here.

[0080] Example 5 Embodiment 5 of the present invention provides a computer program product.

[0081] A computer program product includes software code, wherein the program in the software code performs the steps of the multimodal sentiment analysis method based on cross-modal redundancy resolution as described in Embodiment 1 of the present invention.

[0082] The detailed steps are the same as those of the multimodal sentiment analysis method based on cross-modal redundancy resolution provided in Example 1, and will not be repeated here.

[0083] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0084] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0085] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0086] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0087] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0088] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0089] The above description is merely a preferred embodiment of this practice and is not intended to limit the scope of this practice. Various modifications and variations can be made to this practice by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this practice should be included within the protection scope of this practice.

Claims

1. A multimodal sentiment analysis method based on cross-modal redundancy resolution, characterized in that, include: Obtain the video data to be identified; Extract the multimodal features and their global sequence aggregation vectors from the acquired video data; Cross-modal shared semantic consensus is extracted based on global sequence aggregation vectors. The shared semantic consensus is removed from the global sequence aggregation vectors to separate discriminative residual features of specific modalities. A supervised contrastive loss function is constructed based on the discriminative residual features. Based on the multimodal features extracted by hierarchical cross-modal information bottleneck processing and their global sequence aggregation vector, pure fusion features and information bottleneck loss function are obtained; The obtained pure fusion features are backpropagated for calibration, and the preset maximum number of cross-modal fusion layers is iterated in a loop. Extract the single-modal global sequence aggregation vector and bottleneck pure fusion features at the end of the loop to obtain a joint classification representation; Sentiment prediction is performed based on the obtained joint classification representation, and the sentiment prediction results are obtained, thus completing the multimodal sentiment analysis based on cross-modal redundancy resolution.

2. The multimodal sentiment analysis method based on cross-modal redundancy resolution as described in claim 1, characterized in that, The process of constructing the supervised contrastive loss function is as follows: Feature fusion is performed on the global sequence aggregation vectors of each modality to extract the shared semantic consensus among the multimodalities; By fixing the shared semantic consensus using the stopping gradient operation, the global sequence aggregation vector of each modality is subtracted from the shared semantic consensus to separate the discriminative residual features of a specific modality that do not contain cross-modal global redundancy; Based on the sentiment label, a distance threshold is set to divide positive and negative sample pairs. The contrast loss is calculated using the discriminative residual features to obtain the supervised contrast loss.

3. The multimodal sentiment analysis method based on cross-modal redundancy resolution as described in claim 1, characterized in that, The calculation process of the information bottleneck loss function is as follows: Based on the sequence features of each modality and the bottleneck query vector of the current layer, a multimodal cross-attention mechanism is used to perform cross-modal interaction and aggregate to obtain intermediate fusion features; The intermediate fused features are modeled by probability distribution through random variational approximation and reparameterization, and constrained pure fused features are generated by sampling. The information bottleneck loss function representing the amount of information retained is calculated.

4. The multimodal sentiment analysis method based on cross-modal redundancy resolution as described in claim 1, characterized in that, During the reverse backpropagation calibration of the obtained pure fusion features, a cross-transform encoder is used to construct a bidirectional feedback loop, broadcasting the pure fusion features generated in the current layer back to each single-modal branch. Using the pure fusion features as the key matrix and value matrix, and the single-modal sequence features output by the upper layer as the query matrix, cross-interactive calculation is performed to filter out the temporal redundancy of the single-modal sequence at the bottom layer. The updated and denoised single-modal features are used as the input of the next layer network module, and the above feature purification and bidirectional calibration process is executed cyclically until the set maximum number of cross-modal fusion layers is reached.

5. The multimodal sentiment analysis method based on cross-modal redundancy resolution as described in claim 1, characterized in that, In the sentiment prediction process, a joint classification representation is constructed based on the final updated global sequence aggregation vectors of each modality and the pure fusion features of the last layer. This representation is then input into the prediction network to output the sentiment prediction result, i.e.: Extract the final updated single-modal global sequence aggregation vector at the end of the network loop, and concatenate it with the flattened last layer of pure fusion features in the feature dimension to form the final joint classification representation; The joint classification representation is fed into the regression prediction network to output a continuous sentiment prediction score, and the mean squared error is calculated based on the real sentiment label and the prediction score as the task prediction loss. The overall loss function is obtained by weighting and summing the task prediction loss, the supervised comparison loss multiplied by the first weight coefficient, and the information bottleneck loss of all levels multiplied by the second weight coefficient.

6. The multimodal sentiment analysis method based on cross-modal redundancy resolution as described in claim 1, characterized in that, The process of extracting the global sequence aggregation vector corresponding to each modality is as follows: Pre-trained language models are used to extract initial text sequence features, and Transformer encoders with specific classification labels and positional encodings are used to extract initial speech and visual sequence features. The classification label or equivalent feature representation after pooling operation is extracted from the initial sequence feature header of each modality and used as the global sequence aggregation vector of each modality to characterize the overall contextual semantics of a single modality.

7. A multimodal sentiment analysis system based on cross-modal redundancy resolution, characterized in that, include: The acquisition module is configured to acquire the video data to be identified; The extraction module is configured to extract multimodal features and their global sequence aggregation vectors from the acquired video data; The construction module is configured to extract cross-modal shared semantic consensus based on global sequence aggregation vectors, remove the shared semantic consensus from the global sequence aggregation vectors, separate discriminative residual features of specific modalities, and construct a supervised contrastive loss function based on the discriminative residual features; and obtain pure fusion features and information bottleneck loss function based on the multimodal features extracted by hierarchical cross-modal information bottleneck processing and their global sequence aggregation vectors. The iterative module is configured to backpropagate and calibrate the obtained pure fusion features, iterating cyclically to the preset maximum number of cross-modal fusion layers; extracting the single-modal global sequence aggregation vector and bottleneck pure fusion features at the end of the loop to obtain the joint classification representation; The analysis module is configured to perform sentiment prediction based on the obtained joint classification representation, obtain the sentiment prediction results, and complete multimodal sentiment analysis based on cross-modal redundancy resolution.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the multimodal sentiment analysis method based on cross-modal redundancy resolution as described in any one of claims 1-6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the steps of the multimodal sentiment analysis method based on cross-modal redundancy resolution as described in any one of claims 1-6.

10. A computer program product, comprising software code, characterized in that, The program in the software code performs the steps of the multimodal sentiment analysis method based on cross-modal redundancy resolution as described in any one of claims 1-6.

Images