A multi-modal sentiment analysis method and system

Abstract

This invention belongs to the field of multimodal information processing technology and provides a multimodal sentiment analysis method and system, including: acquiring multimodal sentiment data to be analyzed; extracting multimodal features from the acquired sentiment data; performing multimodal feature fusion and sentiment prediction based on the extracted multimodal features and a preset multimodal sentiment analysis model to complete multimodal sentiment analysis; wherein, the preset multimodal sentiment analysis model includes a dual-path feature decoupling module, an adaptive cross-modal interaction module, and a hierarchical language gating module; wherein, the dual-path feature decoupling module is used to decompose the extracted multimodal features into modality-shared features and modality-specific features; the adaptive cross-modal interaction module is used to enhance the interaction and consistency between modality-shared features; the hierarchical language gating module is used to guide the fusion of audio and video features in the multimodal features using language features to generate a language-enhanced supermodal representation.

Description

Technical Field

[0001] This invention belongs to the field of multimodal information processing technology, specifically relating to a multimodal sentiment analysis method and system. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Multimodal sentiment analysis is a method that integrates information from multiple modalities, such as language, audio, and video, to identify and analyze human emotional states. It typically uses semantic information, audio features, and facial expressions from multi-source heterogeneous data to create a comprehensive model, which yields more accurate sentiment judgments. It can be widely applied in fields such as human-computer interaction, intelligent customer service, public opinion analysis, and mental health assessment.

[0004] Traditional multimodal sentiment analysis methods mostly rely on manual feature design or simple feature fusion strategies, belonging to experience-driven approaches. These methods suffer from low information utilization, limited fusion methods, and insufficient generalization ability. While the development of deep learning technology has enabled existing methods to automatically extract multimodal features and achieve sentiment prediction through feature-level or decision-level fusion, significant heterogeneity exists between different modalities, leading to difficulties in cross-modal alignment. Furthermore, information redundancy and conflict between modalities are prominent issues, especially during the fusion process, where auxiliary modalities may introduce noise, thus weakening model performance. Therefore, existing methods typically fail to effectively address the imbalance of contributions between different modalities, neglecting the dominant role of the language modality in sentiment expression, making it difficult to further improve model stability and accuracy.

[0005] With the rapid development of the Internet and multimedia technologies, language, audio, and video data are experiencing explosive growth, and the complexity and diversity of multimodal data are constantly increasing. Existing methods often fail to effectively distinguish between modal-shared information and modal-specific information when dealing with modal heterogeneity, making it difficult to achieve efficient cross-modal interaction and information fusion. At the same time, insufficient utilization of multi-scale linguistic semantic information makes it difficult for models to capture fine-grained emotional features, severely restricting the performance improvement of multimodal sentiment analysis technology.

[0006] Therefore, there is an urgent need to propose a multimodal sentiment analysis method that can effectively alleviate modal heterogeneity, reduce information redundancy and conflict, and fully utilize language-dominant information to achieve efficient collaborative modeling of multimodal information and improve the accuracy and robustness of sentiment recognition. Existing methods are insufficient to achieve satisfactory multimodal sentiment analysis results, necessitating a new technical solution for improvement. Summary of the Invention

[0007] To address the aforementioned issues, this invention proposes a multimodal sentiment analysis method and system. It introduces a decoupling mechanism between modality sharing and modality-specific information to achieve structured modeling of multimodal features. An adaptive cross-modal interaction mechanism enhances intermodal information consistency, and a hierarchical language gating mechanism fully leverages the dominant role of language modality in sentiment expression, thereby effectively reducing the impact of modal conflicts and redundant information. While improving the multimodal feature fusion effect, it significantly enhances the accuracy and robustness of sentiment analysis, solving the problems of insufficient cross-modal interaction and inadequate utilization of language information in existing technologies.

[0008] According to some embodiments, the first aspect of the present invention provides a multimodal sentiment analysis method, which adopts the following technical solution: A multimodal sentiment analysis method includes: Acquire multimodal sentiment data to be analyzed; Extract multimodal features from the acquired sentiment data; Multimodal feature fusion and sentiment prediction are performed based on the extracted multimodal features and the pre-set multimodal sentiment analysis model to complete multimodal sentiment analysis; The pre-defined multimodal sentiment analysis model includes a dual-path feature decoupling module, an adaptive cross-modal interaction module, and a hierarchical language gating module. The dual-path feature decoupling module decomposes the extracted multimodal features into modality-shared features and modality-specific features. The adaptive cross-modal interaction module enhances the interaction and consistency between modality-shared features. The hierarchical language gating module uses language features to guide the fusion of audio and video features in the multimodal features, generating a language-enhanced supermodal representation.

[0009] As a further technical limitation, the acquired multimodal sentiment data to be analyzed includes language data samples, audio data samples, and video data samples. Each data sample is accompanied by a sentiment tag. The data samples contain language sequences, audio signals, and video frame information. The sentiment tags are discrete sentiment categories or continuous sentiment intensity values.

[0010] Furthermore, in the process of extracting multimodal features from the acquired sentiment data, for language data samples, a BERT-base-uncased model is used to obtain multidimensional language features; for audio data samples, the COVAREP toolkit is used to extract multidimensional low-level acoustic features; for video data samples, Facet is used to indicate facial action units and record facial muscle movements to represent emotions; the obtained features are represented as... ,in, , and These represent the language modality, audio modality, and video modality, respectively. It is the length of the sequence. Indicates the embedding dimension. .

[0011] As a further technical limitation, the dual-path feature decoupling module extracts initial features from multimodal features and decomposes them into modality-shared features and modality-specific features. Specifically, the dual-path feature decomposition architecture inputs language features, audio features, and video features into a shared encoder and three specific encoders, respectively. For the feature-shared part, the shared encoder decomposes the multimodal features into shared language features, shared audio features, and shared video features. For the feature-specific part, the language-specific encoder, audio-specific encoder, and video-specific encoder extract modality-specific information from different modalities to obtain specific language features, specific audio features, and specific video features, respectively. The shared language features, shared audio features, and shared video features are input into the adaptive cross-modal interaction module. The specific language features, specific audio features, and specific video features are input into the hierarchical language gating module.

[0012] As a further technical limitation, the adaptive cross-modal interaction module enhances the consistency and complementarity between modal shared features by quantifying modal differences and adaptive weighting. Specifically: the modal shared features extracted by the shared encoder are input into the fully connected layer and mapped to a unified feature space to obtain modal interaction representations; considering that the importance of modal interactions varies, the interaction weights between networks are generated by predicting weights based on the original shared features of the modalities; based on the difference information and interaction weights, the shared features are iteratively updated to obtain enhanced language features, audio features, and video features; the enhanced language features, audio features, and video features are concatenated to obtain the shared features.

[0013] As a further technical limitation, the loss function of the multimodal sentiment analysis model during training includes at least an information integrity constraint loss function, an adaptive interaction loss function, and a sentiment prediction loss function.

[0014] According to some embodiments, a second aspect of the present invention provides a multimodal sentiment analysis system, employing the following technical solution: A multimodal sentiment analysis system, comprising: The acquisition module is configured to acquire multimodal sentiment data to be analyzed. The extraction module is configured to extract multimodal features from the acquired sentiment data; The analysis module is configured to perform multimodal feature fusion and sentiment prediction based on the extracted multimodal features and a preset multimodal sentiment analysis model, thereby completing multimodal sentiment analysis. The pre-defined multimodal sentiment analysis model includes a dual-path feature decoupling module, an adaptive cross-modal interaction module, and a hierarchical language gating module. The dual-path feature decoupling module decomposes the extracted multimodal features into modality-shared features and modality-specific features. The adaptive cross-modal interaction module enhances the interaction and consistency between modality-shared features. The hierarchical language gating module uses language features to guide the fusion of audio and video features in the multimodal features, generating a language-enhanced supermodal representation.

[0015] According to some embodiments, a third aspect of the present invention provides a computer-readable storage medium, employing the following technical solution: A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of a multimodal sentiment analysis method as described in the first aspect of the present invention.

[0016] According to some embodiments, the fourth aspect of the present invention provides an electronic device, which adopts the following technical solution: An electronic device includes a memory, a processor, and a program stored in the memory and running on the processor, wherein the processor executes the program to implement the steps in a multimodal sentiment analysis method as described in the first aspect of the present invention.

[0017] According to some embodiments, the fifth aspect of the present invention provides a computer program product, which adopts the following technical solution: A computer program product includes software code, wherein the program in the software code performs the steps of a multimodal sentiment analysis method as described in the first aspect of the present invention.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention constructs a multimodal sentiment analysis model, achieving collaborative processing of multi-source information from language, audio, and video, accelerating the response process of the multimodal sentiment analysis system, and improving the efficiency and processing speed of sentiment recognition. By introducing a dual-path feature decoupling mechanism, it effectively alleviates the problems of strong modal heterogeneity, information redundancy, and severe intermodal conflicts in multimodal data, improving the effectiveness and stability of multimodal feature representation. Through an adaptive cross-modal interaction module, it realizes dynamic information interaction and adaptive weight allocation between different modalities, solving the problems of difficult cross-modal information alignment and insufficient interaction in traditional methods, thereby improving the consistency of intermodal information fusion. Through a hierarchical language gating module, it introduces multi-scale linguistic semantic information to guide audio and video modalities, fully utilizing... This study leverages the dominant role of language modality in emotion expression, providing a language-driven modeling approach for multimodal fusion and offering a new technical direction for subsequent multimodal learning research. By constructing language-guided supermodal representations, complementary features from multimodal information can be effectively integrated, thereby improving the model's ability to model complex emotional semantics and enhancing the accuracy of emotion analysis results. Through the synergistic effect of the adaptive cross-modal interaction module and the hierarchical language gating module, the model possesses stronger cross-modal representation learning capabilities, improving the overall performance and robustness in multimodal emotion analysis tasks. By optimizing the information utilization method in the multimodal fusion process, invalid information interference is reduced, and the model's generalization ability under different data scenarios is improved, thereby enhancing the system's stability and practicality. Attached Figure Description

[0019] The accompanying drawings, which form part of this embodiment, are used to provide a further understanding of this embodiment. The illustrative embodiments and their descriptions are used to explain this embodiment and do not constitute an improper limitation of this embodiment.

[0020] Figure 1 This is a flowchart of the multimodal sentiment analysis method in Embodiment 1 of the present invention; Figure 2 This is an architecture diagram of the multimodal sentiment analysis model in Embodiment 1 of the present invention; Figure 3 This is an architecture diagram of the adaptive cross-modal interaction module in Embodiment 1 of the present invention; Figure 4 This is an architecture diagram of the layered language gating module in Embodiment 1 of the present invention; Figure 5 This is an architecture diagram of the gated hypermodal learning module in Embodiment 1 of the present invention; Figure 6 This is a structural block diagram of the multimodal emotion analysis system in Embodiment 2 of the present invention. Detailed Implementation

[0021] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0023] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0024] In this invention, terms such as "upper," "lower," "left," "right," "front," "back," "vertical," "horizontal," "side," and "bottom" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used only to facilitate the description of the structural relationships of the various components or elements of this invention and do not specifically refer to any component or element in this invention. They should not be construed as limiting the invention.

[0025] In this invention, terms such as "fixed connection," "connected," and "linked" should be interpreted broadly, indicating a fixed connection, an integral connection, or a detachable connection; a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can determine the specific meaning of these terms in this invention based on the specific circumstances, and they should not be construed as limitations on the invention.

[0026] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0027] Example 1 Embodiment 1 of this invention introduces a multimodal sentiment analysis method.

[0028] like Figure 1 The multimodal sentiment analysis method shown includes: Acquire multimodal sentiment data to be analyzed; Extract multimodal features from the acquired sentiment data; Multimodal feature fusion and sentiment prediction are performed based on the extracted multimodal features and the pre-set multimodal sentiment analysis model to complete multimodal sentiment analysis; The pre-defined multimodal sentiment analysis model includes a dual-path feature decoupling module, an adaptive cross-modal interaction module, and a hierarchical language gating module. The dual-path feature decoupling module decomposes the extracted multimodal features into modality-shared features and modality-specific features. The adaptive cross-modal interaction module enhances the interaction and consistency between modality-shared features. The hierarchical language gating module uses language features to guide the fusion of audio and video features in the multimodal features, generating a language-enhanced supermodal representation.

[0029] As one or more implementation methods, this embodiment collects data samples containing three modalities: language, audio, and video. Each sample contains a language sequence, an audio signal, and corresponding video frame information, along with an emotion tag. The emotion tag can be a discrete emotion category or a continuous emotion intensity value. The above multimodal data samples are organized to construct a multimodal emotion analysis training dataset.

[0030] It should be noted that this embodiment preprocesses the acquired multimodal data, including: segmenting and encoding the language data; extracting and standardizing the audio data; and aligning and extracting the video data to obtain multimodal input data in a unified format for subsequent model training.

[0031] As one or more implementation methods, the feature extraction module in this embodiment takes language data, audio data and video data in the multimodal dataset as input, performs feature extraction on different modal data respectively, outputs language features, audio features and video features, and sends the language features, audio features and video features together into the dual-path feature decomposition module.

[0032] In this embodiment, language data is used as input, and the language data is encoded using the BERT model to obtain language features that represent the semantic information of the language. Audio data is used as input, and audio modal features are extracted using the COVAREP tool to obtain audio features. Video data is used as input, and video modal features are extracted using the Facet tool to obtain video features. The output of the feature extraction module in this embodiment is language features, audio features, and video features, and the outputs are sent together to the dual-path feature decomposition module.

[0033] It should be noted that the feature extraction in this embodiment is used to extract initial features from multimodal data. For a given multimodal input, their features are extracted respectively; for the language modality, 768-dimensional language features are obtained by using the BERT-base-uncased model; for the audio modality, 74-dimensional low-level acoustic features are extracted using the COVAREP toolkit; for the video modality, Facet is used to indicate 35 facial motion units, recording facial muscle movements to represent emotions; this embodiment represents these features as follows: ,in , and Representing language, audio, and video modalities respectively. It is the length of the sequence. Represents the embedding dimension, where .

[0034] This embodiment uses, as follows: Figure 2 The multimodal sentiment analysis model shown includes a feature extraction module, a dual-path feature decomposition module, an adaptive cross-modal interaction module, a hierarchical language gating module, and a sentiment prediction module.

[0035] As one or more implementation methods, the dual-path feature decomposition module in this embodiment takes language features, audio features and video features as input, separates shared features and specific features of each modality feature, and outputs shared language features, shared audio features, shared video features, as well as specific language features, specific audio features and specific video features.

[0036] This embodiment takes language features, audio features, and video features as input. A shared encoder extracts modal-shared information from different modalities, yielding shared language features, shared audio features, and shared video features. Then, language-specific encoders, audio-specific encoders, and video-specific encoders extract modal-specific information from different modalities, yielding specific language features, specific audio features, and specific video features. The output of the dual-path feature decomposition module is split into two paths: one path, containing shared language features, shared audio features, and shared video features, is fed into the adaptive cross-modal interaction module; the other path, containing specific language features, specific audio features, and specific video features, is fed into the hierarchical language gating module.

[0037] To handle multimodal features, a dual-path feature decomposition architecture is used to input language features, audio features, and video features separately into a shared encoder. and three specific encoders In the middle section, for the feature-sharing part, a shared encoder is used to decompose the multimodal features into shared language features, shared audio features, and shared video features, represented as follows: ,in For feature-specific parts, language-specific encoders, audio-specific encoders, and video-specific encoders are used to extract modality-specific information from different modalities, resulting in specific language features, specific audio features, and specific video features, respectively. Specific features are represented as... ,in, The specific process is as follows:

[0038]

[0039] in, and It consists of 1D convolutional layers. Finally, it includes an adaptive cross-modal interaction module that inputs shared language features, shared audio features, and shared video features; and a hierarchical language gating module that inputs specific language features, specific audio features, and specific video features.

[0040] As one or more implementation methods, such as Figure 3 As shown, the adaptive cross-modal interaction module in this embodiment takes shared language features, shared audio features, and shared video features as inputs. Through cyclic cross-modal interaction and adaptive weighting mechanism, it enhances the consistency of multimodal information in the shared space, outputs enhanced language features, enhanced audio features, enhanced video features, and the final shared fusion feature, and sends the final shared fusion feature to the feature splicing module.

[0041] In this embodiment, shared language features, shared audio features, and shared video features are taken as inputs. Each shared feature is mapped using a fully connected layer and transformed using an activation function to obtain an intermediate shared representation suitable for interactive modeling. The intermediate shared representation is then taken as input to establish a cyclical interaction relationship among the three modalities of language, audio, and video, so that each modality obtains supplementary information during interaction with other modalities, resulting in enhanced language features, enhanced audio features, and enhanced video features. The enhanced language features, enhanced audio features, and enhanced video features are then taken as input, concatenated, and fused to obtain the final shared fused feature, which is then sent to the feature concatenation module.

[0042] This embodiment features an adaptive cross-modal interaction module: This module aims to enhance features by quantifying modal differences and performing adaptive weighting, thereby improving the consistency and complementarity among modally shared features.

[0043] This embodiment will share the encoder. Extracting modal shared features The input to the fully connected layer is mapped to a unified feature space to obtain the modal interaction representation. To quantify the differences between them, a definition is given from modality. To mode Cross-modal variability As an interactive prototype and The L2 distance between them is determined as follows:

[0044]

[0045] in, This represents learnable parameters.

[0046] Considering the varying importance of different modal interactions, this embodiment designs a weight generation network. Based on the original shared features of a pair of modes To predict interaction weights ,Right now

[0047] in, These are learnable parameters.

[0048] Based on differential information and interaction weights, the shared features are iteratively updated to obtain enhanced language features, audio features, and video features, and these three modalities are then shared. By piecing them together, a more expressive shared feature can be formed. ,Right now

[0049] in, It is the total dimension of the connectivity features, which is input into a two-layer fully connected network to generate the final shared fused features. ,Right now

[0050] in, Will Dimensions adjusted to Dimension The final shared fusion feature is generated as the output layer and then fed into the sentiment prediction module.

[0051] As one or more implementation methods, such as Figure 4 As shown, the hierarchical language gating module in this embodiment takes specific language features, specific audio features, specific video features, and initial supermodal features as inputs. Through multi-layer language feature guidance and gating supermodal learning, it outputs enhanced language features and final language-specific fusion features, and sends the final language-specific fusion features to the feature splicing module.

[0052] This embodiment takes specific language features as input and uses multiple Transformer layers to perform layer-by-layer semantic modeling of these features, obtaining language features at different levels, including second-layer and third-layer specific language features. Next, the gated hypermodal learning submodule takes the hierarchical language features, specific audio features, specific video features, and initial hypermodal features as input. Guided by the language modality, it performs gated modulation and iterative updates on the audio and video modalities to obtain the third-layer hypermodal features. Finally, the third-layer specific language features and the third-layer hypermodal features are input together into the gated fusion unit to obtain enhanced language features. These enhanced features are then processed by a fully connected layer and activation function to output the final language-specific fusion features, which are then sent to the feature concatenation module.

[0053] Since language modalities dominate multimodal sentiment analysis, while audio and video modalities contribute relatively little, a hierarchical language gating module was designed for the decoded specific modality, and a supermodal feature was initialized. The goal is to use language features as a guide to gradually generate supermodal features from video and audio modalities; the generated supermodal features have low redundancy and high complementarity with the language modalities.

[0054] This embodiment constructs a language feature hierarchy, which includes the initial modality-specific features. Defined as low-scale features; guided by language modality, from its low-scale features Begin by using the Transformer layer ( Iterative learning of mesoscale ( ) and high-scale ( Language features, namely:

[0055] in, Representing language features at different scales, the scale is... , This represents the parameters of the Transformer layer.

[0056] In this embodiment, the supermodal features are generated by a gated supermodal learning module. The core of the gated supermodal learning module is an iterative framework consisting of three stacked gated layers. Each gated layer utilizes language features at a specific scale to guide the processing of video and audio features, progressively updating the initialized supermodal features. At each GHML layer, language features at the corresponding scale are used. As a dynamic guide, the supermodal features are updated from the previous layer. The input state of the first layer It is a parametric tensor that needs to be initialized through learning, and its shape is... .

[0057] This embodiment requires the use of language features. To generate gating signals for video and audio Specifically, firstly regarding Normalization is performed, using video and audio-specific linear transformations to generate corresponding... Finally, the Sigmoid function is used to compress the result into the (0,1) interval to form a gated signal. The specific process is as follows:

[0058] in, It is the Sigmoid function. It is a weight matrix. It is the deviation vector, where Then, the generated gating signal Compared with the original modal features Element-wise multiplication is performed to obtain the gated features, and the process is as follows:

[0059] in, This represents the Hadamard product. Then, all gated modal features (e.g., video and audio) are combined element-wise:

[0060] This embodiment combines gated features through residual connections. Added to the supermodal feature state of the previous layer Then, layer normalization is applied to obtain the supermodal feature state of the current layer. The updated supermodal features are:

[0061] in, Stored the past Layer-accumulated supermodal information The current layer integrates elements from this foundation. The gating video and audio information is updated iteratively throughout the entire process.

[0062] To enhance the advantages of language features, this embodiment will With language features Combining these features to generate language advantage enhancement features ,Right now

[0063] in, It is a learnable weight matrix. This represents element-wise multiplication. (Enhancing language features) After processing through two layers of fully connected networks, language-specific fusion features are obtained. ,Right now

[0064] In acquiring shared and integrated characteristics and language-specific fusion features Next, a classifier is applied for multimodal fusion and sentiment prediction, i.e.

[0065] in, This indicates sentiment prediction.

[0066] like Figure 5 As shown, the gated supermodal learning submodule in this embodiment takes specific language features, specific audio features, specific video features, and the previous layer supermodal features as input, performs gated signal generation, modality weighting, and supermodal update operations, outputs the current layer supermodal features, and sends the current layer supermodal features into the next layer gated supermodal learning process.

[0067] This embodiment takes specific language features as input and performs layer normalization processing. After audio and video linear transformations, audio and video gating signals are obtained using the Sigmoid function. The audio gating signals and specific audio features are input into an audio weighting unit to obtain audio-gated features. The video gating signals and specific video features are input into a video weighting unit to obtain video-gated features. The audio and video gating features are concatenated or combined to obtain combined gating features. The combined gating features and the previous layer's supermodal features are input into a residual connection unit, and after layer normalization processing, the current layer's supermodal features are output. If the current layer is not the last layer, the current layer's supermodal features are used as input to the next layer's gated supermodal learning submodule.

[0068] As one or more implementation methods, the sentiment prediction module in this embodiment takes the final shared fusion features and the final language-specific fusion features as input, performs feature concatenation, then performs sentiment prediction, and outputs a sentiment classification result or a sentiment intensity prediction result. The sentiment prediction module includes a Transformer, a pooling layer, a multilayer perceptron, and a prediction layer. In this embodiment, the final shared fusion features and the final language-specific fusion features are taken as input and concatenated; the high-level dependencies in the fusion representation are modeled using a Transformer, the feature representation is compressed using a pooling layer, and then nonlinearly mapped using a multilayer perceptron. Finally, the prediction layer outputs a sentiment classification result or a sentiment intensity prediction result.

[0069] This embodiment employs multiple loss functions for joint optimization during the training of the multimodal sentiment analysis model. To ensure the integrity of the decomposed information, an information integrity constraint loss is introduced, which is calculated using the loss generated by the decoder. The mean squared error (MSE) between the reconstructed features and the original features, i.e.:

[0070] The adaptive interaction loss in this embodiment ( Defined as the weighted sum of differences across all interaction paths, it is used to constrain intermodal consistency, i.e.

[0071] in, This represents the actual difference between mode m and mode n. The corresponding weights.

[0072] Sentiment prediction loss is used to measure the difference between the predicted result and the true label; the corresponding sentiment prediction loss is... ,Right now

[0073] in, It is the number of samples in the training set. It is the first The sentiment labels of each sample.

[0074] This embodiment performs a weighted summation of the above loss functions to obtain the total loss function. The gradient descent method is used to optimize and train the model parameters.

[0075] Where α, β, and γ are the corresponding hyperparameters.

[0076] The model proposed in this embodiment achieves better results than other methods on the MOSI dataset. The comparison of experimental results is shown in Table 1.

[0077] Table 1 Comparison of Experimental Results

[0078] The model proposed in this embodiment achieves better results than other methods on the MOSEI dataset. The comparison of the experimental results is shown in Table 2.

[0079] Table 2 Comparison of Experimental Results

[0080] This embodiment constructs a multimodal sentiment analysis model, achieving collaborative processing of multi-source information from language, audio, and video, accelerating the response process of the multimodal sentiment analysis system, and improving the efficiency and processing speed of sentiment recognition. By introducing a dual-path feature decoupling mechanism, it effectively alleviates the problems of strong modal heterogeneity, information redundancy, and severe intermodal conflicts in multimodal data, improving the effectiveness and stability of multimodal feature representation. Through an adaptive cross-modal interaction module, it realizes dynamic information interaction and adaptive weight allocation between different modalities, solving the problems of difficult cross-modal information alignment and insufficient interaction in traditional methods, thereby improving the consistency of intermodal information fusion. Through a hierarchical language gating module, it introduces multi-scale linguistic semantic information to guide audio and video modalities, fully utilizing... This study leverages the dominant role of language modality in emotion expression, providing a language-driven modeling approach for multimodal fusion and offering a new technical direction for subsequent multimodal learning research. By constructing language-guided supermodal representations, complementary features from multimodal information can be effectively integrated, thereby improving the model's ability to model complex emotional semantics and enhancing the accuracy of emotion analysis results. Through the synergistic effect of the adaptive cross-modal interaction module and the hierarchical language gating module, the model possesses stronger cross-modal representation learning capabilities, improving the overall performance and robustness in multimodal emotion analysis tasks. By optimizing the information utilization method in the multimodal fusion process, invalid information interference is reduced, and the model's generalization ability under different data scenarios is improved, thereby enhancing the system's stability and practicality.

[0081] Example 2 Embodiment 2 of this invention introduces a multimodal sentiment analysis system.

[0082] like Figure 6 The multimodal sentiment analysis system shown includes: The acquisition module is configured to acquire multimodal sentiment data to be analyzed. The extraction module is configured to extract multimodal features from the acquired sentiment data; The analysis module is configured to perform multimodal feature fusion and sentiment prediction based on the extracted multimodal features and a preset multimodal sentiment analysis model, thereby completing multimodal sentiment analysis. The pre-defined multimodal sentiment analysis model includes a dual-path feature decoupling module, an adaptive cross-modal interaction module, and a hierarchical language gating module. The dual-path feature decoupling module decomposes the extracted multimodal features into modality-shared features and modality-specific features. The adaptive cross-modal interaction module enhances the interaction and consistency between modality-shared features. The hierarchical language gating module uses language features to guide the fusion of audio and video features in the multimodal features, generating a language-enhanced supermodal representation.

[0083] The detailed steps are the same as those provided in Example 1 for a multimodal sentiment analysis method, and will not be repeated here.

[0084] Example 3 Embodiment 3 of the present invention provides a computer-readable storage medium.

[0085] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of a multimodal sentiment analysis method as described in Embodiment 1 of the present invention.

[0086] The detailed steps are the same as those provided in Example 1 for a multimodal sentiment analysis method, and will not be repeated here.

[0087] Example 4 Embodiment 4 of the present invention provides an electronic device.

[0088] An electronic device includes a memory, a processor, and a program stored in the memory and running on the processor, wherein the processor executes the program to implement the steps of a multimodal sentiment analysis method as described in Embodiment 1 of the present invention.

[0089] The detailed steps are the same as those provided in Example 1 for a multimodal sentiment analysis method, and will not be repeated here.

[0090] Example 5 Embodiment 5 of the present invention provides a computer program product.

[0091] A computer program product includes software code, wherein the program in the software code performs the steps of a multimodal sentiment analysis method as described in Embodiment 1 of the present invention.

[0092] The detailed steps are the same as those provided in Example 1 for a multimodal sentiment analysis method, and will not be repeated here.

[0093] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0094] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0095] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0096] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0097] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0098] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0099] The above description is merely a preferred embodiment of this practice and is not intended to limit the scope of this practice. Various modifications and variations can be made to this practice by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this practice should be included within the protection scope of this practice.

Claims

1. A multimodal sentiment analysis method, characterized in that, include: Acquire multimodal sentiment data to be analyzed; Extract multimodal features from the acquired sentiment data; Multimodal feature fusion and sentiment prediction are performed based on the extracted multimodal features and the pre-set multimodal sentiment analysis model to complete multimodal sentiment analysis; The pre-defined multimodal sentiment analysis model includes a dual-path feature decoupling module, an adaptive cross-modal interaction module, and a hierarchical language gating module. The dual-path feature decoupling module decomposes the extracted multimodal features into modality-shared features and modality-specific features. The adaptive cross-modal interaction module enhances the interaction and consistency between modality-shared features. The hierarchical language gating module uses language features to guide the fusion of audio and video features in the multimodal features, generating a language-enhanced supermodal representation.

2. The multimodal sentiment analysis method as described in claim 1, characterized in that, The acquired multimodal sentiment data to be analyzed includes language data samples, audio data samples, and video data samples. Each data sample is accompanied by a sentiment tag. The data samples contain language sequences, audio signals, and video frame information. The sentiment tags are discrete sentiment categories or continuous sentiment intensity values.

3. The multimodal sentiment analysis method as described in claim 2, characterized in that, In the process of extracting multimodal features from the acquired sentiment data, for language data samples, a BERT-base-uncased model is used to obtain multidimensional language features; for audio data samples, the COVAREP toolkit is used to extract multidimensional low-level acoustic features; for video data samples, Facet is used to indicate facial action units and record facial muscle movements to represent emotions; the obtained features are represented as... ,in, , and These represent the language modality, audio modality, and video modality, respectively. It is the length of the sequence. Indicates the embedding dimension. .

4. The multimodal sentiment analysis method as described in claim 1, characterized in that, The dual-path feature decoupling module extracts initial features from multimodal features and decomposes them into modality-shared features and modality-specific features. Specifically, the dual-path feature decomposition architecture inputs language features, audio features, and video features into a shared encoder and three specific encoders, respectively. For the feature-shared part, the shared encoder decomposes the multimodal features into shared language features, shared audio features, and shared video features. For the feature-specific part, the language-specific encoder, audio-specific encoder, and video-specific encoder extract modality-specific information from different modalities to obtain specific language features, specific audio features, and specific video features, respectively. The shared language features, shared audio features, and shared video features are input into the adaptive cross-modal interaction module. The specific language features, specific audio features, and specific video features are input into the hierarchical language gating module.

5. The multimodal sentiment analysis method as described in claim 1, characterized in that, The adaptive cross-modal interaction module enhances the consistency and complementarity between modal shared features by quantifying modal differences and adaptive weighting. Specifically, the modal shared features extracted by the shared encoder are input into the fully connected layer and mapped to a unified feature space to obtain the modal interaction representation. Considering that the importance of modal interactions varies, the interaction weights between networks are generated by predicting weights based on the original shared features of the modalities. Based on the difference information and interaction weights, the shared features are iteratively updated to obtain enhanced language features, audio features, and video features. The enhanced language features, audio features, and video features are then concatenated to obtain the shared features.

6. The multimodal sentiment analysis method as described in claim 1, characterized in that, The loss function of the multimodal sentiment analysis model during training includes at least an information integrity constraint loss function, an adaptive interaction loss function, and a sentiment prediction loss function.

7. A multimodal sentiment analysis system, characterized in that, include: The acquisition module is configured to acquire multimodal sentiment data to be analyzed. The extraction module is configured to extract multimodal features from the acquired sentiment data; The analysis module is configured to perform multimodal feature fusion and sentiment prediction based on the extracted multimodal features and a preset multimodal sentiment analysis model, thereby completing multimodal sentiment analysis. The pre-defined multimodal sentiment analysis model includes a dual-path feature decoupling module, an adaptive cross-modal interaction module, and a hierarchical language gating module. The dual-path feature decoupling module decomposes the extracted multimodal features into modality-shared features and modality-specific features. The adaptive cross-modal interaction module enhances the interaction and consistency between modality-shared features. The hierarchical language gating module uses language features to guide the fusion of audio and video features in the multimodal features, generating a language-enhanced supermodal representation.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of a multimodal sentiment analysis method as described in any one of claims 1-6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the steps of a multimodal sentiment analysis method as described in any one of claims 1-6.

10. A computer program product, comprising software code, characterized in that, The program in the software code performs the steps of a multimodal sentiment analysis method as described in any one of claims 1-6.

Images