Multimodal feature fusion method and system for sentiment analysis

Abstract

The present disclosure provides a multimodal feature fusion method and system for sentiment analysis, which includes: obtaining text data and its corresponding image data in social data; performing feature extraction on the text data and image data respectively to obtain text features and image features; based on the text features and image features, using a cross-modal attention mechanism to obtain interaction information between the two, and performing noise filtering on the interaction information through a gating mechanism to obtain text features after image filtering and image features after text filtering; splicing the obtained text features, image features, text features after image filtering, and image features after text filtering to obtain fused features; and based on the fused features, obtaining sentiment analysis results through a pre-built sentiment analysis model.

Description

Technical Field

[0001] The present disclosure belongs to the field of sentiment analysis technology, and in particular relates to a multimodal feature fusion method and system for sentiment analysis. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] With the rise of social media platforms, the ways to express emotions across multiple modalities are also becoming increasingly diverse. Previous feature fusion methods have mostly simply combined two modalities without exploring complex correlations. Therefore, how to better utilize both intra-modal information and modal interaction information has become a hot topic of current research.

[0004] There are three deep learning models currently used in multimodal fusion:

[0005] (1) Methods based on CNN, RNN and their variants

[0006] Convolutional neural networks (CNNs) are a typical deep learning model. They can learn feature representation mechanisms at different levels of abstraction for text or image data and are trainable, multi-level feedforward artificial neural networks. For example, two CNN-based models can be used to extract single-modal features. Their combined representation is then fed into a multi-layer CNN to fully exploit the inherent relationships between modalities.

[0007] (2) Attention-based methods

[0008] The attention mechanism is a special structure embedded in machine learning models that automatically learns and calculates the contribution of input data to output data. Models or structures such as BERT, GPT, and Transformer all employ the attention mechanism. Recently, many researchers have begun experimenting with incorporating the Transformer into multimodal tasks. For example, the ViLBERT model combines features from both images and text, transforming BERT into a bidirectional architecture. First, the image is encoded using a Transformer encoder, where the input to the image encoder is the salient object regions extracted by Fast R-CNN. These regions are then fed into a common Transformer layer to achieve sparse interaction and obtain a multimodal joint representation.

[0009] (3) Tensor-based approach

[0010] Tensor representation is a successful multimodal fusion method that first converts multiple inputs into high-dimensional tensors and then maps them back to a low-dimensional output vector space. The tensor representation can be obtained by taking the outer product of the input modalities. For example, the TFN (Tensor Fusion Network) model uses the outer tensor product between modules to measure the relationship between the parts of the module. However, this method greatly increases the dimension of the feature vector and increases the difficulty of training. LMF (Low-rank Multimodal Fusion) is an upgrade of TFN that uses low-rank representation to decompose weights. It reduces the number of model parameters by performing a linear transformation on each mode and then performing a multi-dimensional dot product.

[0011] The main goal of multimodal fusion is to reduce the heterogeneous differences between modalities while maintaining the integrity of the specific semantics of each modality. Previous feature fusion methods have mostly simply combined the two modes without exploring complex correlations. With the continuous rise of social platforms, the ways of expressing emotions in multiple modalities are also constantly diversifying. In order to more accurately extract and fuse multimodal sentiment features, researchers have further explored the cross-modal correlation between image features and text features by capturing similar structures between different modes. How to better utilize intra-modal information and modal interaction information has become a hot topic of current research. For example: a cross-modal attention mechanism is used to obtain the interaction information between the two modes, and a gating mechanism is proposed to filter the noise in the generated interaction information.

[0012] The inventors found that existing multimodal feature fusion methods based on attention mechanisms are mostly bidirectional, aiming to align text entity nouns and local image regions; however, for social data such as blogs and comments, the emotional information expressed in the text is usually the main body, while the image content usually only serves as an auxiliary description or supplementary explanation of the text content. This fusion method often leads to the omission of some image information that independently expresses emotions on current multimedia platforms, resulting in reduced classification accuracy. Summary of the Invention

[0013] In order to solve the above problems, the present disclosure provides a multimodal feature fusion method and system for sentiment analysis. The scheme combines the internal information of the modality with the interactive information between the modalities through cross-modal interaction and gating network based on the attention mechanism, obtains sufficient modal representation, and solves the problem of insufficient mining of multimodal internal information; at the same time, deeper related information between the two modes is obtained through the interaction mechanism to enrich the single-modal information, and the gating mechanism is used to filter out redundant information in the interactive information, further ensuring the accuracy of the fused features.

[0014] According to a first aspect of an embodiment of the present disclosure, a multimodal feature fusion method for sentiment analysis is provided, comprising:

[0015] Obtain text data and its corresponding image data in social data;

[0016] Performing feature extraction on the text data and image data respectively to obtain text features and image features;

[0017] Based on the text features and image features, a cross-modal attention mechanism is used to obtain the interaction information between the two, and the interaction information is noise-filtered through a gating mechanism to obtain the text features after image filtering and the image features after text filtering;

[0018] The obtained text features, image features, text features after image filtering, and image features after text filtering are spliced ​​to obtain fusion features;

[0019] Based on the fusion features, sentiment analysis results are obtained through a pre-built sentiment analysis model.

[0020] Furthermore, the cross-modal attention mechanism is used to obtain the interaction information between the two, specifically:

[0021] Based on the text features and image features, calculating the attention matrix of the two;

[0022] Quantize the attention matrix using the softmax function;

[0023] Calculate the product of the quantized result of the attention matrix and the image features to obtain the interactive information from text to image;

[0024] Calculate the product of the quantized result of the attention matrix and the text feature to obtain the image-to-text interaction information.

[0025] Furthermore, the noise filtering of the interactive information through the gating mechanism includes noise filtering of text-to-image interactive information and noise filtering of image-to-text interactive information.

[0026] Furthermore, the noise filtering of the interactive information from text to image is specifically represented as follows;

[0027]

[0028] Among them, G() represents the gate operation, N() represents the nonlinear operation, and I TV is the interactive information from text to image, H V is the image feature.

[0029] Furthermore, the noise filtering of the image-to-text interaction information is specifically represented as follows;

[0030]

[0031] Among them, I VT is the interactive information from image to text, H T is a text feature.

[0032] According to a second aspect of an embodiment of the present disclosure, a multimodal feature fusion system for sentiment analysis is provided, comprising:

[0033] A data acquisition unit, which is used to acquire text data and corresponding image data in social data;

[0034] A feature extraction unit, configured to extract features from the text data and the image data to obtain text features and image features;

[0035] An interactive information acquisition and filtering unit, which is used to acquire interactive information between the text features and the image features using a cross-modal attention mechanism, and to perform noise filtering on the interactive information using a gating mechanism to obtain text features after image filtering and image features after text filtering;

[0036] A feature fusion unit is used to combine the obtained text features, image features, text features after image filtering, and image features after text filtering to obtain fused features;

[0037] The sentiment analysis unit is used to obtain sentiment analysis results based on the fusion features through a pre-built sentiment analysis model.

[0038] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored and running on the memory, wherein when the processor executes the program, the multimodal feature fusion method for sentiment analysis is implemented.

[0039] According to a fourth aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the multimodal feature fusion method for sentiment analysis is implemented.

[0040] Compared with the prior art, the present invention has the following advantages:

[0041] (1) The present disclosure provides a multimodal feature fusion method and system for sentiment analysis. The scheme combines the internal information of the modality with the interactive information between the modalities through cross-modal interaction and gating network based on the attention mechanism to obtain sufficient modal representation, thereby solving the problem of insufficient mining of multimodal internal information.

[0042] (2) The scheme disclosed in the present invention obtains deeper related information between the two modes through an interactive mechanism to enrich the single-mode information, and uses a gating mechanism to filter out redundant information in the interactive information, thereby further ensuring the accuracy of the fusion features.

[0043] Advantages of additional aspects of the present disclosure will be given in part in the following description and in part will become apparent from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which constitute a part of the present disclosure, are used to provide a further understanding of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation to the present disclosure.

[0045] Figure 1 This is a basic flow chart of a multimodal feature fusion method for sentiment analysis described in an embodiment of the present disclosure;

[0046] Figure 2 Schematic diagram of the experimental results of the method described in the embodiments of the present disclosure. DETAILED DESCRIPTION

[0047] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0048] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure belongs.

[0049] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0050] In the absence of conflict, the embodiments of the present disclosure and the features thereof may be combined with each other.

[0051] Example 1:

[0052] The purpose of this embodiment is to provide a multimodal feature fusion method for sentiment analysis.

[0053] like Figure 1 As shown in FIG, a multimodal feature fusion method for sentiment analysis includes:

[0054] Obtain text data and its corresponding image data in social data;

[0055] Performing feature extraction on the text data and image data respectively to obtain text features and image features;

[0056] Based on the text features and image features, a cross-modal attention mechanism is used to obtain the interaction information between the two, and the interaction information is noise-filtered through a gating mechanism to obtain the text features after image filtering and the image features after text filtering;

[0057] The obtained text features, image features, text features after image filtering, and image features after text filtering are spliced ​​to obtain fusion features;

[0058] Based on the fusion features, sentiment analysis results are obtained through a pre-built sentiment analysis model.

[0059] In a specific implementation, the cross-modal attention mechanism is used to obtain the interaction information between the two, specifically:

[0060] Based on the text features and image features, calculating the attention matrix of the two;

[0061] Quantize the attention matrix using the softmax function;

[0062] Calculate the product of the quantized result of the attention matrix and the image features to obtain the interactive information from text to image;

[0063] Calculate the product of the quantized result of the attention matrix and the text feature to obtain the image-to-text interaction information.

[0064] In a specific implementation, the noise filtering of the interactive information through a gating mechanism includes noise filtering of text-to-image interactive information and noise filtering of image-to-text interactive information.

[0065] In a specific implementation, the noise filtering of the text-to-image interaction information is specifically expressed as follows:

[0066]

[0067] Among them, G() represents the gate operation, N() represents the nonlinear operation, and I TV is the interactive information from text to image, HV is the image feature.

[0068] In a specific implementation, the noise filtering of the image-to-text interaction information is specifically expressed as follows:

[0069]

[0070] Among them, I VT is the interactive information from image to text, H T is a text feature.

[0071] In specific implementation, the scheme of this embodiment utilizes the k-max pooling strategy to retain the relative order of the feature values. At the same time, through the cross-modal interaction and gating network based on the attention mechanism, the internal information of the modality is combined with the interactive information between the modalities to obtain a full modal representation. The scheme adopts the following technical concepts: First, a cross-modal interaction mechanism is used to model the interaction between images and texts, and the interactive information between the two is obtained respectively. Secondly, not all the interactive information obtained is useful for the sentiment analysis task, so the scheme described in this embodiment filters the redundant information in the interaction process through the gating mechanism, retains the part most relevant to the emotion, and obtains the most valuable information. Finally, by using the attention mechanism to automatically calculate the contribution of each modality, the internal modal features are connected with the corresponding learned interactive features to obtain a complete representation of each modal feature, which more effectively integrates the bimodal features.

[0072] Among them, K-Max Pooling is an operation method for globally taking the top-K features. It maintains the relative order of feature values ​​and can express the situation where the same type of features appear multiple times to highlight the importance of a certain type of feature.

[0073] The calculation formula is:

[0074] h' max =k-max(h t )

[0075] Among them, the selection of k value can also be obtained dynamically:

[0076]

[0077] Here, l is the sentence length, L is the total number of convolutional layers, and c is the current number of convolutional layers. Therefore, it can be seen that k changes with the length of the sentence and the depth of the network.

[0078] The contribution value refers to the influence of each modality on the final sentiment classification; the bimodal feature refers to the feature values ​​of the two modalities.

[0079] Specifically, the solution described in this embodiment includes the following processing steps:

[0080] Step 1: Obtain interaction information

[0081] For example, to calculate a set of text H T With image H V The interaction between attention matrix M TV ∈R u×u Expressed as:

[0082]

[0083] The softmax function is then used to quantize the attention matrix:

[0084]

[0085] The final interaction information from text to image is then expressed as:

[0086] I TV =J TV .H V (3)

[0087] Similarly, the final interaction information between image and text is expressed as:

[0088] I VT =J VT .H T (4)

[0089] The obtained text data is processed as follows:

[0090] (1) Text preprocessing: First remove all numbers, special symbols, references, etc., then divide the text into tokens, add special symbols [CLS] and [SEP], and finally truncate the sentences to make attention masks.

[0091] (2) Use BERT to extract text vector representation containing contextual information.

[0092] 1) Position Encoding: Provide the Transformer with the position information of each word in the text to identify the dependencies and temporal features of words in the review text.

[0093] 2) Self-attention mechanism. By calculating the relationship between each word in a sentence and all the words in the sentence, the weight of each word is adjusted to obtain more semantic expressions and solve the polysemy problem. The formula of the self-attention mechanism is as follows

[0094]

[0095] Where Linear(·) represents a linear mapping; E is the annotation text vector; W Q , WK W V is the weight.

[0096]

[0097] Among them, SelfAttention(·) represents self-attention; The attention matrix is ​​transformed into a standard normal distribution; the Softmax(·) is normalized so that the sum of the attention weight of each word in the text and the attention weights of all other words is 1.

[0098] 3) Layer normalization and residual connections. The hidden layers in the layer normalization network are standard normally distributed, which speeds up the training and convergence of the model; residual connections can solve the problems of gradient disappearance and network degradation.

[0099] (3) TextCNN convolution layer. Convolution operation is used to extract high-order local text features of the input word vector matrix. Convolution operation uses convolution kernel to generate feature vector. f∈R l,n h t Where l is the height of the convolution kernel, and n is the width of the convolution kernel, which is the length of the word vector. This paper uses three convolution kernels with convolution window heights of l = (2, 3, 4) to perform convolution and extract text feature vectors of different granularities. The formula is as follows:

[0100] h t =ReLU(Conv(W,f)+b)

[0101] Among them, ReLU is the activation function, Conv is the convolution operation, W is the word vector matrix of this sentence, and b is the bias term.

[0102] The result of the convolution layer output is a feature vector set H generated by multiple convolution kernels:

[0103] H=(h1,h2,...,h n )

[0104] Where n is the number of extracted feature vectors, which is equal to the product of the number of convolution kernels and the convolution kernel type.

[0105] (4) TextCNN pooling layer. Pooling can reduce the number of model parameters, alleviate the overfitting problem, and organize variable-length inputs into fixed-length inputs. Time series maximum pooling only retains the maximum value, which easily leads to the loss of feature position information; and strong features that appear multiple times can only be seen once, so the intensity information is lost. Therefore, this paper uses K-MaxPooling instead of MaxPooling. The core idea is that K-Max Pooling is an operation method for taking the top-K features globally. It maintains the relative order of feature values ​​and can express the situation where the same type of features appear multiple times to highlight the importance of a certain type of feature. The calculation formula is:

[0106] h' max =k-max(h t )

[0107] Among them, the selection of k value can also be obtained dynamically:

[0108]

[0109] Here, l is the sentence length, L is the total number of convolutional layers, and c is the current number of convolutional layers. Therefore, it can be seen that k changes with the length of the sentence and the depth of the network.

[0110] (5) Finally, these aggregated output values ​​are linked into a vector to obtain a richer text feature representation.

[0111]

[0112] In a specific implementation, the acquired image data may be subjected to a SIFT (Scale-invariant feature transform) operator, a HOG (Histogram of Oriented Gradient) feature extraction algorithm, or an LBP (Local Binary Pattern) algorithm, etc. The method described in this embodiment does not limit the method for extracting image features.

[0113] Step 2: Gating mechanism filters noise

[0114] Not all the interaction information obtained above is useful for the sentiment analysis task. Therefore, the solution described in this embodiment proposes a gating mechanism to filter the noise of the generated interaction information and retain the part most relevant to the sentiment.

[0115] Among them, C VT represents the text features after image filtering, C VT Represents the image features after text filtering, which are the fusion of unimodal contextual information and interaction information between modalities.

[0116] CVT =Gate(I TV ,H V ) (5)

[0117] C VT =Gate(I VT ,H T ) (6)

[0118] Among them, we define a kernel function (i.e., gating) Gate() to fuse the modal interaction information and context representation obtained in the previous step.

[0119] We take the interactive information between text and image and the independent representation of image as examples to illustrate the calculation process of the gating mechanism:

[0120]

[0121]

[0122] C TV =G(I TV ,H V )·N(I TV ,H V )+(1-G(I TV ,H V ))·H V (9)

[0123] Among them, N() represents nonlinear operation, G() represents gating, which adjusts the cross-interaction information to the next layer, and W F , b F ,b G is a trainable parameter, ° represents element-wise multiplication, and σ is an activation function. In this embodiment, the sigmoid function is used.

[0124] Step 3: Obtain complete fusion modal features

[0125] The internal modal features (i.e. H T and H V ) and the corresponding interaction features learned (i.e., C VT and C TV ) are connected to obtain the complete representation of each modal feature, and the four fusion feature representations are combined to form a complete feature representation SF=[H T ,H V ,C VT ,C TV ], which is input into the downstream multi-task learning model.

[0126] The downstream multi-task learning model can be a later classification task, specifically:

[0127] The fusion feature is expressed as SF = [H T ,H V ,C VT ,C TV ], by using a feed-forward neural network to learn the attention weights for each feature representation:

[0128]

[0129] in, are the weight parameters of the two fully connected layers, and is the corresponding bias. The attention weight corresponding to each feature representation constitutes the task-specific attention vector attn=[attn T ,attn V ,attn VT ,attn TV ], multiplied by the shared feature representation matrix to obtain the task-specific fusion representation FF task =SF×attn∈R d×d .

[0130] Subsequent predictions are based on the specific task objectives, setting the corresponding number of output layer neurons and activation functions. For all tasks in this paper, the number of output layer neurons is set to 1. The main task and intensity regression task do not use an activation function, while the polarity binary classification task uses a sigmoid activation function.

[0131] The regression task uses the Mean Absolute Error (MAE) as the loss function, while the classification task uses the Cross Entropy as the loss function. The training of the entire model is based on the combined loss of three tasks:

[0132]

[0133] Among them, α, β, and γ are hyperparameters used to adjust the training level of the three tasks. Larger parameter values ​​can guide the model to prioritize learning convergence on the task.

[0134] Example 2:

[0135] The purpose of this embodiment is to provide a multimodal feature fusion system for sentiment analysis.

[0136] A multimodal feature fusion system for sentiment analysis, comprising:

[0137] A data acquisition unit, which is used to acquire text data and corresponding image data in social data;

[0138] A feature extraction unit, configured to extract features from the text data and the image data to obtain text features and image features;

[0139] An interactive information acquisition and filtering unit, which is used to acquire interactive information between the text features and the image features using a cross-modal attention mechanism, and to perform noise filtering on the interactive information using a gating mechanism to obtain text features after image filtering and image features after text filtering;

[0140] A feature fusion unit is used to combine the obtained text features, image features, text features after image filtering, and image features after text filtering to obtain fused features;

[0141] The sentiment analysis unit is used to obtain sentiment analysis results based on the fusion features through a pre-built sentiment analysis model.

[0142] Furthermore, the system described in this embodiment corresponds to the method described in Example 1, and its technical details are described in detail in Example 1, so they are not repeated here.

[0143] In further embodiments, there is also provided:

[0144] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed by the processor. When the computer instructions are executed by the processor, the method described in Example 1 is performed. For the sake of brevity, no further details are given here.

[0145] It should be understood that in this embodiment, the processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), off-the-shelf field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0146] The memory may include a read-only memory and a random access memory, and provides instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0147] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the method described in embodiment 1 is performed.

[0148] The method in Example 1 can be directly implemented as being executed by a hardware processor, or by a combination of hardware and software modules within the processor. The software module can be located in a storage medium well-established in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not given here.

[0149] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in this embodiment can be implemented using electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0150] The multimodal feature fusion method and system for sentiment analysis provided in the above embodiment can be implemented and have broad application prospects.

[0151] The foregoing description is merely a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Those skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present disclosure shall be included within the scope of protection of the present disclosure.

Claims

1. A multimodal feature fusion method for sentiment analysis, characterized in that: include: Obtain text data and its corresponding image data in social data; Performing feature extraction on the text data and image data respectively to obtain text features and image features; Based on the text features and image features, a cross-modal attention mechanism is used to obtain the interaction information between the two, and the interaction information is noise-filtered through a gating mechanism to obtain the text features after image filtering and the image features after text filtering; The obtained text features, image features, text features after image filtering, and image features after text filtering are spliced ​​to obtain fusion features; Based on the fusion features, obtaining sentiment analysis results through a pre-built sentiment analysis model; The noise filtering of the interactive information by the gating mechanism includes noise filtering of text-to-image interactive information and noise filtering of image-to-text interactive information; The noise filtering of the interactive information from text to image is specifically expressed as follows: in, G ( ) indicates a gate operation, N ( ) represents a nonlinear operation, I TV is the interactive information from text to image, H V is the image feature.

2. The multimodal feature fusion method for sentiment analysis according to claim 1, wherein: The cross-modal attention mechanism is used to obtain the interactive information between the two, specifically: Based on the text features and image features, calculating the attention matrix of the two; Quantize the attention matrix using the softmax function; Calculate the product of the quantized result of the attention matrix and the image features to obtain the interactive information from text to image; Calculate the product of the quantized result of the attention matrix and the text feature to obtain the image-to-text interaction information.

3. The multimodal feature fusion method for sentiment analysis according to claim 1, wherein: The noise filtering of the image-to-text interaction information is specifically expressed as follows: in, I VT is the image-to-text interaction information, H T is a text feature.

4. A multimodal feature fusion system for sentiment analysis, which adopts a multimodal feature fusion method for sentiment analysis according to any one of claims 1 to 3, characterized in that: include: A data acquisition unit, which is used to acquire text data and corresponding image data in social data; A feature extraction unit, configured to extract features from the text data and the image data to obtain text features and image features; An interactive information acquisition and filtering unit, which is used to acquire interactive information between the text features and the image features using a cross-modal attention mechanism, and to perform noise filtering on the interactive information using a gating mechanism to obtain text features after image filtering and image features after text filtering; A feature fusion unit is used to combine the obtained text features, image features, text features after image filtering, and image features after text filtering to obtain fused features; The sentiment analysis unit is used to obtain sentiment analysis results based on the fusion features through a pre-built sentiment analysis model.

5. The multimodal feature fusion system for sentiment analysis according to claim 4, wherein: The cross-modal attention mechanism is used to obtain the interactive information between the two, specifically: Based on the text features and image features, calculating the attention matrix of the two; Quantize the attention matrix using the softmax function; Calculate the product of the quantized result of the attention matrix and the image features to obtain the interactive information from text to image; Calculate the product of the quantized result of the attention matrix and the text feature to obtain the image-to-text interaction information.

6. A multimodal feature fusion system for sentiment analysis according to claim 4, characterized in that: The noise filtering of the interactive information through the gating mechanism includes noise filtering of text-to-image interactive information and noise filtering of image-to-text interactive information.

7. An electronic device comprising a memory, a processor, and a computer program stored and running on the memory, characterized in that: When the processor executes the program, it implements a multimodal feature fusion method for sentiment analysis as described in any one of claims 1-3.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a multimodal feature fusion method for sentiment analysis as described in any one of claims 1 to 3 is implemented.

Images