A multi-modal fusion sentiment label determination method and device

Abstract

The application discloses a kind of multi-modal fusion emotional label determination method and device, it is related to multi-modal sentiment analysis technical field, to original multi-modal sequence is carried out feature extraction, obtains multi-modal vector;The multi-modal vector includes text modal vector, visual modal vector and acoustic modal vector;Residual decoupling processing is carried out to the multi-modal vector respectively, obtains enhanced text vector, text explainable vector and difference vector;Difference fusion is carried out to the enhanced text vector, the text explainable vector and the difference vector, obtains target fusion vector;Prediction mapping is carried out to the target fusion vector, obtains target emotional label.Adopting the technical scheme, by residual decoupling to multi-modal vector, non-text vector is decomposed into explainable vector and difference vector, and consensus difference fusion is carried out, strengthens text anchoring effect, realizes the fusion of heterogeneous characteristics, solves the problem that current non-text modal and text modal cannot be balanced.

Description

Technical Field

[0001] This invention relates to the field of multimodal sentiment analysis technology, and in particular to a method and apparatus for determining sentiment labels through multimodal fusion. Background Technology

[0002] With the continuous maturation of deep learning technology, multimodal sentiment analysis, with its more comprehensive sentiment modeling capabilities, has surpassed single-modal methods in applications such as market forecasting and public opinion analysis. Early research on multimodal sentiment analysis mainly focused on explicit cross-modal fusion.

[0003] As the field has developed, existing multimodal sentiment analysis methods have either decomposed features into modality-invariant and modality-specific spaces, or designed specialized self-supervised modules to generate single-modality labels, and further adopted language-centric decoupling strategies to alleviate cross-modal conflicts, or utilized multi-task learning frameworks to capture significant sentiment cues in different modalities.

[0004] However, existing methods often struggle to achieve an effective balance between the high heterogeneity of non-textual signals and the dominant role of textual modalities. Summary of the Invention

[0005] This invention provides a method and apparatus for determining sentiment tags through multimodal fusion, which solves the problem that the current non-textual and textual modalities cannot achieve a balance. By performing residual decoupling on the multimodal vectors, the non-textual vectors are decomposed into interpretable vectors and difference vectors, and consensus difference fusion is performed to strengthen the text anchoring effect and achieve the fusion of heterogeneous features.

[0006] According to one aspect of the present invention, a method for determining sentiment labels through multimodal fusion is provided, comprising: Feature extraction is performed on the original multimodal sequence to obtain multimodal vectors; the multimodal vectors include text modality vectors, visual modality vectors, and acoustic modality vectors; The multimodal vectors are subjected to residual decoupling processing to obtain enhanced text vectors, text interpretable vectors, and difference vectors; the text interpretable vectors include text interpretable visual vectors and text interpretable acoustic vectors; the difference vectors include difference visual vectors and difference acoustic vectors. The enhanced text vector, the text interpretable vector, and the difference vector are fused to obtain the target fused vector. The target fusion vector is predicted and mapped to obtain the target sentiment label.

[0007] According to another aspect of the present invention, a multimodal fusion-based emotion tag determination apparatus is provided, comprising: The feature extraction module is used to extract features from the original multimodal sequence to obtain a multimodal vector; the multimodal vector includes text modality vector, visual modality vector and acoustic modality vector; The residual decoupling module is used to perform residual decoupling processing on the multimodal vectors respectively to obtain enhanced text vectors, text interpretable vectors, and difference vectors; the text interpretable vectors include text interpretable visual vectors and text interpretable acoustic vectors; the difference vectors include difference visual vectors and difference acoustic vectors. The difference fusion module is used to perform difference fusion on the enhanced text vector, the text interpretable vector and the difference vector to obtain the target fusion vector; The prediction mapping module is used to predict and map the target fusion vector to obtain the target sentiment tag.

[0008] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory that is communicatively connected to at least one processor; wherein, The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the multimodal fusion sentiment label determination method according to any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the multimodal fusion sentiment tag determination method of any embodiment of the present invention.

[0010] According to another aspect of the present invention, a computer program product is provided, the computer program product including a computer program that, when executed by a processor, implements a multimodal fusion sentiment tag determination method according to any embodiment of the present invention.

[0011] The technical solution of this invention decomposes non-textual vectors into interpretable vectors and difference vectors by performing residual decoupling on multimodal vectors, and performs consensus difference fusion to solve the problem that the current non-textual modality and textual modality cannot achieve a balance, strengthen the text anchoring role, and realize the fusion of heterogeneous features.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of a multimodal fusion-based sentiment tag determination method provided by an embodiment of the present invention; Figure 2 This is a flowchart of a multimodal fusion-based sentiment tag determination method provided by an embodiment of the present invention; Figure 3 This is a flowchart of a multimodal fusion-based sentiment tag determination method provided by an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a multimodal fusion emotion tag determination device according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device that implements a multimodal fusion-based emotion tag determination method according to an embodiment of the present invention. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0016] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product or device.

[0017] Furthermore, it should be noted that the information collected in the technical solution of this invention is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data all comply with the relevant laws, regulations and standards of relevant countries and regions, necessary confidentiality measures have been taken, and public order and good morals are not violated. Corresponding operation entry points are provided for users to choose to authorize or refuse.

[0018] Figure 1 This invention provides a flowchart of a multimodal fusion-based sentiment label determination method, applicable to multimodal sentiment analysis. The method can be executed by a multimodal fusion-based sentiment label determination device, which can be implemented in hardware and / or software and configured in a server. Figure 1 As shown, the method includes: S110. Extract features from the original multimodal sequence to obtain multimodal vectors; the multimodal vectors include text modal vectors, visual modal vectors, and acoustic modal vectors.

[0019] The original multimodal sequence is the original sequence of multimodal data, including the original text modality sequence, the original visual modality sequence, and the original acoustic modality sequence. The multimodal vector is obtained by preprocessing the original multimodal sequences. The text modality vector is a language identifier with emotion perception capability; the visual modality vector is a facial expression feature; and the acoustic modality vector is a frame-level prosodic feature.

[0020] Specifically, to obtain the original multimodal sequence, one can use... Let t represent the original sequence of the text modality; This is the original sequence of acoustic modes; The original visual modality sequence is used as the basis for preprocessing the original text modality sequence within the original multimodal sequence. Feature preprocessing is performed on both the English and Chinese datasets to extract language representations with emotion perception capabilities, which are then used as text modality vectors. The original visual modality sequences in the original multimodal sequences are preprocessed to extract frame-level facial expression features. These features are then quantized using a Gaussian mixture model codebook, thus mapping continuous visual descriptors to a structured representation space as visual modality vectors. The acoustic modal sequences in the original multimodal sequences are preprocessed to extract frame-level prosodic features. These features are then further quantized using a Gaussian mixture model codebook, thus mapping continuous visual descriptors to a structured representation space as acoustic modal vectors. .

[0021] S120. Perform residual decoupling processing on the multimodal vectors to obtain enhanced text vectors, text interpretable vectors, and difference vectors; the text interpretable vectors include text interpretable visual vectors and text interpretable acoustic vectors; the difference vectors include difference visual vectors and difference acoustic vectors.

[0022] The residual decoupling process involves text augmentation of text modal vectors, decomposing non-text modal vectors, such as visual and acoustic modal vectors, into two parts: one part is information that can be anchored and interpreted by the text modality, and the other part is residual information representing modality-specific information. The augmented text vector is the text modality data after text augmentation; the text interpretable vector is non-textual information that can be interpreted by the text modal vector, which may include text interpretable visual vectors and text interpretable acoustic vectors; the difference vector is used to represent residual information specific to the non-text modal vectors, which may include difference visual vectors and difference acoustic vectors.

[0023] Specifically, text augmentation is performed on the text modality vector in the multimodal vector to obtain augmented text vectors; non-text projection is performed on the visual modality vector and acoustic modality vector in the multimodal vector to obtain transformed non-text modality vectors; and the augmented text vector and the projected non-text modality vector are interacted to obtain the text interpretable part and the text non-interpretable part. The text interpretable part is used as the text interpretable vector, including the text interpretable visual vector and the text interpretable acoustic vector; the text non-interpretable part is used as the difference vector, including the difference visual vector and the difference acoustic vector.

[0024] S130. Perform difference fusion on the enhanced text vector, the text interpretable vector, and the difference vector to obtain the target fused vector.

[0025] Among them, difference fusion is to use the enhanced text vector to calibrate and balance the heterogeneous signals retained in the non-text modality, perform consensus processing on the interpretable part of the text to obtain the cross-modal consistent sentiment representation vector, and perform difference quantization on the uninterpretable part of the text to obtain the non-text residual representation vector; the target fusion vector is a vector that fuses the cross-modal consistent sentiment representation vector with consensus and the non-text residual representation vector of the difference part.

[0026] Specifically, consensus processing is performed on the enhanced text vector and the text interpretable vector to obtain the cross-modal consistent sentiment representation vector. The enhanced text vector is then differentially quantized with the differential visual vector and differential acoustic vector in the differential vector to obtain the non-text residual representation vector, which includes the semantic difference between the text and the differential vector. A linear transformation is performed on the cross-modal consistent sentiment representation vector and the non-text residual representation vector to obtain the consistency-difference gating signal. The cross-modal consistent sentiment representation vector and the non-text residual representation vector are then fused again based on the consistency-difference gating signal to obtain the target fused vector.

[0027] S140. Predict and map the target fusion vector to obtain the target sentiment label.

[0028] The target sentiment label is the sentiment label obtained from the sentiment analysis of the original multimodal sequences.

[0029] Specifically, the target fusion vector can be activated by the activation function of the first linear layer in the multilayer perceptron to obtain the activation vector, and the activation vector can be regularized by the second linear layer in the multilayer perceptron to obtain the regularized vector. Finally, the target sentiment label can be obtained by regularizing the target fusion vector and the regularized vector by the third linear layer in the multilayer perceptron.

[0030] This invention extracts features from the original multimodal sequence to obtain multimodal vectors. These multimodal vectors include textual, visual, and acoustic modal vectors. Residual decoupling is performed on each multimodal vector to obtain enhanced textual vectors, textually interpretable vectors, and differential vectors. The textually interpretable vectors include textually interpretable visual vectors and textually interpretable acoustic vectors; the differential vectors include differentially interpretable visual vectors and differentially acoustic vectors. Differential fusion is performed on the enhanced textual vectors, textually interpretable vectors, and differential vectors to obtain a target fusion vector. Predictive mapping is then performed on the target fusion vector to obtain the target sentiment label. This technical solution, by decomposing non-textual vectors into interpretable and differential vectors through residual decoupling of the multimodal vectors and performing consensus differential fusion, addresses the current problem of the inability to achieve a balance between non-textual and textual modalities, strengthens the text anchoring effect, and achieves the fusion of heterogeneous features.

[0031] Figure 2 This is a flowchart of a multimodal fusion-based sentiment tag determination method according to an embodiment of the present invention. This embodiment supplements the above embodiment by performing residual decoupling processing on the multimodal vectors. It should be noted that for parts not detailed in this embodiment, please refer to the relevant descriptions in other embodiments. Figure 2 As shown, the method includes: S210. Extract features from the original multimodal sequence to obtain multimodal vectors; the multimodal vectors include text modal vectors, visual modal vectors, and acoustic modal vectors.

[0032] S220. Perform text enhancement on the text modality vector to obtain the enhanced text vector.

[0033] Specifically, the text modality vector is sequence extracted to obtain the row vector; and the full vector of the text modality vector, the binary attention mask and the row vector are divided element-wise to obtain the fused vector. The row vector and the fused vector are normalized to obtain the enhanced text vector.

[0034] Optionally, text augmentation can be performed on the text modality vector to obtain augmented text vectors, including: Extract the text modality vector to obtain the target line vector; The fusion vector is obtained from the full vector of the text modality vector and the binarized attention mask; The binarized attention mask and text modality vector of the text modality vector are divided element-wise with the fused vector to obtain the element vector; The element vector and the target row vector are normalized to obtain the enhanced text vector.

[0035] Wherein, the target row vector is the vector extracted from the first row sequence of the text modality vector; the full vector is the number of valid tokens in the text modality vector, used to represent the global semantic representation vector; the binarized attention mask is the vector corresponding to the text modality vector; and the fusion vector is the sequence-level average representation vector calculated based on the attention mask.

[0036] Specifically, the first row sequence of the text modality vector is extracted to obtain the target row vector, and its calculation formula is as follows: in, For the target row vector, This represents the operation of extracting the first line of the text modality vector.

[0037] The fusion vector is obtained by multiplying the full vector of the text modality vector and the binarized attention mask, and the calculation formula is as follows: in, A is the fusion vector; A is the full vector that counts the number of valid tokens. This is a binary attention mask for text modality vectors.

[0038] The binarized attention mask and text modality vector of the text modality vector are divided element-wise with the fused vector to obtain the element vector, and the calculation formula is as follows: in, It is an element vector.

[0039] The element vector and the target row vector are normalized to obtain the enhanced text vector; the calculation formula is shown below: in, To enhance text vectors.

[0040] Understandably, by fusing the global semantic representation of the target row vector and the full vector with the sequence-level average representation calculated based on attention masks, a more stable and representative text representation can be constructed, thereby enhancing the guiding role of the text modality in subsequent cross-modal interactions.

[0041] S230. Treat the visual modal vector and the acoustic modal vector as non-textual modal vectors.

[0042] Specifically, visual modal vectors and acoustic modal vectors are treated as non-textual modal vectors. Non-textual modal vectors can be represented as... .

[0043] S240. Perform linear projection on the non-text modal vector to obtain the non-text projection vector; the non-text projection vector includes the visual projection vector and the acoustic projection vector.

[0044] Among them, linear projection is a projection transformation based on the learnable parameters of the linear layer and the corresponding bias terms; the visual projection vector in the non-text projection vector is obtained by performing a linear projection transformation on the visual modal vector in the non-text modal vector, and the acoustic projection vector in the non-text projection vector is obtained by performing a linear projection transformation on the acoustic modal vector in the non-text modal vector.

[0045] Specifically, based on the learnable parameters of the linear layer and the corresponding bias term Linear projections are performed on the visual modal vector and acoustic modal vector in the non-text modal vector respectively to obtain the visual projection vector and acoustic projection vector, and their calculation formulas are as follows: in, This is a non-text projection vector. The visual projection vector is represented as The acoustic projection vector is represented as .

[0046] S250. By scaling the feature dimension, the non-text projection vector and the text modality vector are decoupled using an activation function to obtain the text interpretable vector and the difference vector.

[0047] The activation function can be the Softmax activation function; the text interpretable vector is the text interpretable part of the non-text projection vector, including the text interpretable visual vector and the text interpretable acoustic vector; the difference vector can be the text non-interpretable part of the non-text projection, including the visual difference vector and the acoustic difference vector.

[0048] Specifically, under the scaling feature dimension, the non-text projection vector is interacted with the text modality vector through the scaling dot product attention by the activation function to obtain the text interpretable part. The text uninterpretable part is determined based on the non-text projection vector and the text interpretable part. Pooling operation and linear projection are performed on the text interpretable part and the text uninterpretable part respectively to obtain the corresponding text interpretable vector and difference vector.

[0049] Optionally, by scaling the feature dimensions, residual decoupling is performed on the non-text projection vector and the text modality vector using an activation function to obtain the text interpretable vector and the difference vector, including: By scaling the feature dimension, the non-text projection vector and the text modality vector are interacted through an activation function to obtain the text vector and the non-text difference vector; Perform average pooling operations on the text vector and the non-text difference vector respectively to obtain the text average pooling vector and the non-text difference average pooling vector; Perform max pooling on the text vector and the non-text difference vector respectively to obtain the text max pooling vector and the non-text difference max pooling vector; The average pooling vector and the max pooling vector of the text are concatenated to obtain the text-interpretable concatenated vector; the text-interpretable concatenated vector includes the text-interpretable acoustic concatenated vector and the text-interpretable visual concatenated vector. The non-textual difference average pooling vector and the non-textual difference max pooling vector are concatenated to obtain the non-textual difference concatenated vector; the non-textual difference concatenated vector includes the non-textual difference acoustic concatenated vector and the non-textual difference visual concatenated vector. Linear projections are performed on the text-interpretable concatenated vector and the non-text-difference concatenated vector to obtain the text-interpretable vector and the difference vector, respectively.

[0050] The text vectors are the interpretable parts of the text, including text visual vectors and text acoustic vectors; the non-text difference vectors are the non-interpretable parts of the text, including non-text difference visual vectors and non-text difference acoustic vectors; the text average pooling vectors include text average pooling visual vectors and text average pooling acoustic vectors; the non-text difference average pooling vectors include non-text difference average pooling visual vectors and non-text difference average pooling acoustic vectors; the text max pooling vectors include text max pooling visual vectors and text max pooling acoustic vectors; and the non-text difference max pooling vectors include non-text difference max pooling visual vectors and non-text difference max pooling acoustic vectors.

[0051] Specifically, by scaling the feature dimension, the non-text projection vector is interacted with the text modality vector through a scaled dot product attention function, using an activation function, to obtain the text vector corresponding to the interpretable part of the text and the non-text difference vector corresponding to the uninterpretable part of the text. The calculation formula is as follows: in, It is a text vector; This is a non-textual difference vector; To scale the feature dimensions.

[0052] The average pooling operation is performed on the text vector and the non-text difference vector separately to obtain the text average pooling vector and the non-text difference average pooling vector. The calculation formula is as follows: in, This is the average pooling vector of the text; This is a non-textual difference average pooling vector.

[0053] Max pooling is performed on the text vector and the non-text difference vector separately to obtain the text max pooling vector and the non-text difference max pooling vector. The calculation formula is as follows: in, This is the text max-pooled vector; This is the non-textual difference max pooling vector.

[0054] The text average pooling vector and the text max pooling vector are concatenated to obtain the text-interpretable concatenated vector. This text-interpretable concatenated vector includes both text-interpretable acoustic and text-interpretable visual concatenated vectors. The non-text difference average pooling vector and the non-text difference max pooling vector are concatenated to obtain the non-text difference concatenated vector. This non-text difference concatenated vector includes both non-text difference acoustic and non-text difference visual concatenated vectors. Linear projections are then performed on both the text-interpretable concatenated vector and the non-text difference concatenated vector to obtain the text-interpretable vector and the difference vector, respectively. The calculation formulas are shown below. in, For text-interpretable vectors For linear layer learnable matrices, A text-interpretable concatenated vector. For linear layer bias vectors, For the difference vector, For linear layer learnable matrices, This is a non-textual difference concatenation vector. This is the linear layer bias vector.

[0055] Understandably, by performing residual decoupling on non-text frame-level information based on text modality vectors, two different components are obtained: text interpretable features and independent residual features. The text interpretable features and independent residual features are then subjected to average pooling and max pooling operations, respectively, to simultaneously preserve the overall distribution information and local peak responses. The processed features are then concatenated and further enhanced in representation and stability through linear projection.

[0056] S260. Perform difference fusion on the enhanced text vector, the text interpretable vector, and the difference vector to obtain the target fused vector.

[0057] S270. Predict and map the target fusion vector to obtain the target sentiment label.

[0058] This invention, through residual decoupling of non-text projection vectors and text modality vectors using an activation function at the scaled feature dimension, yields text interpretable vectors and difference vectors. It decomposes frame-level non-text data into two distinct components: one part is information anchorable and interpretable by the text modality, and the other part is residual information representing modality-specific information. This fully utilizes textual cues to anchor and decompose heterogeneous non-textual information, ensuring an effective balance between the high heterogeneity of non-textual signals and the dominant role of text modality.

[0059] Figure 3 This is a flowchart of a multimodal fusion-based sentiment tag determination method according to an embodiment of the present invention. Based on the above embodiments, this embodiment supplements the differential fusion method of the target fusion vector. It should be noted that for parts not detailed in this embodiment, please refer to the relevant descriptions in other embodiments. Figure 3 As shown, the method includes: S310. Perform feature extraction on the original multimodal sequence to obtain multimodal vectors; the multimodal vectors include text modal vectors, visual modal vectors, and acoustic modal vectors.

[0060] S320. Perform residual decoupling processing on the multimodal vectors to obtain enhanced text vectors, text interpretable vectors, and difference vectors; the text interpretable vectors include text interpretable visual vectors and text interpretable acoustic vectors; the difference vectors include difference visual vectors and difference acoustic vectors.

[0061] S330. Consensus processing is performed on the enhanced text vector and the text interpretable vector, as well as the text interpretable vector and the difference vector, respectively, to obtain the target cross-modal vector and the overall difference vector.

[0062] The consensus processing involves integrating the enhanced text vector and the text interpretable vector, as well as integrating the text interpretable vector and the difference vector; the target cross-modal vector is the overall cross-modal consistent sentiment representation vector; and the overall difference vector is the overall non-textual difference representation vector that cannot be interpreted by text.

[0063] Specifically, consensus processing is performed on the enhanced text vector and the text interpretable vector to obtain the overall cross-modal consistent sentiment representation as the target cross-modal vector; consensus processing is performed on the text interpretable vector and the difference vector to obtain the overall non-text difference representation vector as the overall difference vector.

[0064] Optionally, consensus processing is performed on the enhanced text vector and the text interpretable vector, as well as the text interpretable vector and the difference vector, respectively, to obtain the target cross-modal vector and the overall difference vector, including: The text interpretable vector and the enhanced text vector are concatenated according to the feature dimension to obtain the initial cross-modal vector; Perform a linear operation on the initial cross-modal vector to obtain the target cross-modal vector; Based on the feature dimension, the difference visual vector and the difference acoustic vector in the difference vector are concatenated to obtain the difference concatenation vector; Perform a linear operation on the difference concatenation vector to obtain the overall difference vector.

[0065] The initial cross-modal vector is a text-interpretable vector obtained by concatenating and enhancing text vectors; the linear operation can be performed by a two-layer MLP with learnable parameters; and the difference concatenation vector is a vector that represents the overall non-textual residual features.

[0066] Specifically, the interpretable text vector and the enhanced text vector are concatenated according to their vector feature dimensions to obtain the initial cross-modal vector. The initial cross-modal vector is then linearly operated on using the activation function of the first linear layer in a two-layer MLP with learnable parameters to obtain the intermediate cross-modal vector. Finally, the intermediate cross-modal vector is linearly operated on using the second linear layer to obtain the target cross-modal vector. The calculation formula is shown below: in, This is the initial cross-modal vector; For text-interpretable acoustic vectors; For text-interpretable visual vectors; This is the learnable weight matrix for the first linear layer; This refers to the bias term of the first linear layer; This is the intermediate cross-modal vector; This is the learnable weight matrix for the second linear layer; This is the bias term for the second linear layer; The target cross-modal vector.

[0067] The difference visual vector and difference acoustic vector in the difference vector are concatenated along the vector feature dimension to obtain the fused residual representation vector as the difference concatenation vector. The fused residual representation is then passed through the activation function of the first linear layer of a two-layer multilayer perceptron with learnable parameters to perform a linear operation on the difference concatenation vector to obtain the intermediate difference vector. The intermediate difference vector is then passed through the second linear layer to perform a linear operation on it to obtain the overall difference vector. The calculation formula is shown below: in, For concatenating difference vectors; For differential acoustic vectors; For visual differences; This is the intermediate difference vector; This is the learnable weight matrix for the first linear layer; This refers to the bias term of the first linear layer; This is the overall difference vector; This is the learnable weight matrix for the second linear layer; This is the bias term for the second linear layer.

[0068] Understandably, by concatenating enhanced text vectors and text interpretable vectors along the feature dimension and performing linear projection, an overall cross-modal consistent sentiment representation vector is obtained. Furthermore, by concatenating differential visual vectors and differential acoustic vectors along the feature dimension and performing linear projection, an overall non-textual differential representation vector is obtained. This strengthens the text anchoring effect, maximizes the influence of the text modality, and coordinates the inherent heterogeneous features in the non-text modality, thereby improving the accuracy of subsequent sentiment label determination.

[0069] S340. Perform cosine calculations on the enhanced text vector and the difference acoustic vector and difference visual vector in the difference vector respectively to obtain the semantic difference visual vector and semantic difference acoustic vector.

[0070] Among them, the semantic difference visual vector represents the semantic difference between visual and textual data; the semantic difference acoustic vector represents the semantic difference between acoustic and textual data.

[0071] Specifically, the semantic difference visual vector is obtained by performing cosine calculation on the difference acoustic vectors in the enhanced text vector and the difference vector; the semantic difference acoustic vector is obtained by performing cosine calculation on the difference acoustic vectors in the enhanced text vector and the difference vector, and the calculation formula is as follows: in, For semantic difference visual vectors, For semantic difference acoustic vectors.

[0072] S350. The target cross-modal vector, the overall difference vector, the semantic difference visual vector, and the semantic difference acoustic vector are concatenated to obtain the cross-modal difference vector.

[0073] Specifically, the target cross-modal vector, overall difference vector, semantic difference visual vector, and semantic difference acoustic vector can be concatenated along the feature dimension to obtain the cross-modal difference vector, the calculation formula of which is shown below: in, This is the cross-modal difference vector.

[0074] S360. The cross-modal difference vector is linearly transformed by an activation function to obtain the difference-gated signal.

[0075] The activation function can include the GELU activation function and the Sigmoid activation function.

[0076] Specifically, the cross-modal difference vector is subjected to a first linear transformation using the GELU activation function to obtain a cross-modal linear vector. Then, a second linear transformation is performed on this cross-modal linear vector using the Sigmoid activation function to obtain the difference-gated signal. The calculation formula is shown below: in, It is a cross-modal linear vector; This is the learnable weight matrix for the first linear layer; This refers to the bias term of the first linear layer; For differential gating signals; This is the learnable weight matrix for the second linear layer; This is the bias term for the second linear layer.

[0077] S370. The target cross-modal vector, the overall difference vector, and the difference-gated signal are fused to obtain the target fused vector.

[0078] Specifically, the target cross-modal vector, the overall difference vector, and the difference-gated signal are fused to obtain the target fused vector, and the calculation formula is as follows: Where Z is the target fusion vector.

[0079] S380. Predict and map the target fusion vector to obtain the target sentiment label.

[0080] This invention employs a fusion mechanism to construct a difference intensity display for differences in non-text-interpretable parts, and strengthens the text anchoring effect by fusing target cross-modal data and difference intensity, thereby achieving a balanced fusion of heterogeneous features and adaptively adjusting the contribution of difference branches to the final representation.

[0081] This invention provides a multimodal fusion framework for determining sentiment labels, including a multi-channel feature extraction module, a text-guided residual decoupling module, a consensus-difference balance fusion module, and a prediction output module.

[0082] The multi-channel feature extraction module is used to extract features from the original sequence to obtain the corresponding multimodal vector. Specifically, it extracts features from the original text sequence obtained through text input to obtain the text modality; it extracts features from the original acoustic sequence obtained through acoustic input to obtain the acoustic modality; and it extracts features from the original visual sequence obtained through visual input to obtain the visual modality.

[0083] The text-guided residual decoupling module enhances text representations and decomposes non-textual information into interpretable and non-interpretable parts. Specifically, it performs text enhancement on the text modality to obtain enhanced text vectors; it performs non-textual projection processing on the acoustic and visual modalities to obtain corresponding non-text vectors; and it performs residual decoupling processing on the non-textual projection vectors through the text modality to obtain interpretable text vectors and independent residual vectors.

[0084] The consensus-difference balancing fusion module is used to perform branch-balancing fusion of enhanced text vectors, text interpretable vectors, and independent residual vectors to obtain the target fused vector. Specifically: consensus is reached on the enhanced text vectors and text interpretable vectors through consensus branches to obtain the overall cross-modal sentiment vector; consensus is reached on the independent residual vectors to obtain the non-text difference vector; the difference intensity is determined by the enhanced text vector and the non-text difference vector, and a balancing gating is determined based on the difference intensity, the overall cross-modal sentiment vector, and the non-text difference vector; the target fused vector is obtained by fusing the balancing gating signal, the overall cross-modal sentiment vector, and the non-text difference vector.

[0085] The prediction output module maps the target fusion vector through a multilayer perceptron to obtain the final sentiment label.

[0086] In this embodiment of the invention, features are first extracted from the multimodal raw data to obtain high-level representations of each modality. The extracted features are then input into a text-guided residual decoupling module, which enhances the representation capability of the text modality and divides the non-text frame-level information into two different components: text-interpretable features and independent residual features. After decomposition, the output is further input into a consensus-difference balancing fusion module for fusion. Using text-guided information, the heterogeneous signals retained in the non-text modality are calibrated and balanced to maximize the influence of the text modality while coordinating the inherent heterogeneous features in the non-text modality.

[0087] This invention provides an experimental comparison case. The experimental environment is run on a single NVIDIA GeForce RTX5090 GPU with 32GB of video memory; the software environment is Python 3.10.12 and PyTorch 2.7.0; the Adam optimizer is used during the training phase, with Mean Squared Error (MSE) as the loss function; hyperparameters are determined through discrete grid search, with the following search ranges: batch size {32, 64, 128}, learning rate {1e-5, 2e-5, 3e-5, 4e-5, 5e-5}, weightdecay {0.0, 0.1}, dropout rate {0.1, 0.2, 0.4}, and warmup proportion {0.06, 0.1}; an early stopping mechanism is used during training, stopping training when the validation set performance does not improve for 8 consecutive epochs; datasets and evaluation metrics: the experiment uses three multimodal sentiment analysis datasets: CMU-MOSI, CMU-MOSEI, and CH-SIMS. All three datasets include textual, acoustic, and visual modalities and provide continuous sentiment annotation. CMU-MOSI and CMU-MOSEI are English datasets with sentiment scores ranging from -3 to 3; CH-SIMS is a Chinese dataset with sentiment scores ranging from -1 to 1. Evaluation metrics include mean absolute error (MAE), Pearson correlation coefficient (Corr), binary classification accuracy (Acc-2), and F1-score. A lower MAE indicates smaller errors, while higher Corr, Acc-2, and F1-score indicate better model performance. For CMU-MOSI and CMU-MOSEI, both negative / non-negative and negative / positive settings are reported for Acc-2 and F1-score.

[0088] The experimental results of this invention, on the CMU-MOSI dataset, show that the embodiment of this invention achieves an MAE of 0.580, a Corr of 0.869, an Acc-2 score of 87.76% / 90.24%, and an F1-score of 87.65% / 90.20%. Compared with the suboptimal method TDSF (Temporal Dynamic Fusion Strategy), the MAE is reduced by 0.092, and the Corr is improved by 0.053; under negative / non-negative settings, the Acc-2 score is improved by 1.30%, and the F1-score is improved by 1.13%. On the CMU-MOSEI dataset, the embodiments of the present invention achieved an MAE of 0.526, a Corr of 0.799, an Acc-2 score of 84.98% / 87.15%, and an F1-score of 85.21% / 87.07%. Compared with TDSF, the Corr score was improved by 0.018, the Acc-2 score was improved by 0.16% / 0.78%, and the F1-score was improved by 0.48% / 0.59%.

[0089] The ablation results of this experiment are to verify the contributions of the text-guided residual decoupling module and the consensus-difference balancing fusion module in the embodiments of this invention. Module ablation experiments were conducted on the CMU-MOSI dataset. The MAE, Corr, Acc-2, and F1-score of the complete model were 0.580, 0.869, 87.76% / 90.24%, and 87.65% / 90.20%, respectively. After removing the text-guided residual decoupling module, the MAE increased to 0.591, the Corr decreased to 0.860, and the Acc-2 and F1-scores decreased to 87.32% / 89.48% and 87.26% / 89.47%, respectively. After removing the consensus-difference balancing fusion module, the MAE increased to 0.592, the Corr decreased to 0.865, and the Acc-2 and F1-scores decreased to 87.46% / 89.78% and 87.39% / 89.65%, respectively. Simultaneously removing the text-guided residual decoupling module and the consensus-difference balancing fusion module resulted in an MAE increase to 0.672, a Corr decrease to 0.856, and Acc-2 and F1-scores decrease to 86.88% / 88.72% and 86.85% / 88.73%, respectively. These results indicate that the text-guided residual decoupling module enhances the decoupling capability of non-textual information under text guidance, while the consensus-difference balancing fusion module helps balance consistent and dissimilar information, both contributing to improved overall performance.

[0090] Figure 4This invention provides a schematic diagram of a multimodal fusion-based sentiment label determination device, applicable to multimodal sentiment analysis. The device can be implemented in hardware and / or software and can be configured on a server. Figure 4 As shown, the multimodal fusion sentiment label determination device 400 includes a feature extraction module 410, a residual decoupling module 420, a difference fusion module 430, and a prediction mapping module 440. The feature extraction module 410 is used to extract features from the original multimodal sequence to obtain a multimodal vector; the multimodal vector includes text modality vector, visual modality vector and acoustic modality vector; The residual decoupling module 420 is used to perform residual decoupling processing on the multimodal vectors to obtain enhanced text vectors, text interpretable vectors, and difference vectors; the text interpretable vectors include text interpretable visual vectors and text interpretable acoustic vectors; the difference vectors include difference visual vectors and difference acoustic vectors. The difference fusion module 430 is used to perform difference fusion on the enhanced text vector, the text interpretable vector and the difference vector to obtain the target fusion vector; The prediction mapping module 440 is used to predict and map the target fusion vector to obtain the target sentiment label.

[0091] This invention extracts features from the original multimodal sequence to obtain multimodal vectors. These vectors include textual, visual, and acoustic modal vectors. Residual decoupling is then performed on each multimodal vector to obtain enhanced text vectors, text-interpretable vectors, and difference vectors. The text-interpretable vectors include text-interpretable visual and text-interpretable acoustic vectors; the difference vectors include difference visual and difference acoustic vectors. Difference fusion is then performed on the enhanced text vectors, text-interpretable vectors, and difference vectors to obtain a target fusion vector. Predictive mapping is then performed on the target fusion vector to obtain the target sentiment label. This technical solution, by decomposing non-textual vectors into interpretable and difference vectors through residual decoupling of the multimodal vectors and performing difference consensus fusion, addresses the current problem of the inability to achieve a balance between non-textual and textual modalities, strengthens the text anchoring effect, and achieves the fusion of heterogeneous features.

[0092] Optionally, the residual decoupling module 420 is also used to perform text enhancement on the text modality vector to obtain an enhanced text vector; Visual modal vectors and acoustic modal vectors are used as non-textual modal vectors; Linear projection is performed on the non-text modal vector to obtain the non-text projection vector; the non-text projection vector includes the visual projection vector and the acoustic projection vector. By scaling the feature dimension, the non-text projection vector and the text modality vector are decoupled by residual processing through an activation function to obtain the text interpretable vector and the difference vector.

[0093] Optionally, the residual decoupling module 420 is also used to extract the text modality vector to obtain the target line vector; The fusion vector is obtained from the full vector of the text modality vector and the binarized attention mask; The binarized attention mask and text modality vector of the text modality vector are divided element-wise with the fused vector to obtain the element vector; The element vector and the target row vector are normalized to obtain the enhanced text vector.

[0094] Optionally, the residual decoupling module 420 is also used to interact the non-text projection vector with the text modality vector through an activation function under scaled feature dimensions to obtain the text vector and the non-text difference vector; Perform average pooling operations on the text vector and the non-text difference vector respectively to obtain the text average pooling vector and the non-text difference average pooling vector; Perform max pooling on the text vector and the non-text difference vector respectively to obtain the text max pooling vector and the non-text difference max pooling vector; The average pooling vector and the max pooling vector of the text are concatenated to obtain the text-interpretable concatenated vector; the text-interpretable concatenated vector includes the text-interpretable acoustic concatenated vector and the text-interpretable visual concatenated vector. The non-textual difference average pooling vector and the non-textual difference max pooling vector are concatenated to obtain the non-textual difference concatenated vector; the non-textual difference concatenated vector includes the non-textual difference acoustic concatenated vector and the non-textual difference visual concatenated vector. Linear projections are performed on the text-interpretable concatenated vector and the non-text-difference concatenated vector to obtain the text-interpretable vector and the difference vector, respectively.

[0095] Optionally, the difference fusion module 430 is also used to perform consensus processing on the enhanced text vector and the text interpretable vector, as well as the text interpretable vector and the difference vector, respectively, to obtain the target cross-modal vector and the overall difference vector; The enhanced text vector is cosine-calculated with the difference acoustic vector and difference visual vector in the difference vector to obtain the semantic difference visual vector and semantic difference acoustic vector. The cross-modal vector, the overall difference vector, the semantic difference visual vector, and the semantic difference acoustic vector are concatenated to obtain the cross-modal difference vector; The difference-gated signal is obtained by linearly transforming the cross-modal difference vector using an activation function. The target cross-modal vector, the overall difference vector, and the difference-gated signal are fused to obtain the target fused vector.

[0096] Optionally, the difference fusion module 430 is also used to concatenate the text interpretable vector and the enhanced text vector according to the feature dimension to obtain an initial cross-modal vector; Perform a linear operation on the initial cross-modal vector to obtain the target cross-modal vector; Based on the feature dimension, the difference visual vector and the difference acoustic vector in the difference vector are concatenated to obtain the difference concatenation vector; Perform a linear operation on the difference concatenation vector to obtain the overall difference vector.

[0097] The multimodal fusion sentiment label determination device provided in the embodiments of the present invention can execute the multimodal fusion sentiment label determination method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method.

[0098] According to embodiments of the present invention, the present invention also provides an electronic device, a readable storage medium, and a computer program product.

[0099] Figure 5 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0100] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0101] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0102] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the multimodal fusion sentiment label determination method.

[0103] In some embodiments, the multimodal fusion sentiment labeling method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the multimodal fusion sentiment labeling method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the multimodal fusion sentiment labeling method by any other suitable means (e.g., by means of firmware).

[0104] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0105] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0106] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0107] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0108] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0109] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product within the cloud computing service system. This addresses the shortcomings of traditional physical hosts and dedicated virtual services, such as high management difficulty and weak business scalability.

[0110] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and no limitation is imposed herein.

[0111] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for determining sentiment labels through multimodal fusion, characterized in that, include: Feature extraction is performed on the original multimodal sequence to obtain multimodal vectors; the multimodal vectors include text modality vectors, visual modality vectors, and acoustic modality vectors; The multimodal vectors are subjected to residual decoupling processing to obtain enhanced text vectors, text interpretable vectors, and difference vectors; the text interpretable vectors include text interpretable visual vectors and text interpretable acoustic vectors. The difference vector includes a difference visual vector and a difference acoustic vector; The enhanced text vector, the text interpretable vector, and the difference vector are fused to obtain the target fused vector. The target fusion vector is predicted and mapped to obtain the target sentiment label.

2. The method according to claim 1, characterized in that, The residual decoupling processing of the multimodal vectors to obtain enhanced text vectors, interpretable text vectors, and difference vectors includes: Text enhancement is performed on the text modality vector to obtain the enhanced text vector; The visual modality vector and the acoustic modality vector are used as non-text modality vectors; Linear projection is performed on the non-text modal vector to obtain a non-text projection vector; the non-text projection vector includes a visual projection vector and an acoustic projection vector. By scaling the feature dimension, the non-text projection vector and the text modality vector are decoupled using an activation function to obtain a text interpretable vector and a difference vector.

3. The method according to claim 2, characterized in that, The text enhancement process for the text modality vector to obtain the enhanced text vector includes: The text modality vector is extracted to obtain the target line vector; The fusion vector is obtained based on the full vector of the text modality vector and the binarized attention mask; The binarized attention mask of the text modality vector and the text modality vector are divided element-wise with the fused vector to obtain an element vector; The element vector and the target row vector are normalized to obtain the enhanced text vector.

4. The method according to claim 2, characterized in that, The process of residual decoupling the non-text projection vector and the text modality vector through an activation function under scaled feature dimensions to obtain a text interpretable vector and a difference vector includes: By scaling the feature dimension, the non-text projection vector and the text modality vector are interacted through an activation function to obtain a text vector and a non-text difference vector; Perform average pooling operations on the text vector and the non-text difference vector respectively to obtain the text average pooling vector and the non-text difference average pooling vector; Max pooling is performed on the text vector and the non-text difference vector respectively to obtain the text max pooling vector and the non-text difference max pooling vector; The average pooling vector and the max pooling vector of the text are concatenated to obtain a text-interpretable concatenated vector; the text-interpretable concatenated vector includes a text-interpretable acoustic concatenated vector and a text-interpretable visual concatenated vector. The non-textual difference average pooling vector and the non-textual difference maximum pooling vector are concatenated to obtain a non-textual difference concatenated vector; the non-textual difference concatenated vector includes a non-textual difference acoustic concatenated vector and a non-textual difference visual concatenated vector. Linear projections are performed on the text-interpretable concatenated vector and the non-text difference concatenated vector to obtain the text-interpretable vector and the difference vector, respectively.

5. The method according to claim 1, characterized in that, The step of performing difference fusion on the enhanced text vector, the text interpretable vector, and the difference vector to obtain the target fusion vector includes: Consensus processing is performed on the enhanced text vector and the text interpretable vector, as well as the text interpretable vector and the difference vector, respectively, to obtain the target cross-modal vector and the overall difference vector; The enhanced text vector is cosine-calculated with the difference acoustic vector and difference visual vector in the difference vector to obtain the semantic difference visual vector and semantic difference acoustic vector. The target cross-modal vector, the overall difference vector, the semantic difference visual vector, and the semantic difference acoustic vector are concatenated to obtain the cross-modal difference vector; The difference-gated signal is obtained by linearly transforming the cross-modal difference vector using an activation function. The target cross-modal vector, the overall difference vector, and the difference-gated signal are fused to obtain the target fused vector.

6. The method according to claim 5, characterized in that, The consensus processing performed on the enhanced text vector and the text interpretable vector, and the text interpretable vector and the difference vector, respectively, to obtain the target cross-modal vector and the overall difference vector, includes: The text interpretable vector and the enhanced text vector are concatenated according to the feature dimension to obtain the initial cross-modal vector; A linear operation is performed on the initial cross-modal vector to obtain the target cross-modal vector; The difference visual vector and difference acoustic vector in the difference vector are concatenated according to the feature dimension to obtain the difference concatenation vector; A linear operation is performed on the difference concatenation vector to obtain the overall difference vector.

7. A multimodal fusion-based emotion tag determination device, characterized in that, include: The feature extraction module is used to extract features from the original multimodal sequence to obtain a multimodal vector; the multimodal vector includes text modality vector, visual modality vector and acoustic modality vector; The residual decoupling module is used to perform residual decoupling processing on the multimodal vectors respectively to obtain enhanced text vectors, text interpretable vectors, and difference vectors; the text interpretable vectors include text interpretable visual vectors and text interpretable acoustic vectors; The difference vector includes a difference visual vector and a difference acoustic vector; The difference fusion module is used to perform difference fusion on the enhanced text vector, the text interpretable vector and the difference vector to obtain the target fusion vector; The prediction mapping module is used to predict and map the target fusion vector to obtain the target sentiment tag.

8. An electronic device, characterized in that, Electronic devices include: At least one processor; and A memory that is communicatively connected to at least one processor; wherein, The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the sentiment tag determination method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the sentiment tag determination method of any one of claims 1-6, which is based on multimodal fusion.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the sentiment label determination method of multimodal fusion according to any one of claims 1-6.

Images