Multimodal sentiment analysis method and device for characterizing matching and routing interaction, electronic equipment and medium

Abstract

The invention provides a multi-modal sentiment analysis method for characterization matching and routing interaction. The multi-modal sentiment analysis method comprises the following steps: respectively extracting a text modal representation, an audio modal representation and a visual modal representation from original video data; mapping the text modal representation, the audio modal representation and the visual modal representation to similar distribution spaces through an adversarial loop translation network to obtain uniformly distributed text modal representation, audio modal representation and visual modal representation; transmitting the uniformly distributed text modal representation, audio modal representation and visual modal representation into a dynamic routing interaction network to obtain multi-modal representation; and under the joint loss constraint of the uniformly distributed text modal representation, audio modal representation and visual modal representation, transmitting the multi-modal representation into an emotion prediction layer for emotion prediction, and outputting a final emotion prediction value.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal sentiment analysis, and in particular to a multimodal sentiment analysis method, device, electronic device, and medium for characterizing matching and routing interaction. Background Art

[0002] In the field of sentiment analysis, traditional natural language processing (NLP)-based methods primarily focus on a single text modality to predict the emotional polarity (e.g., negative, positive, neutral) it implies. However, with the rapid development of the internet and the ubiquity of social media, users are expressing their emotions in increasingly diverse ways, encompassing multiple modalities such as text, audio, and video. This emergence of multimodal data poses new challenges to sentiment analysis, as single-modal analysis often fails to fully and accurately capture a user's true emotional state. For example, a seemingly neutral or positive text message, set against the backdrop of sarcastic or mocking audio, may actually convey a negative sentiment.

[0003] To address these challenges, multimodal sentiment analysis technology has emerged. This technology aims to integrate complementary information from different modalities (such as text, audio, and vision) to more accurately infer implicit emotions or opinions. Related technologies have attempted to construct complex multimodal models to mine and fuse the sentiment information from these modalities, thereby generating effective multimodal representations. While these efforts have achieved some success, they still face numerous technical bottlenecks.

[0004] First, the heterogeneity between modal data leads to inconsistencies in the representation space. This inconsistency significantly hinders the extraction of effective sentiment information during modal interaction, thereby affecting the quality of the fused representation. Second, existing methods often fail to fully utilize the complete representation information of each modality when processing multimodal data, including its low-level and high-level features, as well as the consistency and complementarity between modalities. This incomplete utilization of information limits the model's ability to generate high-quality multimodal joint representations, thereby affecting the accuracy of sentiment polarity prediction.

[0005] Furthermore, cross-modal interaction modules based on advanced architectures like the Transformer face the problem of cross-modal interaction homogeneity in practice. As the number of network layers increases, the fused representations of different modalities become more similar, weakening the independence and complementarity between modalities, which are crucial for multimodal fusion. This homogeneity not only reduces the effectiveness of the fused representation but also limits the model's generalization ability in complex emotional scenarios.

[0006] In summary, the main challenges facing current multimodal sentiment analysis technology are the distribution gap between modal representations, insufficient utilization of multimodal representation information, and the homogeneity of cross-modal interactions. These issues severely restrict the development and application of multimodal sentiment analysis technology. Therefore, a method is urgently needed to effectively address these issues and improve the accuracy and robustness of multimodal sentiment analysis. Summary of the Invention

[0007] In view of this, the embodiments of the present disclosure provide a multimodal sentiment analysis method, device, electronic device, and medium for characterizing matching and routing interactions, which at least partially solve the problems existing in the prior art.

[0008] In a first aspect, an embodiment of the present disclosure provides a multimodal sentiment analysis method for characterizing matching and routing interactions, comprising:

[0009] Extracting text modality representation, audio modality representation and visual modality representation from the original video data respectively;

[0010] Mapping the text modality representation, the audio modality representation, and the visual modality representation into a similar distribution space through an adversarial recurrent translation network to obtain evenly distributed text modality representation, audio modality representation, and visual modality representation;

[0011] Passing the evenly distributed text modality representation, audio modality representation, and visual modality representation into a dynamic routing interaction network to obtain a multimodal representation;

[0012] The multimodal representation is passed to the emotion prediction layer for emotion prediction under the joint loss constraint of the evenly distributed text modality representation, audio modality representation and visual modality representation, and the final emotion prediction value is output.

[0013] According to a specific implementation of the disclosed embodiment, mapping the text modality representation, the audio modality representation, and the visual modality representation into a similar distribution space through an adversarial recurrent translation network to obtain uniformly distributed text modality representation, audio modality representation, and visual modality representation includes:

[0014] The distribution of the audio representation is converted into a distribution similar to the text representation through the audio-text distribution conversion network to obtain the uniformly distributed audio modal representation. The specific formula is: Among them, T a→t is an audio-text distribution conversion network, are the parameters of the audio-text distribution conversion network;

[0015] The distribution of the visual representation is converted into a distribution similar to the text representation through the visual-text distribution conversion network to obtain the uniformly distributed visual modality representation. The specific calculation formula is:

[0016] Among them, T v→t is a visual-text distribution conversion network, are the parameters of the visual-text distribution conversion network;

[0017] The distribution matching conversion process of the audio representation and the visual representation is constrained by generator loss, discriminator loss and cycle loss to obtain the uniformly distributed text modality representation, audio modality representation and visual modality representation.

[0018] According to a specific implementation of the embodiment of the present disclosure, the evenly distributed text modality representation, audio modality representation and visual modality representation are introduced into a dynamic routing interaction network to obtain a multimodal representation.

[0019] include:

[0020] In a modality activation component, the evenly distributed text modality representation, audio modality representation, and visual modality representation are processed by a feedforward neural network and an activation function to retain low-level emotional semantics contained in the evenly distributed text modality representation, audio modality representation, and visual modality representation;

[0021] In the intra-modal interaction component, the attention focus of the intra-modal interaction is adjusted through the multi-head self-attention mechanism.

[0022] and mine high-level sentiment information associated within the modality;

[0023] The results of multiple attention heads are concatenated and fed into the feedforward neural network for integration;

[0024] The interactive representation between modalities is obtained through the similarity-difference multi-head attention mechanism;

[0025] In a cross-modal interaction component, a fusion representation between the modalities is optimized using KL divergence and soft orthogonal constraints, where the fusion representation is part of the interaction representation;

[0026] The output results of the modal activation component, the intra-modal interaction component, and the cross-modal interaction component are obtained through a routing strategy, and an aggregation operation is performed on the output results to obtain the multimodal representation.

[0027] According to a specific implementation of the embodiment of the present disclosure, the distribution matching conversion process of the audio representation and the visual representation is constrained by using generator loss, discriminator loss and cycle loss, as follows:

[0028] Among them, G n→t It is a text distribution generator, which is used to convert the distribution of non-text representation into the distribution of text representation. t→n Realize the distribution conversion from text representation to non-text representation. t and D t It is a discriminator used to determine whether the input is a non-text representation or a text representation, n∈{a,v}.

[0029] According to a specific implementation of an embodiment of the present disclosure, the evenly distributed text modality representation, audio modality representation, and visual modality representation are transmitted into a dynamic routing interaction network to obtain a multimodal representation, including: each component of the dynamic routing network can receive a signal from a component in a previous layer. Each component controls the path probability between each component in two adjacent layers by using a routing mechanism.

[0030] According to a specific implementation method of an embodiment of the present disclosure, the multimodal representation is passed into the emotion prediction layer for emotion prediction under the joint loss constraint of the evenly distributed text modal representation, audio modal representation and visual modal representation, and the final emotion prediction value is output, including: splicing the text multimodal representation, audio multimodal representation and visual multimodal representation together, and sending them into a multilayer perceptron under the joint loss constraint to obtain the final emotion prediction value.

[0031] According to a specific implementation of an embodiment of the present disclosure, the joint loss constraint includes: distribution matching loss, similarity loss, difference loss, and sentiment classification loss of the uniformly distributed text modality representation, audio modality representation, and visual modality representation. The joint loss constraint is specifically as follows:

[0032] L = x e L ep +x r L rdm +x s L sim +x d L dis , where L ep is the sentiment prediction loss, L rdm is the representation distribution matching loss, L sim is the similarity loss, L dis is the difference loss, x e ,x r ,x s ,x d Represents the weight coefficient of each loss.

[0033] In a second aspect, an embodiment of the present disclosure provides a multimodal sentiment analysis device for characterizing matching and routing interactions, comprising:

[0034] Multimodal representation extraction module: used to independently extract text modality representation, audio modality representation, and visual modality representation from the original video data;

[0035] A representation distribution mapping module is configured to receive the text modality representation, the audio modality representation, and the visual modality representation from the multimodal representation extraction module, and map the text modality representation, the audio modality representation, and the visual modality representation into a similar distribution space through an adversarial recurrent translation network;

[0036] Dynamic Routing Interaction Network Module: This module receives evenly distributed text, audio, and visual modal representations as input, mines sentiment information within each modality, captures sentiment information across modalities, and ultimately outputs a multimodal representation.

[0037] Sentiment prediction and optimization module: used to receive multimodal representations from the dynamic routing interaction network module, perform sentiment prediction under the joint loss constraint, and output the final sentiment prediction value.

[0038] In a third aspect, an embodiment of the present disclosure provides an electronic device, the electronic device comprising:

[0039] at least one processor; and,

[0040] a memory communicatively connected to the at least one processor; wherein,

[0041] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned multimodal sentiment analysis method of characterization matching and routing interaction.

[0042] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned multimodal sentiment analysis method of characterization matching and routing interaction.

[0043] In summary, compared with the prior art, this embodiment has the following advantages:

[0044] A multimodal sentiment analysis method based on representation distribution matching and dynamic routing interaction first transforms the distribution of audio and visual representations to a distribution similar to that of text representations through a distribution conversion network. Then, a dynamic routing interaction network is used to fully exploit multi-level sentiment information in each modal representation, including low-level and high-level sentiment within the modality and similarities and differences between modalities. Regularization constraints are also used to improve the homogeneity of fused representations during cross-modal interaction. The multimodal sentiment analysis model constructed using this method can fit representation distribution gaps and fully utilize the sentiment information at all levels contained in the modal representations, significantly improving the model's sentiment classification performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with specific embodiments and accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0046] As used herein, the term "including" and its variations represent open inclusion, i.e., "including but not limited to." Unless otherwise stated, the term "or" means "and / or." The term "based on" means "based at least in part on." The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment." The term "another embodiment" means "at least one additional embodiment." The terms "first," "second," etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0047] Figure 1A A schematic flow chart of a multimodal sentiment analysis method for characterizing matching and routing interactions provided in the first embodiment of the present invention;

[0048] Figure 1B A schematic diagram of a multimodal sentiment analysis model framework for characterizing matching and routing interactions provided in the first embodiment of the present invention;

[0049] Figure 2 A schematic diagram of the implementation process of representation distribution matching;

[0050] Figure 3 Implement a flow chart for the routing component;

[0051] Figure 4 Optimize implementation flow for fusion characterization;

[0052] Figure 5 A schematic diagram of the structure of a multimodal sentiment analysis device for characterization matching and routing interaction provided in a second embodiment of the present invention;

[0053] Figure 6 A schematic diagram of an exemplary structure of a device capable of implementing the method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0054] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0055] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0056] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement an apparatus and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this apparatus and / or practice this method.

[0057] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0058] Additionally, in the following description, specific details are provided to provide a thorough understanding of the examples. However, one skilled in the art will appreciate that the aspects described can be practiced without these specific details.

[0059] See also Figure 1A-1B , Figure 1A A flow chart of a multimodal sentiment analysis method for characterizing matching and routing interactions is provided. Figure 1B A schematic diagram of a multimodal sentiment analysis model framework for characterizing matching and routing interactions is provided. Figure 1AThe provided multimodal sentiment analysis method process for characterization matching and routing interaction can be executed by an electronic device, and in particular, by one or more processors within the electronic device, to implement the following steps:

[0060] S101, extracting text modality representation, audio modality representation and visual modality representation from the original video data respectively.

[0061] In order to overcome the limitations of traditional text feature extraction methods in dealing with polysemy, a word embedding method based on the BERT (Bidirectional Encoder Representations from Transformers) pre-trained model is introduced. The BERT model containing a bidirectional Transformer structure is used. With its context understanding ability, the model can capture the bidirectional context information and temporal dependencies of words in a sentence, thereby generating highly discriminative word embeddings. On this basis, the first word vector of the last layer of the model is selected as the overall representation of the text. For example, the dimension of the representation vector is 768, which can fully reflect the deep semantic information of the text content. Text representation F t As shown in formula (1):

[0062]

[0063] in, xt is the original text feature vector, are the BERT network parameters.

[0064] For audio modality, the raw audio data is sampled and framed, and features such as Zero Crossing Rate (ZCR), Mel-frequency cepstral coefficients (MFCC), and Constant-Q Chromagram (CQT) are extracted from the audio frames using the Librosa library. These audio frame features are combined to produce audio features that are closely related to the speaker's emotions. To obtain the temporal information of the audio frame sequence, Bi-LSTM is used to process the above series of audio features to obtain a complete audio representation, the audio representation F a The dimension of is 32 for example. The frequency representation is shown in formula (2):

[0065]

[0066] Among them, x a is the original audio feature vector, are Bi-LSTM network parameters.

[0067] For the visual modality, OpenFace2.0 is used to identify facial features such as head movement, gaze direction, and expression in each video frame, and the amount of data is reduced by frame downsampling based on average pooling. In order to obtain the temporal information of the video frame sequence, Bi-LSTM is used to process the above series of facial features to obtain a visual representation containing rich emotional clues. The visual representation F v The dimension of is 64 for example. The visual representation is shown in formula (3):

[0068]

[0069] Among them, x v is the original visual feature vector, are Bi-LSTM network parameters.

[0070] S102, mapping the text modality representation, the audio modality representation, and the visual modality representation into a similar distribution space through an adversarial recurrent translation network to obtain evenly distributed text modality representation, audio modality representation, and visual modality representation.

[0071] In this embodiment, the characterization distribution matching implementation process includes the following steps, see Figure 2 .

[0072] S201, using the audio-text distribution conversion network to transform the audio representation F a The distribution of the text representation F t distribution and retain the original semantic information of the audio modality, thereby obtaining an audio conversion representation (uniformly distributed audio representation) Specifically as shown in formula (3):

[0073]

[0074] Among them, T a→t It is an audio-text distribution conversion network, are the parameters of the audio-text distribution conversion network;

[0075] S202, using the visual text distribution conversion network to transform the visual representation F v The distribution of the text representation F t distribution and retain the original semantic information of the visual modality, thereby obtaining a visual transformation representation (evenly distributed visual representation) Specifically as shown in formula (4):

[0076]

[0077] Among them, T v→t is a visual-text distribution conversion network, are the parameters of the visual-text distribution conversion network;

[0078] S203, with the help of generator loss L g , discriminator loss L d and circulation loss L c To constrain the distribution matching conversion process of the audio representation and the visual representation, as shown in formula (5):

[0079] L rdm =L g +L d +L c

[0080] =log(1-D t (G n→t (F n )))+log(1-D n (G t→n (F t )))+log(D t (F t ))+log(D n (F n ))+||G n→t (G t→n (F t ))||1+||G t→n (G n→t (F n ))||1 (5)

[0081] Among them, G n→t It is a text distribution generator, which is used to convert the distribution of non-text representations (audio representation and video representation) into the distribution of text representations. t→n Realize the distribution conversion from text representation to non-text representation. t and D t It is a discriminator used to determine whether the input is a non-text representation or a text representation, n∈{a,v}.

[0082] S103: The evenly distributed text modality representation, audio modality representation, and visual modality representation are transferred into a dynamic routing interaction network to obtain a multimodal representation.

[0083] The routing component implementation process provided in this embodiment includes the following steps, see Figure 3 :

[0084] S301, in the modal activation component, a simple feedforward neural network and activation function are used to process the modal representation. Thus, the low-level emotional semantics contained in the representation is preserved. Modal activation representation F1 dSpecifically, as shown in formula (6):

[0085]

[0086] Among them, FFN is a feedforward neural network, ReLU is a nonlinear activation function, m∈{t,a,v}.

[0087] S302, in the intra-modal interaction component, a multi-head self-attention mechanism is used to dynamically adjust the focus of attention of intra-modal interaction and mine the high-level emotional information associated with the modality. For a single attention head, the modality representation is represented by a weight matrix. Convert to query key Sum Then through the vector and vector Calculate the similarity score and add it to vector V2 m The weighted summation is used to obtain the interaction information within the modality, as shown in formula (7):

[0088]

[0089] W2 Q ,W2 K ,W2 V is the corresponding weight matrix, head i For a single attention head, (*) T is the transpose operation, d is the vector dimension;

[0090] The results of multiple attention heads are concatenated and fed into a feedforward neural network for integration. Intra-modal interaction representation F2 d Specifically as shown in formula (8):

[0091]

[0092] Where h is the number of attention heads, concat(*) is concatenation, Dropout and LN are dropout layers and normalization layers respectively;

[0093] S303 uses a similarity-difference multi-head attention mechanism to comprehensively mine interactive representations between modalities. The following uses text representation as an example; the principles for audio and visual features are similar and will not be further elaborated here.

[0094] Specifically, we simultaneously mine the similarities and differences in sentiment information between text audio and text vision, and use it to enhance text representation;

[0095] The text is represented by the weight matrix F t r and audio characterization Convert to query key Sum V3 a Then through the vector and vector After calculating the similarity score, take the absolute value operation and add it to vector V3 a The weighted summation is used to obtain the interaction information between text and audio. Finally, the text-audio interaction representation F3 is obtained by integrating the results of multiple attention heads. d , as shown in formula (9):

[0096]

[0097] Among them W3 Q ,W3 K ,W3 V is the corresponding weight matrix, |*| is the absolute value operation;

[0098] Characterize F through text t r and audio representation F v r Get Query key Sum V4 v . And use the same calculation to get the text-visual interaction representation F4 d , as shown in formula (10):

[0099]

[0100] Among them W4 Q ,W4 K ,W4 V is the corresponding weight matrix.

[0101] Similarly, we can obtain audio-text interaction representation, audio-visual interaction representation, visual-text interaction representation, and visual-audio interaction representation, which will not be described in detail here.

[0102] In S304, in the cross-modal interaction component, as the number of Transformer network layers increases, the fused representations of text-audio (AB) and audio-text (BA) become increasingly similar. Similarly, the fused representations of text-vision and visual-text, as well as audio-vision and visual-audio, also become increasingly similar. Therefore, two regularization methods, KL divergence and soft orthogonal constraints, are used to optimize the fused representations between the modalities.

[0103] It should be noted that the fusion representation is a part of the interaction representation.

[0104] It should be noted that in the cross-modal interaction component, text audio and text vision are processed by two cross-modal interaction components respectively.

[0105] Furthermore, the KL divergence is used to maximize the mutual information between the fusion representation and each modality representation, see Figure 4 Through the constraint of KL divergence, the fusion representation can retain the unique emotional information in the modal representation to the maximum extent. The KL regularization constraint is shown in formula (11):

[0106] L sim =KL(F ta ,F t )+KL(F tv ,F t )+KL(F at ,F a )+KL(F av ,F a )+KL(F vt ,F v )+KL(F va ,F v ) (11)

[0107] Among them, F t ,F a ,F v is the modal representation, F ta ,F tv ,F at ,F av ,F vt ,F va It is a fusion representation;

[0108] Soft orthogonal constraints are used to further enhance the difference between the two fused representations. The soft orthogonal constraints are specifically shown in formula (12):

[0109] L dis =SOC(F ta ,F at )+SOC(F tv ,F vt )+SOC(F av ,F va ) (12)

[0110] S305: Obtain the output results of the modal activation component, the intra-modal interaction component, and the cross-modal interaction component through a routing strategy, and fuse them into a multimodal representation rich in emotional semantics using an aggregation operation. Specifically, as shown in formula (13):

[0111]

[0112] Furthermore, in a dynamic routing network, the dense connections between components in each layer give each component the opportunity to receive signals from the previous layer. In addition, a routing mechanism is used in each component. The routing mechanism can control the path probability between each component in two adjacent layers. Specifically, as shown in formula (14):

[0113]

[0114] Where N represents the number of components in each layer, represents the output of the jth component in the k-1 layer. represents the path probability from the jth component in the k-1 layer to the i-th component in the k layer;

[0115] The specific calculation formula is shown in formula (15):

[0116]

[0117] in express The qth vector of

[0118] S104: passing the multimodal representation to the emotion prediction layer for emotion prediction under the joint loss constraint of the evenly distributed text modal representation, audio modal representation, and visual modal representation, and outputting a final emotion prediction value.

[0119] In this embodiment, the text multimodal representation, audio multimodal representation and visual multimodal representation are spliced ​​together and then fed into a multilayer perceptron (MLP) consisting of a four-layer fully connected network under the joint loss constraint to achieve multimodal sentiment prediction, as shown in formula (16):

[0120] y=MLP(concat(F t d ,F a d ,F v d )) (16)

[0121] Where y∈R is a continuous value representing the predicted emotion;

[0122] Specifically, the joint loss constraint is shown in formula (17):

[0123] L = x e L ep +x r L rdm +x s L sim+x d L dis (17)

[0124] Among them L ep is the sentiment prediction loss, L rdm is the representation distribution matching loss, L sim is the similarity loss, L dis is the difference loss, x e ,x r ,x s ,x d Represents the weight coefficient of each loss.

[0125] In order to illustrate the effect of the multimodal sentiment analysis method for characterization matching and routing interaction provided by the embodiment of the present application, the embodiment of the present application also provides a comparison of the results of various representative methods, see Table (1), which is a comparison of the results of the method provided by the embodiment of the present invention and various representative methods. The comparison was carried out on the public dataset CMU-MOSI in the field of multimodal sentiment analysis, and mainly tested the sentiment 2 classification performance of the model, that is, determining whether the sentiment category of the sample is "positive" or "negative". The test selected the accuracy and F1 score of sentiment 2 classification as evaluation indicators. The accuracy refers to the ratio of the number of correct sentiment classifications in the sample prediction results to the total number of samples. The larger the value, the better the prediction effect. The F1 score is the harmonic mean of the precision and recall rate. The larger the value, the better the prediction effect. As shown in Table 1, the embodiment of the present invention achieved better performance in the sentiment 2 classification test compared with multiple classic models in the field of multimodal sentiment analysis. Compared with the suboptimal model CJTF, the accuracy and F1 score were improved by 1.9% and 2.0% respectively.

[0126] Table (1)

[0127]

[0128]

[0129] In summary, compared with the prior art, this embodiment has the following advantages:

[0130] 1. Multimodal Representation Spatial Alignment: Using a multi-resistance recurrent translation network, we successfully mapped the original audio and visual representations into a high-dimensional space similar to the text representation. This not only narrowed the representation distribution differences between audio, visual, and text modalities but also achieved a uniform distribution of representations, laying the foundation for subsequent multimodal information interaction and fusion.

[0131] 2. Dynamic Sentiment Mining and Interaction: By introducing a dynamic routing interaction network, we can deeply mine sentiment information within each modality (from low-level to high-level) and capture similarities and differences in sentiment information across modalities. This enriches the sentiment semantics of each modality's representation, making the model more nuanced and comprehensive in its understanding of sentiment.

[0132] 3. Optimization of Cross-Modal Fusion Representations: To address the homogeneity of fusion representations that often occurs when traditional Transformers are stacked deeply, we use similarity and difference losses as constraints to optimize fusion representations during cross-modal interaction. This effectively avoids information redundancy and ensures the diversity and effectiveness of fusion representations.

[0133] 4. Joint Loss Optimization Strategy: To further improve the accuracy of sentiment prediction, we combine the classification loss of the sentiment prediction classification layer, the representation distribution matching loss, and the similarity and difference losses of the fused representation to form a joint loss constraint. This joint loss constraint optimizes the training process and ultimately improves the performance of sentiment prediction.

[0134] The second embodiment of the present invention provides a multimodal sentiment analysis device (such as Figure 5 shown), which includes:

[0135] Multimodal representation extraction module 510: used to independently extract text modal representation, audio modal representation and visual modal representation from the original video data to ensure the integrity and accuracy of each modal information.

[0136] Representation Distribution Mapping Module 520: This module receives the text, audio, and visual modality representations from the Multimodal Representation Extraction Module and maps them into a similar distribution space using an adversarial recurrent translation network. This process aims to eliminate distributional differences between the representations of different modalities and obtain uniformly distributed and mutually compatible representations for each modality.

[0137] Dynamic Routing Interaction Network Module 530: This module receives evenly distributed representations of text, audio, and visual modalities as input. Through its internal mechanisms, it deeply mines the emotional information (both low-level and high-level) within each modality and captures the similarities and differences in emotional information across modalities. Ultimately, this module outputs a multimodal representation that contains rich emotional semantics.

[0138] Sentiment Prediction and Optimization Module 540: This module receives the multimodal representations from the dynamic routing interaction network module under the joint loss constraint of evenly distributed textual, audio, and visual modal representations. It processes the multimodal representations through the sentiment prediction layer and outputs the final sentiment prediction value.

[0139] A third embodiment of the present invention further provides an electronic device, comprising:

[0140] at least one processor; and,

[0141] a memory communicatively connected to the at least one processor; wherein,

[0142] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the multimodal sentiment analysis method for characterizing matching and routing interaction in any of the aforementioned embodiments.

[0143] The fourth embodiment of the present invention also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the multimodal sentiment analysis method for characterization matching and routing interaction described in any of the aforementioned embodiments.

[0144] The fifth embodiment of the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the multimodal sentiment analysis method for characterizing matching and routing interactions of any of the aforementioned embodiments.

[0145] The sixth embodiment of the present invention further provides a computer program, which includes program instructions. When the program instructions are executed by a computer, the computer is caused to execute the multimodal sentiment analysis method for characterization matching and routing interaction of any of the aforementioned embodiments.

[0146] Figure 6 A schematic diagram of a method or device 1000 that can implement an embodiment of the present invention is shown. In some embodiments, the method or device 1000 may include more or fewer devices than shown. In some embodiments, the method or device 1000 may be implemented using a single device or multiple devices. In some embodiments, the method or device 1000 may be implemented using cloud-based or distributed devices.

[0147] like Figure 6As shown, device 1000 includes a processor 1001, which can perform various appropriate operations and processes according to the programs and / or data stored in a read-only memory (ROM) 1002 or the programs and / or data loaded from a storage portion 1008 into a random access memory (RAM) 1003. Processor 1001 can be a multi-core processor or can include multiple processors. In some embodiments, processor 1001 can include a general-purpose main processor and one or more special coprocessors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. Various programs and data required for the operation of device 1000 are also stored in RAM 1003. Processor 1001, ROM 1002, and RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to bus 1004.

[0148] The processor and memory are used together to execute the program stored in the memory. When the program is executed by the computer, the methods, steps or functions described in the above embodiments can be implemented.

[0149] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, a touch screen, and the like; an output section 1007 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1008 including a hard disk and the like; and a communication section 1009 including a network interface card such as a LAN card or a modem. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1010 as needed, so that computer programs read therefrom can be installed into the storage section 1008 as needed. Figure 6 Only some components are shown schematically, which does not mean that the device 1000 only includes Figure 6 Components shown.

[0150] The systems, devices, modules, or units described in the above embodiments may be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, a smartphone, a personal computer, a laptop computer, an in-vehicle human-computer interaction device, a personal digital assistant, a media player, a navigation device, a game console, a tablet computer, a wearable device, a smart TV, an Internet of Things system, a smart home, an industrial computer, a server, or a combination thereof.

[0151] Although not shown, in an embodiment of the present invention, a computer-readable storage medium is provided on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the multimodal sentiment analysis method for characterization matching and routing interaction described in Example 1 is implemented.

[0152] Storage media in embodiments of the present invention include permanent and non-permanent, removable and non-removable items that can be used to store information using any method or technology. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0153] Although not shown, an embodiment of the present invention further provides a computer program product, including: a computer program / instruction, which, when executed by a processor, implements the multimodal sentiment analysis method for characterization matching and routing interaction described in Example 1.

[0154] The methods, programs, systems, and apparatuses of the embodiments of the present invention may be executed or implemented in a single or multiple networked computers, or may be practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks may be performed by remote processing devices connected via a communication network.

[0155] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, those skilled in the art will appreciate that the functional modules / units or controllers and related method steps described in the above embodiments may be implemented using software, hardware, or a combination of software / hardware.

[0156] Unless explicitly stated, the actions or steps of the methods, procedures, and methods described in accordance with the embodiments of the present invention do not have to be performed in a specific order and can still achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0157] In this document, multiple embodiments of the present invention are described, but for the sake of brevity, the description of each embodiment is not exhaustive, and the same or similar features or parts between the embodiments may be omitted. In this document, "one embodiment", "some embodiments", "example", "specific example", or "some examples" are intended to apply to at least one embodiment or example according to the present invention, but not all embodiments. The above terms do not necessarily mean to refer to the same embodiment or example. Those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are mutually contradictory.

[0158] While the exemplary systems and methods of the present invention have been specifically shown and described with reference to the foregoing embodiments, these are merely examples of the best modes for implementing the present systems and methods. Those skilled in the art will appreciate that various changes may be made to the embodiments of the systems and methods described herein when implementing the present systems and / or methods without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A multimodal sentiment analysis method for characterizing matching and routing interactions, characterized in that: include: Extracting text modality representation, audio modality representation and visual modality representation from the original video data respectively; Mapping the text modality representation, the audio modality representation, and the visual modality representation into a similar distribution space through an adversarial recurrent translation network to obtain uniformly distributed text modality representation, audio modality representation, and visual modality representation; Passing the evenly distributed text modality representation, audio modality representation, and visual modality representation into a dynamic routing interaction network to obtain a multimodal representation; Under the joint loss constraint of the evenly distributed text modality representation, audio modality representation and visual modality representation, the multimodal representation is passed to the emotion prediction layer for emotion prediction, and a final emotion prediction value is output.

2. The multimodal sentiment analysis method for characterization matching and routing interaction according to claim 1, characterized in that: The step of mapping the text modality representation, the audio modality representation, and the visual modality representation into a similar distribution space through an adversarial recurrent translation network to obtain uniformly distributed text modality representation, audio modality representation, and visual modality representation includes: The distribution of the audio representation is converted into a distribution similar to the text representation through an audio-text distribution conversion network to obtain the uniformly distributed audio modal representation. The specific formula is: Among them, T a→t is an audio-text distribution conversion network, are the parameters of the audio-text distribution conversion network; The distribution of the visual representation is converted into a distribution similar to the text representation through a visual text distribution conversion network to obtain the uniformly distributed visual modality representation. The specific calculation formula is: Among them, T v→t is a visual-text distribution conversion network, are the parameters of the visual-text distribution conversion network; The distribution matching conversion process of the audio representation and the visual representation is constrained by generator loss, discriminator loss and cycle loss to obtain the uniformly distributed text modality representation, audio modality representation and visual modality representation.

3. The multimodal sentiment analysis method for characterization matching and routing interaction according to any one of claims 1-2, characterized in that: The uniformly distributed text modality representation, audio modality representation and visual modality representation are transmitted into a dynamic routing interaction network to obtain a multimodal representation, including: In a modality activation component, the uniformly distributed text modality representation, audio modality representation, and visual modality representation are processed by a feedforward neural network and an activation function to retain low-level emotional semantics contained in the uniformly distributed text modality representation, audio modality representation, and visual modality representation; In the intra-modal interaction component, the multi-head self-attention mechanism is used to adjust the focus of intra-modal interaction and mine the high-level emotional information associated with the modality. Concatenate the results of multiple attention heads, and feed the concatenated results into the feedforward neural network for integration; The interactive representation between modalities is obtained through the similarity-difference multi-head attention mechanism; In a cross-modal interaction component, a fusion representation between the modalities is optimized using KL divergence and soft orthogonal constraints, wherein the fusion representation is a part of the interaction representation; The output results of the modal activation component, the intra-modal interaction component, and the cross-modal interaction component are obtained through a routing strategy, and an aggregation operation is performed on the output results to obtain the multimodal representation.

4. The multimodal sentiment analysis method for characterization matching and routing interaction according to claim 2, characterized in that: The distribution matching conversion process of the audio representation and the visual representation is constrained by the generator loss, the discriminator loss and the cycle loss to obtain the uniformly distributed text modality representation, the audio modality representation and the visual modality representation, as follows: Among them, G n→t is a text distribution generator, which is used to convert the distribution of non-text representations into the distribution of text representations. t→n Realize the distribution transformation from text representation to non-text representation. t and D t is a discriminator used to determine whether the input is a non-text representation or a text representation, n∈{a,v}.

5. The multimodal sentiment analysis method for characterization matching and routing interaction according to claim 1, characterized in that: The evenly distributed text modality representation, audio modality representation and visual modality representation are transmitted into a dynamic routing interaction network to obtain a multimodal representation, including: each component of the dynamic routing network can receive a signal from a component of a previous layer. Each component controls the path probability between each component of two adjacent layers by using a routing mechanism.

6. The multimodal sentiment analysis method for characterization matching and routing interaction as described in claim 1, characterized in that: Under the joint loss constraint of the evenly distributed text modal representation, audio modal representation and visual modal representation, the multimodal representation is passed to the emotion prediction layer for emotion prediction, and the final emotion prediction value is output, including: splicing the text multimodal representation, audio multimodal representation and visual multimodal representation together, and sending them to a multilayer perceptron under the joint loss constraint to obtain the final emotion prediction value.

7. The multimodal sentiment analysis method for characterization matching and routing interaction according to claim 1, characterized in that: The joint loss constraint includes: distribution matching loss, similarity loss, difference loss and sentiment classification loss of the uniformly distributed text modality representation, audio modality representation and visual modality representation. The joint loss constraint is as follows: L = x e L ep +x r L rdm +x s L sim +x d L dis , where L ep is the sentiment prediction loss, L rdm is the representation distribution matching loss, L sim is the similarity loss, L dis is the difference loss, x e ,x r ,x s ,x d Represents the weight coefficient of each loss.

8. A multimodal sentiment analysis device for characterizing matching and routing interaction, characterized in that: include: Multimodal representation extraction module: used to independently extract text modality representation, audio modality representation and visual modality representation from the original video data; A representation distribution mapping module: used to receive the text modality representation, the audio modality representation and the visual modality representation from the multimodal representation extraction module, and map the text modality representation, the audio modality representation and the visual modality representation into a similar distribution space through an adversarial recurrent translation network; Dynamic routing interaction network module: It is used to receive evenly distributed text modal representations, audio modal representations, and visual modal representations as input, mine the sentiment information within each modality, and capture the sentiment information across modalities, and finally output multimodal representations; Sentiment prediction and optimization module: used to receive multimodal representations from the dynamic routing interaction network module, perform sentiment prediction under the joint loss constraint, and output the final sentiment prediction value.

9. An electronic device, characterized in that: The electronic device includes: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multimodal sentiment analysis method for characterization matching and routing interaction as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which are used to enable the computer to perform the multimodal sentiment analysis of characterization matching and routing interaction as described in any one of claims 1 to 7.

Images