A multimodal sentiment analysis method and system based on two-stage heterogeneous hypergraph

Abstract

The present invention discloses a multimodal sentiment analysis method and system based on a dual-stage heterogeneous hypergraph, which belongs to the field of multimodal sentiment analysis prediction. The specific process of the method of the present invention is: obtaining the original data of each modality in a segment of audio and video clips, inputting the original data of each modality into a trained multimodal sentiment analysis model, and outputting the probability of the emotional polarity reflected in the audio and video clips; wherein the original data of each modality is an audio clip, a video clip, and a dialogue text. The present invention uses hyperedges to model the interaction within and between modalities, effectively breaking through the limitation of paired interactions of previous methods. In addition, hyperedges are also used as information of local interactions.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal sentiment analysis and prediction, and in particular relates to a multimodal sentiment analysis method and system based on a dual-stage heterogeneous hypergraph. Background Art

[0002] In the digital communication era, the media for expressing emotions has evolved from simple text information to complex videos, which has brought new challenges and opportunities to multimodal sentiment analysis. In the past, sentiment analysis often relied on a single modality such as text semantics and visual expressions to judge emotions. However, relying on a single modality in multimodal data such as videos may lead to inaccurate sentiment analysis. There is a certain deviation in the emotions expressed by a single modality, which needs to be corrected by using complementary information from other modalities. For example, if only the semantics of the speech of the characters in the video are analyzed, the sarcastic remarks in the video may be mistakenly interpreted as positive remarks. If the speaker's tone and facial expressions are considered at the same time, this misreading can be corrected. Multimodal sentiment analysis provides an effective way to correct the emotion deviation of a single modality, and therefore has received more attention. Constructing an effective multimodal sentiment analysis method can not only improve the accuracy of emotion recognition, but also provide users with a richer emotional communication experience, playing an important role in the development of human-computer interaction.

[0003] At present, multimodal sentiment analysis methods mainly extract low-level features of the three modalities of audio, vision and text from the video, and then obtain complementary information between the modalities through machine learning. Currently, this aspect can be divided into three categories. The first category is feature space-based methods (FSBM), such as contrastive feature decomposition (ConFEDE), hybrid contrastive learning (HyCon), etc. This type of method combines contrastive learning with unimodal sentiment analysis tasks to retain the invariant information and difference information between the modalities. The second type of method is the tensor-based method (TBM), such as the tensor fusion network (TFN), modality-based redundancy reduction fusion (MRRF), etc. These two types of methods have their own advantages and disadvantages: 1) FSBM focuses on retaining the invariant information and specific information between modalities through feature learning, thus lacking the use of intra-modal and inter-modal interactive information. In addition, in the process of constructing positive and negative samples in contrastive learning, heterogeneous multimodal data easily guides the model to select wrong positive and negative samples, which has an adverse effect on feature learning. 2) TBM avoids the risk of wrong sample selection in the contrastive learning process, and models the intra-modal and inter-modal interactions through tensor product, but this modeling method is difficult to use fine-grained modal correlation information, and tensor-based calculations will lead to excessive memory and computing resource consumption when high-dimensional features are involved. The third type of method is the attention-based method (ABM), such as Multimodal Transformer (MulT) and Hybrid cross-modal interaction learning (HCI). This type of method uses LSTM to capture the correlation information within the modality, and then uses the attention mechanism to model the fine-grained correlation information between the modalities. However, this type of method still has defects: (1) ABM uses LSTM to model the interaction information within the modality, but LSTM is a recurrent neural network and cannot be operated in parallel, which is inefficient. (2): ABM uses cross-modal attention to model the interaction information between modalities, but the interaction range modeled by attention each time is limited to two time steps of the two modalities, and the correlation information of the modalities is not effectively utilized. (3): ABM lacks the utilization of local interaction information, resulting in partial information loss. Therefore, the prediction accuracy of these methods for multimodal emotions needs to be improved. Summary of the invention

[0004] The purpose of the present invention is to solve the problems existing in the prior art and provide a multimodal sentiment analysis method and system based on double-stage heterogeneous hypergraphs. In recent years, hypergraphs have achieved good application results in various fields. Therefore, inspired by this, the present invention proposes a multimodal sentiment analysis model based on double-stage heterogeneous hypergraphs (Double-stage heterogeneous hypergraphs based, DH2B) to solve the above defects by applying the high-order modeling capabilities of hypergraphs.

[0005] In order to achieve the above-mentioned invention object, the present invention specifically adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a multimodal sentiment analysis method based on a two-stage heterogeneous hypergraph, comprising the following steps:

[0007] S1. extracting raw data of each modality from a video clip with sound, wherein the module of the raw data includes an audio clip, a video clip, and a dialogue text;

[0008] S2. Input the raw data of each modality into the trained multimodal sentiment analysis model, process the raw data of each modality through the modality-specific embedding method, and obtain a low-level feature with sentiment information corresponding to the raw data of each modality, and the dimensions of the low-level features of each modality are unified;

[0009] S3. Use the low-level features of each modality to construct the homogeneous hyperedges and heterogeneous hyperedges of the corresponding modality, and construct a heterogeneous hypergraph from the nodes, homogeneous hyperedges and heterogeneous hyperedges of all modalities, where homogeneous hyperedges are used to model the interaction within the modality, and heterogeneous hyperedges are used for the interaction between modalities;

[0010] S4. After generating the initial hyperedge features of each homogeneous hyperedge and the initial hyperedge features of each heterogeneous hyperedge in the heterogeneous hypergraph, the heterogeneous hypergraph is input into the heterogeneous hypergraph network, the first node aggregation module outputs the first aggregation representation of all homogeneous hyperedges, the first aggregation representation is input into the first hyperedge diffusion module to obtain the first diffusion representation of all nodes, the first diffusion representation is input into the second node aggregation module to obtain the second aggregation representation of all heterogeneous hyperedges, and the second aggregation representation is input into the second hyperedge diffusion module to obtain the second diffusion representation of all nodes; wherein the first node aggregation module and the first hyperedge diffusion module are used to realize intra-modal interaction modeling based on homogeneous hyperedges, and the second node aggregation module and the second hyperedge diffusion module are used to realize inter-modal interaction modeling based on heterogeneous hyperedges;

[0011] S5. averaging the second aggregate representation and the second diffuse representation in the feature dimension to obtain a graph-level representation;

[0012] S6. Input the image-level representation into the prediction head and output the probability of the emotional polarity reflected in the audio video clip.

[0013] Based on the above scheme, each step can be implemented in the following preferred specific manner.

[0014] As a preferred embodiment of the first aspect, the specific method of obtaining low-level features of each modality through the modality-specific embedding method in S2 is:

[0015] S21. Input the video clip into the pre-trained Vision GNN, extract features from each video frame in the video clip, and splice the features corresponding to each video frame in time sequence to obtain a first feature, and then average pool the first feature according to the timestamp to obtain an initial visual feature;

[0016] S22. Input the audio clip into openSMILE for feature extraction to obtain a second feature, and average pool the second feature according to the timestamp to obtain an initial audio feature;

[0017] S23. Input the conversation text into the pre-trained BERT for embedding to obtain the initial text features;

[0018] S24. The initial text features, initial visual features, and initial audio features are each passed through a convolutional layer to unify the feature dimensions, and correspondingly obtain text low-level features, visual low-level features, and audio low-level features.

[0019] As a preferred embodiment of the first aspect, the specific process of constructing homogeneous hyperedges and heterogeneous hyperedges of each modality in S3 is as follows:

[0020] S31. Take each time step in the audio low-level features as an audio node, and for each audio node, take the k audio nodes closest to itself in time sequence as audio homogeneous associated nodes, and form a homogeneous hyperedge of the audio mode with each audio node and the corresponding audio homogeneous associated node, and construct a homogeneous hyperedge set of the audio mode from the homogeneous hyperedges of all audio modes;

[0021] S32. Take each time step in the visual low-level features as a visual node, and for each visual node, take the k visual nodes closest to itself in time sequence as visual homogeneous associated nodes, and form a homogeneous hyperedge of the visual modality from each visual node and the corresponding visual homogeneous associated node, and construct a homogeneous hyperedge set of the visual modality from the homogeneous hyperedges of all visual modalities;

[0022] S33. Take each time step in the low-level features of the text as a text node, input the conversation text into spaCy, and use spaCy to construct the dependency relationship between each word in the conversation text. For each text node, the k text nodes closest to itself in time sequence are taken as text homogeneous associated nodes, and the text nodes that have a dependency relationship with the words corresponding to each text node are taken as text dependent nodes. Each text node and the corresponding text homogeneous associated node and text dependent node form a homogeneous hyperedge of text modality, and the homogeneous hyperedge set of text modality is constructed from the homogeneous hyperedges of all text modalities;

[0023] S34. Mapping the low-level features of the text into the query space through a query matrix to obtain the query features of the text modality, and then mapping the low-level features of each modality into the pairing space through a modality-specific pairing matrix to obtain the pairing features of each modality;

[0024] S35. Perform vector dot products of the paired features of each modality with the query features of the text modality to obtain the association matrix of each modality;

[0025] S36. Finally, each node in the paired features of the text modality is taken as the text center node. For each text center node, the k nodes with the highest correlation with the text center node are queried in the association matrix of each modality as the heterogeneous associated nodes of the corresponding modality. A heterogeneous hyperedge is formed by each text center node and the heterogeneous associated nodes corresponding to each modality, and a heterogeneous hyperedge set is constructed by all heterogeneous hyperedges.

[0026] As a preferred embodiment of the above-mentioned first aspect, the specific process of the first node aggregation module in S4 outputting the first aggregate representation is as follows: the first, second and third attention heads are used to calculate the correlation of homogeneous hyperedges and nodes in the heterogeneous hypergraph, and the correlation results of each homogeneous hyperedge and each node in the homogeneous hyperedge are obtained in the first, second and third attention heads; the correlation results calculated by the first, second and third attention heads are added as the first correlation between each homogeneous hyperedge and each node in the homogeneous hyperedge, and the first correlation is normalized by the softmax function to obtain a first correlation score; the first correlation score is used as a weight to perform correlation-weighted summation on the pairing features of each node in the homogeneous hyperedge, and the first aggregate representation of each homogeneous hyperedge is obtained.

[0027] Furthermore, in the first attention head, the initial hyperedge feature of each homogeneous hyperedge is multiplied by the first parameter matrix to obtain the first feature, the paired features of each node of each homogeneous hyperedge are multiplied by the second parameter matrix to obtain the second feature, the first feature and the second feature are concatenated to obtain the third feature, the third feature is multiplied by the first vector parameter and then output through the elu activation function to obtain the correlation result output by the first attention head; in the second attention head, the first feature and the second feature are added and then output through the leakyRelu activation function to obtain the correlation result output by the second attention head; in the third attention head, the first feature and the second feature are subjected to the Hadamard product operation and the element-wise sum is calculated and then output through the tanh activation function to obtain the correlation result output by the third attention head.

[0028] As a preferred embodiment of the above-mentioned first aspect, the specific process of the first hyperedge diffusion module in S4 outputting the first diffusion representation is as follows: the fourth, fifth and sixth attention heads are used to calculate the correlation of homogeneous hyperedges and nodes in the heterogeneous hypergraph, and the correlation results of each node and each homogeneous hyperedge containing each node in the fourth, fifth and sixth attention heads are obtained; the correlation results calculated by the fourth, fifth and sixth attention heads are added as the second correlation between each node and each homogeneous hyperedge containing each node, and the second correlation is normalized by the softmax function to obtain a second correlation score; the second correlation score is used as a weight to perform correlation-weighted summation on the first aggregate representation of the homogeneous hyperedge containing each node to obtain an intra-modal interaction representation, and finally the intra-modal interaction representation is residually connected with the paired features of each node to obtain the first diffusion representation of each node.

[0029] Furthermore, in the fourth attention head, the first aggregate representation of each homogeneous hyperedge containing each node is multiplied by the third parameter matrix to obtain the fourth feature, the paired feature of each node is multiplied by the fourth parameter matrix to obtain the fifth feature, the fourth feature and the fifth feature are concatenated to obtain the sixth feature, the sixth feature is multiplied by the second vector parameter and then output through the elu activation function to obtain the correlation result output by the fourth attention head; in the fifth attention head, the fourth feature and the fifth feature are added and then output through the leakyRelu activation function to obtain the correlation result output by the fifth attention head; in the sixth attention head, the fourth feature and the fifth feature are subjected to the Hadamard product operation and the element-wise sum is calculated and then output through the tanh activation function to obtain the correlation result output by the sixth attention head.

[0030] As a preferred embodiment of the above-mentioned first aspect, the specific process of the second node aggregation module in S4 outputting the second aggregate representation is as follows: the seventh, eighth and ninth attention heads are used to calculate the correlation between the heterogeneous hyperedges and nodes in the heterogeneous hypergraph, and the correlation results between each heterogeneous hyperedge and each node in the heterogeneous hyperedge in the seventh, eighth and ninth attention heads are obtained; the correlation results calculated by the seventh, eighth and ninth attention heads are added as the third correlation between each heterogeneous hyperedge and each node in the heterogeneous hyperedge, and the third correlation is normalized by the softmax function to obtain a third correlation score; the third correlation score is used as a weight to perform correlation-weighted summation on the first diffusion representation of each node in the heterogeneous hyperedge, and the second aggregate representation of each heterogeneous hyperedge is obtained.

[0031] Further, in the seventh attention head, the initial hyperedge feature of each heterogeneous hyperedge is multiplied by the fifth parameter matrix to obtain the seventh feature, the first diffusion representation of each node of each heterogeneous hyperedge is multiplied by the sixth parameter matrix to obtain the eighth feature, the seventh feature is concatenated with the eighth feature to obtain the ninth feature, the ninth feature is multiplied by the third vector parameter and then output through the elu activation function to obtain the correlation result of each heterogeneous hyperedge with each node in each heterogeneous hyperedge in the seventh attention head. In the eighth attention head, the seventh feature is added to the eighth feature and then output through the leakyRelu activation function to obtain the correlation result of each heterogeneous hyperedge with each node in each heterogeneous hyperedge in the eighth attention head. In the ninth attention head, the seventh feature and the eighth feature are subjected to the Hadamard product operation and the element sum is calculated and then output through the tanh activation function to obtain the correlation result of each heterogeneous hyperedge with each node in each heterogeneous hyperedge in the ninth attention head.

[0032] As a preferred embodiment of the above-mentioned first aspect, the specific process of the second hyper-edge diffusion module in S4 outputting the second diffusion representation is as follows: the tenth, eleventh and twelfth attention heads calculate the correlation of the heterogeneous hyper-edges and nodes in the heterogeneous hypergraph to obtain the correlation results of each node and each heterogeneous hyper-edge containing each node in the tenth, eleventh and twelfth attention heads; the correlation results calculated by the tenth, eleventh and twelfth attention heads are added as the fourth correlation between each node and each heterogeneous hyper-edge containing each node, and the fourth correlation is normalized by the softmax function to obtain a fourth correlation score; the fourth correlation score is used as a weight to perform correlation-weighted summation on the second aggregation representation of each heterogeneous hyper-edge to obtain an inter-modal interaction representation, and finally the inter-modal interaction representation is residually connected with the first diffusion representation of each node to obtain the second diffusion representation of each node.

[0033] Furthermore, in the tenth attention head, the second aggregate representation of each heterogeneous hyperedge containing each node is multiplied by the seventh parameter matrix to obtain the tenth feature, the first diffusion representation of each node is multiplied by the eighth parameter matrix to obtain the eleventh feature, the tenth feature and the eleventh feature are concatenated to obtain the twelfth feature, the twelfth feature is multiplied by the fourth vector parameter and then output through the elu activation function to obtain the correlation result of each node and the heterogeneous hyperedge containing each node in the tenth attention head. In the eleventh attention head, the tenth feature and the eleventh feature are added and then output through the leakyRelu activation function to obtain the correlation result of each node and the heterogeneous hyperedge containing each node in the eleventh attention head. In the twelfth attention head, the tenth feature and the eleventh feature are subjected to the Hadamard product operation and the element sum is calculated and then output through the tanh activation function to obtain the correlation result of each node and the heterogeneous hyperedge containing each node in the twelfth attention head.

[0034] In a second aspect, the present invention provides a multimodal sentiment analysis system based on a dual-stage heterogeneous hypergraph, comprising:

[0035] A data acquisition module is used to acquire the original data of each mode in a video clip with sound;

[0036] The result acquisition module is used to generate the probability of the emotional polarity reflected in the audio video clip from the original data of each modality according to the multimodal sentiment analysis method based on the two-stage heterogeneous hypergraph as described in any scheme of the first method mentioned above.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1) The present invention proposes a multimodal sentiment analysis model based on a two-stage heterogeneous hypergraph. Considering the intricate interaction relationships within a modality and between different modalities, models such as MRRF and HCI can only capture the interaction information between two time steps within two modalities, and lack effective use of the inter-modal association. The multimodal sentiment analysis model of the present invention can use homogeneous hyperedges in heterogeneous hypergraphs to model super-paired interactions within a modality, and on the other hand, it can use heterogeneous hyperedges in heterogeneous hypergraphs to model super-paired interaction information between modalities, thereby improving the performance of the multimodal sentiment analysis model.

[0039] 2) The multimodal sentiment analysis model of the present invention combines the sentiment expression characteristics of different modalities to construct a heterogeneous hypergraph. Currently, hypergraphs are often constructed through K-nearest neighbor and K-means algorithms, which are not well suited for heterogeneous multimodal sequence data. The multimodal sentiment analysis model of the present invention targets the temporal association between audio and visual modalities, the dependency between text words, and the correlation between multiple modalities, and designs a new heterogeneous hypergraph construction method that is closely integrated with the multimodal sentiment analysis task. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of the steps of the method of the present invention;

[0041] Figure 2 Schematic diagram of the structure of the multimodal sentiment analysis model of the present invention;

[0042] Figure 3 Schematic diagram of the workflow of the multimodal sentiment analysis model of the present invention;

[0043] Figure 4 A schematic diagram of constructing a non-text modality homogeneous hyperedge according to an embodiment of the present invention;

[0044] Figure 5 A schematic diagram of constructing a text modality homogeneous hyperedge according to an embodiment of the present invention;

[0045] Figure 6 A schematic diagram of constructing a heterogeneous hyperedge according to an embodiment of the present invention;

[0046] Figure 7 It is a system block diagram of the present invention. DETAILED DESCRIPTION

[0047] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0048] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0049] like Figure 1 As shown, in a preferred implementation of the present invention, the above-mentioned multimodal sentiment analysis method based on a two-stage heterogeneous hypergraph includes the following steps S1 to S6, which are specifically as follows:

[0050] S1. Extracting raw data of each modality from a video clip with sound, wherein the module of the raw data includes an audio clip, a video clip, and a dialogue text.

[0051] It should be noted that, in order to facilitate the understanding of those skilled in the art, the main assumptions of multimodal sentiment analysis are first introduced in this embodiment. A piece of audio video can be regarded as a sequence data of three modes: vision (such as video clips), audio (such as audio clips) and text (such as dialogue text). The sequence data of each mode consists of multiple time steps. A single time step can be regarded as a node to construct a heterogeneous hypergraph. , used to model the interaction within and between modalities, the heterogeneous hypergraph consists of nodes and hyperedges. Represents a node set; a hyperedge set Represents the association between time steps; Respectively represent the video node set, audio node set and text node set; It means to find the union; Indicates The set of nodes contained in the hyperedge.

[0052] It should be noted that, in the present invention, a multimodal sentiment analysis model is constructed to process the original data of each modality acquired by S1. The specific processing flow in the multimodal sentiment analysis model is described in detail below.

[0053] The multimodal sentiment analysis model of the present invention accepts inputs in three modes: audio clips, video clips, and dialogue texts. The multimodal sentiment analysis model mainly consists of three parts: input-processing-output. Figure 2 The following is a brief introduction to the specific processing of each part.

[0054] The input part of the multimodal sentiment analysis model: The initial input of the multimodal sentiment analysis model is the original data of a single modality corresponding to an audio video clip (such as an audio clip, a dialogue text, a video clip). Through the modality-specific embedding method (audio embedding A-emb, text embedding T-emb, and visual embedding V-emb), low-level features (audio low-level features, text low-level features, and visual low-level features) with uniform feature dimension size and that can reflect emotional information are obtained.

[0055] The processing part of the multimodal sentiment analysis model: construct the low-level features of different modalities (audio low-level features, text low-level features and visual low-level features) into a heterogeneous hypergraph, and then use a multi-layer heterogeneous hypergraph network (HHGN) to model the hyper-pairwise interactions within and between modalities, and then model the graph-level representation through readout, and use the graph-level representation to make predictions.

[0056] Output part of the multimodal sentiment analysis model: Output of the multimodal sentiment analysis model is the probability of predicted sentiment polarity obtained based on the graph-level representation, The probability that the emotion reflected in the audio and video clips is positive, negative or neutral.

[0057] The functions of some of the symbols or functions used above will be explained in the specific implementation described later.

[0058] The processing part of the multimodal sentiment analysis model can be divided into four stages: Figure 3 As shown, they are the embedding stage, composition stage, interaction modeling and prediction stage.

[0059] 1) Embedding stage

[0060] S2. Input the raw data of each modality into the trained multimodal sentiment analysis model, and process the raw data of each modality through the modality-specific embedding method. The raw data of each modality corresponds to a low-level feature with sentiment information, and the dimensions of the low-level features of each modality are unified.

[0061] It should be noted that in S2 of the present invention, the specific process of obtaining low-level features of each modality through a modality-specific embedding method is as follows: the initial inputs of the multimodal sentiment analysis model audio modality, visual modality, and text modality correspond to audio clips, video clips, and corresponding dialogue texts, respectively. First, a feature extraction tool is used to extract low-level features that can reflect emotional information from the inputs of these three modalities (audio clips, video clips, and corresponding dialogue texts). Considering the large differences between the various modalities, different feature extraction tools should be used for different modalities.

[0062] For the visual modality: S21. Input the video clip into the pre-trained Vision GNN (ViG), extract features from each video frame in the video clip, and splice the features corresponding to each video frame in time sequence to obtain the first feature, and then average pool the first feature according to the timestamp to align it with a single word in the dialogue text to obtain the initial visual feature :

[0063]

[0064] in, Indicates alignment operation; is the text length of the dialogue text, i.e. the number of time steps; is the size of the initial visual feature dimension; Represents a video clip.

[0065] For audio modality: S22. Input the audio clip into openSMILE (open-source Speech and Music Interpretation by Large-space Extraction) for feature extraction. OpenSMILE extracts features that can reflect emotional changes, such as smooth fundamental frequency and pitch cycle length deviation, from the audio clip to obtain the second feature. The second feature also needs to be average-pooled according to the timestamp to align with the single word in the dialogue text to obtain the initial audio feature. :

[0066]

[0067] in, Indicates the size of the initial audio feature dimension; Represents an audio clip.

[0068] For text mode: S23. Input the conversation text into the pre-trained BERT for embedding to obtain the initial text features :

[0069]

[0070] in, is the size of the initial text feature dimension; For the dialogue text.

[0071] S24. Then, the features of the three modalities, namely, the initial text features, the initial visual features, and the initial audio features, are each passed through a convolutional layer with a convolution kernel size of 1 and no padding ( ) to unify the feature dimensions and obtain the corresponding low-level features of the text , visual low-level features And audio low-level features :

[0072]

[0073]

[0074]

[0075] in, Indicates the unified feature dimension size.

[0076] 2) Composition stage

[0077] S3. Use the low-level features of each modality to construct the homogeneous hyperedges and heterogeneous hyperedges of the corresponding modality, and construct a heterogeneous hypergraph from the nodes, homogeneous hyperedges and heterogeneous hyperedges of all modalities. Among them, homogeneous hyperedges are used to model the interaction within the modality, and heterogeneous hyperedges are used for the interaction between modalities.

[0078] It should be noted that in S3 of the present invention, considering that the low-level features of each modality are time steps: , , Therefore, each time step can be regarded as a node to construct a heterogeneous hypergraph. The heterogeneous hypergraph of the present invention includes homogeneous hyperedges and heterogeneous hyperedges, wherein homogeneous hyperedges only connect nodes of the same mode and are used to model the interaction within the mode; while heterogeneous hyperedges connect nodes from different modes and are used for the interaction between modes.

[0079] First, homogeneous hyperedges are constructed. Homogeneous hyperedges can be divided into homogeneous hyperedges of text modality, homogeneous hyperedges of audio modality, and homogeneous hyperedges of visual modality.

[0080] The homogeneous hyperedges of audio and visual modalities have similar construction methods, that is, a homogeneous hyperedge of the corresponding modality is constructed with an audio node or a visual node as the center, which means that the audio and visual modalities will each have 𝐿 homogeneous hyperedges. Specifically:

[0081] S31. Take each time step in the audio low-level features as an audio node, and for each audio node, take the k audio nodes closest to itself in time sequence as audio homogeneous associated nodes, and form a homogeneous hyperedge of the audio mode with each audio node and the corresponding audio homogeneous associated node, and construct a homogeneous hyperedge set of the audio mode from the homogeneous hyperedges of all audio modes;

[0082] S32. Take each time step in the visual low-level features as a visual node. For each visual node, take the k visual nodes closest to itself in time sequence as visual homogeneous associated nodes. Each visual node and the corresponding visual homogeneous associated nodes form a homogeneous hyperedge of the visual modality. The homogeneous hyperedges of all visual modalities construct a homogeneous hyperedge set of the visual modality.

[0083] In this embodiment, if Figure 4 As shown, the audio node in the audio low-level feature For example, considering that the audio modality and the visual modality have a major correlation in the time dimension, we need to take k=3 The closest k audio nodes (excluding self) and Together they form a homogeneous hyperedge of the audio mode .in, Respectively represent The closest audio nodes, i.e., homogeneous hyperedges of the audio modality The audio nodes. Similarly, Figure 4 An example of a homogeneous hyperedge of visual modality is also given in the paper, with visual nodes in the visual low-level features As the center, select the k visual nodes closest to it in time sequence, and Together they form a homogeneous hyperedge of visual modality ,in, Respectively represent The closest visual nodes, i.e. homogeneous hyperedges of visual modality The According to this construction method, we can get a homogeneous hyperedge set of audio modality and a homogeneous set of hyperedges of the visual modality ,in, Respectively represent A homogeneous hyperedge built with audio nodes as the center; Respectively represent A homogeneous hyperedge is constructed with visual nodes as the center.

[0084] For text modality, we need to consider not only the temporal association of different words, but also the semantic association between words. Therefore, we first use spaCy to construct the dependencies between words in text modality. Using this dependency, when constructing homogeneous hyperedges, we can not only consider the temporal association, but also the semantic association information. Specifically:

[0085] S33. Take each time step in the low-level features of the text as a text node, input the conversation text into spaCy, and use spaCy to construct the dependency relationship between each word in the conversation text. For each text node, take the k text nodes closest to itself in time sequence as text homogeneous associated nodes, and take the text nodes that have a dependency relationship with the words corresponding to each text node as text dependent nodes. Each text node and the corresponding text homogeneous associated node and text dependent node form a homogeneous hyperedge of the text modality, and the homogeneous hyperedge set of the text modality is constructed by the homogeneous hyperedges of all text modalities.

[0086] In this embodiment, if Figure 5 As shown, the text nodes in the low-level features of the text For example, we also need to take k=3 times in the timing The closest text node (excluding itself), in addition, you need to add The text nodes for which the corresponding words have dependencies (i.e., text-dependent node), thus obtaining a homogeneous hyperedge of text modality .in, Respectively represent The closest text nodes, i.e. homogeneous hyperedges of text modality The According to this construction method, we can get a homogeneous hyperedge set of text modality ,in, Respectively represent A homogeneous hyperedge constructed with text nodes as the center.

[0087] Then we start to build heterogeneous hyperedges. Since the text modality has less noise than the audio modality and the visual modality, each time we build heterogeneous hyperedges with the text nodes of different text modalities as the center, that is, there will be a total of 𝐿 heterogeneous hyperedges. The present invention attempts to build heterogeneous hyperedges by calculating the correlation between the central node and other non-central nodes. Specifically:

[0088] S34. Through a query matrix Set the text low-level features of all text nodes Map into the query space to obtain the query features of the text modality , and then map the text low-level features, visual low-level features, and audio low-level features into the pairing space through the modality-specific pairing matrix to obtain the pairing features corresponding to each modality:

[0089]

[0090]

[0091]

[0092]

[0093] in, Paired features representing text modality; Represents paired features of audio modalities; Representing paired features of visual modalities; A pairing matrix representing the text modality; A pairing matrix representing the audio modality; Represents the pairing matrix of visual modalities.

[0094] S35. Then use the vector dot product to calculate the non-central node of the paired features of each modality and the query features of the text modality The correlation between the central nodes gives the correlation matrix of each mode:

[0095]

[0096]

[0097]

[0098] in, is the association matrix of the text modality; is the correlation matrix of the audio mode; is the correlation matrix of the visual modality; Indicates transpose.

[0099] S36. Finally, each node in the paired features of the text modality is used as the text center node. For each text center node, the k nodes with the highest correlation with the text center node are queried in the association matrix of each modality as the heterogeneous associated nodes of the corresponding modality. Each text center node and the heterogeneous associated nodes corresponding to each modality form a heterogeneous hyperedge, and a heterogeneous hyperedge set is constructed from all heterogeneous hyperedges. .in, Respectively represent A heterogeneous hyperedge is constructed with 1 text node as the text center node.

[0100] In this embodiment, if Figure 6 As shown, the text center node For example, in the association matrix of each modality, k=3 nodes with the highest correlation with the text center node are queried as the heterogeneous association nodes of the corresponding modality: The heterogeneous association nodes of the text modality are , the heterogeneous association nodes of the audio modality are , the heterogeneous association nodes of the visual modality are Each text center node and the heterogeneous associated nodes corresponding to each modality form a heterogeneous hyperedge .

[0101] 3) Interaction modeling stage

[0102] S4. After generating the initial hyperedge features of each homogeneous hyperedge and the initial hyperedge features of each heterogeneous hyperedge in the heterogeneous hypergraph, the heterogeneous hypergraph is input into the heterogeneous hypergraph network, the first node aggregation module outputs the first aggregation representation of all homogeneous hyperedges, the first aggregation representation is input into the first hyperedge diffusion module to obtain the first diffusion representation of all nodes, the first diffusion representation is input into the second node aggregation module to obtain the second aggregation representation of all heterogeneous hyperedges, and the second aggregation representation is input into the second hyperedge diffusion module to obtain the second diffusion representation of all nodes; wherein the first node aggregation module and the first hyperedge diffusion module are used to realize intra-modal interaction modeling based on homogeneous hyperedges, and the second node aggregation module and the second hyperedge diffusion module are used to realize inter-modal interaction modeling based on heterogeneous hyperedges.

[0103] It should be noted that the specific process of generating the initial hyperedge features of homogeneous hyperedges and heterogeneous hyperedges in S4 is as follows: the paired features of all nodes in each homogeneous hyperedge of the heterogeneous hypergraph are averaged pooled in the feature dimension to obtain the initial hyperedge features of each homogeneous hyperedge; the paired features of all nodes in each heterogeneous hyperedge of the heterogeneous hypergraph are averaged pooled in the feature dimension to obtain the initial hyperedge features of each heterogeneous hyperedge.

[0104] Before interactive modeling in this embodiment, it is necessary to generate initial hyperedge features for all hyperedges in the heterogeneous hypergraph. The paired features of all nodes in the homogeneous hyperedge are averaged and pooled in the feature dimension to obtain Initial hyperedge features of homogeneous hyperedges , for heterogeneous hypergraphs The paired features of all nodes in the heterogeneous hyperedge are averaged and pooled in the feature dimension to obtain Initial hyperedge features of heterogeneous hyperedges :

[0105]

[0106]

[0107] in, represents average pooling; , Respectively represent Homogeneous Hyperedge , No. Heterogeneous Hyperedge The node index in ; , Respectively represent homogeneous hyperedge, The set of nodes in a heterogeneous hyperedge; Indicates Homogeneous hyperedge The pairing characteristics of nodes; Indicates Heterogeneous superedge The pairing characteristics of the nodes.

[0108] It should be noted that the multimodal sentiment analysis model models interactive information through a heterogeneous hypergraph network. The operation of the heterogeneous hypergraph network can be divided into two stages in order: the intra-modal interaction modeling stage based on homogeneous hyperedges and the inter-modal interaction modeling stage based on heterogeneous hyperedges. The intra-modal interaction modeling stage can be divided into two processes: homogeneous node aggregation and homogeneous hyperedge diffusion, which are correspondingly implemented by the first node aggregation module and the first hyperedge diffusion module. The inter-modal interaction modeling stage can be divided into two processes: heterogeneous node aggregation and heterogeneous hyperedge diffusion, which are correspondingly implemented by the second node aggregation module and the second hyperedge diffusion module. The above processes are introduced in detail below.

[0109] A) Homogeneous node aggregation process. In the present invention, the node information in the homogeneous hyperedge is used to update the homogeneous hyperedge features through the multi-head attention mechanism, so it is necessary to calculate the correlation between all nodes in the homogeneous hyperedge and the homogeneous hyperedge. Specifically, the first, second and third attention heads first calculate the correlation between the homogeneous hyperedges and nodes in the heterogeneous hypergraph, and obtain the correlation results of each homogeneous hyperedge and each node in the homogeneous hyperedge in the first, second and third attention heads; then the correlation results calculated by the first, second and third attention heads are added as the first correlation between each homogeneous hyperedge and each node in the homogeneous hyperedge, and the first correlation is normalized by the softmax function to obtain the first correlation score; finally, the first correlation score is used as a weight to perform weighted summation of the correlation of the paired features of each node in the homogeneous hyperedge to obtain the first aggregation representation of each homogeneous hyperedge.

[0110] In this embodiment, each attention head has its own calculation method. Specifically, in the first attention head, Initial hyperedge features of homogeneous hyperedges With the first parameter matrix Multiply them to get the first characteristic, and Homogeneous hyperedge Pairing characteristics of nodes With the second parameter matrix Multiply them to get the second feature, concatenate the first feature with the second feature to get the third feature, and concatenate the third feature with the first vector parameter After multiplication, the output is passed through the elu activation function to obtain the homogeneous hyperedge and Homogeneous hyperedge The relevance results of nodes in the first attention head ,in, In the second attention head, the first feature is added to the second feature and then output through the leakyRelu activation function to obtain the first homogeneous hyperedge and Homogeneous hyperedge The correlation results of nodes in the second attention head In the third attention head, the first feature and the second feature are subjected to the Hadamard product operation and the element sum is calculated and then output through the tanh activation function to obtain the first homogeneous hyperedge and Homogeneous hyperedge The correlation results of nodes in the third attention head ,in, is the Haddam product; sum means the sum of the elements.

[0111] Then add the correlation results calculated by the first, second and third attention heads as the first homogeneous hyperedge and Homogeneous hyperedge The first relevance of the node , and then normalize the first correlation through the softmax function to get the homogeneous hyperedge and Homogeneous hyperedge The first correlation score between nodes :

[0112]

[0113] in, Indicates A homogeneous hyperedge and Homogeneous hyperedge The first relevance of the nodes.

[0114] Finally, the homogeneous hyperedge and Homogeneous hyperedge The first correlation score between nodes As a weight, Homogeneous hyperedge Pairing characteristics of nodes Perform weighted summation of correlation to update the The characteristics of homogeneous hyperedges are obtained The first aggregate characterization of homogeneous hyperedges:

[0115]

[0116] B) Homogeneous hyperedge diffusion process. In the present invention, hyperedge information is used to update node features through homogeneous hyperedge diffusion. Therefore, it is necessary to calculate the correlation between each node and the homogeneous hyperedge containing each node for each node, and the multi-head attention mechanism is also used. Specifically, the fourth, fifth and sixth attention heads first calculate the correlation between the homogeneous hyperedges and nodes in the heterogeneous hypergraph, and obtain the correlation results of each node and each homogeneous hyperedge containing each node in the fourth, fifth and sixth attention heads; then the correlation results calculated by the fourth, fifth and sixth attention heads are added as the second correlation between each node and each homogeneous hyperedge containing each node, and then the second correlation is normalized by the softmax function to obtain the second correlation score; finally, the second correlation score is used as a weight to perform a weighted summation of the correlation of the first aggregate representation of the homogeneous hyperedge containing each node to obtain an intra-modal interaction representation, and the intra-modal interaction representation is residually connected with the paired features of each node to obtain the first diffusion representation of each node.

[0117] In this embodiment, each attention head has its own calculation method. Specifically, in the fourth attention head, the first The node The first aggregate representation of homogeneous hyperedges With the third parameter matrix Multiply them together to get the fourth characteristic, and Pairing characteristics of nodes With the fourth parameter matrix Multiply them to get the fifth feature, concatenate the fourth and fifth features to get the sixth feature, and then add the sixth feature to the second vector parameter After multiplication, the output is passed through the elu activation function to obtain the The node contains The node The correlation results of homogeneous hyperedges in the fourth attention head In the fifth attention head, the fourth feature and the fifth feature are added and then output through the leakyRelu activation function to obtain the first The node contains The node The correlation results of homogeneous hyperedges in the fifth attention head In the sixth attention head, the fourth feature and the fifth feature are processed by the Hadamard product operation and then output by the tanh activation function to obtain the The node contains The node The correlation results of homogeneous hyperedges in the sixth attention head .

[0118] Then add the correlation results calculated by the fourth, fifth and sixth attention heads as the first The node contains The node The second correlation of homogeneous hyperedges , and then normalize the second correlation through the softmax function to get the first The node contains The node The second correlation score between homogeneous hyperedges :

[0119]

[0120] in, Indicates that all Hyperedge index of homogeneous hyperedges of nodes; Indicates that all The hyperedge set of homogeneous hyperedges of nodes; Indicates The node contains The node The second correlation of homogeneous hyperedges.

[0121] Then the The node contains The node The second correlation score between homogeneous hyperedges As weight, The node The first aggregate representation of homogeneous hyperedges Perform weighted summation of correlation to update the The features of each node are used to obtain the intra-modal interaction representation.

[0122] Finally, the present invention also adds a residual connection to retain the Specifically, the intra-modal interaction representation is combined with the The paired features of the nodes are residually connected to obtain the The first diffusion representation of nodes :

[0123]

[0124] C) Heterogeneous node aggregation process. The calculation method of heterogeneous node aggregation is similar to that of homogeneous node aggregation, except that the processing of homogeneous hyperedges is replaced by the processing of heterogeneous hyperedges. The multi-head attention mechanism is also used to update the heterogeneous hyperedge features using the node information in the heterogeneous hyperedge, so it is necessary to calculate the correlation between all nodes in the heterogeneous hyperedge and the heterogeneous hyperedge. It can be inferred from the processing flow of the first node aggregation module that the specific process of the second node aggregation module outputting the second aggregate representation is as follows: the seventh, eighth and ninth attention heads are used to calculate the correlation between the heterogeneous hyperedges and nodes in the heterogeneous hypergraph, and the correlation results between each heterogeneous hyperedge and each node in the heterogeneous hyperedge in the seventh, eighth and ninth attention heads are obtained; the correlation results calculated by the seventh, eighth and ninth attention heads are added as the third correlation between each heterogeneous hyperedge and each node in the heterogeneous hyperedge, and the third correlation is normalized by the softmax function to obtain the third correlation score; the third correlation score is used as a weight to perform correlation-weighted summation on the paired features of each node in the heterogeneous hyperedge, and the second aggregate representation of each heterogeneous hyperedge is obtained.

[0125] Similarly, in this embodiment, the seventh, eighth and ninth attention heads have the same processing flow as the first, second and third attention heads, except that the parameter matrices of the former and the latter are different, and the first, second and third attention heads process homogeneous hyperedges, while the seventh, eighth and ninth attention heads process heterogeneous hyperedges. Initial hyperedge features of heterogeneous hyperedges With the fifth parameter matrix Multiply them together to get the seventh characteristic, and Heterogeneous Superedge The first diffusion representation of nodes With the sixth parameter matrix Multiply them to get the eighth feature, concatenate the seventh and eighth features to get the ninth feature, and then concatenate the ninth feature with the third vector parameter After multiplication, the output is passed through the elu activation function to obtain the Heterogeneous hyperedge and Heterogeneous superedge The correlation results of the nodes in the seventh attention head In the eighth attention head, the seventh feature is added to the eighth feature and then output through the leakyRelu activation function to obtain Heterogeneous hyperedge and Heterogeneous superedge The correlation results of the nodes in the eighth attention head In the ninth attention head, the seventh feature and the eighth feature are processed through the Hadamard product operation and then output through the tanh activation function to obtain the Heterogeneous hyperedge and Heterogeneous superedge The correlation results of the nodes in the ninth attention head .

[0126] Then add the correlation results calculated by the seventh, eighth and ninth attention heads as the Heterogeneous hyperedge and Heterogeneous superedge The third relevance of nodes , and then normalize the third correlation through the softmax function to get the Heterogeneous hyperedge and Heterogeneous superedge The third correlation score between nodes :

[0127]

[0128] in, Indicates Heterogeneous hyperedge and Heterogeneous superedge The third relevance of the nodes.

[0129] Finally, the Heterogeneous hyperedge and Heterogeneous superedge The third correlation score between nodes As a weight, Heterogeneous Superedge The first diffusion representation of nodes Perform weighted summation of correlation to update the The characteristics of heterogeneous hyperedges are obtained The second aggregate representation of heterogeneous hyperedges :

[0130]

[0131] D) Heterogeneous hyperedge diffusion process. The calculation method of heterogeneous hyperedge diffusion is similar to that of homogeneous hyperedge diffusion, except that the processing of homogeneous hyperedges is replaced by the processing of heterogeneous hyperedges. The hyperedge information is also used to update the node features through the multi-head attention mechanism, so it is necessary to calculate the correlation between each node and the heterogeneous hyperedge containing each node. It can be inferred from the processing flow of the first hyperedge diffusion module that the specific process of the second hyperedge diffusion module outputting the second diffusion representation is as follows: the tenth, eleventh and twelfth attention heads are used to calculate the correlation of the heterogeneous hyperedges and nodes in the heterogeneous hypergraph, and the correlation results of each node and each heterogeneous hyperedge containing each node in the tenth, eleventh and twelfth attention heads are obtained; the correlation results calculated by the tenth, eleventh and twelfth attention heads are added as the fourth correlation between each node and each heterogeneous hyperedge containing each node, and then the fourth correlation is normalized by the softmax function to obtain a fourth correlation score; the fourth correlation score is used as a weight to perform correlation weighted summation on the second aggregation representation of each heterogeneous hyperedge to obtain an inter-modal interaction representation, and finally the inter-modal interaction representation is residually connected with the first diffusion representation of each node to obtain the second diffusion representation of each node.

[0132] Similarly, in this embodiment, the processing flow of the tenth, eleventh and twelfth attention heads is the same as that of the fourth, fifth and sixth attention heads, except that the parameter matrices of the former and the latter are different, and the fourth, fifth and sixth attention heads process homogeneous hyperedges, while the tenth, eleventh and twelfth attention heads process heterogeneous hyperedges. Specifically, in the tenth attention head, the first node containing the jth node is The second aggregate representation of heterogeneous hyperedges With the seventh parameter matrix Multiply them together to get the tenth characteristic, and The first diffusion representation of nodes With the eighth parameter matrix Multiply them to get the eleventh feature, concatenate the tenth feature with the eleventh feature to get the twelfth feature, and concatenate the twelfth feature with the fourth vector parameter After multiplication, the output is passed through the elu activation function to obtain the The node contains The node The correlation results of heterogeneous hyperedges in the tenth attention head In the eleventh attention head, the tenth feature and the eleventh feature are added and then output through the leakyRelu activation function to obtain the first The node contains The node The correlation results of heterogeneous hyperedges in the eleventh attention head In the twelfth attention head, the tenth feature and the eleventh feature are processed through the Hadamard product operation and then output through the tanh activation function to obtain the first The node contains The node The correlation results of heterogeneous hyperedges in the twelfth attention head .

[0133] Then add the correlation results calculated by the tenth, eleventh and twelfth attention heads as the first The node contains The node The fourth correlation of heterogeneous hyperedges , and then normalize the fourth correlation through the softmax function to get the The node contains The node The fourth correlation score between heterogeneous hyperedges :

[0134]

[0135] in, Indicates that all Hyperedge index of heterogeneous hyperedges of nodes; Indicates that all The hyperedge set of heterogeneous hyperedges with nodes; Indicates The node contains The node The fourth correlation of a heterogeneous hyperedge.

[0136] Then the The node contains The node The fourth correlation score between heterogeneous hyperedges As weight, The node The second aggregate representation of heterogeneous hyperedges Perform weighted summation of correlation to update the The features of each node are used to obtain the interaction representation between modalities.

[0137] Finally, the inter-modal interaction representation is combined with the The first diffusion representation of the nodes is residually connected to obtain the The second diffusion representation of nodes :

[0138]

[0139] S5. Averaging the second aggregate representation and the second diffuse representation in the feature dimension to obtain a graph-level representation.

[0140] It should be noted that in S5 of the present invention, after the second diffusion representations of all nodes and the second aggregation representations of all heterogeneous hyperedges are averaged in the feature dimension for reading, the following graph-level representation is obtained:

[0141]

[0142] 4) Prediction stage

[0143] S6. Input the image-level representation into the prediction head to obtain the probability of the emotional polarity reflected in the audio video clip.

[0144] It should be noted that in S6 of the present invention, the multimodal sentiment analysis model takes the final graph-level representation for prediction, first passes it through a relu activation function to obtain a new graph-level representation, and then passes the new graph-level representation through a feedforward layer and the softmax function outputs the final prediction probability vector, that is, the probability of the emotional polarity reflected in the audio video clip:

[0145]

[0146] in, represents a feed-forward layer.

[0147] It should also be noted that when training the multimodal sentiment analysis model DH2B, the present invention takes into account that it is a fully differentiable model. Therefore, the present invention optimizes the multimodal sentiment analysis model through the gradient propagation algorithm. When training downstream tasks, the cross entropy loss can be used as the loss function of the multimodal sentiment analysis model. The training of the multimodal sentiment analysis model can be completed by reducing the value of the cross entropy loss. The above cross entropy loss The functional form is:

[0148]

[0149] in, For the Samples belong to emotion One-hot encoded labels of ; For the samples are predicted to be sentiment probability.

[0150] The present invention will now use a specific example to demonstrate the application effect of a multimodal sentiment analysis method based on a two-stage heterogeneous hypergraph described in the above embodiment on a specific data set, so as to facilitate understanding of the essence of the present invention.

[0151] Example

[0152] The specific implementation process of the multimodal sentiment analysis method based on a two-stage heterogeneous hypergraph adopted in this embodiment is as described above and will not be repeated here.

[0153] In order to prove the practical effect of the present invention, the present invention is verified on the MOSI and MOSEI datasets and compared with other methods. Table 1 and Table 2 show the effect of the present invention on sentiment analysis of these two datasets and the comparison results with other methods.

[0154] Table 1. Actual effect of the present invention on the MOSI dataset

[0155]

[0156] Table 2. Actual effect of the present invention on the MOSEI dataset

[0157]

[0158] The methods compared with the present invention include a text-enhanced transformer-based model (TeTFA), a transformer-based multimodal binding learning model (TMBL), a cross-modal feature translation and dynamic propagation-based model (PEST), and a visual language pre-training-based model (VLP2MSA). These four methods compared with the present invention are all prior arts and will not be described in detail.

[0159] From the comparison of the results with other methods in Tables 1 and 2, it can be seen that the present invention has improved in all four indicators, which shows that the present invention can more effectively perform sentiment analysis tasks by utilizing the hyperpaired interaction information within and between modalities. In summary, the present invention uses hyperedges to model the interaction within and between modalities, effectively breaking through the limitation of paired interactions of previous methods. In addition, hyperedges are also used as local interaction information.

[0160] It should also be noted that the multimodal sentiment analysis method based on the dual-stage heterogeneous hypergraph in the above embodiment can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a multimodal sentiment analysis system based on a dual-stage heterogeneous hypergraph corresponding to the multimodal sentiment analysis method based on the dual-stage heterogeneous hypergraph provided in the above embodiment, such as Figure 7 As shown, it includes:

[0161] A data acquisition module is used to acquire the original data of each mode in a video clip with sound;

[0162] The result acquisition module is used to generate the probability of the emotional polarity reflected in the audio video clip from the original data of each modality according to the multimodal sentiment analysis method based on the two-stage heterogeneous hypergraph described in the above embodiment.

[0163] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A multimodal sentiment analysis method based on a two-stage heterogeneous hypergraph, characterized in that: The following steps are involved: S1. extracting raw data of each modality from a video clip with sound, wherein the module of the raw data includes an audio clip, a video clip, and a dialogue text; S2. Input the raw data of each modality into the trained multimodal sentiment analysis model, process the raw data of each modality through the modality-specific embedding method, and obtain a low-level feature with sentiment information corresponding to the raw data of each modality, and the dimensions of the low-level features of each modality are unified; S3. Use the low-level features of each modality to construct the homogeneous hyperedges and heterogeneous hyperedges of the corresponding modality, and construct a heterogeneous hypergraph from the nodes, homogeneous hyperedges and heterogeneous hyperedges of all modalities, where homogeneous hyperedges are used to model the interaction within the modality, and heterogeneous hyperedges are used for the interaction between modalities; S4. After generating the initial hyperedge features of each homogeneous hyperedge and the initial hyperedge features of each heterogeneous hyperedge in the heterogeneous hypergraph, the heterogeneous hypergraph is input into the heterogeneous hypergraph network, the first node aggregation module outputs the first aggregation representation of all homogeneous hyperedges, the first aggregation representation is input into the first hyperedge diffusion module to obtain the first diffusion representation of all nodes, the first diffusion representation is input into the second node aggregation module to obtain the second aggregation representation of all heterogeneous hyperedges, and the second aggregation representation is input into the second hyperedge diffusion module to obtain the second diffusion representation of all nodes; wherein the first node aggregation module and the first hyperedge diffusion module are used to realize intra-modal interaction modeling based on homogeneous hyperedges, and the second node aggregation module and the second hyperedge diffusion module are used to realize inter-modal interaction modeling based on heterogeneous hyperedges; S5. averaging the second aggregate representation and the second diffuse representation in the feature dimension to obtain a graph-level representation; S6. Input the image-level representation into the prediction head, and output the probability of the emotional polarity reflected in the audio video clip; The specific way to obtain the low-level features of each modality through the modality-specific embedding method in S2 is: S21. Input the video clip into the pre-trained Vision GNN, extract features from each video frame in the video clip, and splice the features corresponding to each video frame in time sequence to obtain a first feature, and then average pool the first feature according to the timestamp to obtain an initial visual feature; S22. Input the audio clip into openSMILE for feature extraction to obtain a second feature, and average pool the second feature according to the timestamp to obtain an initial audio feature; S23. Input the conversation text into the pre-trained BERT for embedding to obtain the initial text features; S24. The initial text features, initial visual features, and initial audio features are each passed through a convolutional layer to unify the feature dimensions, and correspondingly obtain text low-level features, visual low-level features, and audio low-level features.

2. A multimodal sentiment analysis method based on a two-stage heterogeneous hypergraph as claimed in claim 1, characterized in that: The specific process of constructing homogeneous hyperedges and heterogeneous hyperedges of each modality in S3 is as follows: S31. Take each time step in the audio low-level features as an audio node, and for each audio node, take the k audio nodes closest to itself in time sequence as audio homogeneous associated nodes, and form a homogeneous hyperedge of the audio mode with each audio node and the corresponding audio homogeneous associated node, and construct a homogeneous hyperedge set of the audio mode from the homogeneous hyperedges of all audio modes; S32. Take each time step in the visual low-level features as a visual node, and for each visual node, take the k visual nodes closest to itself in time sequence as visual homogeneous associated nodes, and form a homogeneous hyperedge of the visual modality from each visual node and the corresponding visual homogeneous associated node, and construct a homogeneous hyperedge set of the visual modality from the homogeneous hyperedges of all visual modalities; S33. Take each time step in the low-level features of the text as a text node, input the conversation text into spaCy, and use spaCy to construct the dependency relationship between each word in the conversation text. For each text node, the k text nodes closest to itself in time sequence are taken as text homogeneous associated nodes, and the text nodes that have a dependency relationship with the words corresponding to each text node are taken as text dependent nodes. Each text node and the corresponding text homogeneous associated node and text dependent node form a homogeneous hyperedge of text modality, and the homogeneous hyperedge set of text modality is constructed from the homogeneous hyperedges of all text modalities; S34. Mapping the low-level features of the text into the query space through a query matrix to obtain the query features of the text modality, and then mapping the low-level features of each modality into the pairing space through the pairing matrices of the audio modality, the visual modality and the text modality, respectively, to obtain the pairing features of each modality; S35. Perform vector dot products of the paired features of each modality with the query features of the text modality to obtain the association matrix of each modality; S36. Finally, each node in the paired features of the text modality is taken as the text center node. For each text center node, the k nodes with the highest correlation with the text center node are queried in the association matrix of each modality as the heterogeneous associated nodes of the corresponding modality. A heterogeneous hyperedge is formed by each text center node and the heterogeneous associated nodes corresponding to each modality, and a heterogeneous hyperedge set is constructed by all heterogeneous hyperedges.

3. A multimodal sentiment analysis method based on a two-stage heterogeneous hypergraph as claimed in claim 2, characterized in that: The specific process of the first node aggregation module in S4 outputting the first aggregate representation is as follows: the first, second and third attention heads are used to calculate the correlation between homogeneous hyperedges and nodes in the heterogeneous hypergraph, and the correlation results between each homogeneous hyperedge and each node in the homogeneous hyperedge in the first, second and third attention heads are obtained; the correlation results calculated by the first, second and third attention heads are added as the first correlation between each homogeneous hyperedge and each node in the homogeneous hyperedge, and the first correlation is normalized by the softmax function to obtain a first correlation score; the first correlation score is used as a weight to perform weighted summation of the correlation of the paired features of each node in the homogeneous hyperedge, and the first aggregate representation of each homogeneous hyperedge is obtained.

4. The multimodal sentiment analysis method based on a two-stage heterogeneous hypergraph as claimed in claim 3, characterized in that: In the first attention head, the initial hyperedge feature of each homogeneous hyperedge is multiplied by the first parameter matrix to obtain the first feature, the paired features of each node of each homogeneous hyperedge are multiplied by the second parameter matrix to obtain the second feature, the first feature is concatenated with the second feature to obtain the third feature, the third feature is multiplied by the first vector parameter and then output through the elu activation function to obtain the correlation result output by the first attention head; in the second attention head, the first feature is added to the second feature and then output through the leakyRelu activation function to obtain the correlation result output by the second attention head; in the third attention head, the first feature and the second feature are subjected to the Hadamard product operation and the element sum is calculated and then output through the tanh activation function to obtain the correlation result output by the third attention head.

5. The multimodal sentiment analysis method based on a two-stage heterogeneous hypergraph as claimed in claim 2, characterized in that: The specific process of the first hyperedge diffusion module in S4 outputting the first diffusion representation is as follows: the fourth, fifth and sixth attention heads calculate the correlation of homogeneous hyperedges and nodes in the heterogeneous hypergraph to obtain the correlation results of each node and each homogeneous hyperedge containing each node in the fourth, fifth and sixth attention heads; the correlation results calculated by the fourth, fifth and sixth attention heads are added as the second correlation between each node and each homogeneous hyperedge containing each node, and then the second correlation is normalized by the softmax function to obtain a second correlation score; the second correlation score is used as a weight to perform correlation weighted summation on the first aggregate representation of the homogeneous hyperedge containing each node to obtain the intra-modal interaction representation, and finally the intra-modal interaction representation is residually connected with the paired features of each node to obtain the first diffusion representation of each node.

6. A multimodal sentiment analysis method based on a two-stage heterogeneous hypergraph as claimed in claim 5, characterized in that: In the fourth attention head, the first aggregate representation of each homogeneous hyperedge containing each node is multiplied by the third parameter matrix to obtain the fourth feature, the paired feature of each node is multiplied by the fourth parameter matrix to obtain the fifth feature, the fourth feature and the fifth feature are concatenated to obtain the sixth feature, the sixth feature is multiplied by the second vector parameter and then output through the elu activation function to obtain the correlation result output by the fourth attention head; in the fifth attention head, the fourth feature and the fifth feature are added and then output through the leakyRelu activation function to obtain the correlation result output by the fifth attention head; in the sixth attention head, the fourth feature and the fifth feature are subjected to the Hadamard product operation and the element-wise sum is calculated and then output through the tanh activation function to obtain the correlation result output by the sixth attention head.

7. A multimodal sentiment analysis method based on a two-stage heterogeneous hypergraph as claimed in claim 2, characterized in that: The specific process of the second node aggregation module in S4 outputting the second aggregate representation is as follows: the seventh, eighth and ninth attention heads are used to calculate the correlation between the heterogeneous hyperedges and nodes in the heterogeneous hypergraph, and the correlation results between each heterogeneous hyperedge and each node in the heterogeneous hyperedge in the seventh, eighth and ninth attention heads are obtained; the correlation results calculated by the seventh, eighth and ninth attention heads are added as the third correlation between each heterogeneous hyperedge and each node in the heterogeneous hyperedge, and the third correlation is normalized by the softmax function to obtain a third correlation score; the third correlation score is used as a weight to perform correlation weighted summation on the first diffusion representation of each node in the heterogeneous hyperedge, and the second aggregate representation of each heterogeneous hyperedge is obtained.

8. The multimodal sentiment analysis method based on a two-stage heterogeneous hypergraph as claimed in claim 2, characterized in that: The specific process of the second hyperedge diffusion module in S4 outputting the second diffusion representation is as follows: the tenth, eleventh and twelfth attention heads calculate the correlation of the heterogeneous hyperedges and nodes in the heterogeneous hypergraph to obtain the correlation results of each node and each heterogeneous hyperedge containing each node in the tenth, eleventh and twelfth attention heads; the correlation results calculated by the tenth, eleventh and twelfth attention heads are added as the fourth correlation between each node and each heterogeneous hyperedge containing each node, and then the fourth correlation is normalized by the softmax function to obtain a fourth correlation score; the fourth correlation score is used as a weight to perform correlation weighted summation on the second aggregation representation of each heterogeneous hyperedge to obtain an inter-modal interaction representation, and finally the inter-modal interaction representation is residually connected with the first diffusion representation of each node to obtain the second diffusion representation of each node.

9. A multimodal sentiment analysis system based on a two-stage heterogeneous hypergraph, characterized in that: include: A data acquisition module is used to acquire the original data of each mode in a video clip with sound; The result acquisition module is used to generate the probability of the emotional polarity reflected in the audio video clip from the original data of each modality according to the multimodal sentiment analysis method based on the two-stage heterogeneous hypergraph as described in any one of claims 1 to 8.

Images