A sentiment recognition method and system based on a double hypergraph neural network

Abstract

The application discloses a kind of based on double hypergraph neural network sentiment recognition method and system, it is related to multimodal sentiment recognition technical field, the application is enhanced to each modality initial feature by multi-scale attention mechanism, and the modality enhanced feature obtained is constructed sentiment enhanced hypergraph, by the modality enhanced feature node in sentiment enhanced hypergraph and the sentiment concept node of connection aggregation, make sentiment enhanced hypergraph can capture the relationship between modality enhanced feature and sentiment feature, realize the sentiment enhancement of modality enhanced feature, by the intermodal hyperedge information between each sentiment enhanced modality feature, modal interaction hypergraph is constructed, so that the multimodal fusion interaction feature generated by modal interaction hypergraph can reflect the high-order interaction relationship between different modality enhanced features, can more comprehensively reflect sentiment information, effectively improve the accuracy of sentiment recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal sentiment recognition, and particularly relates to a sentiment recognition method and system based on a double hypergraph neural network. BACKGROUND

[0002] With the rapid development of Internet technology, the expression of users has changed significantly, gradually shifting from traditional single text form to diversified content presentation modes such as video and audio. This diversification of expression makes the sentiment recognition task on social media platforms more complex and challenging.

[0003] Existing sentiment recognition introduces video and audio modalities and strengthens the learning of the text modality, achieving multi-modal sentiment recognition. However, the graph neural network used in current multi-modal sentiment recognition performs sentiment recognition based on low-order connection relationships between points and points or edges and edges, which can only capture simple relationships between single modality features and sentiment features, ignoring the interaction between multi-modal features, resulting in low accuracy of sentiment recognition. SUMMARY

[0004] The main purpose of the present application is to provide a sentiment recognition method and system based on a double hypergraph neural network, aiming to solve the technical problem of low accuracy of existing technology sentiment recognition.

[0005] To achieve the above purpose, the present application provides a sentiment recognition method based on a double hypergraph neural network, which comprises:

[0006] Performing feature extraction on multi-modal data to obtain modality initial features corresponding to each modality data;

[0007] Enhancing each modality initial feature through a multi-scale attention mechanism to obtain modality enhanced features corresponding to each modality initial feature;

[0008] Constructing a sentiment enhanced hypergraph based on each modality enhanced feature, wherein the nodes of the sentiment enhanced hypergraph include sentiment concept nodes and modality enhanced feature nodes corresponding to the modality enhanced features, and the modality enhanced feature nodes and the sentiment concept nodes are connected through sentiment concept hyperedges;

[0009] Aggregating the modality enhanced feature nodes and the sentiment concept nodes connected thereto in the sentiment enhanced hypergraph to obtain sentiment enhanced modality features corresponding to each modality enhanced feature;

[0010] Determining inter-modal hyperedge information between each sentiment enhanced modality feature, and constructing a modality interaction hypergraph based on the inter-modal hyperedge information;

[0011] Generating multi-modal fusion interaction features through the modality interaction hypergraph;

[0012] outputting a sentiment prediction probability according to the multimodal fusion interaction feature.

[0013] In an embodiment, the step of enhancing each of the modal initial features through a multi-scale attention mechanism to obtain modal enhanced features corresponding to each of the modal initial features comprises:

[0014] enhancing each of the modal initial features through a preset temporal attention formula to obtain temporal attention enhanced features corresponding to each of the modal initial features, the preset temporal attention formula being:

[0015] ;

[0016] ;

[0017] ;

[0018] ;

[0019] wherein, denotes the temporal attention enhanced feature, denotes a query matrix, denotes a key matrix, denotes a value matrix, denotes a dimension of a key vector, F denotes a modal initial feature, denotes a random initial query weight matrix, denotes a random initial key weight, denotes a random initial value weight matrix;

[0020] enhancing each of the modal initial features through a preset channel attention formula to obtain channel attention enhanced features corresponding to each of the modal initial features, the channel attention formula being:

[0021] ;

[0022] wherein, denotes the channel attention enhanced feature, denotes a modal initial feature, denotes a global average pooling operation, denotes a global maximum pooling operation, denotes a multi-layer perception, denotes sigmoid an activation function;

[0023] The modal initial features are enhanced by a preset spatial attention formula to obtain spatial attention enhanced features corresponding to the modal initial features, and the spatial attention formula is:

[0024] ;

[0025] wherein, represents the spatial attention enhanced feature, represents a convolution operation, represents a global average pooling, represents a global maximum pooling output, represents sigmoid an activation function, represents the modal initial feature;

[0026] The temporal attention enhanced features, the channel attention enhanced features and the spatial attention enhanced features corresponding to the same modal initial feature are fused to obtain modal enhanced features corresponding to the modal initial features.

[0027] In an embodiment, the step of constructing the emotion enhanced supergraph based on the modal enhanced features comprises:

[0028] An emotion embedding matrix is generated based on the input sequence by a preset emotion embedding formula, and the preset emotion embedding formula is:

[0029] ;

[0030] wherein, represents the emotion embedding matrix, represents a global shared external knowledge base, represents the input sequence;

[0031] A reshaped feature matrix corresponding to each of the modal enhanced features is generated by a first preset feature flattening formula, and the first preset feature flattening formula is:

[0032] ;

[0033] wherein, represents the reshaped feature matrix, represents the modal enhanced feature, represents a batch size, represents the number of time points or frames contained in each sequence, represents a feature dimension;

[0034] The similarity between the emotion embedding matrix and each of the reshaped feature matrices is calculated by a preset similarity formula to obtain a similarity matrix corresponding to each of the reshaped feature matrices, and the preset similarity formula is:

[0035] ;

[0036] wherein, denotes a similarity matrix, denotes an L2 normalization function, denotes a reshaped feature matrix, denotes an emotion embedding matrix;

[0037] a similarity value matrix and an index matrix are determined based on each of the similarity matrices by a preset similarity value formula, the preset similarity value formula being:

[0038] ;

[0039] wherein, denotes a similarity value matrix, denotes an index matrix, is a function for finding the K largest values and corresponding indices in the last dimension of a tensor, and is set to K = 3, denotes a similarity matrix;

[0040] an emotion hyperedge weight set is determined according to the similarity value matrix;

[0041] an emotion hyperedge set is determined according to the index matrix;

[0042] an emotion-enhanced hypergraph is generated based on the emotion hyperedge weight set and the emotion hyperedge set by a preset hypergraph construction formula, the preset hypergraph construction formula being:

[0043] ;

[0044] wherein, denotes an emotion-enhanced hypergraph, denotes a constructor of a hypergraph object, denotes a set of all nodes in the emotion-enhanced hypergraph, denotes a hyperedge set, denotes a set of hyperedge weights.

[0045] In an embodiment, the step of aggregating the modality-enhanced feature nodes and the emotion concept nodes connected thereto in the emotion-enhanced hypergraph to obtain emotion-enhanced modality features corresponding to each of the modality-enhanced features comprises:

[0046] The long and short term dependency relationship in each of the modal enhanced features is captured by a preset timing context awareness formula to generate a hidden state tensor corresponding to each of the modal enhanced features, the hidden state tensor including context information of each time point in the modal enhanced feature, and the preset timing context awareness formula is:

[0047] ;

[0048] wherein, represents the hidden state tensor, GRU represents a gated recurrent unit network layer, represents a text modal enhanced feature, represents a video modal enhanced feature, an audio modal enhanced feature;

[0049] A gating signal tensor corresponding to each of the hidden state tensors is generated by a preset long and short term network gating formula, and the preset long and short term network gating formula is:

[0050] ;

[0051] wherein, represents the gating signal tensor, represents sigmoid an activation function, represents the hidden state tensor, represents a fully connected layer;

[0052] Each of the modal enhanced features is combined with the corresponding gating signal tensor by a preset gating feature fusion formula to obtain a gated modal feature corresponding to each of the modal enhanced features, and the preset gating feature fusion formula is:

[0053] ;

[0054] wherein, represents the gated modal feature, represents the gating signal tensor, represents a text modal enhanced feature, represents a video modal enhanced feature, represents an audio modal enhanced feature;

[0055] The modal enhanced feature nodes and the connected emotional concept nodes in the emotional enhancement supergraph are aggregated based on each of the gated modal features to obtain an emotional enhancement modal feature corresponding to each of the modal enhanced features.

[0056] In an embodiment, the step of aggregating the modality enhanced feature nodes and the sentiment concept nodes connected thereto in the sentiment enhanced supergraph based on each of the gated post-modality features, to obtain a sentiment enhanced modality feature corresponding to each of the modality enhanced features, comprises:

[0057] generating a gated feature flattening matrix corresponding to each of the gated post-modality features by a second preset feature flattening formula, the second preset feature flattening formula being:

[0058] ;

[0059] wherein, denotes the gated feature flattening matrix, denotes a flattening operation, denotes the gated post-modality feature, denotes a batch size, denotes a number of time points or frames contained in each sequence, denotes an input feature dimension;

[0060] concatenating each of the gated feature flattening matrices and the sentiment embedding matrix to obtain a node feature matrix containing all nodes of the sentiment enhanced supergraph by a preset feature concatenation formula, the preset feature concatenation formula being:

[0061] ;

[0062] wherein, denotes the node feature matrix of the sentiment enhanced supergraph, denotes a concatenation operation, denotes the gated feature flattening matrix, denotes the sentiment embedding matrix, is a specified dimension;

[0063] inputting the node feature matrix of the sentiment enhanced supergraph into a plurality of supergraph convolution layers in a supergraph neural network for processing, aggregating each of the modality enhanced feature nodes and the sentiment concept nodes connected thereto in the node feature matrix by each of the supergraph convolution layers to obtain an updated node feature matrix;

[0064] performing feature enhancement on the updated node feature matrix by a preset graph domain to time domain feature conversion formula to obtain a sentiment enhanced modality feature corresponding to each of the modality enhanced features, the preset graph domain to time domain feature conversion formula being:

[0065] ;

[0066] wherein, denotes the sentiment enhanced modality feature, denotes a flattening operation, denotes an updated node feature matrix, denotes a slicing operation, denotes a batch size, denotes the number of time points or frames contained in each sequence, denotes an output feature dimension.

[0067] In an embodiment, the step of generating a multi-modal fusion interaction feature through the modality interaction hypergraph comprises:

[0068] inputting the node feature matrix of the modality interaction hypergraph into a multi-layer hypergraph convolution layer for processing to obtain a target node feature matrix after depth fusion and residual connection;

[0069] determining a reshaped fusion feature tensor based on the target node feature matrix through a preset reshaping formula, the preset reshaping formula being:

[0070] ;

[0071] wherein, denotes a reshaped fusion feature tensor, denotes that the flattened node feature is restored to a four-dimensional structure, denotes a target node feature matrix, denotes a batch size, denotes a number of modalities, denotes a sequence length, denotes an output feature dimension.

[0072] determining a post-fusion sentiment-enhanced modality feature based on the reshaped fusion feature tensor through a preset fusion formula, the preset fusion formula being:

[0073] ;

[0074] wherein, denotes a post-fusion sentiment-enhanced modality feature, denotes a reshaped fusion feature tensor, denotes a mean pooling along the modality dimension ;

[0075] normalizing the post-fusion sentiment-enhanced modality feature to obtain a multi-modal fusion interaction feature.

[0076] In an embodiment, the step of outputting a sentiment prediction probability according to the multi-modal fusion interaction feature comprises:

[0077] extracting modality initial shared features of each of the sentiment-enhanced modality features through a shared encoder, and splicing each of the modality initial shared features to obtain a modality shared feature.

[0078] each of the emotion-enhanced modality features is decomposed into a corresponding modality unique feature by a unique encoder;

[0079] candidate prediction probabilities of the multi-modal fusion interaction feature, the modality shared feature, and each of the modality unique features are respectively generated;

[0080] attention weights of each of the candidate prediction probabilities are respectively calculated;

[0081] an emotion prediction probability is outputted based on each candidate prediction probability and a corresponding attention weight by a main tasker calling a preset fusion prediction formula, the preset fusion prediction formula being:

[0082] ;

[0083] wherein, the emotion prediction probability is represented by p, a set composed of the multi-modal fusion interaction feature, the modality shared feature, and the modality unique feature is represented by X, the candidate prediction probability is represented by q, the attention weight is represented by a.

[0084] In an embodiment, after the step of outputting the emotion prediction probability based on each candidate prediction probability and a corresponding attention weight by the main tasker calling the preset fusion prediction formula, the method further comprises:

[0085] a total regularization loss is determined by each modality initial shared feature;

[0086] a source prediction probability is obtained by predicting the source of the modality shared feature by a modality discriminator;

[0087] an adversarial modality classification loss is determined based on the source prediction probability by a preset adversarial loss formula, the preset adversarial loss formula being:

[0088] ;

[0089] wherein, the adversarial modality classification loss is represented by L, a number of training samples is represented by N, a number of modality categories is represented by K, a read-hot vector of a i -th sample belonging to a modality j is represented by y, a modality shared feature of a i -th sample is represented by x, a prediction probability of a j -th modality by the discriminator is represented by d.

[0090] determine a main task sentiment classification loss according to the sentiment prediction probability, and determine a joint loss based on the adversarial modal classification loss, the main task sentiment classification loss, and the regularization term total loss;

[0091] update the main tasker, the modal discriminator, and the shared feature extractor respectively through the joint loss.

[0092] In addition, to achieve the above object, the present application also proposes a sentiment recognition system based on a double hypergraph neural network, which comprises:

[0093] A feature extraction module is configured to extract features from the multi-modal data to obtain modal initial features corresponding to each modal data.

[0094] An attention module is configured to enhance each of the modal initial features through a multi-scale attention mechanism to obtain modal enhanced features corresponding to each of the modal initial features.

[0095] A sentiment enhanced hypergraph module is configured to construct a sentiment enhanced hypergraph based on each of the modal enhanced features, wherein the nodes of the sentiment enhanced hypergraph include sentiment concept nodes and modal enhanced feature nodes corresponding to the modal enhanced features, and the modal enhanced feature nodes and the sentiment concept nodes are connected through sentiment concept hyperedges.

[0096] The sentiment enhanced hypergraph module is further configured to aggregate the modal enhanced feature nodes and the sentiment concept nodes connected thereto in the sentiment enhanced hypergraph to obtain sentiment enhanced modal features corresponding to each of the modal enhanced features.

[0097] A modal interaction hypergraph module is configured to determine inter-modal hyperedge information between each of the sentiment enhanced modal features, and construct a modal interaction hypergraph based on the inter-modal hyperedge information.

[0098] The modal interaction hypergraph module is further configured to generate multi-modal fusion interaction features through the modal interaction hypergraph.

[0099] A sentiment recognition module is configured to output a sentiment prediction probability according to the multi-modal fusion interaction features.

[0100] In addition, to achieve the above object, the present application also proposes a sentiment recognition device based on a double hypergraph neural network, which comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the sentiment recognition method based on the double hypergraph neural network as described above.

[0101] The one or more technical solutions proposed in the present application have at least the following technical effects:

[0102] The application obtains modal initial features corresponding to each modal data by feature extraction on the multi-modal data, enhances each modal initial feature through a multi-scale attention mechanism to obtain modal enhanced features corresponding to each modal initial feature, constructs a sentiment enhanced supergraph based on each modal enhanced feature, the nodes of the sentiment enhanced supergraph include sentiment concept nodes and modal enhanced feature nodes corresponding to the modal enhanced features, and the modal enhanced feature nodes and the sentiment concept nodes are connected through sentiment concept superedges, aggregates the modal enhanced feature nodes and the connected sentiment concept nodes in the sentiment enhanced supergraph to obtain sentiment enhanced modal features corresponding to each modal enhanced feature, determines inter-modal superedge information between each sentiment enhanced modal feature, and constructs a modal interaction supergraph based on the inter-modal superedge information, generates a multi-modal fusion interaction feature through the modal interaction supergraph, and outputs a sentiment prediction probability according to the multi-modal fusion interaction feature. The modal enhanced features obtained by enhancing each modal initial feature through the multi-scale attention mechanism are used to construct the sentiment enhanced supergraph, the modal enhanced feature nodes and the connected sentiment concept nodes in the sentiment enhanced supergraph are aggregated, so that the sentiment enhanced supergraph can capture the relationship between the modal enhanced features and the sentiment features, the sentiment enhancement of the modal enhanced features is realized, the inter-modal superedge information between each sentiment enhanced modal feature is used to construct the modal interaction supergraph, so that the multi-modal fusion interaction feature generated through the modal interaction supergraph can reflect the high-order interaction relationship between different modal enhanced features, the sentiment information can be more comprehensively reflected, and the accuracy of sentiment recognition is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0103] The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0104] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings required to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0105] Figure 1 A flowchart of a first embodiment of a sentiment recognition method based on a double supergraph neural network of the present application;

[0106] Figure 2 A flowchart of a second embodiment of a sentiment recognition method based on a double supergraph neural network of the present application;

[0107] Figure 3 A flowchart of a third embodiment of a sentiment recognition method based on a double supergraph neural network of the present application;

[0108] Figure 4A module structure schematic diagram of a sentiment recognition system based on a double hypergraph neural network of the present application;

[0109] Figure 5 A structure schematic diagram of a sentiment recognition device based on a double hypergraph neural network of the present application.

[0110] The object implementation, functional features and advantages of the present application will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0111] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0112] In order to better understand the technical solutions of the present application, the following will be described in detail in conjunction with the drawings and specific embodiments of the specification.

[0113] The main solution of the embodiment of the present application is: feature extraction is performed on multi-modal data to obtain modal initial features corresponding to each modal data; each modal initial feature is enhanced through a multi-scale attention mechanism to obtain modal enhanced features corresponding to each modal initial feature; a sentiment enhanced hypergraph is constructed based on each modal enhanced feature, the nodes of the sentiment enhanced hypergraph include sentiment concept nodes and modal enhanced feature nodes corresponding to the modal enhanced features, and the modal enhanced feature nodes and the sentiment concept nodes are connected through sentiment concept hyperedges; the modal enhanced feature nodes and the connected sentiment concept nodes in the sentiment enhanced hypergraph are aggregated to obtain sentiment enhanced modal features corresponding to each modal enhanced feature; inter-modal hyperedge information between each sentiment enhanced modal feature is determined, and a modal interaction hypergraph is constructed based on the inter-modal hyperedge information; a multi-modal fusion interaction feature is generated through the modal interaction hypergraph; and a sentiment prediction probability is output according to the multi-modal fusion interaction feature.

[0114] Since the prior art adopts a graph neural network, sentiment recognition is performed based on point-to-point or edge-to-edge low-order connection relationships, only simple relationships between single modal features and sentiment features can be captured, the interaction between multi-modal features is ignored, and the accuracy of sentiment recognition is low.

[0115] The application provides a solution. The modal enhanced features are obtained by enhancing the initial features of each modality through a multi-scale attention mechanism, a sentiment enhanced supergraph is constructed from the modal enhanced features, the modal enhanced features in the sentiment enhanced supergraph are aggregated with the sentiment concept nodes connected thereto, so that the sentiment enhanced supergraph can capture the relationship between the modal enhanced features and the sentiment features, realize sentiment enhancement of the modal enhanced features, and construct a modal interaction supergraph through the inter-modal superedge information between the sentiment enhanced modal features, so that the multi-modal fusion interaction features generated through the modal interaction supergraph can reflect the high-order interaction relationship between different modal enhanced features, can more comprehensively reflect the sentiment information, and effectively improve the accuracy of sentiment recognition.

[0116] It should be noted that the execution subject of the embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a sentiment recognition device based on a double supergraph neural network, etc. The following will take the sentiment recognition device based on the double supergraph neural network (referred to as the sentiment recognition device) as an example to describe the embodiment and the following embodiments.

[0117] Based on this, the embodiment of the application provides a sentiment recognition method based on a double supergraph neural network, referring to Figure 1 , Figure 1 The flowchart of the first embodiment of the sentiment recognition method based on the double supergraph neural network of the application is shown in the figure.

[0118] In the embodiment, the sentiment recognition method based on the double supergraph neural network includes steps S10-S70:

[0119] Step S10, feature extraction is performed on the multi-modal data to obtain modal initial features corresponding to each modal data.

[0120] It should be noted that the above multi-modal data at least includes text data, video data and audio data.

[0121] It can be understood that the above modal initial features at least include text initial features, video initial features and audio initial features.

[0122] In specific implementation, the embodiment and the following embodiments are described by taking multi-modal data including text data, video data and audio data, but the application is not limited thereto.

[0123] The above-mentioned emotion recognition device can perform word segmentation, stop word removal, etc. on the text data, and convert it into pre-trained model compatible processed text data. The pre-trained model can be a model for extracting text initial features, such as the BERT-base-uncased model. Taking the BERT-base-uncased model as the pre-trained model for illustration, the BERT-base-uncased model can generate text initial features with multi-dimensional (such as 768 dimensions) hidden states based on the input processed text data. The formula corresponding to the pre-trained model is:

[0124] ;

[0125] wherein, represents the text initial feature, represents BERT-base-uncased the model, X and the processed text data.

[0126] The above-mentioned emotion recognition device can capture the facial action units in the video data, the subtle changes of the eyebrows, eyes, mouth, etc., obtain video extraction features, and the video extraction features at least include the height and width of the required extraction features. The key points are input into the preset video feature extraction formula to generate video initial features. The preset video feature extraction formula is:

[0127] ;

[0128] wherein, represents the video initial feature, h represents the height of the required extraction feature, w represents the width of the required extraction feature, c represents the number of channels.

[0129] The above-mentioned emotion recognition device can extract audio extraction features such as fundamental frequency, MFCC, amplitude from audio data, and then perform dimensionality reduction processing on the audio extraction features through a preset audio feature extraction formula to obtain audio initial features, whose expression is as follows:

[0130] ;

[0131] wherein, represents the audio initial feature, represents the audio extraction feature, MFCC represents the calculated mel-frequency cepstrum coefficient, E represents the calculated amplitude feature.

[0132] Step S20, each of the modal initial features is enhanced by a multi-scale attention mechanism to obtain a modal enhanced feature corresponding to each of the modal initial features.

[0133] In a specific implementation, the emotion recognition device can enhance each modality initial feature through a multi-scale attention mechanism, capture key information in a long sequence of each modality initial feature, obtain modality enhanced features corresponding to each modality initial feature, enhance the spatio-temporal perception ability of each modality initial feature at the feature representation level, and significantly improve the modeling effect of emotional dynamics such as time sequence mutation and delayed excitation.

[0134] In a feasible implementation, the multi-scale attention mechanism at least includes a time sequence attention mechanism, a channel attention mechanism, and a spatial attention mechanism. Step S20 includes the following steps:

[0135] Step S201, enhancing each modality initial feature through a preset time sequence attention formula to obtain time sequence attention enhanced features corresponding to each modality initial feature.

[0136] The preset time sequence attention formula is:

[0137] ;

[0138] ;

[0139] ;

[0140] ;

[0141] wherein, represents the time sequence attention enhanced feature, represents a query matrix, represents a key matrix, represents a value matrix, represents the dimension of the key vector, F represents the modality initial feature, represents a random initial query weight matrix, represents a random initial key weight, represents a random initial value weight matrix.

[0142] In a specific implementation, the formula corresponding to the time sequence attention mechanism is a preset time sequence attention formula. The emotion recognition device can input each modality initial feature into the preset time sequence attention formula to perform time sequence attention enhancement on each modality initial feature, and obtain time sequence attention enhanced features corresponding to each modality initial feature.

[0143] Step S202, enhancing each modality initial feature through a preset channel attention formula to obtain channel attention enhanced features corresponding to each modality initial feature.

[0144] The channel attention formula is:

[0145] ;

[0146] wherein, denotes a channel attention enhanced feature, denotes a modal initial feature, denotes a global average pooling operation, denotes a global max pooling operation, denotes a multi-layer perceptron, denotes sigmoid an activation function.

[0147] In a specific implementation, the formula corresponding to the channel attention mechanism described above is a preset channel attention formula. The emotion recognition device can input each modal initial feature into the preset temporal attention formula to perform channel attention enhancement on each modal initial feature, enhance the channel information in each modal initial feature, and obtain a channel attention enhanced feature corresponding to each modal initial feature.

[0148] In step S203, each of the modal initial features is enhanced by a preset spatial attention formula to obtain a spatial attention enhanced feature corresponding to each of the modal initial features.

[0149] The spatial attention formula is:

[0150] ;

[0151] wherein, denotes a spatial attention enhanced feature, denotes a convolution operation, denotes a global average pooling, denotes a global max pooling output, denotes sigmoid an activation function, denotes a modal initial feature.

[0152] In a specific implementation, the formula corresponding to the spatial attention mechanism described above is a preset spatial attention formula. The emotion recognition device can input each modal initial feature into the preset spatial attention formula to perform spatial attention enhancement on each modal initial feature, enhance the spatial position information in each modal initial feature, and obtain a spatial attention enhanced feature corresponding to each modal initial feature.

[0153] In step S204, the temporal attention enhanced feature, the channel attention enhanced feature, and the spatial attention enhanced feature corresponding to the same modal initial feature are fused to obtain a modal enhanced feature corresponding to each of the modal initial features.

[0154] In a specific implementation, the emotion recognition device can concatenate the time sequence attention enhanced features, the channel attention enhanced features, and the spatial attention enhanced features corresponding to the same modality initial features in a set dimension, thereby fusing the time sequence attention enhanced features, the channel attention enhanced features, and the spatial attention enhanced features corresponding to the same modality initial features. The modality initial features can be fused in the manner of feature concatenation, thereby obtaining modality enhanced features corresponding to the modality initial features.

[0155] In step S30, an emotion enhanced hypergraph is constructed based on the modality enhanced features.

[0156] The nodes of the emotion enhanced hypergraph include emotion concept nodes and modality enhanced feature nodes corresponding to the modality enhanced features, and the modality enhanced feature nodes and the emotion concept nodes are connected by emotion concept hyperedges.

[0157] In a specific implementation, the emotion recognition device can take the modality enhanced features as modality enhanced feature nodes, introduce emotion concept features, and take the emotion concept features as emotion concept nodes. For each modality enhanced feature node, the similarity between the modality enhanced feature node and each emotion concept node is calculated. For any modality enhanced feature node, a number of emotion concept nodes (e.g., 3) with the highest similarity to the modality enhanced feature node are selected to reach a set value. A hyperedge is created as an emotion concept hyperedge for each modality enhanced feature node, and the emotion concept hyperedge connects one end of the modality enhanced feature node and the other end of all the emotion concept nodes selected for the modality enhanced feature node, thereby being able to represent more complex high-order relationships. If a text enhanced feature contains words or semantic information representing “happiness” and “joy”, the text enhanced feature node is connected to the emotion concept nodes corresponding to “happiness” and “joy” through the emotion concept hyperedge. By the above manner, the modality enhanced feature nodes and the corresponding emotion concept nodes are connected, thereby constructing an emotion enhanced hypergraph that can explicitly capture the high-order semantic relationships between the modalities and the emotions, and thereby realizing the fusion of the modality enhanced features and the emotion concept features.

[0158] It should be understood that the emotion concept features can be derived from a global shared external knowledge base or learned through emotion annotated data. For example, the emotion concepts of “happiness”, “sadness”, and “anger”.

[0159] In step S40, the modality enhanced feature nodes and the connected emotion concept nodes in the emotion enhanced hypergraph are aggregated, thereby obtaining emotion enhanced modality features corresponding to the modality enhanced features.

[0160] In a specific implementation, the above-mentioned sentiment recognition device can aggregate each modality enhanced feature node in the sentiment enhanced supergraph and the connected sentiment feature node through a supergraph convolution operation, that is, aggregate the modality enhanced feature nodes at both ends of the same sentiment concept superedge and the sentiment concept nodes. Each modality enhanced feature node absorbs the most relevant sentiment concept feature through the above-mentioned aggregation operation, thereby enhancing the sentiment semantics of each modality enhanced feature, and obtaining a sentiment enhanced modality feature. After the above-mentioned aggregation of each modality enhanced feature, the obtained sentiment enhanced modality feature not only reflects its own feature, but also fuses the information of the related sentiment concept, so that the sentiment enhanced modality feature is more consistent with the context and reflects the sentiment semantic context information.

[0161] In step S50, inter-modal superedge information between each of the sentiment enhanced modality features is determined, and a modality interaction supergraph is constructed based on the inter-modal superedge information.

[0162] In step S60, a multi-modal fusion interaction feature is generated through the modality interaction supergraph.

[0163] In step S70, a sentiment prediction probability is output according to the multi-modal fusion interaction feature.

[0164] In a specific implementation, the above-mentioned sentiment recognition device can calculate the similarity between each sentiment enhanced modality feature node as a sentiment enhanced modality feature node, obtain a similarity matrix, and use the similarity in the similarity matrix as a superedge weight. The superedge weight represents the strength or importance of the superedge connection. The inter-modal superedge can be constructed through the similarity matrix. The above-mentioned inter-modal superedge information is used to connect different sentiment enhanced modality feature nodes and represent the interaction relationship between different sentiment enhanced modalities, such as complementary information between text and video. The inter-modal superedge and the superedge weight constitute the inter-modal superedge information.

[0165] Further, the modality interaction supergraph can be constructed through the inter-modal superedge information. The interaction relationship between different modality features can be captured through the modality interaction supergraph. The information from different modalities is integrated through the interaction relationship to generate a multi-modal fusion interaction feature that can reflect high-order emotional co-occurrence and complex context dependence between multi-modal data. The multi-modal fusion interaction feature is multiplied by the corresponding weight to output a sentiment prediction probability.

[0166] The embodiment extracts features from the multi-modal data to obtain initial modal features corresponding to the modal data, enhances the initial modal features through a multi-scale attention mechanism to obtain enhanced modal features corresponding to the initial modal features, constructs an emotion-enhanced hypergraph based on the enhanced modal features, the nodes of the emotion-enhanced hypergraph include emotion concept nodes and enhanced modal feature nodes corresponding to the enhanced modal features, the enhanced modal feature nodes are connected with the emotion concept nodes through emotion concept hyperedges, aggregates the enhanced modal feature nodes and the connected emotion concept nodes in the emotion-enhanced hypergraph to obtain emotion-enhanced modal features corresponding to the enhanced modal features, determines inter-modal hyperedge information between the emotion-enhanced modal features, and constructs a modal interaction hypergraph based on the inter-modal hyperedge information, generates multi-modal fusion interaction features through the modal interaction hypergraph, and outputs emotion prediction probability according to the multi-modal fusion interaction features. The multi-scale attention mechanism is used to enhance the initial modal features to obtain the enhanced modal features for constructing the emotion-enhanced hypergraph, the enhanced modal feature nodes and the connected emotion concept nodes in the emotion-enhanced hypergraph are aggregated, so that the emotion-enhanced hypergraph can capture the relationship between the enhanced modal features and the emotion features, the emotion-enhanced modal features are enhanced, the inter-modal hyperedge information between the emotion-enhanced modal features is used to construct the modal interaction hypergraph, so that the multi-modal fusion interaction features generated through the modal interaction hypergraph can reflect the high-order interaction relationship between different enhanced modal features, can more comprehensively reflect the emotion information, and effectively improves the accuracy of emotion recognition.

[0167] Based on the first embodiment of the present application, the second embodiment of the present application is proposed. In the second embodiment of the present application, the same or similar contents as the above-mentioned first embodiment can be referred to the above introduction, and will not be described in detail. On this basis, please refer to Figure 2 , Figure 2 is a flowchart of the second embodiment of the present application based on the dual-hypergraph neural network emotion recognition method.

[0168] In the embodiment, step S30 includes steps S301-S307:

[0169] In step S301, an emotion embedding matrix is generated based on an input sequence through a preset emotion embedding formula.

[0170] The preset emotion embedding formula is:

[0171] ;

[0172] Wherein, represents the emotion embedding matrix, represents a global shared external knowledge base, represents the input sequence.

[0173] It should be noted that the global shared external knowledge base can include sentiment concepts of the initialized sentiment semantic representation.

[0174] It can be understood that the input sequence can be an integer vector representing the ID of a dynamic and specific input sequence. Each ID corresponds to a sentiment concept in the global shared external knowledge base.

[0175] In a specific implementation, the sentiment recognition device can query each sentiment concept related to the input sequence in the global shared external knowledge base based on the input sequence through the preset sentiment embedding formula, and map each queried sentiment concept to a sentiment embedding space to obtain a sentiment embedding matrix.

[0176] In step S302, a reshaped feature matrix corresponding to each modality enhanced feature is generated through a first preset feature flattening formula.

[0177] The first preset feature flattening formula is:

[0178] ;

[0179] wherein, the reshaped feature matrix is represented by, the modality enhanced feature is represented by, the batch size is represented by, the number of time points or frames contained in each sequence is represented by, the feature dimension is represented by.

[0180] It should be noted that the batch size can be the number of samples processed at a time.

[0181] In a specific implementation, the sentiment recognition device can reshape each modality enhanced feature into a reshaped feature matrix with a shape of [ ] through the first preset feature flattening formula. Each row vector in the reshaped feature matrix represents the features of a time point or a frame.

[0182] In step S303, a similarity matrix corresponding to each reshaped feature matrix is obtained by calculating the similarity between the sentiment embedding matrix and each reshaped feature matrix through a preset similarity formula.

[0183] The preset similarity formula is:

[0184] ;

[0185] wherein, the similarity matrix is represented by, the reshaped feature matrix is represented by, L2 the normalization function is represented by, the reshaped feature matrix is represented by, The emotion embedding matrix is denoted as.

[0186] In a specific implementation, the emotion recognition device can calculate the similarity between the normalized emotion embedding matrix and the normalized reshaped feature matrix by using the preset similarity formula, and obtain a similarity matrix.

[0187] In step S304, a similarity value matrix and an index matrix are determined based on the similarity matrices by using a preset similarity value formula.

[0188] The preset similarity value formula is as follows:

[0189] ;

[0190] wherein, the similarity value matrix is denoted as, the index matrix is denoted as, is a function for finding the K largest values and corresponding indexes in the last dimension of the tensor, and is set as K = 3, the similarity matrix is denoted as.

[0191] In a specific implementation, the emotion recognition device can extract the most important top K similarity maximum values and their corresponding indexes from the similarity matrix by using the preset similarity value formula, and obtain a similarity index matrix and an index matrix. The similarity matrix contains the maximum similarity values corresponding to each feature, which represents the association strength between the feature and the emotion concept. The index matrix contains the indexes (i.e., the IDs of the emotion concepts) of the emotion concepts corresponding to each feature.

[0192] In step S305, a set of emotion hyperedge weights is determined according to the similarity value matrix.

[0193] In a specific implementation, the emotion recognition device can obtain emotion concept hyperedge weights by calculating the average values of the corresponding similarity values in the similarity value matrix, and add the emotion concept hyperedge weights to the same blank set to obtain the set of emotion hyperedge weights. The emotion concept hyperedge weight represents the association strength between the modal enhanced feature node and the emotion concept node.

[0194] In step S306, a set of emotion hyperedges is determined according to the index matrix.

[0195] In a specific implementation, the emotion recognition device can determine emotion concept hyperedges according to the index matrix, and add the emotion concept hyperedges to the same blank set to obtain the set of emotion hyperedges, which represent the connection relationship between the modal enhanced feature node and the emotion concept node. For example, if =[c1,c2,c3], the emotional concept hyperedge can be a hyperedge connecting the modal enhanced feature node i and the emotional concept nodes c1, c2, and c3.

[0196] In step S307, an emotional enhancement hypergraph is generated based on the set of emotional hyperedge weights and the set of emotional hyperedges by using a preset hypergraph construction formula.

[0197] The preset hypergraph construction formula is:

[0198] ;

[0199] wherein, represents the emotional enhancement hypergraph, represents a constructor of a hypergraph object, represents a set of all nodes in the emotional enhancement hypergraph, represents a set of hyperedges, represents a set of hyperedge weights.

[0200] In a specific implementation, the above-mentioned emotional recognition device can construct an emotional enhancement hypergraph including emotional concept nodes and modal enhanced feature nodes based on the set of hyperedges and the set of hyperedge weights by using the preset hypergraph construction formula.

[0201] In this embodiment, step S40 includes steps S401-S404:

[0202] In step S401, long and short-term dependency relationships in each of the modal enhanced features are captured by using a preset time sequence context perception formula, and a hidden state tensor corresponding to each of the modal enhanced features is generated.

[0203] The hidden state tensor includes context information of each time point in the modal enhanced feature, and the preset time sequence context perception formula is:

[0204] ;

[0205] wherein, represents the hidden state tensor, GRU represents a gated recurrent unit network layer, represents a text modal enhanced feature, represents a video modal enhanced feature, represents an audio modal enhanced feature.

[0206] It should be noted that initial modal features include at least text modal features, video modal features, and audio modal features. Text modal enhancement features can be features obtained by enhancing text modal features through a multi-scale attention mechanism; video modal enhancement features can be features obtained by enhancing video modal features through a multi-scale attention mechanism; and audio modal enhancement features can be features obtained by enhancing audio modal features through a multi-scale attention mechanism.

[0207] Understandably, a gated recurrent unit (GRU) is a variant of a recurrent neural network used to process sequential data.

[0208] In a specific implementation, the aforementioned emotion recognition device can substitute text modality enhancement features, video modality enhancement features, and audio modality enhancement features into the aforementioned preset temporal context perception formula. The device processes the text modality enhancement features, video modality enhancement features, and audio modality enhancement features through the gating mechanism inside the GRU, fuses feature information from different modalities, captures the long-short-term dependencies in each modality enhancement feature, generates the hidden state tensor corresponding to each modality enhancement feature, and the generated hidden state tensor includes the hidden state tensor of the context information at each time point in the modality enhancement feature.

[0209] Step S402: Generate the gate signal tensor corresponding to each hidden state tensor using a preset long short-term network gating formula.

[0210] The preset long-short-term network gating formula is:

[0211] ;

[0212] in, G Represents the gated signal tensor. express sigmoid Activation function Represents the hidden state tensor. This indicates a fully connected layer.

[0213] In its implementation, the aforementioned emotion recognition device can substitute each hidden state tensor into the preset long short-term network gating formula, perform a linear transformation on each hidden state tensor through a fully connected layer, and adjust the dimension of each hidden state tensor to make the generated gating signal tensor easier to use. sigmoid Activation function processing. sigmoid The activation function maps the output of the fully connected layer to the range [0,1], generating a gated signal tensor corresponding to each hidden state tensor. Each element in the gated signal tensor represents the importance weight of the corresponding modality enhancement feature.

[0214] It should be understood that the features of important time points can be dynamically amplified and irrelevant or noise information can be inhibited by the gating signal tensor, so as to generate a higher quality feature flow.

[0215] In step S403, each modality enhanced feature is combined with the corresponding gating signal tensor by a preset gating feature fusion formula, to obtain a gated post-modality feature corresponding to each modality enhanced feature.

[0216] The preset gating feature fusion formula is:

[0217] ;

[0218] wherein, the gated post-modality feature, G the gating signal tensor, the text modality enhanced feature, the video modality enhanced feature, the audio modality enhanced feature.

[0219] In a specific implementation, the above-mentioned emotion recognition device can substitute each modality enhanced feature, i.e., the text modality enhanced feature, the video modality enhanced feature, and the audio modality enhanced feature, into the above-mentioned preset gating feature fusion formula, use element-by-element multiplication to multiply each modality enhanced feature with the corresponding gating signal tensor, and filter and weight the modality enhanced feature by the gating signal tensor, to obtain a gated post-modality feature corresponding to each modality enhanced feature. The gated post-modality feature fuses the modality enhanced feature and the gating signal tensor, highlights important features, and suppresses unimportant features.

[0220] In step S404, the modality enhanced feature nodes and the connected emotion concept nodes in the emotion enhanced supergraph are aggregated based on each gated post-modality feature, to obtain an emotion enhanced modality feature corresponding to each modality enhanced feature.

[0221] In a specific implementation, the above-mentioned emotion recognition device can flatten each gated post-modality feature, splice the flattened gated feature flattening matrix with the emotion embedding matrix, to obtain a node feature matrix of the emotion enhanced supergraph, and input the node feature matrix into a multi-layer supergraph convolution layer, to aggregate the modality enhanced feature nodes and the connected emotion concept nodes in the emotion enhanced supergraph, to obtain an emotion enhanced modality feature corresponding to each modality enhanced feature.

[0222] In a feasible implementation, step S404 includes steps S4041-S4044:

[0223] In step S4041, a gated feature flattening matrix corresponding to each gated post-modality feature is generated by a second preset feature flattening formula.

[0224] The second preset feature flattening formula is:

[0225]

[0226] wherein, represents a gating feature flattening matrix, represents a flattening operation, represents a gated modality feature, represents a batch size, represents the number of time points or frames contained in each sequence, represents an input feature dimension.

[0227] In a specific implementation, the above-mentioned emotion recognition device can substitute each gated modality feature into the above-mentioned second preset feature flattening formula to flatten each gated modality feature, and obtain a gating feature flattening matrix corresponding to each gated modality feature.

[0228] Step S4042, each of the gating feature flattening matrix and the emotion embedding matrix is spliced by a preset feature splicing formula to obtain a node feature matrix containing all nodes of the emotion-enhanced hypergraph.

[0229] The preset feature splicing formula is:

[0230]

[0231] wherein, represents a node feature matrix of the emotion-enhanced hypergraph, represents a splicing operation, represents a gating feature flattening matrix, represents an emotion embedding matrix, represents an input feature dimension.

[0232] In a specific implementation, the above-mentioned emotion recognition device can substitute each gated feature flattening matrix and the emotion embedding matrix into the above-mentioned preset feature splicing formula, and splice the gated feature flattening matrix and the emotion embedding matrix in a specified dimension (e.g., the input feature dimension) to integrate each modality-enhanced feature node and the emotion concept node into a unified matrix, and obtain the node feature matrix.

[0233] Step S4043, the node feature matrix of the emotion-enhanced hypergraph is input into a multi-layer hypergraph convolution layer in the hypergraph neural network for processing, and each of the modality-enhanced feature nodes and the emotion concept nodes connected thereto in the node feature matrix is aggregated by each layer of the hypergraph convolution layer to obtain an updated node feature matrix.

[0234] ​​​In a specific implementation, the emotion recognition device can input the node feature matrix of the emotion-enhanced hypergraph into a plurality of layers of hypergraph convolution layers in the hypergraph neural network for processing. Each layer of convolution layer adopts the following first preset node update formula:

[0235] ;

[0236] wherein, represents the output node feature matrix of the current hypergraph convolution layer, represents a rectified linear unit activation function, represents a hypergraph convolution layer function, represents the input node feature matrix of the current hypergraph convolution layer, represents an emotion-enhanced hypergraph, represents a residual feature.

[0237] The current hypergraph convolution layer receives the node feature matrix of the emotion-enhanced hypergraph input by the emotion recognition device as the input node feature matrix thereof, or receives the node feature matrix output by the previous hypergraph convolution layer as the input node feature matrix thereof. Through the above-mentioned first preset node update formula, the hypergraph convolution layer function is adopted, the neighbor node in the emotion-enhanced hypergraph is aggregated, that is, each modality-enhanced feature node in the input node feature matrix is aggregated with the connected emotion concept node, and the rectified linear unit activation function is introduced to enhance the expression ability of the output node feature matrix. The above-mentioned residual feature is the residual connection of the input node feature matrix, which is used to help the gradient flow and alleviate the gradient vanishing problem in the deep network. The emotion recognition device can take the node feature matrix output by the last layer of hypergraph convolution layer as the updated node feature matrix.

[0238] In step S4044, the updated node feature matrix is enhanced in feature through a preset graph domain to time domain feature conversion formula, to obtain emotion-enhanced modality features corresponding to each modality-enhanced feature.

[0239] The preset graph domain to time domain feature conversion formula is:

[0240] ;

[0241] wherein, represents an emotion-enhanced modality feature, represents a flattening operation, represents an updated node feature matrix, represents a slicing operation, represents a batch size, represents the number of time points or frames contained in each sequence, represents an output feature dimension.

[0242] In a specific implementation, the emotion recognition device can substitute the updated node feature matrix into a preset graph domain to time domain feature conversion formula, use a slicing operation, extract features corresponding to the aggregated node pairs, and convert the extracted features from a two-dimensional shape to a three-dimensional shape through a flattening operation , restore the time sequence structure, output the emotion-enhanced modal features, and realize reverse conversion of the updated node feature matrix from the graph domain to the time domain. The features of each time point in the output emotion-enhanced modal features are fused with information of related emotional concepts, and the emotional semantic expression of the features is enhanced.

[0243] In this embodiment, step S60 includes steps S601-S604:

[0244] Step S601 inputs the node feature matrix of the modal interaction hypergraph into a multi-layer hypergraph convolution layer for processing to obtain a target node feature matrix after depth fusion and residual connection.

[0245] In a specific implementation, the emotion recognition device can input the node feature matrix of the modal interaction hypergraph into a multi-layer hypergraph convolution layer in the hypergraph neural network for processing. Each layer of convolution layer adopts the following second preset node update formula:

[0246] ;

[0247] ;

[0248] wherein, represents the node feature matrix input into the i-th layer of hypergraph convolution layer, represents a rectified linear unit activation function, represents a hypergraph convolution layer function, represents the node feature matrix input into the i-th layer of hypergraph convolution layer, represents a modal interaction hypergraph, represents a target node feature matrix after depth fusion and residual connection, represents a node feature matrix input into the i-th layer of hypergraph convolution layer, represents residual features;

[0249] ​​​​The current hypergraph convolution layer receives the node feature matrix of the modality interaction hypergraph input by the emotion recognition device as the input node feature matrix thereof, or receives the node feature matrix output by the previous hypergraph convolution layer as the input node feature matrix thereof, aggregates neighbor nodes in the hyperedge relationship in the modality interaction hypergraph by using the second preset node update formula and the hypergraph convolution layer function, updates the input node feature matrix, and introduces a rectified linear unit activation function to enhance the expression ability of the output node feature matrix. The residual feature is a residual connection of the input node feature matrix, which is used to help gradient flow and alleviate the gradient vanishing problem in a deep network. The emotion recognition device can take the node feature matrix output by the last layer of hypergraph convolution layer as the target node feature matrix after deep fusion and residual connection.

[0250] In step S602, a reshaped fusion feature tensor is determined based on the target node feature matrix by using a preset reshaping formula.

[0251] The preset reshaping formula is:

[0252] ;

[0253] wherein, the reshaped fusion feature tensor is denoted as, the flattened node feature is restored to a four-dimensional structure, the target node feature matrix is denoted as, the batch size is denoted as, the number of modalities is denoted as, the sequence length is denoted as, the output feature dimension is denoted as.

[0254] In a specific implementation, the emotion recognition device can substitute the target node feature matrix into the preset reshaping formula, restore the flattened target node feature matrix to a four-dimensional structure with physical meaning, reshape it, and obtain the reshaped fusion feature tensor.

[0255] In step S603, a fused emotion enhancement modality feature is determined based on the reshaped fusion feature tensor by using a preset fusion formula.

[0256] The preset fusion formula is:

[0257] ;

[0258] wherein, the fused emotion enhancement modality feature is denoted as, the reshaped fusion feature tensor is denoted as, the mean pooling along the modality dimension is denoted as.

[0259] ​In a specific implementation, the emotion recognition device can substitute the reshaped fusion feature tensor into a preset fusion formula, complete the final fusion of all modalities into a single representation through a pooling operation, and obtain a fused emotion-enhanced modality feature.

[0260] In step S604, the fused emotion-enhanced modality feature is standardized to obtain a multi-modal fusion interaction feature.

[0261] In a specific implementation, the emotion recognition device can standardize the fused emotion-enhanced modality feature to obtain a multi-modal fusion interaction feature.

[0262] In this embodiment, the preset temporal attention formula is used to enhance each modality initial feature to obtain a temporal attention-enhanced feature corresponding to each modality initial feature; the preset channel attention formula is used to enhance each modality initial feature to obtain a channel attention-enhanced feature corresponding to each modality initial feature; the preset spatial attention formula is used to enhance each modality initial feature to obtain a spatial attention-enhanced feature corresponding to each modality initial feature; and the temporal attention-enhanced feature, the channel attention-enhanced feature, and the spatial attention-enhanced feature corresponding to the same modality initial feature are fused to obtain a modality-enhanced feature corresponding to each modality initial feature. In this embodiment, the feature is enhanced by fusing the attention mechanism, which can accurately capture key information in a long sequence, significantly improve the modeling effect of emotional dynamics such as time sequence mutation and delayed excitation, and overcome the problem of weak feature expression caused by ignoring the time sequence structure in traditional methods.

[0263] Based on the first and second embodiments of the present application, the third embodiment of the present application is proposed. In the third embodiment of the present application, the same or similar contents as the above first and second embodiments can be referred to the above introduction, and will not be described in detail. On this basis, please refer to Figure 3 , Figure 3 The flowchart of the third embodiment of the emotion recognition method based on the double supergraph neural network of the present application is shown.

[0264] In this embodiment, step S70 includes steps S701-S705:

[0265] In step S701, the modality initial shared features of the emotion-enhanced modality features are extracted through a shared encoder, and the modality initial shared features are spliced to obtain a modality shared feature.

[0266] In step S702, the emotion-enhanced modality features are decomposed into corresponding modality unique features through a unique encoder.

[0267] In specific implementation, the above-mentioned emotion recognition device can extract common features of each emotion-enhanced modality feature respectively by using a shared encoder through a dual-channel encoding strategy, then splice the respective common features to obtain modality initial shared features, and use a unique encoder corresponding to each modality to decompose each emotion-enhanced modality feature into a corresponding modality unique feature, that is, use a unique encoder corresponding to the text modality to decompose the text emotion-enhanced modality feature into a text modality unique feature, use a unique encoder corresponding to the video modality to decompose the video emotion-enhanced modality feature into a video modality unique feature, and use a unique encoder corresponding to the audio modality to decompose the audio emotion-enhanced modality feature into an audio modality unique feature.

[0268] In step S703, candidate prediction probabilities of the multi-modal fusion interaction feature, the modality shared feature, and each modality unique feature are respectively generated.

[0269] In specific implementation, the above-mentioned emotion recognition device can generate the candidate prediction probability of the multi-modal fusion interaction feature through a preset modality fusion probability formula, generate the candidate prediction probability corresponding to the modality shared feature through a preset modality shared probability formula, and generate the candidate prediction probability corresponding to the modality unique feature through a preset modality unique probability formula.

[0270] The preset modality fusion probability formula is:

[0271] ;

[0272] wherein, the candidate prediction probability corresponding to the multi-modal fusion interaction feature is represented by p (Y|X), the multi-modal fusion interaction feature is represented by Y.

[0273] The preset modality shared probability formula is:

[0274] ;

[0275] wherein, the candidate prediction probability corresponding to the modality shared feature is represented by p (Y|X), the modality shared feature is represented by Y.

[0276] The preset modality unique probability formula is:

[0277] ;

[0278] wherein, the candidate prediction probability corresponding to the modality unique feature is represented by p (Y|X), the text modality, the video modality, and the audio modality are respectively represented by T, V, and A, the modality unique feature is represented by Y.

[0279] Step S704: Calculate the attention weights for each of the candidate prediction probabilities.

[0280] In its implementation, the aforementioned emotion recognition device can use a preset attention weight formula to treat modality-shared features as contextual information. Based on this contextual information, it can assign attention weights to each candidate prediction probability using a gating mechanism. The preset attention weight formula is as follows:

[0281] ;

[0282] ;

[0283] in, This represents a vector containing 5 scores, each score corresponding to the importance of different candidate prediction probabilities. Indicates a gating network. Indicates modal sharing features, Representing different attention weights, each Values ​​are between 0 and 1, and all The sum is 1. This means converting the importance score vector into a probability distribution.

[0284] Step S705: The main tasker calls the preset fusion prediction formula to output the sentiment prediction probability based on the prediction probability of each candidate and the corresponding attention weight.

[0285] The preset fusion prediction formula is:

[0286] ;

[0287] in, Indicates the probability of sentiment prediction. It represents the set consisting of multimodal fusion interaction features, modality-shared features, and modality-unique features. Indicates the candidate prediction probability. This represents the attention weight.

[0288] In a specific implementation, the aforementioned emotion recognition device can call a preset fusion prediction formula through the main tasker, substitute the prediction probabilities of each candidate and the corresponding attention weights into the preset fusion prediction formula, and output the emotion prediction probability.

[0289] In this embodiment, steps S7051 to S7055 are included after step S705:

[0290] Step S7051: Determine the total loss of the regularization term through the initial shared features of each modality.

[0291] In practical implementation, the aforementioned emotion recognition device can introduce a regularization term and calculate the total loss of the regularization term using a preset regularization formula. The preset regularization formula is as follows:

[0292] ;

[0293] ;

[0294] in, This represents the total loss of the regularization term. Indicates a compactness penalty term. Indicates the difference-promoting item. , Hyperparameters representing the strength of regularization Representing modes i Specific feature vectors, express i Modal eigenvectors Norm, Representing modes i and modality j Differences in shared feature vectors between them This represents the difference between the shared eigenvectors of mode i and mode j. Norm, l Representing text modality, a Representing audio modality, v Indicates video modality, ε This represents a positive coefficient to prevent the denominator from being zero.

[0295] Step S7052: The modality discriminator predicts the source of the modality-shared features to obtain the source prediction probability.

[0296] In a practical implementation, the aforementioned emotion recognition device can input modality-shared features into a modality discriminator, construct a label classification task through the modality discriminator, predict the source of the modality-shared features, and obtain the source prediction probability.

[0297] Step S7053: Determine the adversarial modality classification loss based on the source prediction probability using a preset adversarial loss formula.

[0298] The preset adversarial loss formula is:

[0299] ;

[0300] in, This represents the adversarial modality classification loss. Indicates the number of training samples. Indicates the number of modal categories. Indicates the first i Each sample belongs to the modality ja reading heat vector, a modal shared feature representing the i th sample, a prediction probability of the discriminator for the j th class of modal.

[0301] In a specific implementation, the above-mentioned sentiment recognition device can substitute the source prediction probability into a preset adversarial modal classification loss formula to obtain the adversarial modal classification loss.

[0302] Step S7054, determine a main task sentiment classification loss according to the sentiment prediction probability, and determine a joint loss based on the adversarial modal classification loss, the main task sentiment classification loss, and the regularization term total loss.

[0303] In a specific implementation, the above-mentioned sentiment recognition device can calculate the main task sentiment classification loss by using a preset sentiment classification loss formula, which is as follows:

[0304] ;

[0305] wherein, the main task sentiment classification loss, CrossEntropyLoss the cross-entropy loss function, the sentiment prediction probability, T true the true label.

[0306] Further, the above-mentioned sentiment recognition device can substitute the adversarial modal classification loss, the main task sentiment classification loss, and the regularization term total loss into a preset joint loss formula to obtain the joint loss. The preset joint loss formula is as follows:

[0307] ;

[0308] wherein, the joint loss, the main task sentiment classification loss, the adversarial modal classification loss, the regularization term total loss, a weight coefficient for controlling the adversarial modal classification loss in the total loss, a weight coefficient for controlling the regularization term total loss in the total loss.

[0309] Step S7055, update the main tasker, the modal discriminator, and the shared feature extractor respectively by using the joint loss.

[0310] In a specific implementation, the emotion recognition device can update parameters of the main tasker, the modal discriminator, and the shared feature extractor through the joint loss, thereby improving the accuracy of the emotion prediction probability.

[0311] The embodiment extracts modal initial shared features of each emotion-enhanced modal feature through a shared encoder, and splices each modal initial shared feature to obtain modal shared features; decomposes each emotion-enhanced modal feature into a corresponding modal unique feature through a unique encoder; generates candidate prediction probabilities of the multi-modal fusion interaction feature, the modal shared feature, and each modal unique feature respectively; calculates attention weights of each candidate prediction probability respectively; and outputs an emotion prediction probability based on each candidate prediction probability and the corresponding attention weight through a preset fusion prediction formula by the main tasker, which can effectively integrate common and unique features in multi-modal data, improve the accuracy of emotion prediction and the robustness of the model through multi-angle prediction and attention mechanism.

[0312] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the emotion recognition method based on the double hypergraph neural network of the present application. More forms of simple transformation based on this technical concept are within the protection scope of the present application.

[0313] The present application also provides an emotion recognition system based on a double hypergraph neural network, which is described in detail in the following Figure 4 , Figure 4 is a module structure schematic diagram of the emotion recognition system based on the double hypergraph neural network of the present application. The emotion recognition system based on the double hypergraph neural network comprises:

[0314] The feature extraction module 10 is configured to extract features from the multi-modal data to obtain modal initial features corresponding to each modal data.

[0315] The attention module 20 is configured to enhance each modal initial feature through a multi-scale attention mechanism to obtain a modal enhanced feature corresponding to each modal initial feature.

[0316] The emotion-enhanced hypergraph module 30 is configured to construct an emotion-enhanced hypergraph based on each modal enhanced feature. The nodes of the emotion-enhanced hypergraph include emotion concept nodes and modal enhanced feature nodes corresponding to the modal enhanced features. The modal enhanced feature nodes and the emotion concept nodes are connected through emotion concept hyperedges.

[0317] The emotion-enhanced hypergraph module 30 is further configured to aggregate the modal enhanced feature nodes and the connected emotion concept nodes in the emotion-enhanced hypergraph to obtain emotion-enhanced modal features corresponding to each modal enhanced feature.

[0318] The modal interaction hypergraph module 40 is configured to determine inter-modal hyperedge information between the emotion-enhanced modal features, and construct a modal interaction hypergraph based on the inter-modal hyperedge information.

[0319] The modal interaction hypergraph module 40 is further configured to generate a multi-modal fusion interaction feature based on the modal interaction hypergraph.

[0320] The emotion recognition module 50 is configured to output an emotion prediction probability based on the multi-modal fusion interaction feature.

[0321] The emotion recognition system based on the dual-hypergraph neural network provided in the present application adopts the emotion recognition method based on the dual-hypergraph neural network in the above embodiments, and can solve the technical problem of low accuracy of emotion recognition in the prior art. Compared with the prior art, the emotion recognition system based on the dual-hypergraph neural network provided in the present application has the same beneficial effects as the emotion recognition method based on the dual-hypergraph neural network provided in the above embodiments, and other technical features in the emotion recognition system based on the dual-hypergraph neural network are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0322] The present application provides an emotion recognition device based on a dual-hypergraph neural network. The emotion recognition device based on a dual-hypergraph neural network comprises at least one processor and a memory in communication connection with the at least one processor. The memory stores instructions executable by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the emotion recognition method based on a dual-hypergraph neural network in the above embodiment one.

[0323] Reference will be made to the following Figure 5 , Figure 5 FIG. 1 is a structural schematic diagram of an emotion recognition device based on a dual-hypergraph neural network according to an embodiment of the present application. The emotion recognition device based on a dual-hypergraph neural network in the embodiment of the present application can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle terminals (such as vehicle navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 5 The emotion recognition device based on a dual-hypergraph neural network shown is only an example, and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0324] As Figure 5As shown, the emotion recognition device based on dual hypergraph neural network can include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. In the RAM 1004, various programs and data required for the operation of the emotion recognition device based on dual hypergraph neural network are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the emotion recognition device based on dual hypergraph neural network to communicate with other devices wirelessly or by wire to exchange data. Although the emotion recognition device based on dual hypergraph neural network with various systems is shown in the figure, it should be understood that all the systems shown are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0325] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0326] The emotion recognition device based on the dual hypergraph neural network provided in the application adopts the emotion recognition method based on the dual hypergraph neural network in the above embodiment, and can solve the technical problem of low accuracy of emotion recognition in the prior art. Compared with the prior art, the emotion recognition device based on the dual hypergraph neural network provided in the application has the same beneficial effects as the emotion recognition method based on the dual hypergraph neural network provided in the above embodiment, and other technical features in the emotion recognition device based on the dual hypergraph neural network are the same as the features disclosed in the above embodiment method, and will not be repeated here.

[0327] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0328] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0329] The above is only part of the embodiments of the present application, and does not limit the patent scope of the present application, and any equivalent structural transformation made by using the content of the present application specification and drawings, or direct / indirect application in other related technical fields is included in the patent protection scope of the present application.

Claims

1. A method for emotion recognition based on a dual hypergraph neural network, characterized in that, The method comprises: feature extraction is performed on multi-modal data to obtain modal initial features corresponding to each modal data, wherein the multi-modal data at least includes text data, video data and audio data; each of the modal initial features is enhanced through a multi-scale attention mechanism to obtain modal enhanced features corresponding to each of the modal initial features; each of the modal initial features is enhanced through a preset temporal attention formula to obtain temporal attention enhanced features corresponding to each of the modal initial features; each of the modal initial features is enhanced through a preset channel attention formula to obtain channel attention enhanced features corresponding to each of the modal initial features; each of the modal initial features is enhanced through a preset spatial attention formula to obtain spatial attention enhanced features corresponding to each of the modal initial features; the temporal attention enhanced features, the channel attention enhanced features and the spatial attention enhanced features corresponding to the same modal initial feature are fused to obtain modal enhanced features corresponding to each of the modal initial features; a sentiment enhanced supergraph is constructed based on each of the modal enhanced features, nodes of the sentiment enhanced supergraph include sentiment concept nodes and modal enhanced feature nodes corresponding to the modal enhanced features, and the modal enhanced feature nodes and the sentiment concept nodes are connected through sentiment concept superedges; the modal enhanced feature nodes and the sentiment concept nodes connected thereto in the sentiment enhanced supergraph are aggregated to obtain sentiment enhanced modal features corresponding to each of the modal enhanced features; inter-modal superedge information between each of the sentiment enhanced modal features is determined, and a modal interaction supergraph is constructed based on the inter-modal superedge information; a multi-modal fusion interaction feature is generated through the modal interaction supergraph; a sentiment prediction probability is output according to the multi-modal fusion interaction feature.

2. The sentiment recognition method based on a dual-hypergraph neural network according to claim 1, wherein, The step of enhancing each of the modal initial features through a multi-scale attention mechanism to obtain modal enhanced features corresponding to each of the modal initial features comprises: each of the modal initial features is enhanced through a preset temporal attention formula to obtain temporal attention enhanced features corresponding to each of the modal initial features, the preset temporal attention formula is: wherein, denotes a temporal attention enhanced feature, denotes a query matrix, denotes a key matrix, denotes a value matrix, denotes a dimension of a key vector, F denotes a modal initial feature, denotes a random initial query weight matrix, denotes a random initial key weight, denotes a random initial value weight matrix; each of the modal initial features is enhanced through a preset channel attention formula to obtain channel attention enhanced features corresponding to each of the modal initial features, the channel attention formula is: wherein, denotes a channel attention enhanced feature, denotes a modality initial feature, denotes a global average pooling operation, denotes a global max pooling operation, denotes a multi-layer perceptron, denotes sigmoid activation function; each of the modal initial features is enhanced through a preset spatial attention formula to obtain spatial attention enhanced features corresponding to each of the modal initial features, the spatial attention formula is: wherein, denotes spatial attention enhanced features, denotes a convolution operation, denotes a global average pooling, denotes a global max pooling output, denotes sigmoid an activation function, denotes modal initial features; the temporal attention enhanced features, the channel attention enhanced features and the spatial attention enhanced features corresponding to the same modal initial feature are fused to obtain modal enhanced features corresponding to each of the modal initial features.

3. The sentiment recognition method based on double hypergraph neural network according to claim 2, wherein, The step of constructing a sentiment enhanced supergraph based on each of the modal enhanced features comprises: a sentiment embedding matrix is generated based on an input sequence through a preset sentiment embedding formula, the preset sentiment embedding formula is: wherein, denotes an emotion embedding matrix, denotes a globally shared external knowledge base, denotes an input sequence; generate a remodeled feature matrix corresponding to each of the modal enhanced features through a first preset feature flattening formula, the first preset feature flattening formula being: wherein, represents a reshaped feature matrix, represents a modal enhanced feature, represents a batch size, represents a number of time points or frames contained in each sequence, represents a feature dimension; calculate the similarity between the sentiment embedding matrix and each of the remodeled feature matrices through a preset similarity formula to obtain a similarity matrix corresponding to each of the remodeled feature matrices, the preset similarity formula being: wherein, denotes a similarity matrix, denotes an L2 normalization function, denotes a reshaped feature matrix, denotes an emotion embedding matrix; determine a similarity value matrix and an index matrix based on each of the similarity matrices through a preset similarity value formula, the preset similarity value formula being: wherein, denotes a similarity value matrix, denotes an index matrix, is a function for finding the K largest values in the last dimension of the tensor and the corresponding indices, set K = 3, denotes a similarity matrix; determine a sentiment hyperedge weight set according to the similarity value matrix; determine a sentiment hyperedge set according to the index matrix; generate a sentiment enhanced hypergraph based on the sentiment hyperedge weight set and the sentiment hyperedge set through a preset hypergraph construction formula, the preset hypergraph construction formula being: wherein, represents an affect-enhanced hypergraph, represents a constructor for a hypergraph object, represents a set of all nodes in an affect-enhanced hypergraph, represents a set of hyperedges, represents a set of hyperedge weights.

4. The sentiment recognition method based on double hypergraph neural network according to claim 3, wherein, The step of aggregating the modal enhanced feature nodes and the sentiment concept nodes connected thereto in the sentiment enhanced hypergraph to obtain sentiment enhanced modal features corresponding to each of the modal enhanced features, comprising: capture long and short term dependencies in each of the modal enhanced features through a preset time sequence context perception formula to generate a hidden state tensor corresponding to each of the modal enhanced features, the hidden state tensor including context information at each time point in the modal enhanced feature, the preset time sequence context perception formula being: wherein, denotes a hidden state tensor, GRU denotes a gated recurrent unit network layer, denotes a text modality enhanced feature, denotes a video modality enhanced feature, an audio modality enhanced feature; generate a gating signal tensor corresponding to each of the hidden state tensors through a preset long and short term network gating formula, the preset long and short term network gating formula being: wherein, denotes a gating signal tensor, denotes a sigmoid activation function, denotes a hidden state tensor, denotes a fully connected layer; combine each of the modal enhanced features and the corresponding gating signal tensor through a preset gating feature fusion formula to obtain a gated modal feature corresponding to each of the modal enhanced features, the preset gating feature fusion formula being: wherein, denotes the gated post-modal feature, denotes the gating signal tensor, denotes the text modal augmented feature, denotes the video modal augmented feature, denotes the audio modal augmented feature; aggregate the modal enhanced feature nodes and the sentiment concept nodes connected thereto in the sentiment enhanced hypergraph based on each of the gated modal features to obtain sentiment enhanced modal features corresponding to each of the modal enhanced features.

5. The sentiment recognition method based on double hypergraph neural network according to claim 4, wherein, The step of aggregating the modal enhanced feature nodes and the sentiment concept nodes connected thereto in the sentiment enhanced hypergraph based on each of the gated modal features to obtain sentiment enhanced modal features corresponding to each of the modal enhanced features, comprising: generate a gated feature flattening matrix corresponding to each of the gated modal features through a second preset feature flattening formula, the second preset feature flattening formula being: wherein, represents a gating feature flattening matrix, represents a flattening operation, represents gated post-modality features, represents a batch size, represents the number of time points or frames contained in each sequence, represents an input feature dimension; splice each of the gated feature flattening matrices and the sentiment embedding matrix through a preset feature splicing formula to obtain a node feature matrix containing all nodes of the sentiment enhanced hypergraph, the preset feature splicing formula being: wherein, represents a node feature matrix of the sentiment-enhanced hypergraph, represents a concatenation operation, represents a gating feature flattening matrix, represents a sentiment embedding matrix, is a specified dimension; input the node feature matrix of the sentiment enhanced hypergraph into a multi-layer hypergraph convolution layer in a hypergraph neural network for processing, aggregate each of the modal enhanced feature nodes and the sentiment concept nodes connected thereto in the node feature matrix through each layer of the hypergraph convolution layer to obtain an updated node feature matrix; perform feature enhancement on the updated node feature matrix through a preset graph domain to time sequence domain feature conversion formula to obtain sentiment enhanced modal features corresponding to each of the modal enhanced features, the preset graph domain to time sequence domain feature conversion formula being: wherein, represents an emotion enhancement modal feature, represents a flattening operation, represents an updated node feature matrix, represents a slicing operation, represents a batch size, represents a number of time points or frames contained in each sequence, represents an output feature dimension.

6. The sentiment recognition method based on double hypergraph neural network according to claim 1, wherein, The step of generating the multi-modal fusion interaction feature through the modal interaction supergraph comprises: inputting the node feature matrix of the modal interaction supergraph into a multi-layer supergraph convolution layer for processing to obtain a target node feature matrix after depth fusion and residual connection; determining a reshaped fusion feature tensor based on the target node feature matrix through a preset reshaping formula, the preset reshaping formula being: wherein, represents the reshaped fusion feature tensor, represents the flattened node feature restored to a four-dimensional structure, represents the target node feature matrix, represents the batch size, represents the number of modalities, represents the sequence length, represents the output feature dimension; determining a post-fusion sentiment-enhanced modal feature based on the reshaped fusion feature tensor through a preset fusion formula, the preset fusion formula being: wherein, represents the fused affective augmentation modal feature, represents the reshaped fused feature tensor, represents the mean-pooling along the modal dimension ; normalizing the post-fusion sentiment-enhanced modal feature to obtain a multi-modal fusion interaction feature.

7. The dual-hypergraph neural network-based sentiment recognition method of any one of claims 1 to 6, wherein, The step of outputting a sentiment prediction probability according to the multi-modal fusion interaction feature comprises: extracting modal initial shared features of each of the sentiment-enhanced modal features through a shared encoder, and splicing each of the modal initial shared features to obtain a modal shared feature; decomposing each of the sentiment-enhanced modal features into a corresponding modal unique feature through a unique encoder; generating candidate prediction probabilities of the multi-modal fusion interaction feature, the modal shared feature, and each of the modal unique features, respectively; calculating attention weights of each of the candidate prediction probabilities, respectively; outputting a sentiment prediction probability based on each candidate prediction probability and the corresponding attention weight through a main tasker calling a preset fusion prediction formula, the preset fusion prediction formula being: wherein, denotes the emotion prediction probability, denotes a set of multi-modal fusion interaction features, modal shared features, and modal unique features, denotes the candidate prediction probability, denotes the attention weight.

8. The sentiment recognition method based on double hypergraph neural network according to claim 7, wherein, After the step of outputting a sentiment prediction probability based on each candidate prediction probability and the corresponding attention weight through a main tasker calling a preset fusion prediction formula, the step further comprises: determining a total regularization term loss through each of the modal initial shared features; predicting the source of the modal shared feature through a modal discriminator to obtain a source prediction probability; determining an adversarial modal classification loss based on the source prediction probability through a preset adversarial loss formula, the preset adversarial loss formula being: wherein, represents the adversarial modal classification loss, represents the number of training samples, represents the number of modal categories, represents the read heat vector of the i th sample belonging to the modal j , represents the modal shared feature of the i th sample, represents the predicted probability of the j th modal by the discriminator; determining a main task sentiment classification loss according to the sentiment prediction probability, and determining a joint loss based on the adversarial modal classification loss, the main task sentiment classification loss, and the total regularization term loss; updating the main tasker, the modal discriminator, and the shared feature extractor through the joint loss, respectively.

9. A sentiment recognition system based on a dual hypergraph neural network, characterized in that, The method of claim 1 is executed, and the system comprises: a feature extraction module configured to perform feature extraction on multi-modal data to obtain modal initial features corresponding to each modal data, wherein the multi-modal data at least includes text data, video data, and audio data; an attention module configured to enhance each of the modal initial features through a multi-scale attention mechanism to obtain modal enhanced features corresponding to each of the modal initial features; an attention module configured to enhance each of the modal initial features through a preset temporal attention formula to obtain temporal attention enhanced features corresponding to each of the modal initial features; an attention module configured to enhance each of the modal initial features through a preset channel attention formula to obtain channel attention enhanced features corresponding to each of the modal initial features; The attention module is configured to enhance each of the initial modal features by a preset spatial attention formula to obtain spatial attention enhanced features corresponding to the initial modal features; The attention module is configured to fuse the temporal attention enhanced features, the channel attention enhanced features and the spatial attention enhanced features corresponding to the same initial modal feature to obtain modal enhanced features corresponding to each of the initial modal features; The emotion enhanced hypergraph module is configured to construct an emotion enhanced hypergraph based on the modal enhanced features, wherein nodes of the emotion enhanced hypergraph include emotion concept nodes and modal enhanced feature nodes corresponding to the modal enhanced features, and the modal enhanced feature nodes and the emotion concept nodes are connected by emotion concept hyperedges; The emotion enhanced hypergraph module is further configured to aggregate the modal enhanced feature nodes and the connected emotion concept nodes in the emotion enhanced hypergraph to obtain emotion enhanced modal features corresponding to each of the modal enhanced features; The modal interaction hypergraph module is configured to determine inter-modal hyperedge information between the emotion enhanced modal features, and construct a modal interaction hypergraph based on the inter-modal hyperedge information; The modal interaction hypergraph module is further configured to generate multi-modal fusion interaction features through the modal interaction hypergraph; The emotion recognition module is configured to output emotion prediction probabilities according to the multi-modal fusion interaction features.

10. A sentiment recognition device based on a dual hypergraph neural network, characterized in that, The emotion recognition based on the dual-hypergraph neural network includes a memory, a processor and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the emotion recognition method based on the dual-hypergraph neural network in any one of claims 1 to 8.

Images