Sentiment Analysis Method Based on Modal Heterogeneity Bridging and Multimodal Dynamic Graph Fusion

Abstract

This invention discloses a sentiment analysis method based on modal heterogeneity bridging and multimodal dynamic graph fusion. The method includes: acquiring sentiment data of different modalities, including video modality, audio modality, and text modality; constructing a sentiment analysis model based on modal heterogeneity bridging and multimodal dynamic graph fusion; and performing sentiment analysis on the data to be analyzed according to the sentiment analysis model to obtain the sentiment analysis results. This invention has high accuracy and can be widely applied in the field of computer technology.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a sentiment analysis method based on modal heterogeneity bridging and multimodal dynamic graph fusion. Background Technology

[0002] Emotions are closely related to daily life, and many people frequently express their emotions and opinions through text, images, and videos on social media platforms. Currently, most researchers use three modalities for sentiment analysis: facial expression information, text information, and speech information. Multimodal sentiment analysis effectively overcomes the limitations of single-modal analysis, and the complementary information between different modalities can improve the accuracy of sentiment analysis. The main challenges in multimodal sentiment analysis tasks include multimodal data representation, multimodal fusion, and multimodal alignment. Information from different modalities differs significantly in data format and processing methods. Many current methods process each modality's data into a sequential structure, failing to effectively consider the unstructured nature of the data. Furthermore, the differences between the three modalities necessitate consideration of intermodal heterogeneity, noisy data, and semantic relationships within and between modalities before and during multimodal data fusion. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a highly accurate sentiment analysis method, apparatus, electronic device, and storage medium.

[0004] One aspect of this invention provides a sentiment analysis method based on modal heterogeneity bridging and multimodal dynamic graph fusion, comprising:

[0005] Acquire sentiment data in different modalities, including video, audio, and text modalities;

[0006] A sentiment analysis model is constructed based on intermodal heterogeneity bridging and multimodal dynamic graph fusion;

[0007] Based on the aforementioned sentiment analysis model, sentiment analysis is performed on the data to be analyzed, and the sentiment analysis results are obtained.

[0008] Optionally, acquiring sentiment data in different modalities, including video, audio, and text modalities, includes:

[0009] Acquire video data;

[0010] Extract audio features, facial features, and text features from the video data;

[0011] Configure the emotional polarity for each video; wherein the emotional polarity includes negative emotions, positive emotions, and neutral attitudes;

[0012] The audio features include tone and emotion information; the facial features include facial information of the target object.

[0013] Optionally, the construction of the sentiment analysis model based on intermodal heterogeneity bridging and multimodal dynamic graph fusion includes:

[0014] Sentiment data from different modalities are input into different network models, and feature extraction and standardization are performed on the sentiment data of each modality to obtain the node features of each modality.

[0015] Based on the node features of each mode, a unimodal graph is constructed for each mode;

[0016] Based on the constructed single-modal graph, the difference between modes is bridged to obtain the target modal features;

[0017] A multimodal graph is constructed based on the target modal features, and dynamic graph fusion processing under a multi-head attention mechanism is performed on this graph to obtain a multimodal fusion graph.

[0018] Based on the multimodal fusion graph, a sentiment analysis module is constructed.

[0019] Optionally, for sentiment data in text modalities, the construction of a unimodal graph for each modality based on the node features of each modality includes:

[0020] Node representation information is extracted using a bidirectional long short-term memory neural network;

[0021] Based on the node representation information, a unimodal graph of the sentiment data of the text modality is constructed using bidirectional edges.

[0022] Optionally, for sentiment data in audio and video modalities, the construction of a unimodal graph for each modality based on the node features of each modality includes:

[0023] By processing emotional data of different modalities through modality-specific feedforward neural networks, corresponding video and audio feature codes are obtained;

[0024] Based on the video and audio feature encoding, a single-peak graph of the video and audio is constructed using unweighted bidirectional edges.

[0025] Optionally, the step of performing intermodal difference bridging processing based on the constructed single-modal graph to obtain target modal features includes:

[0026] The single-modal graphs of different modalities are input into a two-layer graph convolution for graph convolution processing to complete the initial bridging;

[0027] After the graph convolution process, the graphs are bridged again through a fully connected feedforward layer and a nonlinear exponential unit, projecting the three unimodal graphs into a common embedding space to bridge the heterogeneity gap between modes.

[0028] Optionally, the step of constructing a multi-modal graph based on the target modal features and performing dynamic graph fusion processing on this graph under a multi-head attention mechanism to filter and obtain a multi-modal fused graph includes:

[0029] After constructing the unimodal graphs of the three modalities, a multimodal graph was constructed by performing a full connection operation on each text, audio, and video node using unweighted bidirectional edges.

[0030] A multi-layer dynamic fusion approach is adopted to dynamically fuse multimodal graphs;

[0031] The method of dynamically fusing multimodal graphs using a multi-layer dynamic fusion approach includes the following steps:

[0032] Multi-head attention mechanism is used to fuse multimodal graphs. For each node, there are several neighboring nodes. The edges of the neighboring nodes are incident on the central node. The initial attention score is calculated based on the two modalities of the node's edges.

[0033] Each node is regularized using Softmax;

[0034] Calculate the output features of each node;

[0035] The output information of all output features is concatenated to obtain the node representation;

[0036] Through fusion calculation, a node is transformed from a node containing only single-modal information into a multi-modal node with information from other modalities, thereby achieving intermodal fusion.

[0037] After the fusion operation at each layer, the average attention weight of all attention heads for each edge is calculated;

[0038] The average attention weights of all edges are sorted, and the edges with the highest weight values ​​are selected. Only these selected edges are included in the calculation in the next layer of fusion operation.

[0039] After one layer of dynamic fusion is completed, the output of the new multimodal graph is input into the next layer. After multiple fusions and filterings, the final multimodal fused graph is obtained.

[0040] Another aspect of this invention provides a sentiment analysis device based on modal heterogeneity bridging and multimodal dynamic graph fusion, comprising:

[0041] The first module is used to acquire sentiment data in different modalities, including video modality, audio modality, and text modality;

[0042] The second module is used to construct a sentiment analysis model based on intermodal heterogeneity bridging and multimodal dynamic graph fusion;

[0043] The third module is used to perform sentiment analysis on the data to be analyzed based on the sentiment analysis model, and obtain the sentiment analysis results.

[0044] Another aspect of the present invention provides an electronic device, including a processor and a memory;

[0045] The memory is used to store programs;

[0046] The processor executes the program to implement the method described above.

[0047] Another aspect of this invention provides a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.

[0048] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0049] The embodiments of the present invention first acquire sentiment data in different modalities, including video modality, audio modality, and text modality; based on the bridging of intermodal heterogeneity and the fusion of multimodal dynamic graphs, a sentiment analysis model is constructed; according to the sentiment analysis model, sentiment analysis is performed on the data to be analyzed to obtain the sentiment analysis results. The present invention has high accuracy. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating the overall steps of an embodiment of the present invention;

[0052] Figure 2 This is a schematic diagram of the structure of the sentiment analysis model provided in an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] To address the problems existing in the prior art, one aspect of this invention provides a sentiment analysis method based on modal heterogeneity bridging and multimodal dynamic graph fusion, comprising:

[0055] Acquire sentiment data in different modalities, including video, audio, and text modalities;

[0056] A sentiment analysis model is constructed based on intermodal heterogeneity bridging and multimodal dynamic graph fusion;

[0057] Based on the aforementioned sentiment analysis model, sentiment analysis is performed on the data to be analyzed, and the sentiment analysis results are obtained.

[0058] Optionally, acquiring sentiment data in different modalities, including video, audio, and text modalities, includes:

[0059] Acquire video data;

[0060] Extract audio features, facial features, and text features from the video data;

[0061] Configure the emotional polarity for each video; wherein the emotional polarity includes negative emotions, positive emotions, and neutral attitudes;

[0062] The audio features include tone and emotion information; the facial features include facial information of the target object.

[0063] Optionally, the construction of the sentiment analysis model based on intermodal heterogeneity bridging and multimodal dynamic graph fusion includes:

[0064] Sentiment data from different modalities are input into different network models, and feature extraction and standardization are performed on the sentiment data of each modality to obtain the node features of each modality.

[0065] Based on the node features of each mode, a unimodal graph is constructed for each mode;

[0066] Based on the constructed single-modal graph, the difference between modes is bridged to obtain the target modal features;

[0067] A multimodal graph is constructed based on the target modal features, and dynamic graph fusion processing under a multi-head attention mechanism is performed on this graph to obtain a multimodal fusion graph.

[0068] Based on the multimodal fusion graph, a sentiment analysis module is constructed.

[0069] Optionally, for sentiment data in text modalities, the construction of a unimodal graph for each modality based on the node features of each modality includes:

[0070] Node representation information is extracted using a bidirectional long short-term memory neural network;

[0071] Based on the node representation information, a unimodal graph of the sentiment data of the text modality is constructed using bidirectional edges.

[0072] Optionally, for sentiment data in audio and video modalities, the construction of a unimodal graph for each modality based on the node features of each modality includes:

[0073] By processing emotional data of different modalities through modality-specific feedforward neural networks, corresponding video and audio feature codes are obtained;

[0074] Based on the video and audio feature encoding, a single-peak graph of the video and audio is constructed using unweighted bidirectional edges.

[0075] Optionally, the step of performing intermodal difference bridging processing based on the constructed single-modal graph to obtain target modal features includes:

[0076] The single-modal graphs of different modalities are input into a two-layer graph convolution for graph convolution processing to complete the initial bridging;

[0077] After the graph convolution process, the graphs are bridged again through a fully connected feedforward layer and a nonlinear exponential unit, projecting the three unimodal graphs into a common embedding space to bridge the heterogeneity gap between modes.

[0078] Optionally, the step of constructing a multi-modal graph based on the target modal features and performing dynamic graph fusion processing on this graph under a multi-head attention mechanism to filter and obtain a multi-modal fused graph includes:

[0079] After constructing the unimodal graphs of the three modalities, a multimodal graph was constructed by performing a full connection operation on each text, audio, and video node using unweighted bidirectional edges.

[0080] A multi-layer dynamic fusion approach is adopted to dynamically fuse multimodal graphs;

[0081] The method of dynamically fusing multimodal graphs using a multi-layer dynamic fusion approach includes the following steps:

[0082] Multi-head attention mechanism is used to fuse multimodal graphs. For each node, there are several neighboring nodes. The edges of the neighboring nodes are incident on the central node. The initial attention score is calculated based on the two modalities of the node's edges.

[0083] Each node is regularized using Softmax;

[0084] Calculate the output features of each node;

[0085] The output information of all output features is concatenated to obtain the node representation;

[0086] Through fusion calculation, a node is transformed from a node containing only single-modal information into a multi-modal node with information from other modalities, thereby achieving intermodal fusion.

[0087] After the fusion operation at each layer, the average attention weight of all attention heads for each edge is calculated;

[0088] The average attention weights of all edges are sorted, and the edges with the highest weight values ​​are selected. Only these selected edges are included in the calculation in the next layer of fusion operation.

[0089] After one layer of dynamic fusion is completed, the output of the new multimodal graph is input into the next layer. After multiple fusions and filterings, the final multimodal fused graph is obtained.

[0090] Another aspect of this invention provides a sentiment analysis device based on modal heterogeneity bridging and multimodal dynamic graph fusion, comprising:

[0091] The first module is used to acquire sentiment data in different modalities, including video modality, audio modality, and text modality;

[0092] The second module is used to construct a sentiment analysis model based on intermodal heterogeneity bridging and multimodal dynamic graph fusion;

[0093] The third module is used to perform sentiment analysis on the data to be analyzed based on the sentiment analysis model, and obtain the sentiment analysis results.

[0094] Another aspect of the present invention provides an electronic device, including a processor and a memory;

[0095] The memory is used to store programs;

[0096] The processor executes the program to implement the method described above.

[0097] Another aspect of this invention provides a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.

[0098] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0099] The specific implementation process of the present invention will now be described in detail with reference to the accompanying drawings:

[0100] like Figure 1 As shown, the method of the present invention generally includes the following steps:

[0101] S1. Processing multimodal sentiment data; S2. Model construction based on intermodal heterogeneity bridging and multimodal dynamic graph fusion; S3. Multimodal sentiment analysis.

[0102] Specifically, the processing of multimodal sentiment data is described as follows:

[0103] With the development of internet technology and social networking, more and more people are posting videos online to showcase their lifestyles and various opinions. This video data typically includes the speaker's facial expressions, the spoken text, and the emotions and tone of voice used. These can generally correspond to the three modalities of video, text, and audio in multimodal research. To extract these three modalities from the video, this embodiment uses OpenSMIL to extract audio features containing tone and emotional information. Facial expression information is crucial for understanding and analyzing emotions. In this embodiment, the video is segmented into frames, facial information is obtained using the OpenFace facial behavior analysis tool and principal component analysis, and facial features are extracted using a convolutional recurrent neural network. This embodiment uses an automatic speech recognition system to extract text information from video and uses GloVe word embeddings to obtain pre-trained word vectors of the text. Each video has its own emotional polarity. In this embodiment, the emotional characteristics of each video are manually represented by a number between -3 and +3. The magnitude of the number represents the positivity of the emotional state, with negative numbers representing negative emotions (-3 representing highly negative emotions), positive numbers representing positive emotions (3 representing highly positive emotions), and 0 representing a neutral attitude.

[0104] The following describes in detail the model construction based on bridging the heterogeneity gap between modes and dynamically fusing multimodal graphs:

[0105] The structure of the model is as follows Figure 2 As shown, the model's workflow includes:

[0106] 1. Feature processing of single modes

[0107] First, to obtain node features for the three heterogeneous modalities and ensure that the size and dimensionality of each node feature are uniform, this embodiment inputs the feature vectors of each modality into different network models to perform feature extraction and standardization on the multimodal data. After obtaining the corresponding node features for each modality, a unimodal graph is constructed for each modality. At this point, the node dimensions of the same modality are the same, but the dimensions between different modalities are not yet uniform.

[0108] Text modality:

[0109] 1) Node construction:

[0110] The meaning and sentiment of a text are strongly dependent on its context; the same word appearing in different sentences often conveys different meanings. To address the context-dependent problem, a bidirectional long short-term memory neural network is chosen when extracting node representation information. A sentence containing multiple words can be represented as follows: Where n is the number of words. The text feature encoding is as follows:

[0111]

[0112] Among them, t i Represents the feature of the i-th node in the text modality. This represents the i-th word in the sentence. and This represents the use of LSTM for forward and reverse feature extraction, respectively. The concatenation of the two extracted feature vectors yields node t. i Feature representation.

[0113] 2) Construct a unimodal graph using bidirectional edges:

[0114] A text unimodal plot can be represented as graph G. T =(N T E T ), node N T ={t1,t2,···t n The text unimodal graph is obtained by fully connecting nodes with unweighted bidirectional edges using text feature representations extracted from the text model.

[0115] Video and audio modalities:

[0116] 1) Node construction:

[0117] The processed original feature vectors are then passed through modality-specific feedforward neural networks to obtain the corresponding video and audio feature codes, as shown below:

[0118]

[0119]

[0120] Where v i a i These represent the feature representations of video and audio nodes, respectively, where ω is the weight matrix and b i It is a bias.

[0121] 2) Construct a unimodal graph using bidirectional edges:

[0122] A single-peak plot of video and audio can be represented as G j =(N j E j ), where j∈(V,A). When j=V, node N v ={v1,v2,···v n} represents the extracted video node feature representation. When j = A, node N A ={a1,a2,···a n} represents the extracted audio node feature representation. To maintain consistency with the text unimodal graph, this embodiment uses unweighted bidirectional edges to fully connect all video and audio nodes respectively, obtaining the final audio unimodal graph and video unimodal graph.

[0123] 2. Modal difference bridging based on unimodal plots

[0124] There is semantic correlation between the three modalities. In order to better utilize the information between heterogeneous modalities and reduce the gap between heterogeneous modal data, this embodiment designs a weight-sharing network layer in this module. Through this network layer, the three unimodal graphs are embedded into the same space, and all node features are processed to a uniform dimension.

[0125] First, the three unimodal graphs are each processed through a two-layer graph convolution. After passing through the graph convolution layer, the nodes of the three modalities undergo preliminary processing, which not only bridges the gaps but also increases the inter-modal information connections.

[0126]

[0127] Where gcn(·) represents the graph convolution operation, matrix A represents the normalized relationship matrix between nodes, and W () W () X represents a parameter that is randomly initialized. mThe features of the nodes form an N×D matrix, where D is the dimension of the node features, and the activation functions used are ReLU and Softmax, respectively.

[0128] Then, in this embodiment, a fully connected feedforward layer and a nonlinear exponential unit (ELU) are used after the convolutional layer to further bridge the gap between the modes. In this step, the parameter weights in this embodiment are shared, that is, the same weight matrix is ​​used for processing the information of each node. This allows the three unimodal plots to be projected into a common embedding space, thus bridging the heterogeneity gap between modes. The modal features after bridging are represented as follows.

[0129]

[0130]

[0131]

[0132] Among them, W M and b M These are parameters of the projection layer, v i a i , t i These are the node features after preliminary convolution processing. This is the feature representation of the final output video, audio, and text nodes, where all node features are in the same dimension.

[0133] 3. Dynamic fusion and filtering of multi-peak plots under a multi-head attention mechanism

[0134] After obtaining unimodal graphs of the three modalities through initial modal projection embedding, a new multimodal graph is constructed, and the unimodal nodes are fused to obtain the final multimodal fused graph. By dynamically fusing the multimodal graphs under a multi-head attention mechanism, the representational information between and within modalities is integrated, and the semantic relationships between modalities are further utilized to obtain richer feature data. This step consists of two parts: 1. Constructing the multimodal graph; 2. Dynamically fusing and filtering the multimodal graphs.

[0135] Step 1: Constructing a multi-peak plot

[0136] After constructing and processing the unimodal graphs of the three modalities, this embodiment uses unweighted bidirectional edges to perform a fully connected operation on each text, audio, and video node, thus constructing a multimodal graph G. m =(N m E m Among them, nodes These represent the features of video, audio, and text nodes, respectively. The total number of nodes is j+k+l+3, and E mThese are the edges connecting the nodes. A fully connected multimodal graph connects all the data information, which allows for better handling of semantic relationships between different modalities and enables learning from dependencies generated within and between modalities at a finer level.

[0137] Step 2: Dynamically fuse and filter multimodal graphs

[0138] When fusing multimodal graphs, this embodiment adopts a multi-layer dynamic fusion approach. Dynamic fusion can gradually focus the model's attention on key information, reducing attention to noise and irrelevant information, thereby improving the accuracy of sentiment analysis and reducing memory consumption. In this module, the operation at each layer is the same calculation process, each including one fusion and filtering operation. The process of one layer of fusion and filtering will be described in detail below.

[0139] 1) Dynamic fusion of multi-peak maps under multi-head attention mechanism

[0140] First, to capture richer node information and increase the robustness and stability of the network, this embodiment uses a multi-head attention mechanism for multimodal graph fusion. For each node N... i ∈N m There are several neighbor nodes N j Connected to it, the edges of neighboring nodes are incident on the central node N. i This embodiment calculates the original attention score based on the two modalities at the edge.

[0141]

[0142] e [], Represents node N j For node N i The importance of [h] represents the calculation of the h-th attention head, and [·||·] indicates concatenating two vectors. [] It is a weight matrix.

[0143] Next, this embodiment uses Softmax to apply N i Regularize the adjacent nodes of N, where N is the number of nodes in the set. p Represents the set of neighboring nodes, where [h] indicates that the h-th attention head is being calculated.

[0144]

[0145] Then calculate the h-th output feature for each node.

[0146]

[0147] Finally, the output information of all attention heads is concatenated to obtain the node representation.

[0148]

[0149] After fusion calculation, N i A node containing only single-modal information is transformed into a multi-modal node with information from other modalities, thereby achieving intermodal fusion.

[0150] 2) Dynamic edge filtering

[0151] Because the nodes in the multimodal graph designed in this embodiment are fully connected, there are interconnected edges both between and within modalities. This results in an excessive amount of edge data to be calculated, which is time-consuming and consumes a lot of storage space. In addition, some edges containing interfering information are also included in the calculation, distracting the model from focusing on important information and affecting the accuracy of the final sentiment prediction. To solve this problem, this embodiment filters edges by removing some unimportant edge information during the multimodal fusion process.

[0152] First, after the fusion operation at each layer, the average attention weight of all attention heads for each edge is calculated.

[0153]

[0154] Where τ [],, Representing edge e i, The h-th attention weight, where H represents the number of attention heads.

[0155] Subsequently, the average attention weights of all edges are sorted, and the k% of edges with the highest weight values ​​are selected, where k is a hyperparameter. In the next layer's fusion operation, only these selected edges participate in the calculation, while other edges are discarded.

[0156] After one layer of dynamic fusion is completed, the output of the new multimodal graph is input into the next layer. After multiple fusions and filterings, the final multimodal fused graph is obtained.

[0157] The multimodal sentiment analysis process is described in detail below:

[0158] Before sentiment prediction, the final multimodal fusion graph needs to be processed into vector feature form and output. Because the fusion process involves pruning the edges of the multimodal graph, some nodes may end up with no connecting edges, indicating that these nodes did not participate in subsequent node update operations and their information can be discarded. Therefore, only the feature information of all nodes with connecting edges is output into a single vector S, which is then passed to a 3-layer perceptron to predict sentiment characteristics.

[0159]

[0160] Where σ(·) represents the ReLU activation function, denoted by , and W represents the predicted sentiment label.

[0161] The model training process is described in detail below:

[0162] This invention designs an end-to-end regression model. When training the model, the mean absolute error will be used to calculate the loss of the multimodal sentiment analysis task. The loss function of the entire model is as follows.

[0163]

[0164] Among them, y i and Let represent the true sentiment and predicted sentiment polarity of the i-th sample, respectively, and m be the number of samples participating in the training.

[0165] In summary, this invention utilizes graph convolutional neural networks and multi-head attention mechanisms to design a sentiment analysis method based on bridging intermodal heterogeneity and multimodal dynamic graph fusion. This model transforms three types of sequence data into graph structures with nodes and edges, and adds a weight-sharing network layer to bridge the heterogeneity gap between the various modalities, ensuring that all modalities reside in the same embedding space. Simultaneously, a multi-head attention mechanism is used to fuse the multimodal graphs, continuously pruning and updating the graph during the fusion process to increase the model's focus on key information, thereby improving the accuracy of multimodal sentiment analysis.

[0166] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0167] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0168] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0169] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0170] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0171] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0172] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0173] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0174] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A sentiment analysis method based on modal heterogeneity bridging and multimodal dynamic graph fusion, characterized in that, include: Acquire sentiment data in different modalities, including video, audio, and text modalities; A sentiment analysis model is constructed based on intermodal heterogeneity bridging and multimodal dynamic graph fusion; Based on the aforementioned sentiment analysis model, sentiment analysis is performed on the data to be analyzed, and the sentiment analysis results are obtained. The aforementioned construction of a sentiment analysis model based on intermodal heterogeneity bridging and multimodal dynamic graph fusion includes: Sentiment data from different modalities are input into different network models, and feature extraction and standardization are performed on the sentiment data of each modality to obtain the node features of each modality. Based on the node features of each mode, a unimodal graph is constructed for each mode; Based on the constructed single-modal graph, the difference between modes is bridged to obtain the target modal features; A multimodal graph is constructed based on the target modal features, and dynamic graph fusion processing under a multi-head attention mechanism is performed on this graph to obtain a multimodal fusion graph. Based on the multimodal fusion graph, a sentiment analysis model is constructed; The modal difference bridging process is used to project the single modal graphs of the three modes into the same embedding space. The step of constructing a multi-modal graph based on the target modal features and performing dynamic graph fusion processing on this graph using a multi-head attention mechanism to obtain a multi-modal fused graph includes: After constructing the unimodal graphs of the three modalities, a multimodal graph was constructed by performing a full connection operation on each text, audio, and video node using unweighted bidirectional edges. A multi-layer dynamic fusion approach is adopted to dynamically fuse multimodal graphs; The method of dynamically fusing multimodal graphs using a multi-layer dynamic fusion approach includes the following steps: Multi-head attention mechanism is used to fuse multimodal graphs. For each node, there are several neighboring nodes. The edges of the neighboring nodes are incident on the central node. The initial attention score is calculated based on the two modalities of the node's edges. Each node is regularized using Softmax; Calculate the output features of each node; The output information of all output features is concatenated to obtain the node representation; Through fusion calculation, a node is transformed from a node containing only single-modal information into a multi-modal node with information from other modalities, thereby achieving intermodal fusion. After the fusion operation at each layer, the average attention weight of all attention heads for each edge is calculated; The average attention weights of all edges are sorted, and the edges with the highest weight values ​​are selected. Only these selected edges are included in the calculation in the next layer of fusion operation. After one layer of dynamic fusion is completed, the output of the new multimodal graph is input into the next layer. After multiple fusions and filterings, the final multimodal fused graph is obtained.

2. The sentiment analysis method based on modal heterogeneity bridging and multimodal dynamic graph fusion according to claim 1, characterized in that, The acquisition of sentiment data in different modalities, including video, audio, and text modalities, includes: Acquire video data; Extract audio features, facial features, and text features from the video data; Configure the emotional polarity for each video; wherein the emotional polarity includes negative emotions, positive emotions, and neutral attitudes; The audio features include tone and emotion information; the facial features include facial information of the target object.

3. The sentiment analysis method based on modal heterogeneity bridging and multimodal dynamic graph fusion according to claim 1, characterized in that, For sentiment data in text modalities, the construction of a unimodal graph for each modality based on the node features of each modality includes: Node representation information is extracted using a bidirectional long short-term memory neural network; Based on the node representation information, a unimodal graph of the sentiment data of the text modality is constructed using bidirectional edges.

4. The sentiment analysis method based on modal heterogeneity bridging and multimodal dynamic graph fusion according to claim 1, characterized in that, For sentiment data in audio and video modalities, the construction of a unimodal graph for each modality based on the node features of each modality includes: By processing emotional data of different modalities through modality-specific feedforward neural networks, corresponding video and audio feature codes are obtained; Based on the video and audio feature encoding, a single-peak graph of the video and audio is constructed using unweighted bidirectional edges.

5. The sentiment analysis method based on modal heterogeneity bridging and multimodal dynamic graph fusion according to claim 1, characterized in that, The process of constructing a single-modal graph and then performing inter-modal difference bridging to obtain target modal features includes: The single-modal graphs of different modalities are input into a two-layer graph convolution for graph convolution processing to complete the initial bridging; After the graph convolution process, the graphs are bridged again through a fully connected feedforward layer and a nonlinear exponential unit, projecting the three unimodal graphs into a common embedding space to bridge the heterogeneity gap between modes.

6. A sentiment analysis device based on modal heterogeneity bridging and multimodal dynamic graph fusion, characterized in that, include: The first module is used to acquire sentiment data in different modalities, including video modality, audio modality, and text modality; The second module is used to construct a sentiment analysis model based on intermodal heterogeneity bridging and multimodal dynamic graph fusion; The third module is used to perform sentiment analysis on the data to be analyzed based on the sentiment analysis model, and obtain the sentiment analysis results. The aforementioned construction of a sentiment analysis model based on intermodal heterogeneity bridging and multimodal dynamic graph fusion includes: Sentiment data from different modalities are input into different network models, and feature extraction and standardization are performed on the sentiment data of each modality to obtain the node features of each modality. Based on the node features of each mode, a unimodal graph is constructed for each mode; Based on the constructed single-modal graph, the difference between modes is bridged to obtain the target modal features; A multimodal graph is constructed based on the target modal features, and dynamic graph fusion processing under a multi-head attention mechanism is performed on this graph to obtain a multimodal fusion graph. Based on the multimodal fusion graph, a sentiment analysis model is constructed; The modal difference bridging process is used to project the single modal graphs of the three modes into the same embedding space. The step of constructing a multi-modal graph based on the target modal features and performing dynamic graph fusion processing on this graph using a multi-head attention mechanism to obtain a multi-modal fused graph includes: After constructing the unimodal graphs of the three modalities, a multimodal graph was constructed by performing a full connection operation on each text, audio, and video node using unweighted bidirectional edges. A multi-layer dynamic fusion approach is adopted to dynamically fuse multimodal graphs; The method of dynamically fusing multimodal graphs using a multi-layer dynamic fusion approach includes the following steps: Multi-head attention mechanism is used to fuse multimodal graphs. For each node, there are several neighboring nodes. The edges of the neighboring nodes are incident on the central node. The initial attention score is calculated based on the two modalities of the node's edges. Each node is regularized using Softmax; Calculate the output features of each node; The output information of all output features is concatenated to obtain the node representation; Through fusion calculation, a node is transformed from a node containing only single-modal information into a multi-modal node with information from other modalities, thereby achieving intermodal fusion. After the fusion operation at each layer, the average attention weight of all attention heads for each edge is calculated; The average attention weights of all edges are sorted, and the edges with the highest weight values ​​are selected. Only these selected edges are included in the calculation in the next layer of fusion operation. After one layer of dynamic fusion is completed, the output of the new multimodal graph is input into the next layer. After multiple fusions and filterings, the final multimodal fused graph is obtained.

7. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 5.

Images