Multimodal emotion detection method, device, computer equipment and storage medium
By introducing text to image and image to text in multi-modal emotion detection, the problem that static networks are difficult to capture multi-modal emotion information is solved, and more efficient emotion detection is achieved.
Patent Information
- Application Number
- CN202411139349.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-20
AI Technical Summary
The existing multimodal emotion detection methods use static networks and are difficult to capture complex and diverse emotional information in multimodal, resulting in insufficient detection accuracy and flexibility.
Using multi-layer modal interaction layers in text-to-image modal interaction network and image-to-text modal interaction network, the text and image modal interaction network are realized through encoding modules, dual-branch modal interaction modules and feature fusion modules. Interaction of emotional information within improves the accuracy and efficiency of emotional detection.
Through feature extraction of the multi-layer modal interaction layer, complex emotional information is captured, the accuracy and efficiency of multi-modal emotion detection is improved, and the understanding of multi-modal data is enhanced.
Smart Images

Figure CN118656791B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of natural language processing technology, and in particular to a multimodal emotion detection method, device, computer equipment and storage medium. Background Art
[0002] With the development of the Internet, people are increasingly posting multimodal information on social media platforms to share opinions and express emotions. As a result, multimodal sentiment detection has attracted great attention in recent academic and industrial research and has been shown to be beneficial for tasks such as product review analysis and political opinion mining. Unlike unimodal data, multimodal data provides richer information that can reveal a person's true emotions.
[0003] Current methods for multimodal emotion detection usually introduce memory networks, or construct multi-channel graph neural networks to achieve interaction between modalities, or use sparse attention mechanisms to address modal heterogeneity to enhance fusion for multimodal emotion detection. However, the above methods mainly apply existing static networks for sample processing. The fixed structure of static networks leads to their lack of flexibility in adapting to different multimodal inputs, making it difficult to capture complex and diverse emotional information in multimodal information, thereby reducing the accuracy of multimodal emotion detection. Summary of the invention
[0004] Based on this, the present invention provides a multimodal emotion detection method, device, computer equipment and storage medium, which extracts features through multiple modal interaction layers in a text-to-image modal interaction network and an image-to-text modal interaction network, captures complex and diverse emotional information in text and image modalities, and realizes the interaction of emotional information between modalities and within modalities, so as to perform emotion detection and improve the accuracy and efficiency of multimodal emotion detection. The technical method is as follows:
[0005] In a first aspect, an embodiment of the present application provides a multimodal emotion detection method, comprising the following steps:
[0006] Obtaining document data to be tested and a multimodal emotion detection model; wherein the document data to be tested includes text to be tested and an image to be tested, and the multimodal emotion detection model includes an encoding module, a dual-branch modal interaction module, a feature fusion module, and an emotion detection module; the dual-branch modal interaction module includes a text-to-image modal interaction network and an image-to-text modal interaction network;
[0007] Inputting the text to be tested and the image to be tested into the encoding module for encoding processing to obtain a document encoding representation, wherein the document encoding representation includes a text encoding representation and a visual encoding representation;
[0008] The text encoding representation and the visual encoding representation are respectively used as the same-modal input representation and the different-modal input representation of the text-to-image modal interaction network, and are input into the text-to-image modal interaction network for modal interaction processing to obtain a text-to-image modal interaction feature representation;
[0009] The visual encoding representation and the text encoding representation are respectively used as the same-modal input representation and the different-modal input representation of the image-to-text modal interaction network, and are input into the image-to-text modal interaction network for modal interaction processing to obtain an image-to-text modal interaction feature representation;
[0010] Inputting the text-to-image modality interaction feature representation and the image-to-text modality interaction feature representation into the feature fusion module for feature fusion to obtain a feature fusion representation;
[0011] The feature fusion representation is input into the emotion detection module to perform emotion detection, and the emotion detection result of the document data to be tested is obtained.
[0012] In a second aspect, an embodiment of the present application provides a multimodal emotion detection device, comprising:
[0013] A data acquisition module, used to obtain the document data to be tested and the multimodal emotion detection model; wherein the document data to be tested includes the text to be tested and the image to be tested, and the multimodal emotion detection model includes an encoding module, a dual-branch modal interaction module, a feature fusion module and an emotion detection module; the dual-branch modal interaction module includes a text-to-image modal interaction network and an image-to-text modal interaction network;
[0014] A document encoding module, used for inputting the text to be tested and the image to be tested into the encoding module for encoding processing to obtain a document encoding representation, wherein the document encoding representation includes a text encoding representation and a visual encoding representation;
[0015] A first modal interaction module, configured to use the text encoding representation and the visual encoding representation as a homomodal input representation and a heteromodal input representation of the text-to-image modal interaction network, respectively, and input them into the text-to-image modal interaction network for modal interaction processing to obtain a text-to-image modal interaction feature representation;
[0016] A second modal interaction module is used to use the visual encoding representation and the text encoding representation as the same-modal input representation and the different-modal input representation of the image-to-text modal interaction network, respectively, and input them into the image-to-text modal interaction network for modal interaction processing to obtain an image-to-text modal interaction feature representation;
[0017] A fusion module, used for inputting the text-to-image modality interaction feature representation and the image-to-text modality interaction feature representation into the feature fusion module for feature fusion to obtain a feature fusion representation;
[0018] The detection module is used to input the feature fusion representation into the emotion detection module to perform emotion detection and obtain the emotion detection result of the document data to be tested.
[0019] In a third aspect, an embodiment of the present application provides a computer device comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, the steps of the multimodal emotion detection method described in the first aspect are implemented.
[0020] In a fourth aspect, an embodiment of the present application provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the multimodal emotion detection method as described in the first aspect are implemented.
[0021] In this embodiment, a multimodal emotion detection method, apparatus, computer device and storage medium are provided, which perform feature extraction through multiple modal interaction layers in a text-to-image modal interaction network and an image-to-text modal interaction network, capture complex and diverse emotion information in text and image modalities, and realize interaction of emotion information between and within modalities for emotion detection, thereby improving the accuracy and efficiency of multimodal emotion detection.
[0022] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 A schematic diagram of a multimodal emotion detection method provided in one embodiment of the present application;
[0024] Figure 2 A schematic diagram of the process of S2 in the multimodal emotion detection method provided in one embodiment of the present application;
[0025] Figure 3 A schematic diagram of the process of S3 in the multimodal emotion detection method provided in one embodiment of the present application;
[0026] Figure 4 A schematic diagram of the process of S31 in the multimodal emotion detection method provided in one embodiment of the present application;
[0027] Figure 5 A schematic diagram of the process of S32 in the multimodal emotion detection method provided in one embodiment of the present application;
[0028] Figure 6A schematic diagram of the process of S5 in the multimodal emotion detection method provided in one embodiment of the present application;
[0029] Figure 7 A schematic diagram of the process of S6 in the multimodal emotion detection method provided in one embodiment of the present application;
[0030] Figure 8 A schematic diagram of the flow of S7 in the multimodal emotion detection method provided in another embodiment of the present application;
[0031] Fig. 9 A schematic diagram of the structure of a multimodal emotion detection device provided in one embodiment of the present application;
[0032] Fig.10 A schematic diagram of the structure of a computer device provided for one embodiment of the present application. DETAILED DESCRIPTION
[0033] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0034] The terms used in this application are for the purpose of describing the target embodiments only and are not intended to limit the application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.
[0035] It should be understood that, although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" / "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".
[0036] See also Figure 1 , Figure 1 A schematic flow chart of a multimodal emotion detection method provided in one embodiment of the present application includes the following steps:
[0037] S1: Obtain the document data to be tested and the preset multimodal sentiment detection model.
[0038] The executor of the multimodal emotion detection method of the present application is a detection device of the multimodal emotion detection method (hereinafter referred to as the detection device).
[0039] In an optional embodiment, the detection device may be a computer device, a server, or a server cluster formed by a combination of multiple computer devices.
[0040] The detection device obtains the document data to be tested. In an optional embodiment, the document data to be tested originates from social media, wherein social media is a content production and exchange platform based on user relationships on the Internet, mainly including social networking sites, Weibo, WeChat, Facebook, Twitter, blogs, forums, podcasts, etc. The training document data includes the opinions, insights, experiences, and views shared by users on social media.
[0041] Specifically, the detection device can be connected to the social media to obtain the document data to be tested published on the social media, wherein the training document data includes the text to be tested and the image to be tested.
[0042] The detection device obtains a preset multimodal emotion detection model, wherein the multimodal emotion detection model includes an encoding module, a dual-branch modal interaction module, a feature fusion module, and an emotion detection module; the dual-branch modal interaction module includes a text-to-image modal interaction network and an image-to-text modal interaction network. The text-to-image modal interaction network and the image-to-text modal interaction network both include several modal interaction layers, and the modal interaction layers are connected by a preset soft router; the modal interaction layer includes several modal processing units.
[0043] S2: Inputting the text to be tested and the image to be tested into the encoding module for encoding processing to obtain a document encoding representation.
[0044] The encoding module adopts the CLIP (Contrastive Language-Image Pre-Training) contrastive language-image pre-training model. In this embodiment, the detection device inputs the text to be tested and the image to be tested into the encoding module for encoding processing to obtain a document encoding representation, wherein the document encoding representation includes a text encoding representation and a visual encoding representation.
[0045] The encoding module includes a text encoder and an image encoder. Figure 2 , Figure 2 The flowchart of S2 in the multimodal emotion detection method provided in one embodiment of the present application includes steps S21 to S22, which are as follows:
[0046] S21: Input the text to be tested into the word embedding module for encoding processing to obtain the text encoding representation.
[0047] The word embedding module adopts the BERT (Bidirectional Encoder Representation from Transformers) word embedding model, which is used to convert the vectors of several words in the sentence to be tested into corresponding state vectors.
[0048] In this embodiment, the detection device inputs the text to be tested into the word embedding module for encoding processing to obtain the text encoding representation, wherein the text encoding representation includes word embedding vectors of several words.
[0049] Specifically, the detection device inputs the sentence to be tested into the BERT word embedding model, maps each word in the text to be tested into a low-dimensional vector space, and obtains the hidden layer vectors of several words of the text to be tested output by the BERT word embedding model after querying the pre-trained BERT matrix as the word embedding vector, and performs encoding processing to obtain the text encoding representation.
[0050] S22: Divide the image to be tested into a plurality of image sub-regions, input the plurality of image sub-regions into the target detection module for target detection, and obtain the visual coding representation.
[0051] The target detection module adopts the ViT model for multi-target detection and can determine predefined objects related to entities in the image.
[0052] In this embodiment, the detection device divides the image to be tested into several image sub-regions, inputs the several image sub-regions into the target detection module for target detection, and obtains the visual coding representation, wherein the visual coding representation includes detection vectors of several image sub-regions.
[0053] S3: Using the text encoding representation and the visual encoding representation as the homomodal input representation and heteromodal input representation of the text-to-image modal interaction network respectively, inputting them into the text-to-image modal interaction network for modal interaction processing, and obtaining a text-to-image modal interaction feature representation.
[0054] In this embodiment, the detection device uses the text encoding representation and the visual encoding representation as the homomodal input representation and heteromodal input representation of the text-to-image modal interaction network, respectively, and inputs them into the text-to-image modal interaction network for modal interaction processing to obtain a text-to-image modal interaction feature representation.
[0055] See also Figure 3 , Figure 3 The flowchart of S3 in the multimodal emotion detection method provided in one embodiment of the present application includes steps S31 to S33, which are as follows:
[0056] S31: Inputting the same-modal input representation and the different-modal input representation into the first-layer modal interaction layer of the text-to-image modal interaction network for modal processing to obtain the output representation of the first-layer modal interaction layer.
[0057] In this embodiment, the detection device inputs the homomodal input representation and the heteromodal input representation into the first modal interaction layer of the text-to-image modal interaction network for modal processing to obtain the output representation of the first modal interaction layer, wherein the homomodal input representation and the heteromodal input representation both include feature vectors of several segments, and the feature vectors of the segments are used to indicate the feature vectors of the corresponding modalities. Taking the text-to-image modal interaction network as an example, the feature vectors of the segments in the homomodal input representation are word embedding vectors of words in the text encoding representation, and the feature vectors of the segments in the heteromodal input representation are detection vectors of image sub-regions in the visual encoding representation.
[0058] The modality processing unit includes a semantic extraction unit, a semantic reasoning unit, a semantic matching unit, a semantic alignment unit, a semantic filtering unit, and a semantic perception unit, which are used to capture the complex and diverse emotional information in the document encoding representation and realize the interaction of emotional information between modalities and within modalities. Figure 4 , Figure 4 The flowchart of S31 in the multimodal emotion detection method provided in one embodiment of the present application includes steps S311 to S316, which are as follows:
[0059] S311: Input the same-modal input representation to the semantic extraction unit of the first-layer modal interaction layer to extract semantic features and obtain semantic feature representation.
[0060] In this embodiment, the detection device inputs the same modal input representation into the semantic extraction unit of the first modal interaction layer to extract semantic features and obtain a semantic feature representation.
[0061] Specifically, the detection device extracts semantic features from the text encoding representation as the homomodal input representation according to the homomodal input representation and a preset semantic feature extraction algorithm to obtain a semantic feature representation to simplify the original emotional information and alleviate the gradient vanishing problem, as described below:
[0062]
[0063] In the formula, is the semantic feature representation, is the same modal input representation, is the activation function.
[0064] S312: Input the same-modal input representation to the semantic reasoning unit of the first-level modal interaction layer for semantic reasoning to obtain a feature reasoning representation.
[0065] Starting from a single modality, there are also emotional and semantic similarities between local segments (such as between words or between local areas of pictures). These dependencies improve the accuracy of multimodal emotion detection to a certain extent. In this embodiment, the detection device inputs the same modal input representation into the semantic reasoning unit of the first modal interaction layer for semantic reasoning to obtain feature reasoning representation.
[0066] Specifically, the detection device constructs an attention matrix set of the homomodal input representation based on the homomodal input representation, and the attention matrix set includes a first attention matrix, a second attention matrix, and a third attention matrix. The detection device uses a multi-head self-attention mechanism to extract attention features from the homomodal input representation and obtain a multi-head self-attention feature representation to capture the local sentiment associations within the modalities of different subspaces, as described below:
[0067]
[0068] In the formula, is the multi-head self-attention feature representation corresponding to the same modal input representation, is the concatenation function, is the first i The output representation of each head is, R is the number of heads of the multi-head self-attention mechanism, is the normalized exponential function, As the first i The first attention matrix, the second attention matrix, and the third attention matrix represented by the input of each head, is the dimension parameter, T is the bias sign.
[0069] The detection device uses a feedforward neural network to perform feature fitting on the multi-head self-attention feature representation to obtain a feature inference representation as described below:
[0070]
[0071] In the formula, is the feature inference representation, It is a feed-forward neural network.
[0072] S313: Input the same-modal input representation and the heteromodal input representation into the semantic matching unit of the first-layer modal interaction layer to perform local feature matching to obtain a cross-modal local feature matching representation.
[0073] There may be certain correspondence or semantic conflict relationships between different modal local segments. In this embodiment, the detection device inputs the same-modal input representation and the heteromodal input representation into the semantic matching unit of the first-level modal interaction layer to perform local feature matching to obtain a cross-modal local feature matching representation.
[0074] Specifically, the detection device adopts a cosine similarity calculation method to construct a cosine similarity matrix between the homomodal input representation and the heteromodal input representation, wherein the cosine similarity matrix includes the cosine similarities between the feature vectors of several segments in the homomodal input representation and the feature vectors of several segments in the heteromodal input representation.
[0075] The detection device constructs a feature association matrix corresponding to the homomodal input representation according to the cosine similarity matrix, the heteromodal input representation and the preset feature association matrix construction algorithm, and enriches the local emotional information representation by exploring the association between the segment features from a cross-modal perspective, wherein the feature association matrix corresponding to the homomodal input representation includes feature association vectors of several segments, and the matrix construction algorithm is:
[0076]
[0077] In the formula, For the i The feature association vector of each segment, K is the number of segments in the heteromodal input representation, is the first i The feature vector of the segment is the same as the feature vector of the heteromodal input representation. j The cosine similarity between the feature vectors of the segments, is the inverse temperature coefficient, is the first j The feature vector of each segment.
[0078] The detection device adopts a fully connected layer mapping method to construct a feature scaling matrix and a feature transfer matrix corresponding to the feature association matrix to enhance the semantic features and emotional information of the fragment, wherein the feature scaling matrix includes scaling vectors of several fragments, the feature transfer matrix includes transfer vectors of several fragments, and the scaling vector is:
[0079]
[0080] In the formula, For the i The scaling vector of the fragments, is the hyperbolic tangent function, is the first fully connected function.
[0081] The transfer vector is:
[0082]
[0083] In the formula, For the i The transfer vector of each fragment, is the second fully connected function.
[0084] The detection device adopts a multi-layer perceptron to perform local feature matching according to the feature scaling matrix, feature transfer matrix, same-modal input representation and preset local feature matching algorithm, obtain a cross-modal local feature matching representation, and obtain a more refined local segment feature, wherein the cross-modal local feature matching representation includes a cross-modal local vector of several segments, and the local feature matching algorithm is:
[0085]
[0086] In the formula, For the i The cross-modal local vector of segments, is the first j The feature vector of each segment.
[0087] S314: Input the same-modal input representation and the heteromodal input representation into the semantic alignment unit of the first-layer modal interaction layer to perform cross-modal global feature alignment to obtain a cross-modal global feature alignment representation.
[0088] Compared with local fragment information, global overall information can reflect the overall emotion of the image-text pair from a broader level. In this embodiment, the detection device inputs the homomodal input representation and the heteromodal input representation into the semantic alignment unit of the first-level modal interaction layer, performs cross-modal global feature alignment, and obtains a cross-modal global feature alignment representation.
[0089] Specifically, the detection device adopts an average pooling method to perform global feature extraction on the homomodal input representation and the heteromodal input representation, respectively, to obtain a first global feature representation corresponding to the homomodal input representation and a second global feature representation corresponding to the heteromodal input representation.
[0090] The detection device adopts a gated fusion mechanism to perform cross-modal global feature alignment according to the first global feature representation, the second global feature representation and a preset global feature alignment algorithm to integrate the global sentiment information of different segments and obtain a cross-modal global feature alignment representation, wherein the global feature alignment algorithm is:
[0091]
[0092] In the formula, is a cross-modal global feature alignment representation, is the first global feature representation, is the second global feature representation, z is the intermediate representation, is the first learnable parameter matrix, is the second learnable parameter matrix.
[0093] S315: Input the same-modal input representation and the heteromodal input representation into the semantic filtering unit of the first-layer modal interaction layer to perform cross-modal feature filtering to obtain a cross-modal feature filtering representation.
[0094] For some complex image-text pairs, relying solely on cross-modal global or local information is still not enough to judge emotions. This is because the local emotional tendency (words or regions) may not be consistent with the global emotional tendency (sentences or pictures). In this embodiment, the detection device inputs the homomodal input representation and the heteromodal input representation into the semantic filtering unit of the first-level modal interaction layer, performs cross-modal feature filtering, and obtains a cross-modal feature filtering representation to make full use of information from different fragments.
[0095] Specifically, the detection device adopts a vector similarity calculation method to obtain a cross-modal global similarity representation according to the first global feature representation corresponding to the homomodal input representation, the second global feature representation corresponding to the heteromodal input representation, and a preset global similarity calculation algorithm, wherein the global similarity calculation algorithm is:
[0096]
[0097] In the formula, is the cross-modal global similarity representation, is the third learnable parameter matrix.
[0098] The detection device constructs a feature association matrix corresponding to the homomodal input representation according to the homomodal input representation and the heteromodal input representation. For a specific embodiment, please refer to step S313, which will not be described in detail here.
[0099] The detection device obtains a cross-modal local similarity representation according to the same-modal input representation, the feature association matrix corresponding to the same-modal input representation, and a preset local similarity calculation algorithm, thereby capturing rich inconsistency information between different modal feature representations to explore the correspondence between local features of images and texts, thereby identifying more complex cross-modal sentiment features and cross-modal level inconsistency information, wherein the cross-modal local similarity representation includes a cross-modal local similarity vector of several fragments, and the local similarity calculation algorithm is:
[0100]
[0101] In the formula, For the i The cross-modal local similarity vector of the segments, is the fourth learnable parameter matrix.
[0102] The detection device splices the cross-modal global similarity representation and the cross-modal local similarity representation to construct a cross-modal similarity splicing representation, and adopts an aggregation weight matrix calculation method to obtain an aggregation weight matrix corresponding to the cross-modal similarity splicing representation according to the cross-modal similarity splicing representation and a preset aggregation weight matrix calculation algorithm, wherein the aggregation weight matrix calculation algorithm is:
[0103]
[0104] In the formula, is the aggregation weight matrix, for sigmod Activation function, is the normalization function, is the linear transformation parameter matrix, N is the number of vectors in the cross-modal similarity concatenation representation, , are the first u and v vectors.
[0105] The detection device adopts a similarity score calculation method, and performs cross-modal feature filtering on the same-modal input representation and the heteromodal input representation according to the cross-modal similarity splicing representation, the aggregation weight matrix corresponding to the cross-modal similarity splicing representation, and a preset similarity calculation algorithm to obtain the cross-modal feature filtering representation, effectively suppressing invalid comparisons with low emotional contribution, thereby improving the accuracy of multimodal emotion detection, wherein the similarity calculation algorithm is:
[0106]
[0107] In the formula, Filtering representations for cross-modal features.
[0108] S316: Input the same-modal input representation and the heteromodal input representation into the semantic perception unit of the first-layer modal interaction layer to perform cross-modal feature perception to obtain a cross-modal feature perception representation.
[0109] Although fusing the sentiment information of different fragments can obtain good cross-modal association, relying solely on cross-modal features to make the final sentiment prediction for the example may also be inaccurate. In this embodiment, the detection device inputs the homomodal input representation and the heteromodal input representation into the semantic perception unit of the first-level modal interaction layer to perform cross-modal feature perception and obtain a cross-modal feature perception representation.
[0110] Specifically, the detection device constructs a feature association matrix corresponding to the homomodal input representation according to the homomodal input representation and the heteromodal input representation. For a specific embodiment, reference may be made to step S313, which will not be described in detail here.
[0111] The detection device adopts a feature mapping method, and constructs a mapping feature representation of the homomodal input representation and a mapping feature representation of the feature association matrix according to the homomodal input representation, the feature association matrix corresponding to the homomodal input representation, and a preset feature mapping algorithm, as a unimodal feature representation and a multimodal feature representation, respectively, to learn cross-modal emotional features with rich unimodal contexts from two different visual perspectives of unimodality and cross-modality, aiming to promote interactive reasoning between unimodal and multimodal emotional information, wherein the feature mapping algorithm is:
[0112]
[0113] In the formula, is a unimodal feature representation, is a multimodal feature representation, is the fifth learnable parameter matrix, is the sixth learnable parameter matrix, is the first bias parameter, is the second bias parameter.
[0114] Considering that there may be difference features between unimodal emotion information and cross-modal emotion information, the detection device adopts a feature perception method, and according to the unimodal feature representation, multimodal feature representation and a preset feature perception algorithm, a gating mechanism is used to filter these difference noises, thereby integrating the common features in the unimodal emotion information and the multimodal emotion information, and then learning the cross-modal emotion features with rich unimodal context, performing feature perception, and obtaining the cross-modal feature perception representation, wherein the feature perception algorithm is:
[0115]
[0116] In the formula, is a cross-modal feature-aware representation, is the seventh learnable parameter matrix, is the eighth learnable parameter matrix.
[0117] S32: Input the output representation of the first modal interaction layer into the connected soft router for path decision, obtain the target input representation of the next modal interaction layer as the updated same-modal input representation, input the updated same-modal input representation and the heteromodal input representation into the next modal interaction layer for modal processing, obtain the output representation of the next modal interaction layer, until the output representation of the last modal interaction layer is obtained.
[0118] In this embodiment, the detection device inputs the output representation of the first modal interaction layer into the connected soft router for path decision, obtains the input representation of the next modal interaction layer as an updated same-modal input representation, inputs the updated same-modal input representation and the heteromodal input representation into the next modal interaction layer for modal processing, obtains the output representation of the next modal interaction layer, and so on. The output representation of the last modal interaction layer is obtained.
[0119] The modal interaction layers are connected by preset soft routers, and the soft routers are set in parallel in adjacent modal interaction layers, thereby ensuring the richness and flexibility of the routing space, being able to explore many undeveloped interaction modes, and giving full play to the respective advantages of several modal processing units in each modal interaction layer.
[0120] See also Figure 5 , Figure 5 The flowchart of S32 in the multimodal emotion detection method provided in one embodiment of the present application includes steps S321 to S322, which are specifically as follows:
[0121] S321: Obtain path probabilities from a plurality of modal processing units in the first modal interaction layer to a plurality of modal processing units in the next modal interaction layer.
[0122] In this embodiment, the detection device obtains the path probability from several modal processing units of the first modal interaction layer to several modal processing units of the next modal interaction layer, wherein the path probability is used to indicate the probability that the output representation of the modal processing unit of the previous modal interaction layer is used as the input representation of the modal processing unit of the next modal interaction layer.
[0123] Specifically, the detection device uses the same modal input representation as the target input representation of several modal processing units in the first modal interaction layer, performs path probability calculation according to the target input representation of several modal processing units in the first modal interaction layer and a preset path probability calculation algorithm, and obtains the path probability from several modal processing units in the first modal interaction layer to several modal processing units in the next modal interaction layer, wherein the path probability calculation algorithm is:
[0124]
[0125] In the formula, For the n Layer modal interaction layer m The probability of the input representation of the modal processing unit, For the n Layer modal interaction layer m The routing function of each modal processing unit, For the n Layer modal interaction layer m The target input representation of each modality processing unit is For the n Layer modal interaction layer m The input representation of the modal processing unit r The vector of rows, is a linear activation function, is a multi-layer perceptron function.
[0126] S322: Calculate the input representation according to the output representations of the plurality of modal processing units of the first modal interaction layer and the path probability to obtain the target input representation of the next modal interaction layer.
[0127] In this embodiment, the detection device calculates the input representation according to the output representations of the plurality of modal processing units of the first modal interaction layer and the path probability to obtain the target input representation of the next modal interaction layer.
[0128] Specifically, the detection device calculates the input representation according to the output representations of the several modal processing units in the modal interaction layer and the path probabilities from the several modal processing units in the modal interaction layer to the several modal processing units in the next modal interaction layer, and obtains the input representations of the several modal processing units in the next modal interaction layer, wherein the input representation of the modal processing unit is:
[0129]
[0130] In the formula, C is the number of modal processing units in the modal interaction layer, For then -1 modal interaction layer j modal processing unit to the next layer, i.e. n Layer modal interaction layer m The path probability of a modal processing unit, For the n -1 modal interaction layer j The output representation of a modal processing unit.
[0131] S33: Perform feature aggregation on the output representation of the last modal interaction layer of the text-to-image modal interaction network to obtain a text-to-image modal interaction feature representation.
[0132] In this embodiment, the detection device performs feature aggregation on the output representations of several modal processing units of the last modal interaction layer of the text-to-image modal interaction network to obtain a text-to-image modal interaction feature representation.
[0133] S4: Using the visual encoding representation and the text encoding representation as the homomodal input representation and heteromodal input representation of the image-to-text modal interaction network respectively, inputting them into the image-to-text modal interaction network for modal interaction processing, and obtaining an image-to-text modal interaction feature representation.
[0134] In this embodiment, the detection device uses the visual encoding representation and the text encoding representation as the homomodal input representation and heteromodal input representation of the image-to-text modal interaction network, respectively, and inputs them into the image-to-text modal interaction network for modal interaction processing, obtains the output representations of several modal processing units of the last modal interaction layer of the image-to-text modal interaction network, performs feature aggregation on the output representations of several modal processing units of the last modal interaction layer of the image-to-text modal interaction network, and obtains image-to-text modal interaction feature representation. The specific implementation method can refer to steps S31~S33, which will not be repeated here.
[0135] S5: Inputting the text-to-image modal interaction feature representation and the image-to-text modal interaction feature representation into the feature fusion module for feature fusion to obtain a feature fusion representation.
[0136] In this embodiment, the detection device inputs the text-to-image modal interaction feature representation and the image-to-text modal interaction feature representation into the feature fusion module for feature fusion to obtain a feature fusion representation.
[0137] See also Figure 6 , Figure 6 The flowchart of S5 in the multimodal emotion detection method provided in one embodiment of the present application includes step S51, which is as follows:
[0138] S51: adopting a block fusion strategy, according to a preset association tensor, projecting the text-to-image modal interaction feature representation and the image-to-text modal interaction feature representation into a preset feature space, performing feature fusion, and obtaining a feature fusion representation.
[0139] In this embodiment, the detection device adopts a block fusion strategy, and according to a preset feature fusion algorithm and a preset association tensor, projects the text-to-image modality interaction feature representation and the image-to-text modality interaction feature representation into a preset feature space, performs feature fusion, and obtains a feature fusion representation, wherein the feature fusion algorithm is:
[0140]
[0141] In the formula, is the feature fusion representation, T is the preset associated tensor, is the feature representation of text-to-image modality interaction, Interactive feature representation for image-to-text modality.
[0142] S6: Input the feature fusion representation into the emotion detection module to perform emotion detection, and obtain the emotion detection result of the document data to be tested.
[0143] In this embodiment, the detection device inputs the feature fusion representation into the emotion detection module to perform emotion detection, and obtains the emotion detection result of the document data to be tested.
[0144] Feature extraction is performed through multiple modal interaction layers in the text-to-image modal interaction network and the image-to-text modal interaction network, as well as multiple modal processing units in the modal interaction layer. By setting up soft routers in parallel in adjacent modal interaction layers, the richness and flexibility of the routing space are ensured, and many undeveloped interaction modes can be explored. The respective advantages of several modal processing units in each modal interaction layer are fully utilized. The gap between text and image modalities is taken into account, and the complex and diverse emotional information in text and image modalities is captured. The emotional information interaction between and within modalities is realized for emotion detection, thereby improving the accuracy and efficiency of multimodal emotion detection.
[0145] See also Figure 7 , Figure 7 The flowchart of S6 in the multimodal emotion detection method provided in one embodiment of the present application includes step S61, which is as follows:
[0146] S61: Obtain a predicted emotion probability vector according to the feature fusion representation and a preset emotion probability vector algorithm, and obtain an emotion detection result of the document data to be tested according to the predicted emotion probability vector.
[0147] The emotion probability vector algorithm is:
[0148]
[0149] In the formula, is the predicted emotion probability vector, is the normalized exponential function, is a multi-layer perceptron function.
[0150] In this embodiment, the detection device obtains a predicted emotion probability vector based on the feature fusion representation and a preset emotion probability vector algorithm, and obtains the emotion detection result of the document data to be tested based on the predicted emotion probability vector. Specifically, the detection device obtains the emotion detection result corresponding to the probability vector with the largest dimension based on the emotion probability vector as the emotion detection result of the document data to be tested.
[0151] In an optional embodiment, the step S7 is also included: training the multimodal emotion detection model, see Figure 8 , Figure 8 The flowchart of S7 in the multimodal emotion detection method provided in another embodiment of the present application includes steps S71 to S73, which are as follows:
[0152] S71: Obtain a training document data set, input the training document data set into the multimodal emotion detection model, obtain the document encoding representation of the several training document data output by the encoding module, the predicted emotion probability vector of the several training document data output by the emotion detection module, the text-to-image modal interaction network, and the total path probability output by the image-to-text modal interaction network.
[0153] In this embodiment, the detection device obtains a training document data set, inputs the training document data set into the multimodal emotion detection model, obtains the document encoding representation of several training document data output by the encoding module, the predicted emotion probability vector of several training document data output by the emotion detection module, the text-to-image modal interaction network, and the total path probability output by the image-to-text modal interaction network, wherein the total path probability is the cumulative result of the path probabilities of the soft routers between several layers of modal interaction layers. For the specific embodiment, please refer to steps S2 to S6, which will not be repeated here.
[0154] S72: Obtain a first global feature representation and a second global feature representation corresponding to the text encoding representation and the visual encoding representation in the document encoding representation, and use the first global feature representation as the homomodal global feature representation of the text-to-image modal interaction network, and use the second global feature representation as the homomodal global feature representation of the image-to-text modal interaction network; adopt a path regularization loss function to obtain the path regularization loss values corresponding to the text-to-image modal interaction network and the image-to-text modal interaction network according to the total path probability and the homomodal global feature representation.
[0155] In this embodiment, the detection device obtains the first global feature representation and the second global feature representation corresponding to the text encoding representation and the visual encoding representation in the document encoding representation. Specifically, the detection device performs average pooling processing on the text encoding representation and the visual encoding representation respectively to obtain the first global feature representation and the second global feature representation.
[0156] The detection device uses the first global feature representation as a homomodal global feature representation of the text-to-image modal interaction network, and uses the second global feature representation as a homomodal global feature representation of the image-to-text modal interaction network.
[0157] In order to achieve emotional path consistency, the detection device adopts a path regularization loss function. According to the total path probability and the same-modal global feature representation, the path regularization loss values corresponding to the text-to-image modal interaction network and the image-to-text modal interaction network are obtained. Under the guidance of fuzziness, a more reasonable input representation can be constructed for the modal processing units of different modal interaction layers. The path regularization loss function is:
[0158]
[0159] In the formula, is the path regularization loss value, is the total path probability, is the divergence function.
[0160] S73: Obtain a number of true emotion probability vectors of the training document data, adopt a cross entropy loss function, and obtain a cross entropy loss value based on the predicted emotion probability vectors and the true emotion probability vectors of the training document data; train the multimodal emotion detection model based on the path regularization loss value and the cross entropy loss value corresponding to the text-to-image modal interaction network and the image-to-text modal interaction network.
[0161] In this embodiment, the detection device obtains the true emotion probability vectors of the training document data, and adopts the cross entropy loss function to obtain the cross entropy loss value according to the predicted emotion probability vectors and the true emotion probability vectors of the training document data, wherein the cross entropy loss function is:
[0162]
[0163] In the formula, is the cross entropy loss value, y is the true emotion probability vector.
[0164] The detection device obtains a total loss value according to the path regularization loss value, the cross entropy loss value and the preset total loss function corresponding to the text-to-image modal interaction network and the image-to-text modal interaction network, and trains the multimodal emotion detection model according to the total loss value, and collects multiple levels of emotion information to improve the performance of multimodal emotion detection, wherein the total loss value is:
[0165]
[0166] In the formula, is the total loss value, is the path regularization loss value corresponding to the text-to-image modality interaction network, is the path regularization loss value corresponding to the image-to-text modality interaction network, is the first ratio parameter, is the second ratio parameter.
[0167] See also Fig. 9 , Fig. 9 A schematic diagram of the structure of a multimodal emotion detection device provided in one embodiment of the present application, the device can implement all or part of the multimodal emotion detection method through software, hardware or a combination of both, the device 9 includes:
[0168] A data acquisition module 91 is used to obtain the document data to be tested and the multimodal emotion detection model; wherein the document data to be tested includes the text to be tested and the image to be tested, and the multimodal emotion detection model includes an encoding module, a dual-branch modal interaction module, a feature fusion module and an emotion detection module; the dual-branch modal interaction module includes a text-to-image modal interaction network and an image-to-text modal interaction network;
[0169] A document encoding module 92, used for inputting the text to be tested and the image to be tested into the encoding module for encoding processing to obtain a document encoding representation, wherein the document encoding representation includes a text encoding representation and a visual encoding representation;
[0170] A first modal interaction module 93 is used to use the text encoding representation and the visual encoding representation as the same modal input representation and the different modal input representation of the text-to-image modal interaction network, respectively, and input them into the text-to-image modal interaction network for modal interaction processing to obtain a text-to-image modal interaction feature representation;
[0171] A second modal interaction module 94 is used to input the visual encoding representation and the text encoding representation as the same modal input representation and the different modal input representation of the image-to-text modal interaction network, respectively, into the image-to-text modal interaction network for modal interaction processing, and obtain an image-to-text modal interaction feature representation;
[0172] A fusion module 95, configured to input the text-to-image modality interaction feature representation and the image-to-text modality interaction feature representation into the feature fusion module for feature fusion to obtain a feature fusion representation;
[0173] The detection module 96 is used to input the feature fusion representation into the emotion detection module to perform emotion detection and obtain the emotion detection result of the document data to be tested.
[0174] In an embodiment of the present application, the document data to be tested and the multimodal emotion detection model are obtained through a data acquisition module; wherein, the document data to be tested includes the text to be tested and the image to be tested, and the multimodal emotion detection model includes an encoding module, a dual-branch modal interaction module, a feature fusion module and an emotion detection module; the dual-branch modal interaction module includes a text-to-image modal interaction network and an image-to-text modal interaction network; through a document encoding module, the text to be tested and the image to be tested are input into the encoding module for encoding processing to obtain a document encoding representation, wherein the document encoding representation includes a text encoding representation and a visual encoding representation; through a first modal interaction module, the text encoding representation and the visual encoding representation are input into the encoding module for encoding processing to obtain a document encoding representation, wherein the document encoding representation includes a text encoding representation and a visual encoding representation; The representations are respectively used as the homomodal input representation and heteromodal input representation of the text-to-image modal interaction network, and are input into the text-to-image modal interaction network for modal interaction processing to obtain a text-to-image modal interaction feature representation; through a second modal interaction module, the visual encoding representation and the text encoding representation are respectively used as the homomodal input representation and heteromodal input representation of the image-to-text modal interaction network, and are input into the image-to-text modal interaction network for modal interaction processing to obtain an image-to-text modal interaction feature representation; through a fusion module, the text-to-image modal interaction feature representation and the image-to-text modal interaction feature representation are input into the feature fusion module for feature fusion to obtain a feature fusion representation;
[0175] Through the detection module, the feature fusion representation is input into the emotion detection module for emotion detection, and the emotion detection result of the document data to be tested is obtained. Feature extraction is performed through multiple modal interaction layers in the text-to-image modal interaction network and the image-to-text modal interaction network, and multiple modal processing units in the modal interaction layer. By setting soft routers in parallel in adjacent modal interaction layers, the richness and flexibility of the routing space are ensured, and many undeveloped interaction modes can be explored. The advantages of several modal processing units in each modal interaction layer are fully utilized. The gap between text and image modalities is taken into account, and the complex and diverse emotional information in text and image modalities is captured, and the emotional information interaction between modalities and within modalities is realized for emotion detection, thereby improving the accuracy and efficiency of multimodal emotion detection.
[0176] Please refer to Fig.10 , Fig.10 The computer device 100 is a schematic diagram of a computer device according to an embodiment of the present application. The computer device 100 includes: a processor 101, a memory 102, and a computer program 103 stored in the memory 102 and executable on the processor 101; the computer device may store a plurality of instructions, which are suitable for being loaded and executed by the processor 101. Figures 1 to 8 For the method steps of the embodiment, the specific execution process can be found in Figures 1 to 8 The specific description of the embodiment will not be repeated here.
[0177] Among them, the processor 101 may include one or more processing cores. The processor 101 uses various interfaces and lines to connect various parts in the server, and executes various functions and processes data of the multimodal emotion detection device 8 by running or executing instructions, programs, code sets or instruction sets stored in the memory 102, and calling the data in the memory 102. Optionally, the processor 101 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programble Logic Array, PLA). The processor 101 can integrate one or a combination of a central processing unit 101 (Central Processing Unit, CPU), an image processor 101 (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the touch display screen; the modem is used to process wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 101, and it can be implemented by a single chip.
[0178] The memory 102 may include a random access memory 102 (RAM) or a read-only memory 102 (Read-Only Memory). Optionally, the memory 102 includes a non-transitory computer-readable storage medium. The memory 102 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 102 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch instructions, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 102 may also be optionally at least one storage device located away from the aforementioned processor 101.
[0179] The present application also provides a storage medium, which can store multiple instructions. The instructions are suitable for the processor to load and execute the method steps of the first to third embodiments shown. The specific execution process can be seen in the Figures 1 to 8 The specific description of the embodiment will not be repeated here.
[0180] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0181] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0182] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the target application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each target application, but such implementation should not be considered to be beyond the scope of the present invention.
[0183] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0184] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0185] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0186] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc.
[0187] The present invention is not limited to the above-mentioned embodiments. If various changes or modifications to the present invention do not depart from the spirit and scope of the present invention, and if these changes and modifications fall within the scope of the claims and equivalent technologies of the present invention, the present invention is also intended to include these changes and modifications.
Claims
1. A multimodal emotion detection method, characterized in that: The following steps are involved: Obtaining document data to be tested and a multimodal emotion detection model; wherein the document data to be tested includes text to be tested and an image to be tested, and the multimodal emotion detection model includes an encoding module, a dual-branch modal interaction module, a feature fusion module, and an emotion detection module; the dual-branch modal interaction module includes a text-to-image modal interaction network and an image-to-text modal interaction network; the text-to-image modal interaction network and the image-to-text modal interaction network both include several modal interaction layers, and the modal interaction layers are connected by a preset soft router; Inputting the text to be tested and the image to be tested into the encoding module for encoding processing to obtain a document encoding representation, wherein the document encoding representation includes a text encoding representation and a visual encoding representation; The text encoding representation and the visual encoding representation are used as the homomodal input representation and heteromodal input representation of the text-to-image modal interaction network respectively, and the homomodal input representation and heteromodal input representation are input into the first modal interaction layer of the text-to-image modal interaction network for modal processing to obtain the output representation of the first modal interaction layer; The output representation of the first modal interaction layer is input into the connected soft router for path decision, and the target input representation of the next modal interaction layer is obtained as the updated same-modal input representation. The updated same-modal input representation and the heteromodal input representation are input into the next modal interaction layer for modal processing, and the output representation of the next modal interaction layer is obtained, until the output representation of the last modal interaction layer is obtained; Performing feature aggregation on the output representation of the last modal interaction layer of the text-to-image modal interaction network to obtain a text-to-image modal interaction feature representation; The visual encoding representation and the text encoding representation are respectively used as the same-modal input representation and the different-modal input representation of the image-to-text modal interaction network, and are input into the image-to-text modal interaction network for modal interaction processing to obtain an image-to-text modal interaction feature representation; Inputting the text-to-image modality interaction feature representation and the image-to-text modality interaction feature representation into the feature fusion module for feature fusion to obtain a feature fusion representation; The feature fusion representation is input into the emotion detection module to perform emotion detection, and the emotion detection result of the document data to be tested is obtained.
2. The multimodal emotion detection method according to claim 1, characterized in that: The encoding module includes a word embedding module and a target detection module; The step of inputting the text to be tested and the image to be tested into the encoding module for encoding processing to obtain a document encoding representation comprises the following steps: Inputting the test text into the word embedding module for encoding processing to obtain the text encoding representation, wherein the text encoding representation includes word embedding vectors of several words; The image to be tested is divided into a plurality of image sub-regions, and the plurality of image sub-regions are input into the target detection module for target detection to obtain the visual coding representation, wherein the visual coding representation includes detection vectors of the plurality of image sub-regions.
3. The multimodal emotion detection method according to claim 1, characterized in that: The modal interaction layer includes several modal processing units; the output representation of the modal interaction layer includes the output representations of several modal processing units; the modal processing units include a semantic extraction unit, a semantic reasoning unit, a semantic matching unit, a semantic alignment unit, a semantic filtering unit and a semantic perception unit; the output representation of the modal processing unit includes a semantic feature representation, a feature reasoning representation, a cross-modal local feature matching representation, a cross-modal global feature alignment representation, a cross-modal feature filtering representation and a cross-modal feature perception representation; The step of inputting the same-modal input representation and the different-modal input representation into the first-layer modal interaction layer of the text-to-image modal interaction network for modal processing to obtain the output representation of the first-layer modal interaction layer comprises the following steps: Inputting the same-modal input representation into the semantic extraction unit of the first-layer modal interaction layer to extract semantic features and obtain semantic feature representation; Inputting the same-modal input representation into the semantic reasoning unit of the first-layer modal interaction layer to perform semantic reasoning to obtain a feature reasoning representation; Inputting the same-modal input representation and the different-modal input representation into the semantic matching unit of the first-layer modal interaction layer to perform local feature matching to obtain a cross-modal local feature matching representation; Inputting the same-modal input representation and the different-modal input representation into the semantic alignment unit of the first-layer modal interaction layer to perform cross-modal global feature alignment to obtain a cross-modal global feature alignment representation; Inputting the same-modal input representation and the different-modal input representation into the semantic filtering unit of the first-layer modal interaction layer to perform cross-modal feature filtering to obtain a cross-modal feature filtering representation; The same-modal input representation and the heteromodal input representation are input into the semantic perception unit of the first-layer modal interaction layer to perform cross-modal feature perception to obtain a cross-modal feature perception representation.
4. The multimodal emotion detection method according to claim 1, characterized in that: The output representation of the first modal interaction layer is input into the connected soft router for path decision, and the target input representation of the next modal interaction layer is obtained, including the steps of: Obtaining path probabilities from a plurality of modal processing units of the first modal interaction layer to a plurality of modal processing units of the next modal interaction layer, wherein the path probability is used to indicate the probability that the output representation of the modal processing unit of the previous modal interaction layer is used as the input representation of the modal processing unit of the next modal interaction layer; An input representation calculation is performed based on the output representations of the plurality of modal processing units of the first modal interaction layer and the path probability to obtain a target input representation of the next modal interaction layer.
5. The multimodal emotion detection method according to any one of claims 1 to 4, characterized in that: The step of inputting the text-to-image modal interaction feature representation and the image-to-text modal interaction feature representation into the feature fusion module for feature fusion to obtain a feature fusion representation comprises the following steps: A block fusion strategy is adopted to project the text-to-image modal interaction feature representation and the image-to-text modal interaction feature representation into a preset feature space according to a preset association tensor, perform feature fusion, and obtain a feature fusion representation.
6. The multimodal emotion detection method according to claim 5, characterized in that: The step of inputting the feature fusion representation into the emotion detection module for emotion detection to obtain the emotion detection result of the document data to be tested comprises the following steps: According to the feature fusion representation and a preset emotion probability vector algorithm, a predicted emotion probability vector is obtained, and according to the predicted emotion probability vector, an emotion detection result of the document data to be tested is obtained.
7. The multimodal emotion detection method according to claim 6, characterized in that: The method further comprises the steps of: training the multimodal emotion detection model; the training of the multimodal emotion detection model comprises the steps of: Obtain a training document data set, input the training document data set into the multimodal emotion detection model, obtain document encoding representations of a plurality of training document data output by the encoding module, predicted emotion probability vectors of a plurality of training document data output by the emotion detection module, the text-to-image modal interaction network, and the total path probability output by the image-to-text modal interaction network, wherein the total path probability is the cumulative result of the path probabilities of the soft routers between a plurality of modal interaction layers; Obtaining a first global feature representation and a second global feature representation corresponding to the text encoding representation and the visual encoding representation in the document encoding representation, using the first global feature representation as the homomodal global feature representation of the text-to-image modal interaction network, and using the second global feature representation as the homomodal global feature representation of the image-to-text modal interaction network; adopting a path regularization loss function, and obtaining path regularization loss values corresponding to the text-to-image modal interaction network and the image-to-text modal interaction network according to the total path probability and the homomodal global feature representation; Obtain a number of true emotion probability vectors of the training document data, adopt a cross entropy loss function, and obtain a cross entropy loss value based on the predicted emotion probability vectors and the true emotion probability vectors of the training document data; train the multimodal emotion detection model based on the path regularization loss value and the cross entropy loss value corresponding to the text-to-image modal interaction network and the image-to-text modal interaction network.
8. A multimodal emotion detection device, characterized in that: include: A data acquisition module, used to obtain document data to be tested and a multimodal emotion detection model; wherein the document data to be tested includes text to be tested and an image to be tested, and the multimodal emotion detection model includes an encoding module, a dual-branch modal interaction module, a feature fusion module and an emotion detection module; the dual-branch modal interaction module includes a text-to-image modal interaction network and an image-to-text modal interaction network; the text-to-image modal interaction network and the image-to-text modal interaction network both include several layers of modal interaction layers, and the modal interaction layers are connected by a preset soft router; A document encoding module, used for inputting the text to be tested and the image to be tested into the encoding module for encoding processing to obtain a document encoding representation, wherein the document encoding representation includes a text encoding representation and a visual encoding representation; A first modal interaction module, configured to use the text encoding representation and the visual encoding representation as a homomodal input representation and a heteromodal input representation of the text-to-image modal interaction network, respectively, input the homomodal input representation and the heteromodal input representation into a first modal interaction layer of the text-to-image modal interaction network for modal processing, and obtain an output representation of the first modal interaction layer; The output representation of the first modal interaction layer is input into the connected soft router for path decision, and the target input representation of the next modal interaction layer is obtained as the updated same-modal input representation. The updated same-modal input representation and the heteromodal input representation are input into the next modal interaction layer for modal processing, and the output representation of the next modal interaction layer is obtained, until the output representation of the last modal interaction layer is obtained; Performing feature aggregation on the output representation of the last modal interaction layer of the text-to-image modal interaction network to obtain a text-to-image modal interaction feature representation; A second modal interaction module is used to use the visual encoding representation and the text encoding representation as the same-modal input representation and the different-modal input representation of the image-to-text modal interaction network, respectively, and input them into the image-to-text modal interaction network for modal interaction processing to obtain an image-to-text modal interaction feature representation; A fusion module, used for inputting the text-to-image modality interaction feature representation and the image-to-text modality interaction feature representation into the feature fusion module for feature fusion to obtain a feature fusion representation; The detection module is used to input the feature fusion representation into the emotion detection module to perform emotion detection and obtain the emotion detection result of the document data to be tested.
9. A computer device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the multimodal emotion detection method as described in any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Multi-modal aspect-level sentiment analysis method and device, equipment and storage medium
CN117235261A
Multi-mode irony detection method and device, computer equipment and storage medium
CN117891940A
Multi-modal online comment sentiment analysis method based on mutual information and cross-modal interaction diagram
CN118468883A
