A document-level image-text review sentiment classification method fusing common sense knowledge

Abstract

The application belongs to the fields of artificial intelligence, computer vision, natural language processing and the like, and is a document-level image-text review sentiment classification method fusing common sense knowledge. First, review data collected from the Internet is abstracted into structured data, and after data preprocessing work such as Chinese word segmentation and image normalization, the feature representation of the review data is obtained by sequentially passing through single-modal sentiment feature representation learning and multi-modal sentiment feature fusion, and finally the fusion features are input into a classifier or a multi-layer perceptron is directly used to obtain a sentiment probability distribution. The method can effectively improve the accuracy of sentiment classification on the one hand by fusing common sense knowledge to strengthen text semantic understanding, and on the other hand by introducing related image information to assist in improving text sentiment expression.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of artificial intelligence, computer vision, natural language processing, etc., and mainly relates to deep learning technology, and particularly proposes a document-level image-text review sentiment classification method fusing common sense knowledge. BACKGROUND

[0002] With the development of the Internet and the popularity of intelligent terminal devices, network media as a new form of information dissemination has penetrated into people's daily life. People express their views on media platforms, spread their ideas, and post reviews on various goods, virtual products or online and offline services, and participate in discussions. Online users are not only content browsers but also creators. A large number of online users instantly upload reviews on e-commerce products, hotels, catering services and other service fields. With the continuous progress of mobile devices and media platforms, review content has evolved from single-text description to multi-modal and multi-form comprehensive evaluation system containing pictures, videos and ratings. Such review content is more intuitive and persuasive. Studies have shown that multi-modal reviews combining text and images can better predict the usefulness of online reviews than single-modal reviews containing only text or only images. Therefore, in order to fully exploit the potential value of user-generated content, multi-modal sentiment analysis technology can effectively mine the emotional expression of online image-text review data to guide further business analysis and decision-making.

[0003] Although multi-modal data contains more rich emotional information, different modal types of data play different roles in overall emotional expression. On the one hand, each modality has its unique characteristics in emotional expression; on the other hand, the interaction between different modal features can make up for the lack of emotional information of single-modal data and further improve the accuracy of sentiment recognition. Review data mainly relies on text information to convey emotions, so it is important to strengthen the emotional semantic understanding of text in single-modal modeling; and text and corresponding images generally revolve around the same emotional theme and have certain emotional semantic relevance, so the two modalities have consistency in a specific emotional feature space and can be measured for similarity. Therefore, for document-level image-text review data, the two key problems to be solved are: how to improve the quality of text data and learn more deep feature representation; and how to learn the associated information and complementary information between images and text and establish a more effective modal fusion mechanism. SUMMARY

[0004] To solve the above problems, the application provides a document-level image-text review sentiment classification method fusing common sense knowledge. By combining text content with relevant background knowledge, a coherent representation of the event and the reason can be obtained, and better text feature representation can be obtained. For additional images, consider their semantic relevance to the text, and use images highly relevant to the review content as supplementary features. Compared with the prior art, the accuracy of document-level image-text review sentiment classification can be effectively improved.

[0005] The technical solution adopted by the application is: a document-level image-text review sentiment classification method fusing common sense knowledge, comprising the following steps:

[0006] S1. Obtain original review data with text and pictures from a target (related to hotels, restaurants or other products) review website, preprocess the original review data of text and pictures to obtain preprocessed review data of text and pictures, and then perform preliminary feature extraction on the preprocessed review data of text and pictures to obtain word-level text features and global visual features;

[0007] S2. Input the preprocessed review data of text into the ATOMIC-COMET language model, and the ATOMIC-COMET language model performs common sense reasoning on the events contained in the preprocessed review data of text to obtain common sense knowledge features;

[0008] S3. Input the word-level text features into the sequence neural network Bi-GRU to obtain context-enhanced sentence-level text features, and input the sentence-level text features and the common sense knowledge features into the common sense knowledge fusion layer to obtain sentence-level text sentiment features;

[0009] S4. Input the global visual features and the sentence-level text sentiment features into the multi-modal information interaction layer to learn the inter-modal fine-grained correlation, and obtain sentence-level image-text related fine-grained fusion features;

[0010] S5. Aggregate the sentence-level text sentiment features and the sentence-level image-text related fine-grained fusion features to obtain document-level features of two channels respectively, fuse the document-level features of the two channels by a gating mechanism to obtain the final document-level feature representation, use a multi-layer perception machine to calculate the probability distribution of the sentiment category of the final document-level feature representation, and select the category with the highest probability value as the final classification result.

[0011] As a preferred mode: the preprocessing of the original review data of text and images in step S1 refers to the processing of sentence segmentation, word segmentation and stop word removal of the review data of text, and the processing of converting all images in the original review data of images into RGB images with a size of 224x224.

[0012] As a preferred mode: the preliminary feature extraction of the pre-processed review data of text and image in step S1 refers to extracting text features by using a pre-trained GloVe model to obtain word-level text features, and extracting global visual features of the pre-processed review data of the picture by using a pre-trained VGG-16 framework.

[0013] As a preferred mode: the ATOMIC-COMET language model in step S2 is derived from the ATOMIC knowledge triple pre-trained ATOMIC-COMET model.

[0014] As a preferred mode: the common sense reasoning in step S2 refers to three kinds of common sense reasoning of the intention of the principal part (xIntent), the emotional reaction of the principal part (xReact) and the emotional reaction of others (oReact); and the common sense knowledge features obtained in step S2 refer to the common sense knowledge features of the intention of the principal part, the emotional reaction of the principal part and the emotional reaction of others.

[0015] In order to extract effective common sense knowledge features, the language model ATOMIC-COMET based on the common sense library is used for common sense reasoning. Considering the intention of the reviewer and the emotional reaction of himself and the viewer, additional supervision signals can be provided for emotion classification, so three of the nine reasoning conditions related to psychological state, namely the intention of the principal part (xIntent), the emotional reaction of the principal part (xReact) and the emotional reaction of others (oReact) are selected. From each text review data in S1, the three common sense knowledge features of the intention of the principal part (xIntent), the emotional reaction of the principal part (xReact) and the emotional reaction of others (oReact) are reasoned out.

[0016] As a preferred mode: in step S3, the word-level text features are input into the sequence neural network Bi-GRU to obtain context-enhanced sentence-level text features, and the sentence-level text features and the common sense knowledge features are input into the common sense knowledge fusion layer to obtain sentence-level text emotion features, including the following steps:

[0017] Step 1: two levels of bidirectional gated recurrent neural network (Bi-GRU) are used in turn to obtain the context-enhanced sentence-level text feature representation in S3. Bi-GRU receives an input vector ft and outputs a new hidden state vector:

[0018]

[0019]

[0020]

[0021] wherein is the current word-level or sentence-level hidden state vector, is the previous word-level or sentence-level hidden state vector, is the next word-level or sentence-level hidden state vector. () represents a positive gate, () represents a reverse gate, BiG-DRU(f t ) represents a bidirectional gated recurrent unit function.

[0022] The word vector of the jth word x j in the given S1 original review is obtained by a word-level Bi-GRU However, the meaning of a word to a sentence does not carry equal amount of information, so a self-attention mechanism with the following formula is used to give more weight to more important words, and all word vectors in a sentence are aggregated into a sentence vector:

[0023]

[0024]

[0025]

[0026] wherein W w and b w are randomly initialized weights and bias parameters, is the hidden state vector output by an MLP, u w is a randomly initialized context word vector, a j is a self-attention weight, and s is a sentence vector aggregated according to the weight.

[0027] In order to introduce inter-sentence context information, a sentence-level bidirectional gated recurrent unit Bi-GRU takes the sentence vector s as input and then outputs a hidden state vector h = Bi-GRU(s). If there are L sentences (L events) in the S1 Chinese text review data, the context-enhanced text feature representation {h1, h2, …, h L} is obtained, wherein h1 represents the hidden state vector of the first sentence, h2 represents the hidden state vector of the second sentence, and h L represents the hidden state vector of the Lth sentence.

[0028] Step 2: The sentence-level text features and common sense knowledge features are input into a common sense knowledge fusion layer constructed based on a soft attention mechanism, the relevance of the sentence-level text features and the common sense knowledge features in common sense reasoning is calculated respectively, selective fusion is performed, and sentence-level text sentiment features are obtained.

[0029] For the original review data containing L events, the extracted common sense knowledge features {q1, q2, …, q L}, {r1, r2, …, r L} and {r'1, r'2, …, r' L} are introduced into the review document modeling process as supplementary information, q, r, r' are used to distinguish three types of knowledge (representing one type of knowledge, respectively, the subscript hides the state vector label). First, the common sense knowledge features are mapped to a new feature space K r，i with the same dimension as the inherent text features. The inherent text features h i are taken as queries, and the h i and k r，i are combined to calculate the attention distribution A r，i of different external knowledge, taking the r knowledge type as an example, the other two types of knowledge are processed in the same way, and the common sense knowledge features k i are connected with the context-enhanced sentence-level text features h i to obtain the sentence-level text features T = {t1, t2, …, t L} that fuse common sense knowledge, t1 represents the first event sentence-level text features that fuse common sense knowledge, t2 represents the second event sentence-level text features that fuse common sense knowledge, and t L represents the Lth event sentence-level text features that fuse common sense knowledge. The specific process is shown in the following formula:

[0030] A r，i = U K T tanh(W r [k r，j ; h i ]+b r )

[0031]

[0032]

[0033]

[0034] In the formula, W r and b r are randomly initialized weights and bias parameters, U K T is a randomly initialized global matrix parameter, β r，i is a self-attention weight, and t i represents the i-th event sentence-level text features that fuse common sense knowledge.

[0035] As a preferred mode: the multimodal information interaction layer in step S4 is a text-guided interaction attention mechanism, and the learning of the inter-modal fine-grained correlation in step S4 refers to taking the sentence-level text sentiment feature as a query to calculate the correlation with each image in the global visual feature, the stronger the correlation, the greater the weight of the image, thereby aggregating the features of multiple images, and then fusing the sentence-level text sentiment feature.

[0036] The multimodal information interaction layer in S4 is used to capture the interaction between modal features. The sentence-level text sentiment feature T = {t1, t2,..., t L} in S3 and the global visual feature V = {v1, v2,..., v M} extracted by the VGG-16 framework in S1 are taken as inputs, which are first projected into a shared feature space to be converted into sentence-level text sentiment feature T' and global visual feature V' with the same dimension.

[0037] Then, based on the text-guided interaction attention mechanism, the sentence-level text sentiment feature in S3 is taken as a query to calculate the correlation with each image:

[0038] C = (T') T · V'

[0039]

[0040] In the formula, C is a fine-grained attention matrix, M i，j is an attention weight.

[0041] The stronger the correlation, the greater the weight of the image, thereby aggregating the features of multiple images, and then fusing the text features:

[0042] O = V' · M T

[0043] G = tanh(W g [T; O])

[0044] In the formula, O is a feature matrix after aggregating the text and the picture, W g is a randomly initialized weight, and G is a fused feature matrix.

[0045] As a preferred mode: the sentence-level text sentiment feature and the sentence-level fine-grained fusion feature of the text-image correlation are respectively aggregated to obtain document-level features of two channels, and the document-level features of the two channels are fused by a gating mechanism to obtain the final document-level feature representation, including the following steps

[0046] Step 1: The sentence-level text sentiment feature is aggregated into a document-level text feature representation d T by a self-attention mechanism.

[0047] Step 2: The sentence-level fine-grained fusion features related to the image are aggregated into document-level fusion feature representation d by a self-attention mechanism VT ;

[0048] Step 3: The document-level feature representations of the two channels are fused by using a gating mechanism M(·) to control the information flow, and the final document-level feature representation d is obtained:

[0049] d = M(d T , d VT ) ⊙ d T + (1 - M(d T , d VT )) ⊙ d vT

[0050] In the above steps, ⊙ represents the multiplication of elements, and the gating mechanism function M(·) is specifically represented as:

[0051] M(d T , d VT ) = sigmoid(U T tanh(W q (d T + d VT ) + b q ))

[0052] where W q and b q are randomly initialized weight and bias parameters, U T is a randomly initialized global matrix parameter, and sigmoid and tanh are activation functions.

[0053] Further, after obtaining the final document-level image-text feature d, the probability distribution of the sentiment category of the final document-level feature representation is calculated using a multi-layer perception, and the category with the highest probability value is selected as the final classification result. The probability distribution calculation formula is:

[0054] p = softmax(W c d + b c )

[0055] where W c and b c are randomly initialized weight and bias parameters, softmax is an activation function, and p is a probability.

[0056] Beneficial effects: the present application proposes a kind of document-level image-text review sentiment classification method fusing common sense knowledge, compared with the implementation effect of prior art, progress has been made.In the representation learning stage of text feature, it is not novel to introduce the context information of original text, but if the inference event not described in original text segment is used to enhance semantic understanding, it is beneficial to obtain stronger text features.As the text and picture in image-text review are not sequence-aligned, for this, the present application proposes text-guided visual attention in the stage of modal information interaction, and the image highly related to review text is used as supplementary information, so as to reduce the noise interference of irrelevant features in image, while the robust text features are maximally retained. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 The document-level image-text review sentiment classification method fusing common sense described in the present application is a flow chart.

[0058] Figure 2 The model structure diagram of the document-level image-text review sentiment classification method fusing common sense described in the present application. DETAILED DESCRIPTION

[0059] The technical solutions are specifically and exhaustively described below in combination with the model diagram and specific embodiments of the method of the present application, so that the technical features of the present application can be more easily understood by those skilled in the art.

[0060] Sentiment classification belongs to a supervised learning task, and its basic idea is to train a classification model through known data, and then use the model to predict the sentiment category of other input data.In this embodiment, classification accuracy is selected as the evaluation index of the model.

[0061] Embodiment: the model is trained and classified by using Yelp restaurant review data.The data set contains image-text reviews of some restaurants in five cities in the United States, and the data statistics are shown in the following table:

[0062]

[0063] The document-level image-text review sentiment classification method fusing common sense described in the present application is executed as follows:

[0064] Step 1: in order to play the auxiliary role of image information, each comment data collected in S1 contains at least three pictures.In order to ensure data consistency, three images are randomly selected and normalized to 224x224 RGB images before preliminary feature extraction is performed;the comment document is segmented and standardized to length L, and then tokenization and stop word removal processing are performed.Pre-trained VGG-16 framework is used to extract global visual features, and pre-trained GloVe model is used to extract word-level text features.

[0065] Step 2: The intent of the reviewer and its own and the emotional reactions of the browser can provide additional supervision signals for sentiment classification, considering the three if-then relations in ATOMIC related to psychological states, namely xIntent, xReact and oReact. Specifically, the i-th sentence s i is taken as the subject phrase and concatenated with the relation phrase r. Next, the concatenated is passed to the pre-trained ATOMIC-COMET model, and the output of its last time step activation function is extracted as the inferred commonsense knowledge features, resulting in three different feature vectors q i , r i , r′ i , corresponding to the commonsense knowledge features of the i-th sentence in the review document under the three reasoning relations, respectively.

[0066] Step 3: The commonsense knowledge features in S2 are fused with the word-level text features in S1, which includes the following steps:

[0067] Step 3.1: Bi-directional gated recurrent unit (Bi-GRU) is used at both word and sentence levels to obtain context-enhanced sentence-level text feature representations in S3. Bi-GRU receives input vector f t and outputs a new hidden state vector:

[0068]

[0069]

[0070]

[0071] where h is the hidden state vector.

[0072] The word vector of a given word x j in the original review S1 is obtained through the word-level Bi-GRU. However, words do not carry equal amounts of information about the meaning of a sentence, so a self-attention mechanism is used to give more weight to more important words and aggregate all word vectors in a sentence into a sentence vector:

[0073]

[0074]

[0075]

[0076] where W w and b w are randomly initialized weights and bias parameters, and uj It is the hidden state vector output by a single MLP layer, uw is the randomly initialized context word vector, and α is the hidden state vector. j is the self-attention weight, and s is the sentence vector aggregated based on the weight.

[0077] Subsequently, to incorporate inter-sentence contextual information, the sentence-level Bi-GRU takes the sentence vector s as input and outputs the hidden state vector h = Bi-GRU(s). If the text comment data in S1 has a length of L, i.e., L sentences, then the context-enhanced text feature representation {h1, h2, ..., h...} in S3 is obtained. L}

[0078] Step 3.2: For the S1 Chinese text comment data containing L events, extract the common sense knowledge features {q1, q2, ..., q} from S2. L}, {r1, r2, ..., r L} and {r′1, r′2, ..., r′ L This information is introduced as supplementary information into the comment document modeling process. First, the common sense knowledge features in S2 are mapped to a new feature space k with the same dimension as the inherent text features. r，i r is used to distinguish knowledge types. The inherent textual features h i As a query, merge h i and k r，i Then, the tanh activation function is used to calculate the attention distribution A for different external knowledge. r，i The common sense knowledge feature k in S2 is obtained by weighted calculation. i Compared with context-enhanced sentence-level text features h in S3 i Connect the sentences to obtain the sentence-level text features T = {t1, t2, ..., t3} in S3 that incorporate common-sense knowledge. L The specific process is shown in the following formula:

[0079] A r，i =U K T tanh(W r [k r，i h i ]+b r )

[0080]

[0081]

[0082]

[0083] In the formula W r and b r It randomly initializes the weights and bias parameters, U KT are randomly initialized global matrix parameters, β r，i are self-attention weights.

[0084] Step 4: Modal information interaction layer, taking the sentence-level text features T = {t1, t2, …, t L} obtained in step 3 and the global visual features V = {v1, v2, …, v M} extracted by the VGG-16 framework in step 1 as inputs, first project them into the same shared space:

[0085] T' = tanh(W s T + b s )

[0086] V' = tanh(W a V + b a )

[0087] where W s and W a are randomly initialized weights, and b s and b a are randomly initialized bias parameters.

[0088] Then, based on a text-guided interactive attention mechanism, calculate the relevance to each image:

[0089] C = (R')T T · V'

[0090]

[0091] where C is a fine-grained attention matrix, M ，j is an attention weight.

[0092] The stronger the relevance, the greater the weight of the image, thereby aggregating the features of multiple images and then fusing them with the text features:

[0093] O = V' · M T

[0094] G = tanh(W g [T; O])

[0095] where O is the feature matrix after aggregating the text and images, W g is a randomly initialized weight, and G is the fused feature matrix.

[0096] Step 5: Through a self-attention mechanism, aggregate the knowledge-rich sentence-level text sentiment features obtained in step S3 into document-level text feature representation d TThe sentence-level image-text related fine-grained fusion features obtained in step S4 are aggregated into a document-level fusion feature representation d. VT Then, a gating mechanism M(·) is used to control the information flow, fusing the document-level feature representations of the two channels to obtain the final document-level feature representation d in S5:

[0097] d=M(d T d VT )*d T +(1-M(d T ,d VT ))*d VT

[0098] In the above steps, * represents element-wise multiplication, and the gating mechanism function M(·) is specifically represented as follows:

[0099] M(d T d VT ) = sigmoid(U T tanh(W q (d T +d VT )+b q ))

[0100] In the formula W q and b q These are randomly initialized weights and bias parameters, U T These are randomly initialized global matrix parameters, and sigmoid and tanh are activation functions.

[0101] After obtaining the final document-level image and text features d, a multilayer perceptron is used to calculate the probability distribution of the sentiment category in the final document-level feature representation. The category with the highest probability value is selected as the final classification result. The probability distribution calculation formula is:

[0102] p = softmax(W c d+b c )

[0103] In the formula W c and b c These are randomly initialized weights and bias parameters, and softmax is the activation function.

[0104] In this example, approximately 15% of the data from each city was used to test and evaluate the model's performance, 80% was used as training samples, and 5% was reserved as a validation set. The model was trained using the cross-entropy loss function.

[0105]

[0106] In the formula p cis the predicted probability that the sample belongs to class c, when the true label value takes c, y c is 1, otherwise 0.

[0107] The test sample is input into the trained model, the classification accuracy is used to evaluate the document-level image-text review sentiment classification model fused with common sense knowledge, and the existing model without introducing common sense knowledge and without considering the fine-grained interaction of image-text information is compared, and the experimental results are as follows:

[0108]

[0109] In the above embodiment, the sentiment classification accuracy is improved, and the effect of the method is verified. It should be noted that the specific embodiments listed here are only exemplary descriptions of the present application, and are not limitations on the scope of protection of the present application, and the scope of protection of the present application is defined by the claims. The technical solutions of the present application are modified or replaced with tools without departing from the purpose of the technical solutions, which should fall within the scope of protection of the present application.

Claims

1. A document-level image and text comment sentiment classification method that integrates common sense knowledge, characterized in that: Includes the following steps: S1. Obtain raw comment data with text and images from the target comment website, preprocess the raw comment data with text and images to obtain preprocessed comment data with text and images, and then perform preliminary feature extraction on the preprocessed comment data with text and images to obtain word-level text features and global visual features. S2. Input the preprocessed comment data into the ATOMIC-COMET language model. The ATOMIC-COMET language model performs common-sense reasoning on events contained in preprocessed comment data to obtain common-sense knowledge features; S3. Inputting word-level text features into a sequence-type neural network Bi-GRU to obtain context-enhanced sentence-level text features, and inputting sentence-level text features together with common-sense knowledge features into a common-sense knowledge fusion layer to obtain sentence-level text sentiment features; the process of inputting word-level text features into a sequence-type neural network Bi-GRU to obtain context-enhanced sentence-level text features, and inputting sentence-level text features together with common-sense knowledge features into a common-sense knowledge fusion layer to obtain sentence-level text sentiment features includes: inputting word-level text features into a word-level bidirectional gated recurrent neural network Bi-GRU, and aggregating them into sentence vectors using a self-attention mechanism, then inputting the sentence vectors into a sentence-level bidirectional gated recurrent neural network Bi-GRU, and outputting the hidden states of each unit as context-enhanced sentence-level text features; inputting sentence-level text features together with common-sense knowledge features into a common-sense knowledge fusion layer constructed based on a soft attention mechanism, calculating the correlation between sentence-level text features and common-sense knowledge features in common-sense reasoning respectively, and selectively fusing them to obtain sentence-level text sentiment features; S4. Input the global visual features and sentence-level text sentiment features into the multimodal information interaction layer to learn the fine-grained correlation between modalities and obtain sentence-level image-text related fine-grained fusion features. The multimodal information interaction layer is a text-guided interactive attention mechanism. In step S4, learning the fine-grained correlation between modalities means using sentence-level text sentiment features as queries and calculating their correlation with each image in the global visual features. The stronger the correlation, the greater the weight of the image. This aggregates the features of multiple images and then fuses them with the sentence-level text sentiment features. S5. The sentence-level text sentiment features and the sentence-level image-text related fine-grained fusion features are aggregated to obtain document-level features for two channels. After fusing the document-level features of the two channels by a gating mechanism, the final document-level feature representation is obtained. The probability distribution of the sentiment category of the final document-level feature representation is calculated using a multilayer perceptron, and the category with the highest probability value is selected as the final classification result.

2. The document-level image and text comment sentiment classification method integrating common sense knowledge as described in claim 1, characterized in that: The preprocessing of the original comment data of text and images mentioned in step S1 refers to the processing of the text comment data into sentences, words, and stop words, and the processing of converting all images in the original comment data of images into RGB images of size 224×224.

3. The document-level image and text comment sentiment classification method integrating common sense knowledge as described in claim 1, characterized in that: The preliminary feature extraction of preprocessed comment data for text and images mentioned in step S1 refers to using a pre-trained GloVe model to extract text features to obtain word-level text features, and using a pre-trained VGG-16 framework to extract global visual features from preprocessed comment data for images.

4. The document-level image and text comment sentiment classification method integrating common sense knowledge as described in claim 1, characterized in that: The ATOMIC-COMET language model mentioned in step S2 is derived from the ATOMIC-COMET model pre-trained with knowledge triples in ATOMIC.

5. The document-level image and text comment sentiment classification method integrating common sense knowledge as described in claim 1, characterized in that: In step S2, common sense reasoning refers to the three types of common sense reasoning: the party's intention, the party's emotional reaction, and the emotional reactions of others. In step S2, obtaining common sense knowledge features refers to obtaining the three types of common sense knowledge features: the party's intention, the party's emotional reaction, and the emotional reactions of others.

6. The document-level image and text comment sentiment classification method integrating common sense knowledge as described in claim 1, characterized in that: In step S5, the sentence-level text sentiment features and the sentence-level image-text related fine-grained fusion features are aggregated to obtain document-level features for two channels. After fusing the document-level features of the two channels by a gating mechanism, the final document-level feature representation is obtained, including the following steps. Step 1: Using a self-attention mechanism, aggregate sentence-level text sentiment features into document-level text feature representations. T ; Step 2: Through a self-attention mechanism, aggregate the sentence-level image-text related fine-grained fusion features into a document-level fusion feature representation d. VT ; Step 3: Use a gating mechanism M(·) to control the information flow and fuse the document-level feature representations of the two channels to obtain the final document-level feature representation d: d=M(d T ,d VT )☉d T +(1-M(d T ,d VT ))☉d VT In the above steps, ⊙ represents element-wise multiplication, and the gating mechanism function M(.) is specifically represented as follows: M(d T ,d VT )=sigmoid(U T tanh(W q (d T +d VT )+b q )) In the formula W q and b q These are randomly initialized weights and bias parameters, U T These are randomly initialized global matrix parameters, and sigmoid and tanh are activation functions.

Images