A method and system for image-text sentiment classification based on heterogeneous fusion and symmetric translation
By adopting heterogeneous fusion and symmetric translation technologies in the classification of graphic and text emotions, the problems of insufficient feature extraction and modal heterogeneity in the existing technology are solved, which significantly improves the effect of graphic and text emotions, especially through the symmetric translation module. The generation of text feature vectors is enhanced, taking into account the two-way fusion of text and pictures.
Patent Information
- Application Number
- CN202210580293.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-25
- Publication Date
- 2025-01-03
- Estimated Expiration
- 2042-05-25
AI Technical Summary
The existing graphic and text emotion classification methods have problems in insufficient feature extraction ability, modal heterogeneity and insufficient consideration of the role of text modalities, resulting in poor emotional classification effects.
Using a method based on heterofusion and symmetric translation, the feature vectors of text and pictures are extracted through attention mechanism and Transformer architecture, and spliced and encoded. The symmetric translation module is used to translate text and pictures into the same vector space to enhance text The generation of feature vectors takes into account the two-way fusion of text and pictures.
The effect of graphic and text emotional classification is improved, the role of text modality is more considered, the heterogeneity problem between guidance vectors and text vectors is solved, and the fusion effect is improved.
Smart Images
Figure CN114969338B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of natural language processing and deep learning technology, and in particular to a method and system for image-text sentiment classification based on heterogeneous fusion and symmetric translation. Background Art
[0002] With the rapid development of the Internet and computer science, a large number of social platforms and commercial websites have gradually emerged, such as Weibo, Douyin, Xiaohongshu, Meituan, etc. Various video websites headed by Bilibili and shopping platforms such as Taobao are widely used, which has greatly affected people's lives and the way they communicate. People's expressions on various platforms are becoming more and more diverse, not just limited to text, but also through pictures and videos to express their opinions. For example, users will upload their own videos on Bilibili and attach some text, and will also evaluate the products they have used on shopping websites in the form of photos and text. Consumers can decide whether to buy based on existing product reviews, and merchants can judge users' preferences for products and the popularity of products through user reviews, and decide how to better serve users.
[0003] Each source or form of information can be called a modality, and text, pictures, and audio constitute the three most common modalities in reality. With the continuous growth of multimodal content on the Internet, the task of multimodal sentiment analysis has emerged. Multimodal sentiment analysis aims to use multimodal content such as text, pictures, and audio, make full use of their complementarity, and analyze the user emotions contained therein. In the past, sentiment analysis was limited to a single modality. For example, text sentiment analysis only needs to focus on mining and reasoning the emotions contained in the text. Multimodal sentiment analysis requires processing data from multiple modalities, which brings many challenges. Compared with single-modal data, multimodal data contains more information and can complement each other to help machines better understand emotions. The task of multimodal sentiment analysis is an important research topic in the field of social computing and sentiment analysis, and has become a research hotspot in recent years.
[0004] Existing image-text sentiment classification methods include the VistaNet model proposed by Truong et al. and the deep multimodal fusion model proposed by Huang et al. The main innovation of the VistaNet model is to use the attention mechanism to let the image feature vector guide the generation of the text feature vector, which solves the problem of inconsistency between the two modal vector spaces, that is, data heterogeneity. It can also learn the direct correlation information between text and image invisibly, and the fusion effect is better than the previous model. However, this model simply uses the image feature vector to guide the text feature vector. Although it can be implicitly fused, the image feature vector used for guidance is essentially heterogeneous with the text feature vector, which will also limit the fusion effect. The deep multimodal fusion model first uses two independent attention mechanism modules to extract features from text and images respectively, and then uses the multimodal attention mechanism module to fuse text and images, and finally performs sentiment classification. However, this model does not fully consider the role of text modality, and the sentiment classification effect is average. Summary of the invention
[0005] In order to solve the problems of insufficient feature extraction capability, modal heterogeneity, insufficient consideration of the role of text modality, and single image-text fusion method in existing image-text sentiment classification methods, the present invention provides an image-text sentiment classification method and system based on heterogeneous fusion and symmetric translation.
[0006] To achieve the above object, according to a first aspect of the present invention, a method for image-text sentiment classification based on heterogeneous fusion and symmetric translation is provided, the method comprising:
[0007] Input text and pictures, and obtain sentiment classification through the sentiment classification model. The training method of the sentiment classification model includes:
[0008] S1. Perform feature extraction on the text and image data in the data set to obtain feature vector representations of words in the text and feature vector representations of images respectively;
[0009] S2. Based on the attention mechanism, the word feature vector is encoded to obtain the sentence feature vector;
[0010] S3, based on the Transformer architecture, set the sentences and pictures in the text as the source modality and target modality for encoding respectively, and take the fusion vector output by the Transformer encoder for splicing as the guidance vector;
[0011] S4. Based on the attention mechanism and the guidance vector, the sentence feature vector is encoded to obtain the text feature vector.
[0012] S5. Concatenate the output vector of the Transformer encoder and the feature vector of the document to obtain the final vector representation, and perform sentiment classification and parameter adjustment of the sentiment classification model.
[0013] Furthermore, in step S1, the Bert model is used to obtain the feature vector representation x of the word i,t , use VGG16 to obtain the feature vector representation P of the picture j .
[0014] Furthermore, the encoding process in step S2 includes the following steps:
[0015] S21, for the selected word feature vector x i,t , use bidirectional LSTM to analyze x from both the forward and backward directions i,t Encode and concatenate the hidden layer vectors obtained in the two directions to obtain the final hidden layer vector h i,t ;
[0016] S22, for the hidden layer vector h i,t , calculated using the attention mechanism and then normalized:
[0017] v i,t =V·tanh(W v h i,t +b v )
[0018]
[0019]
[0020] Where V represents a randomly initialized matrix, v i,t represents the weight of the word for the entire sentence, tanh represents the activation function, exp represents the exponential function, W v and b v is a randomly initialized value that is automatically adjusted during training, α i,t Represents the weight of the normalized word for the entire sentence, s i Represents the obtained sentence feature vector.
[0021] Furthermore, in step S3, the encoding method of the Transformer encoder is:
[0022] ε s→p =f s→p (X s )
[0023] ε p→s =f p→s (X p )
[0024]
[0025] Among them, X s Indicates that the text is used as the source mode, X p represents the image as the source modality, ε s→p and ε p→s are the two output vectors of the encoder, f represents the activation function of the Transformer architecture, represents the concatenation operation and ε represents the guidance vector.
[0026] Furthermore, in step S4, encoding the sentence feature vector includes the following steps:
[0027] S41, for the selected sentence feature vector s i , use bidirectional LSTM to analyze s from the front and back directions respectively i Encode and concatenate the hidden layer vectors obtained from the two directions to obtain the sentence hidden layer vector h i ;
[0028] S42, use the tanh activation function to activate the guidance vector ε and the sentence hidden layer vector h i Use a nonlinear method to map to the same vector space, so as to obtain the feature vector representation f of the guidance vector generated after the image and text fusion, and the text feature vector representation g i ;
[0029] S43, based on the attention mechanism, f and g i Perform inner product operation. In order to consider text factors more, add g separately. i , we get the vector u representing the sentence attention weight i , the calculation formula is:
[0030] u i =U·(f⊙g i +g i )
[0031] Among them, U represents the parameter matrix obtained by random initialization, and ⊙ represents the inner product operation;
[0032] S44, weight vector u i Perform normalization calculation, and then use the normalized weight γ i and the sentence hidden layer vector h i The weighted summation is used to obtain the text feature vector d; where h i represents the hidden layer vector of the sentence, γ i Represents the normalized weight of the ith sentence relative to the entire text.
[0033] Furthermore, in step S5, the obtained guidance vector and text feature vector are first concatenated to obtain a final vector representation, and then the final vector representation is classified using a fully connected network, and the parameters of the sentiment classification model are adjusted through a back propagation algorithm.
[0034] According to a second aspect of the present invention, a picture-text sentiment classification system based on heterogeneous fusion and symmetric translation is provided, the system comprising a sentiment classification model and a training module, wherein the training module is used to input pictures and texts in a data set into the sentiment classification model, and compare the sentiment classifications corresponding to the pictures and texts in the data set with the classifications output by the sentiment classification model, and adjust the parameters of the sentiment classification model through a back propagation algorithm;
[0035] The sentiment classification model includes: a modal feature extraction module, which is used to extract features from the data of two modalities in the data set, and obtain feature vectors of words in the text and feature vectors of pictures respectively;
[0036] Heterogeneous fusion attention module generates a sentence feature vector based on the attention mechanism combined with the influence of words on the sentence sentiment polarity, and generates a text feature vector based on the attention mechanism combined with the influence of sentences on the text sentiment polarity;
[0037] The symmetric translation fusion module is used to set sentences and pictures as the source modality and target modality respectively, use Transformer to translate the source modality into the target modality, and take the output of the encoder as the vector representation after the fusion of the two modalities;
[0038] The sentiment classification module concatenates the output vectors of the two Transformer encoders and the feature vector of the text to obtain the final vector representation, and outputs the sentiment classification through a fully connected network.
[0039] The image-text sentiment classification method and system based on heterogeneous fusion and symmetric translation provided by the present invention have the following beneficial effects compared with the existing image-text sentiment classification method and system:
[0040] 1. The present invention takes the role of text modality into consideration more, and uses a heterogeneous fusion attention mechanism to consider the influence of each word in a sentence on sentiment polarity, as well as the influence of each sentence in a text on sentiment polarity.
[0041] 2. The present invention uses the vector guidance generated by the fusion of the symmetric translation module to generate the text vector, which solves the problem of heterogeneity between the guidance vector and the text vector. At the same time, in the process of text vector generation, the fusion of text and picture is considered again, rather than a single fusion method, which further improves the effect of picture and text fusion. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0043] Figure 1 A schematic diagram of a flow chart of a method for image-text sentiment classification based on heterogeneous fusion and symmetric translation according to an embodiment of the present invention;
[0044] Figure 2 A schematic flow chart of a method for image-text sentiment classification based on heterogeneous fusion and symmetric translation according to another embodiment of the present invention;
[0045] Figure 3 This is a flow chart of step S2 of a method for image-text sentiment classification based on heterogeneous fusion and symmetric translation according to an embodiment of the present invention;
[0046] Figure 4 This is a flow chart of step S3 of a method for image-text sentiment classification based on heterogeneous fusion and symmetric translation according to an embodiment of the present invention;
[0047] Figure 5 This is a flow chart of step S4 of a method for image-text sentiment classification based on heterogeneous fusion and symmetric translation according to an embodiment of the present invention;
[0048] Figure 6 The system structure diagram of a picture-text sentiment classification system based on heterogeneous fusion and symmetric translation according to an embodiment of the present invention. DETAILED DESCRIPTION
[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0050] The present invention provides a method for image-text sentiment classification based on heterogeneous fusion and symmetric translation. Figure 1 and 2 As shown, the method includes:
[0051] Input text and pictures, and obtain sentiment classification through the sentiment classification model. The training method of the sentiment classification model includes:
[0052] S1. Extract features from the text and image data in the dataset and obtain word feature vectors x i,t and the image feature vector P j . Where i represents the i-th sentence in the selected text, t represents the t-th word in the i-th sentence, assuming that the i-th sentence contains a total of C words, and j represents the j-th image in the sample.
[0053] S2, based on the attention mechanism, the selected word feature vector x i,t Encode to get the feature vector s of each sentence i . Where i represents the i-th sentence in the selected text, assuming that the selected text contains a total of L sentences.
[0054] S3, based on Transformer, the sentence feature vector s i and the image feature vector P j Set them as source mode and target mode for encoding respectively, and take the ε output by the encoder respectively s→p and ε p→s The concatenation is performed as the guidance vector ε.
[0055] S4, based on the attention mechanism and the guidance vector ε, the sentence feature vector s i Encode to obtain the text feature vector d.
[0056] S5. Concatenate the output vector of the Transformer encoder and the feature vector of the document to obtain the final vector representation, and perform sentiment classification and parameter adjustment of the sentiment classification model.
[0057] In step S1, the Bert model is used to obtain the feature vector representation x of the word i,t , use VGG16 to obtain the feature vector representation P of the picture j .
[0058] The specific process is as follows: feature extraction is performed on the input text and image data to obtain word feature vectors and image feature vectors respectively.
[0059] The word feature vector is x i,t Indicates that the image feature vector is represented by P j where i represents the i-th sentence in the selected text, t represents the t-th word in the i-th sentence, assuming that the i-th sentence contains a total of C words, and j represents the j-th image in the sample. Taking the i-th sentence "They have a large selection of macaroons too which are delectable" as an example, the word vector x i,t is the vector representation of the tth word in the sentence.
[0060] In step S2, during the encoding process, for each word feature vector x to be input i,t , use bidirectional LSTM to calculate from the front and back directions respectively, and take the hidden layer vectors of the two directions at each moment to splice to get the final hidden layer vector output h i,t .
[0061] Then, the attention mechanism is calculated for the vector encoded by the bidirectional LSTM. The purpose is to obtain the weight of each word relative to the entire sentence. The weight value represents the influence of the current word on the sentiment of the entire sentence. After obtaining the weight value, the final vector representation s of the current sentence is calculated. i ,like Figure 3 shown.
[0062] For example, "They have a large selection of macaroons too which are delectable", after encoding, we can get the weight of each word in the sentence relative to the entire sentence. The weight value represents the influence of the current word on the sentiment polarity of the entire sentence. The specific encoding method is:
[0063] S21, for the selected word feature vector x i,t , use bidirectional LSTM to analyze x from both the forward and backward directions i,t To encode:
[0064]
[0065] Among them, x i,t represents the feature vector of the tth word in the ith sentence, and Respectively represent x i,t Hidden layer vectors in both left-to-right and right-to-left directions, h i,t represents the final hidden layer vector obtained by concatenating the above two hidden layer vectors. At each moment, the current hidden layer vector h i,t The calculation depends on the hidden layer vector h at the previous moment. i,t-1 and the current input x i,t .
[0066] S22, the obtained hidden layer vector h i,t Perform the attention mechanism calculation, and then calculate the final feature vector representation of the current sentence:
[0067] v i,t =V·tanh(W v h i,t +b v )
[0068]
[0069]
[0070] Where V represents a randomly initialized matrix, v i,t represents the weight of the word for the entire sentence, tanh represents the activation function, exp represents the exponential function, W v and b v is a randomly initialized value that is automatically adjusted during training, α i,t It represents the weight of the normalized word for the entire sentence, with a value range of [0,1], s i Represents the obtained sentence feature vector.
[0071] The sentence vector obtained after encoding takes into account the emotional impact of each word in the sentence on the entire sentence, rather than a sentence vector obtained by simple concatenation.
[0072] In step S3, the sentence feature vector and the picture feature vector in the selected text are set as the source modality and the target modality respectively, and then the source modality is translated into the target modality using the encoder-decoder architecture. The two output vectors of the encoder are concatenated as the guidance vector to guide the generation of the text vector in the next step, which solves the problem of heterogeneity between the guidance vector and the text vector. At the same time, the fusion of text and picture is considered again in the process of text vector generation, which further improves the effect of picture-text fusion. Figure 4 As shown. Translation refers to encoding and decoding, the process is as follows:
[0073] (1) Determine the sentence vector and image vector: the sentence feature vector s obtained by encoding in S2 i and the image feature vector P obtained in S1 j .
[0074] (2) The process of calculating the fusion vector: Use two Transformers to set the two feature vectors as the source mode and the target mode respectively to encode the fusion vector ε s→p and ε p→s :
[0075] ε s→p =f s→p (X s )
[0076] ε p→s =f p→s (X p )
[0077] Among them, X s Indicates that the text is used as the source mode, X p represents the image as the source modality, εs→p and ε p→s is the fused vector of the encoder’s output, and f represents the activation function in the Transformer architecture.
[0078] (3) The process of obtaining the guidance vector: The two fusion vectors are concatenated to obtain the guidance vector ε:
[0079]
[0080] in, Represents a concatenation operation.
[0081] Through the above steps (1)-(3), the image-text fusion vector ε can be obtained and used to guide the next step.
[0082] In step S4, according to the sentence feature vector s i Combined with the guidance vector ε, the text feature vector d is generated, such as Figure 5 As shown, the specific process is as follows:
[0083] S41. Sentence vector s i Encoding: Use bidirectional LSTM to encode s from both the forward and backward directions i To encode:
[0084]
[0085] Among them, s i represents the feature vector of the i-th sentence, and are the hidden layer vectors from left to right and from right to left for the i-th sentence, respectively. i Represents the final hidden layer vector obtained by concatenating the two hidden layer vectors obtained by the bidirectional LSTM.
[0086] S42. Encode the hidden layer vector and the guidance vector: Use the tanh activation function to map the guidance vector and the sentence hidden layer vector to the same vector space in a nonlinear manner. The calculation formula is:
[0087] f = tanh(W f ε+b f )
[0088] g i =tanh(W g h i +b g )
[0089] Among them, W f , W g 、b f and b gis a randomly initialized value that is automatically adjusted during training. f represents the feature vector representation of the guidance vector generated after image and text fusion. g i Represents the text feature vector representation.
[0090] S43. Calculate the sentence attention weight vector u i :f and g i Perform inner product operation. In order to consider text factors more, add g separately. i , we get the vector u representing the sentence attention weight i , the calculation formula is:
[0091] u i =U·(f⊙g i +g i )
[0092] Among them, U represents the parameter matrix, f and g i Perform inner product operation to express the similarity between them. i The result after the inner product only represents the interaction between them, not the information of the text, so it is necessary to add g i , thus taking the influence of text into consideration more. The vector u obtained above i , represents the attention weight of the ith sentence. The larger the weight, the greater the influence of the sentence on the text sentiment.
[0093] S44, weight vector u i Normalization calculation is performed to obtain text representation. First, the weight vector u i Perform normalization calculation, and then use the normalized weight γ i and the sentence hidden layer vector h i The text feature vector d is obtained by weighted summation. The specific process is as follows:
[0094]
[0095]
[0096] where γ i The attention score is represented by the weighted sum of the attention score and the hidden layer vector of each sentence to obtain the final text vector representation, which is used as one of the sources of the sentiment classification representation vector. This text vector contains not only the information of the text, but also the information of the fusion of text and image, because in the second-layer attention mechanism, the fusion vector ε generated during symmetric translation is used to guide the generated attention weight.
[0097] In step S5, the output vectors of the two Transformer encoders and the text feature vector are first concatenated to obtain the final vector representation, and then a fully connected network is used to classify the vector representation. The specific process is as follows:
[0098] a=concat(d,ε s→p ,ε p→s )
[0099]
[0100] Among them, d represents the text feature vector, ε s→p and ε p→s Represents the two output vectors of the encoder, concat represents the concatenation operation, and Linear represents the fully connected network. The final sentiment classification result is represented, and then the parameters of the sentiment classification model are adjusted through the classification and back propagation algorithms of the pictures and texts.
[0101] The present invention also provides a picture-text sentiment classification system based on heterogeneous fusion and symmetric translation, such as Figure 6 As shown, it includes a training module and a sentiment classification model, wherein the training module is used to input the pictures and texts in the data set into the sentiment classification model, and compare the sentiment classifications corresponding to the pictures and texts in the data set with the classifications output by the sentiment classification model, and adjust the parameters of the sentiment classification model through a back propagation algorithm;
[0102] The sentiment classification model includes a modal feature extraction module, which is used to extract features from the data of two modalities in the data set to obtain feature vectors of words in the text and feature vectors of pictures respectively; see step S1 above for details.
[0103] The heterogeneous fusion attention module generates a feature vector of a sentence based on the attention mechanism combined with the influence of words on the sentiment polarity of the sentence. At the same time, it generates a feature vector of the text based on the attention mechanism combined with the influence of sentences on the sentiment polarity of the text. For details, see steps S2 and S4 above.
[0104] The symmetric translation fusion module is used to set the sentence and the picture as the source modality and the target modality respectively, use the Transformer to translate the source modality into the target modality, and take the output of the encoder as the vector representation after the fusion of the two modalities; see step S3 above for details.
[0105] The sentiment classification module concatenates the two output vectors of the Transformer encoder and the feature vector of the text to obtain the final vector representation for sentiment classification. For details, see step S5 above.
[0106] The technical contents not elaborated in detail in the present invention belong to the common knowledge of those skilled in the art.
[0107] Although the above describes the illustrative specific embodiments of the present invention to facilitate the understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.
Claims
1. A method for image-text sentiment classification based on heterogeneous fusion and symmetric translation, characterized in that: The method comprises: Input text and pictures, and obtain sentiment classification through the sentiment classification model. The training method of the sentiment classification model includes: S1. Perform feature extraction on the text and image data in the data set to obtain feature vector representations of words in the text and feature vector representations of images respectively; S2. Based on the attention mechanism, the word feature vector is encoded to obtain the sentence feature vector; S3, based on the Transformer architecture, set the sentences and pictures in the text as the source modality and target modality for encoding respectively, and take the fusion vector output by the Transformer encoder for splicing as the guidance vector; S4, based on the attention mechanism and the guidance vector, the sentence feature vector is encoded to obtain the text feature vector; S5, concatenating the output vector of the Transformer encoder and the feature vector of the document to obtain the final vector representation, and performing sentiment classification and parameter adjustment of the sentiment classification model; In step S4, encoding the sentence feature vector comprises the following steps: S41, for the selected sentence feature vector s i , use bidirectional LSTM to analyze s from the front and back directions respectively i Encode and concatenate the hidden layer vectors obtained from the two directions to obtain the sentence hidden layer vector h i ; S42, use the tanh activation function to activate the guidance vector ε and the sentence hidden layer vector h i Use a nonlinear method to map to the same vector space, so as to obtain the feature vector representation f of the guidance vector generated after the image and text fusion, and the text feature vector representation g i ; S43, based on the attention mechanism, f and g i Perform inner product operation. In order to consider text factors more, add g separately. i , we get the vector u representing the sentence attention weight i , the calculation formula is: in i =U·(f⊙g i +g i ) Among them, U represents the parameter matrix obtained by random initialization, and ⊙ represents the inner product operation; S44, weight vector u i Perform normalization calculation, and then use the normalized weight γ i and the sentence hidden layer vector h i The weighted summation is used to obtain the text feature vector d; where h i represents the hidden layer vector of the sentence, γ i Represents the normalized weight of the ith sentence relative to the entire text.
2. The method according to claim 1, characterized in that In step S1, the Bert model is used to obtain the feature vector representation x of the word i，t , use VGG16 to obtain the feature vector representation P of the picture j .
3. The method according to claim 1, characterized in that The encoding process in step S2 includes the following steps: S21, for the selected word feature vector x i，t , use bidirectional LSTM to analyze x from both the forward and backward directions i,t Encode and concatenate the hidden layer vectors obtained in the two directions to obtain the final hidden layer vector h i，t ; S22, for the hidden layer vector h i，t , calculated using the attention mechanism and then normalized: v i，t =V·tanh(W v h i，t +b v ) Among them, V represents a randomly initialized matrix, v i，t represents the weight of the word for the entire sentence, tanh represents the activation function, exp represents the exponential function, W v and b v is a randomly initialized value that is automatically adjusted during training, α i，t Represents the weight of the normalized word for the entire sentence, s i Represents the obtained sentence feature vector.
4. The method according to claim 1, characterized in that In step S3, the encoding method of the Transformer encoder is: e s→p =f s→p (X s ) e p→s =f p→s (X p ) Among them, X s Indicates that the text is used as the source mode, X p represents the image as the source modality, ε s→p and ε p→s are the two output vectors of the encoder, f represents the activation function of the Transformer architecture, represents the concatenation operation and ε represents the guidance vector.
5. The method according to claim 1, characterized in that In step S5, the obtained guidance vector and text feature vector are first concatenated to obtain a final vector representation, and then the final vector representation is classified using a fully connected network, and the parameters of the sentiment classification model are adjusted through a back propagation algorithm.
6. A picture-text sentiment classification system based on heterogeneous fusion and symmetric translation, characterized in that: The system includes a sentiment classification model and a training module, wherein the training module is used to input pictures and texts in a data set into the sentiment classification model, and compare the sentiment classifications corresponding to the pictures and texts in the data set with the classifications output by the sentiment classification model, and adjust the parameters of the sentiment classification model through a back propagation algorithm; The sentiment classification model includes: a modal feature extraction module, which is used to extract features from the data of two modalities in the data set, and obtain feature vectors of words in the text and feature vectors of pictures respectively; Heterogeneous fusion attention module generates a sentence feature vector based on the attention mechanism combined with the influence of words on the sentence sentiment polarity, and generates a text feature vector based on the attention mechanism combined with the influence of sentences on the text sentiment polarity; The symmetric translation fusion module is used to set sentences and pictures as the source modality and target modality respectively, use Transformer to translate the source modality into the target modality, and take the output of the encoder as the vector representation after the fusion of the two modalities; The sentiment classification module concatenates the two output vectors of the Transformer encoder and the feature vector of the text to obtain the final vector representation, and outputs the sentiment classification through a fully connected network; In the heterogeneous fusion attention module, generating a feature vector of the text includes the following steps: S41, for the selected sentence feature vector s i , use bidirectional LSTM to analyze s from the front and back directions respectively i Encode and concatenate the hidden layer vectors obtained from the two directions to obtain the sentence hidden layer vector h i ; S42, use the tanh activation function to activate the guidance vector ε and the sentence hidden layer vector h i Use a nonlinear method to map to the same vector space, so as to obtain the feature vector representation f of the guidance vector generated after the image and text fusion, and the text feature vector representation g i ; S43, based on the attention mechanism, f and g i Perform inner product operation. In order to consider text factors more, add g separately. i , we get the vector u representing the sentence attention weight i , the calculation formula is: in i =U·(f⊙g i +g i ) Among them, U represents the parameter matrix obtained by random initialization, and ⊙ represents the inner product operation; S44, weight vector u i Perform normalization calculation, and then use the normalized weight γ i and the sentence hidden layer vector h i The weighted summation is used to obtain the text feature vector d; where h i represents the hidden layer vector of the sentence, γ i Represents the normalized weight of the ith sentence relative to the entire text.