Joint multi-modal cycle complementary attention aspect-level sentiment analysis method and device

Abstract

The application discloses a combined multi-modal cycle complementary attention aspect-level sentiment analysis method and device. The method comprises the following steps: tokenizing input text, performing text representation extraction to obtain text representation; extracting input image features, projecting the image features to the dimension of the matching text representation to obtain visual representation; performing highlighting processing to obtain text highlighted visual representation; performing a multi-modal aspect word extraction task MATE to predict aspect positions, thereby extracting aspect words; according to the aspect positions, truncating the text representation to obtain aspect text representation, and then performing a multi-modal aspect sentiment classification task MASC to obtain sentiment classification results of each aspect word. The device comprises five modules of text representation extraction, visual representation extraction, visual representation highlighting, aspect word extraction and aspect word sentiment classification. The application can reduce visual noise, complement modal information, and simultaneously pay attention to multi-modal aspect-level sentiment analysis of aspect word extraction and related aspect polarity classification.

Description

Technical Field

[0001] This invention belongs to the field of electronic information technology, specifically relating to a joint multimodal cyclic complementary attention aspect-level sentiment analysis method and device. Background Technology

[0002] With the widespread use of social media platforms like WeChat Moments and others, a large number of comments about specific events or products emerge daily. These comments often contain rich multimedia elements, such as images and text. In-depth analysis of the emotional information contained in these comments can quickly and accurately detect abnormal or negative sentiments, providing researchers with a powerful tool for timely feedback. Sentiment analysis plays a crucial role in this process. By analyzing the text and images in comments, a more comprehensive understanding of users' attitudes and emotional tendencies towards specific events or products can be achieved. Aspect-level sentiment analysis on WeChat Moments and other social media platforms further enhances the detailed analysis of the emotions encompassed by different attributes of the same product or event. This refined sentiment analysis provides a more reliable guarantee for detecting abnormal comments, helping to identify potential problems and take corresponding measures. By meticulously analyzing the sentiment tendency of each attribute, a more comprehensive understanding of users' attitudes towards specific aspects on social media can be achieved, allowing for more targeted handling of potential problems or challenges. This application of sentiment analysis not only helps monitor users' overall feelings about a product or event but also provides important information support for developing precise feedback and improvement strategies. Overall, sentiment analysis on social media provides businesses and organizations with deeper insights, enabling them to respond more flexibly to market changes, improve products and services, and enhance user satisfaction.

[0003] Multimodal aspect sentiment analysis (MMASA) is an emerging task in sentiment mining, aiming to extract aspect terms and their corresponding sentiment polarities from multimodal inputs. It comprises three subtasks: Multimodal Aspect Extraction (MATE), Multimodal Aspect Sentiment Classification (MASC), and end-to-end JMASA. For MATE, most existing methods typically construct it as a sequence labeling problem, proposing various deep learning approaches, including recurrent neural network-based, Transformer-based, and graph neural network-based methods. For MASC, since each sample usually provides aspect categories or aspect terms, existing methods primarily focus on modeling pairwise interactions between aspects, text, and images. Because MATE and MASC are highly interdependent, Ju et al. explored the end-to-end JMASA task to jointly extract aspect terms and their corresponding sentiments in an end-to-end manner. In MASC, for textual information, Ju et al. only considered aspect representations, limiting the model's understanding of relationships between aspects and making aspect analysis challenging without direct description. In multimodal fusion, Ju et al., Ling et al., and Yang et al. directly correlated the entire visual representation with the textual content. However, coarse overall image visualization intuitively introduces aspect-independent visual noise, potentially hindering sentiment analysis. Furthermore, the inconsistency in the amount and level of semantic information between images and text, as mentioned in previous research, also poses a challenge. The asymmetry in the amount of image and text information across various subtasks makes modality fusion susceptible to modality superposition. Moreover, MASC requires more detailed information compared to MATE. To address this challenge, Ju et al. and Zhou et al. introduced an image-text relationship detection module to determine the appropriate level of image information for a given task and directly fuse features. However, establishing such image-text relationships in the real world is both difficult and resource-intensive, and direct modality fusion may lead to information loss. Summary of the Invention

[0004] The purpose of this invention is to provide a method and apparatus for combined multimodal cyclic complementary attention aspect-level sentiment analysis, which can reduce visual noise and perform modal information complementarity, while focusing on aspect word extraction and related aspect polarity classification in multimodal aspect-level sentiment analysis.

[0005] This invention is achieved through the following technical solutions:

[0006] A combined multimodal, cyclic, complementary attention-based sentiment analysis approach includes the following steps:

[0007] Step 1: Tokenize the input text and extract text representations using the BERT pre-trained model to obtain the text representations.

[0008] Step 2: Use a pre-trained ResNet to extract features from the input image. To ensure that the dimensions of the image and text representations are consistent, project the image features onto the dimensions that match the text representation to obtain the visual representation.

[0009] Step 3: Use the extracted text representation to highlight the visual representation to obtain a highlighted visual representation of the text.

[0010] Step 4: Perform a multimodal aspect word extraction task (MATE) using text representation, visual representation, and visual representation of text highlighting. Through MATE, we can understand the key aspects in the text and images, predict the aspect positions, and extract aspect words. The optimization objective of this MATE is the sum of the predicted probability distribution vectors and the corresponding loss values ​​of the label vectors.

[0011] Step 5: Based on the aspect positions predicted by MATE, truncate the text representation to obtain the aspect text representation. Then, input the text representation, aspect text representation, and the visual representation of text highlighting into the multimodal aspect sentiment classification task MASC to obtain the sentiment classification results of each aspect word.

[0012] A further improvement of the present invention is that, in step 1, the wordpiece algorithm is used to tokenize the input text, and the BERT pre-trained model is selected as BERT-base.

[0013] A further improvement of the present invention is that, in step 2, a 152-layer ResNet is used to extract image features, and a multi-layer linear neural network is used to project the extraction results to match the text representation dimension.

[0014] A further improvement of the present invention is that, in step 3, cross-modal attention (CA) is used for text highlighting, with the text representation serving as the query and the visual representation serving as the key and value.

[0015] A further improvement of this invention is that the multimodal aspect word extraction method in step 4 employs a multimodal recurrent attention module (CIRA) to perform closed-loop attention fusion of text representation, visual representation, and visual representation of text highlighting, including the following steps:

[0016] S41, Multimodal Mutual Attention Fusion Stage. This stage is cascaded by CAs, consisting of three CAs: CAs H CA V CA T Corresponding to CA guided by the visual representation H, visual representation V, and text representation T of text highlighting, CA achieves the fusion of two modalities; for the first CA H The text-highlighted visual representation H serves as the query, and the text representation T serves as the key and value. The output is as follows:

[0017]

[0018] In the formula W Q W K , In the multimodal mutual attention fusion stage, the fusion order is CA. H (),CA V (),CA T ();

[0019] S42, the multimodal reverse mutual attention stage, consists of a cascaded multimodal recurrent attention module (CA). The order of this stage is the reverse of S41. Due to the different fusion order of each stage, the fusion between any two modalities is constrained by the other modality, ultimately yielding the CIRA result D. c ;

[0020] S43, by combining the input features with F c The connection is processed by a feedforward layer to produce F', which serves as the input to the aspect extraction algorithm. in Indicates dimensional connection;

[0021] S44, F' is mapped through a linear layer to form a non-normalized vector f representing the starting position of an aspect, and then used to generate the corresponding probability distribution p:

[0022]

[0023] p s =softmax(f s ),p e =softmax(f e ).

[0024] S45, During training, considering that each input sentence may contain multiple aspects, the start position label vector ys and the end position vector ye are derived from the dataset based on the aspect term sequence A = {a_1, a_2, ..., a_k}; the start position vector y_n^s of the N-dimensional vector indicates whether the nth position represents the start of an aspect term, while y_n^e indicates whether the nth position represents the start of an aspect term; the label vector ye indicates whether it represents the end of an aspect term; the optimization objective is expressed as the sum of the predicted probability distribution vectors and the corresponding loss values ​​of the label vectors:

[0025]

[0026] Where l is the length of the text term sequence, and p_n^s and p_n^e represent the n-dimensional representations of ps and pe, respectively;

[0027] S46, the aspect word extraction algorithm Table 1 is used to extract aspect words. First, the Top-M score values ​​are selected from f^s and f^e of each sample. The starting position that meets the conditions becomes the starting position of the candidate aspect item. The corresponding candidate aspect item score u_l and candidate aspect item position r_l are merged into their respective sets. γ is used as the score threshold.

[0028] S47, the r_l corresponding to the highest score will be iteratively removed from the set and added to the aspect term position set O; at the same time, the corresponding data will be removed from the candidate aspect term position set R; the aspect term position set R exceeds a predetermined value K or the candidate aspect term position set R is empty; The definition indicates the purpose of selecting the starting and ending positions of aspect items with higher non-standardized scores and shorter aspect item lengths; each time a new position is merged, the remaining candidate set and score set are checked for overlap, and if there is any overlap, it is deleted.

[0029] A further improvement of this invention is that, in step 5, the multimodal aspect word sentiment classification employs a multimodal complementary attention module (MCA), which truncates the text representation using the aspect position predicted by MATE to obtain the aspect text representation. Let T represent a d×n dimensional real space, where d represents the dimension of each word vector and n is the number of words. Then, the text representation T and aspect text representation T' are... a In addition to the visual representation of text highlighting (H) as input, the MASC sentiment classification task complements the text's modality, specifically including the following steps:

[0030] S51, using the aspect positions predicted by MATE, truncate the text representation to obtain the aspect text representation. n represents the number of aspects;

[0031] S52, In order to improve information density, firstly, in aspect text representation T a Cross-modal fusion is performed between the text highlight visual representation H and the text highlight visual representation H to obtain the fused highlight visual representation H. a Next, aspect text representation T a The text representation T and the fused highlighted visual representation H serve as the query. a They then serve as the keys and values ​​for cross-modal complementary fusion;

[0032] S53, merge the results and feed them into the feedforward layer to obtain sentiment features.

[0033] S54, based on emotional characteristics Emotional polarity f p The score can be obtained through a linear network defined as follows:

[0034]

[0035] S55, the resulting score is then normalized using the softmax function, and the polarity probability is obtained as follows:

[0036] p p =softmax(f p );

[0037] The optimization objective of the S56 MASC subtask is as follows:

[0038]

[0039] Where m is the number of aspect terms, ∈ represents the count of part-of-speech elements in the aspect term, and represents the morpheme count in the aspect term; sentiment tag The nth morpheme corresponding to the j-th aspect word, and It is an n-dimensional vector representing the probability distribution of sentiment polarity of aspect words; during inference, the sentiment polarity probability of each aspect word in set O within the target span is calculated; in p p The sentiment category with the highest median value will be selected as the sentiment category for the current aspect word.

[0040] A combined multimodal cyclic complementary attention-level sentiment analysis device includes:

[0041] The text representation extraction module is used to tokenize the input text and extract text representations using the BERT pre-trained model to obtain the text representations.

[0042] The visual representation extraction module is used to extract features from the input image using a pre-trained ResNet. To ensure that the dimensions of the image and text representations are consistent, the image features are projected onto the matching dimensions of the text representation to obtain the visual representation.

[0043] The visual representation highlighting module is used to highlight the visual representation using the extracted text representation, resulting in a highlighted visual representation of the text.

[0044] The aspect word extraction module is used to perform multimodal aspect word extraction (MATE) using text representation, visual representation, and visual representation of text highlighting. Through MATE, key aspects in text and images are understood, aspect positions are predicted, and aspect words are extracted. The optimization objective of this MATE is the sum of the predicted probability distribution vectors and the corresponding loss values ​​of the label vectors.

[0045] The aspect-word sentiment classification module is used to truncate the text representation based on the aspect position predicted by MATE to obtain the aspect text representation. Then, the text representation, aspect text representation and the visual representation of text highlighting are input into the multimodal aspect sentiment classification task MASC to obtain the sentiment classification results of each aspect word.

[0046] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the joint multimodal cyclic complementary attention aspect-level sentiment analysis method.

[0047] The present invention has at least the following beneficial technical effects:

[0048] The joint multimodal cyclic complementary attention aspect-level sentiment analysis method and apparatus provided by this invention extracts features from both visual and text modalities, and uses text for visual highlighting to reduce the influence of visual noise. Simultaneously, visual, text, and highlighted visual features are input into the Model-Aspect-Oriented (MASC) module for cyclic attention aspect word extraction. This allows the model to notice coarse-grained detailed features for better aspect word extraction. The obtained aspect word features, highlighted visual features, and global text representation are then input into the MASC module, enabling the model to understand the relationships between aspect words and strengthening its sentiment discrimination ability for aspects without directly descriptive aspect words. Finally, the final classification result is obtained. It has the following advantages:

[0049] First, it is a joint multimodal aspect-level sentiment analysis framework that takes into account both aspect word extraction and aspect sentiment classification.

[0050] Second: Visual highlighting of text effectively reduces the impact of visual noise;

[0051] Third: Global text was introduced into the MASC task, which effectively recognized the relationships between various aspects, thereby strengthening the model's ability to discriminate sentiments without words that directly describe aspects;

[0052] Fourth: CIRA and MCA are introduced for modal fusion to focus on features with different levels of detail, which is different from previous relationship detection methods and improves practicality. Attached Figure Description

[0053] Figure 1 This is a flowchart of the method of the present invention.

[0054] After extracting multimodal information using BERT-base and ResNet152, the text is visually highlighted. Then, the three features are input into the MATE task, and the resulting aspect features are combined with the global text and the text highlight visual representation to perform aspect-level sentiment classification.

[0055] Figure 2This is a comparison chart of the output of a specific instance with the output of other methods.

[0056] Figure 3 This is a structural block diagram of the device of the present invention. Detailed Implementation

[0057] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0058] like Figure 1 As shown, the joint multimodal cyclic complementary attention aspect-level sentiment analysis method provided by this invention includes the following steps:

[0059] Step 1: Tokenize the input text and extract text representations using a BERT pre-trained model, where Tok = {tok1, ..., tok} N} represents the input, T = {t1, ...,t} N} represents the generated text representation;

[0060] Step 2: Extract input image features using a pre-trained ResNet. To ensure consistency in dimensionality between the image and text representations, project the image features onto the matching text representation dimensions to obtain the visual representation V, denoted as...

[0061] Step 3: Highlight the visual representation V using the extracted text representation T to obtain the highlighted visual features of the text.

[0062] Step 4: Perform the multimodal aspect word extraction task MATE using text representation T, visual representation V, and visual representation H of text highlighting. Through MATE, we can understand the key aspects in the text and image, predict the aspect positions, and extract aspect words. The optimization objective of this MATE is the sum of the predicted probability distribution vectors and the corresponding loss values ​​of the label vectors.

[0063] Step 5: Based on the aspect positions predicted by MATE, truncate the text representation to obtain the aspect text representation. Subsequently, the text representation T and the aspect text representation T are... aThe visual representation H of the highlighted text is used as input for the multimodal sentiment classification task MASC, and the sentiment classification results of words in each aspect are obtained.

[0064] In step 1, the wordpiece algorithm is used for text tokenization, and the BERT pre-trained model is BERT-base.

[0065] In step 2, a 152-layer ResNet is used to extract image features, and a multi-layer linear neural network is used to project the extraction results to match the text representation dimension.

[0066] In step 3, cross-modal attention (CA) is used to highlight the text, making the text representation a query and the visual representation a key and value.

[0067] The multimodal aspect word extraction method in step 4 employs a multimodal recurrent attention module (CIRA) to perform closed-loop attention fusion of text representation, visual representation, and visual representation of text highlighting, including the following steps:

[0068] S41, Multimodal Mutual Attention Fusion Stage. This stage is cascaded by CAs, consisting of three CAs: CAs H CA V CA T This corresponds to CA, guided by the visual representation H (highlighted by text), visual representation V, and text representation T. CA achieves the fusion of information from two modalities, for example, in the visual-text cross-modal attention fusion network CA. T The text representation and the output features of the previous network are used as input to generate the main text fusion features. (Using the first CA...) H For example, its visual representation H of text highlighting is the query, and its text representation T is the key and value. The output is as follows:

[0069]

[0070] In the formula W Q W K , In the multimodal mutual attention fusion stage, the fusion order is CA. H (),CA V (),CA T ();

[0071] S42, the multimodal reverse mutual attention stage, consists of a cascaded multimodal recurrent attention module (CA). The order of this stage is the reverse of S41. Due to the different fusion order of each stage, the fusion between any two modalities is constrained by the other modality, ultimately yielding the CIRA result F. c ;

[0072] S43, by combining the input features with Fc The connection is processed by a feedforward layer to produce F', which serves as the input to the aspect extraction algorithm. in Indicates dimensional connection;

[0073] S44, F' is mapped through a linear layer to form a non-normalized vector f representing the starting position of an aspect, and then used to generate the corresponding probability distribution p:

[0074]

[0075] p s =softmax(f s ),p e =softmax(f e ).

[0076] S45, During training, considering that each input sentence may contain multiple aspects, the start position label vector ys and the end position vector ye are derived from the dataset based on the aspect term sequence A = {a_1, a_2, ..., a_k}. The start position vector y_n^s of the N-dimensional vector indicates whether the nth position represents the start of an aspect term, while y_n^e indicates whether the nth position represents the start of an aspect term. The label vector ye indicates whether it represents the end of an aspect term. The optimization objective is expressed as the sum of the predicted probability distribution vectors and the corresponding loss values ​​of the label vectors:

[0077]

[0078] Where l is the length of the text term sequence, and p_n^s and p_n^e represent the n-dimensional representations of ps and pe, respectively.

[0079] S46, the aspect term extraction and inference algorithm (Table 1) is used for aspect term extraction. First, the Top-M scores are selected from the f^s and f^e of each sample. The starting positions that meet the conditions become the starting positions of candidate aspect terms. The corresponding candidate aspect term scores u_l and candidate aspect term positions r_l are merged into their respective sets. γ is used as the score threshold.

[0080] S47, the highest score r_l is iteratively removed from the set and added to the aspect term position set O. Simultaneously, the corresponding data is removed from the candidate aspect term position set R. The aspect term position set R exceeds a predetermined value K or the candidate aspect term position set R is empty. The definition indicates the purpose of selecting the start and end positions of aspect terms with higher unstandardized scores and shorter aspect term lengths. Each time a new position is merged, the remaining candidate set and score set are checked for overlap. If there is overlap, it is removed to minimize redundancy in the predicted aspect terms.

[0081] In step 5, the multimodal aspect-based sentiment classification method is an independently designed multimodal complementary attention module (MCA). The aspect position predicted by MATE is used to truncate the text representation, resulting in the aspect text representation. Subsequently, the representation T and the aspect text representation T will be... a In addition to the visual representation of text highlighting (H) as input, the MASC sentiment classification task complements the modalities, including the following steps:

[0082] S51, using the aspect positions predicted by MATE, truncate the text representation to obtain the aspect text representation. n represents the number of aspects;

[0083] S52, In order to improve information density, firstly, in aspect text representation T a Cross-modal fusion is performed between the text highlight visual representation H and the text highlight visual representation H to obtain the fused highlight visual representation H. a Next, the aspect text representation T a The text representation T and the fused highlighted visual representation H serve as the query. a They then serve as the keys and values ​​for cross-modal complementary fusion;

[0084] S53, merge the results and feed them into the feedforward layer to obtain sentiment features.

[0085] S54, based on emotional characteristics Sentiment polarity scores are obtained through a linear network defined as follows:

[0086]

[0087] S55, the resulting score is then normalized using the softmax function, and the polarity probability is obtained as follows:

[0088] p p =softmax(f p );

[0089] The optimization objective of the S56 MASC subtask is as follows:

[0090]

[0091] Where m is the number of aspect terms, ∈ represents the count of part-of-speech elements in the aspect term. represents the morpheme count in the aspect term. Sentiment tag The nth morpheme corresponding to the j-th aspect word, and It is an n-dimensional vector representing the probability distribution of sentiment polarity for aspect words. During inference, the sentiment polarity probability of each aspect word in set O within the target span is calculated. In pp The sentiment category with the highest median value will be selected as the sentiment category for the current aspect word.

[0092] Example

[0093] like Figure 1 and Figure 2 As shown, the joint multimodal cyclic complementary attention aspect-level sentiment analysis method provided by this invention includes the following steps:

[0094] Step 1: Using the Twitter-15 and Twitter-17 datasets as specific test objects, tokenize the input text and extract text representations using a BERT pre-trained model, letting Tok = {tok1, ..., tok}. N} represents the input, T = {t1, ...,t} N} represents the generated text representation;

[0095] Step 2: Extract features from the input image using a pre-trained ResNet, represented as... Where 49 corresponds to the image region, and 2048 is the feature dimension. To ensure consistency in dimensions between the image and text representations, the image features are projected onto the matching text representation dimensions, represented as follows:

[0096] Step 3: Highlight the visual representation V using the extracted text representation T to obtain the highlighted visual features of the text.

[0097] Step 4: Perform the multimodal aspect word extraction task MATE using text representation T, visual representation V, and visual representation H of text highlighting. Through MATE, we can understand the key aspects in the text and image, predict the aspect positions, and extract aspect words. The optimization objective of this network is the sum of the predicted probability distribution vectors and the corresponding loss values ​​of the label vectors.

[0098] Step 5: Based on the aspect positions predicted by MATE, truncate the text representation to obtain the aspect text representation. Subsequently, the text representation T and the aspect text representation T are... a The visual representation H of the highlighted text is input into the multimodal sentiment classification task MASC to obtain the final sentiment classification results of words in each aspect.

[0099] Step 6: Test all instances of the Twitter-15 and Twitter-17 test sets, and finally compare them with the labels. The specific test results are shown in the table below. JCC is the invention of this invention, and it can be seen that its results are significantly improved compared with previous algorithms.

[0100] Table 1 Comparative experimental results of the joint aspect-level sentiment analysis algorithm on the Twitter-15 and Twitter-17 datasets.

[0101]

[0102] like Figure 3 As shown, the joint multimodal cyclic complementary attention aspect-level sentiment analysis device provided by the present invention includes:

[0103] The text representation extraction module is used to tokenize the input text and extract text representations using the BERT pre-trained model to obtain the text representations.

[0104] The visual representation extraction module is used to extract features from the input image using a pre-trained ResNet. To ensure that the dimensions of the image and text representations are consistent, the image features are projected onto the matching dimensions of the text representation to obtain the visual representation.

[0105] The visual representation highlighting module is used to highlight the visual representation using the extracted text representation, resulting in a highlighted visual representation of the text.

[0106] The aspect word extraction module is used to perform multimodal aspect word extraction (MATE) using text representation, visual representation, and visual representation of text highlighting. Through MATE, key aspects in text and images are understood, aspect positions are predicted, and aspect words are extracted. The optimization objective of this MATE is the sum of the predicted probability distribution vectors and the corresponding loss values ​​of the label vectors.

[0107] The aspect-word sentiment classification module is used to truncate the text representation based on the aspect position predicted by MATE to obtain the aspect text representation. Then, the text representation, aspect text representation and the visual representation of text highlighting are input into the multimodal aspect sentiment classification task MASC to obtain the sentiment classification results of each aspect word.

[0108] The present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the joint multimodal cyclic complementary attention aspect-level sentiment analysis method.

[0109] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0110] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0111] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0112] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0113] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A combined multimodal cyclic complementary attention-based sentiment analysis method, characterized in that, Includes the following steps: Step 1: Tokenize the input text and extract text representations using the BERT pre-trained model to obtain the text representations. Step 2: Use a pre-trained ResNet to extract features from the input image. To ensure that the dimensions of the image and text representations are consistent, project the image features onto the dimensions that match the text representation to obtain the visual representation. Step 3: Use the extracted text representation to highlight the visual representation to obtain a highlighted visual representation of the text. Step 4: Perform a multimodal aspect word extraction task (MATE) using text representation, visual representation, and visual representation of text highlighting. MATE identifies key aspects in the text and image, predicts aspect locations, and extracts aspect words. The optimization objective of MATE is the sum of the predicted probability distribution vectors and the corresponding loss values ​​of the label vectors. The multimodal aspect word extraction method employs a multimodal recurrent attention module (CIRA) to perform closed-loop attention fusion of text representation, visual representation, and visual representation of text highlighting, including the following steps: S41, Multimodal Mutual Attention Fusion Stage, this stage is cascaded by CAs, consisting of three CAs: Corresponding to CA guided by the visual representation H, visual representation V, and text representation T of text highlighting, CA achieves the fusion of two modalities; for the first The text-highlighted visual representation H serves as the query, and the text representation T serves as the key and value. The output is as follows: In the formula In the multimodal mutual attention fusion stage, the fusion order is as follows: ; S42, the multimodal reverse mutual attention stage, consists of a multimodal recurrent attention module composed of cascaded CAs. The order of this stage is the reverse of S41. Due to the different fusion order of each stage, the fusion between any two modalities is constrained by the other modality, ultimately yielding the CIRA result. ; S43, by combining input features with The connection is processed by the feedforward layer to produce F', which serves as the input to the aspect extraction algorithm. F' = FFN(Fc⊕T), where ⊕ represents the dimension connection. S44, F' is mapped through a linear layer to form a non-normalized vector f representing the starting position of an aspect, and then used to generate the corresponding probability distribution p: S45, During training, considering that each input sentence may contain multiple aspects, the start position label vector ys and the end position vector ye are derived from the dataset based on the aspect term sequence A={a_1,a_2,…a_k}; the start position vector of the N-dimensional vector... This indicates whether the nth position represents the beginning of an aspect term, and The nth position indicates whether it represents the beginning of an aspect term; the label vector ye indicates whether it represents the end of an aspect term; the optimization objective is expressed as the sum of the predicted probability distribution vectors and the corresponding loss values ​​of the label vectors: Where l is the length of the text term sequence, and Let ps and pe represent the n-dimensional dimensions respectively; S46, the aspect term extraction algorithm (Table 1) is used for aspect term extraction. First, from each sample... and The Top-M scores are selected, and the starting positions that meet the conditions become the starting positions of the candidate aspects; the corresponding candidate aspect scores u_l and candidate aspect positions r_l are merged into their respective sets; γ is used as the score threshold. S47, the r_l corresponding to the highest score will be iteratively removed from the set and added to the aspect term position set O; at the same time, the corresponding data will be removed from the candidate aspect term position set R; the aspect term position set R exceeds a predetermined value K or the candidate aspect term position set R is empty; The definition indicates the purpose of selecting the starting and ending positions of aspect items with higher non-standardized scores and shorter aspect item lengths; each time a new position is merged, the remaining candidate set and score set are checked for overlap, and if there is any overlap, it is deleted; Step 5: Based on the aspect positions predicted by MATE, truncate the text representation to obtain aspect text representations. Then, input the text representations, aspect text representations, and highlighted visual representations into the multimodal aspect sentiment classification task MASC to obtain the sentiment classification results for each aspect word. Multimodal aspect word sentiment classification uses the Multimodal Complementary Attention Module (MCA), which truncates the text representation based on the aspect positions predicted by MATE to obtain aspect text representations. Let represent a d×n dimensional real space, where d represents the dimension of each word vector and n is the number of words. The text representation will then be used. The input to the multimodal sentiment classification task MASC complements the modalities, specifically including the following steps: S51, using the aspect positions predicted by MATE, truncate the text representation to obtain the aspect text representation. ; S52, In order to improve information density, firstly in aspect text representation. and Cross-modal fusion is performed between the two modes to obtain the fused high-brightness visual representation. Next, aspect text representation The text representation T serves as the query, while the fused highlighted visual representation... They then serve as the keys and values ​​for cross-modal complementary fusion; S53, merge the results and feed them into the feedforward layer to obtain sentiment features. ; S54, based on emotional characteristics Emotional polarity The score can be obtained through a linear network defined as follows: S55, the resulting score is then normalized using the softmax function, and the polarity probability is obtained as follows: The optimization objective for the S56 MASC subtask is as follows: Where m is the number of aspect terms, The count of parts of speech in the representative terms, and the count of morphemes in the representative terms; sentiment tags. The nth morpheme corresponding to the j-th aspect word, and It is an n-dimensional vector representing the probability distribution of sentiment polarity of aspect words; during the reasoning process, the sentiment polarity probability of each aspect word in set O within the target span is calculated; The sentiment category with the highest median value will be selected as the sentiment category for the current aspect word.

2. The joint multimodal cyclic complementary attention aspect-level sentiment analysis method according to claim 1, characterized in that, In step 1, the wordpiece algorithm is used to tokenize the input text, and the BERT pre-trained model is BERT-base.

3. The joint multimodal cyclic complementary attention aspect-level sentiment analysis method according to claim 1, characterized in that, In step 2, a 152-layer ResNet is used to extract image features, and a multi-layer linear neural network is used to project the extraction results to match the text representation dimension.

4. The joint multimodal cyclic complementary attention aspect-level sentiment analysis method according to claim 1, characterized in that, In step 3, cross-modal attention (CA) is used for text highlighting, with text representation serving as the query and visual representation serving as the key and value.

5. A combined multimodal cyclic complementary attention-level emotion analysis device, characterized in that, The device is based on the joint multimodal cyclic complementary attention aspect-level sentiment analysis method described in claim 1, comprising: The text representation extraction module is used to tokenize the input text and extract text representations using the BERT pre-trained model to obtain the text representations. The visual representation extraction module is used to extract features from the input image using a pre-trained ResNet. To ensure that the dimensions of the image and text representations are consistent, the image features are projected onto the matching dimensions of the text representation to obtain the visual representation. The visual representation highlighting module is used to highlight the visual representation using the extracted text representation, resulting in a highlighted visual representation of the text. The aspect word extraction module is used to perform multimodal aspect word extraction (MATE) using text representation, visual representation, and visual representation of text highlighting. Through MATE, key aspects in text and images are understood, aspect positions are predicted, and aspect words are extracted. The optimization objective of this MATE is the sum of the predicted probability distribution vectors and the corresponding loss values ​​of the label vectors. The aspect-word sentiment classification module is used to truncate the text representation based on the aspect position predicted by MATE to obtain the aspect text representation. Then, the text representation, aspect text representation and the visual representation of text highlighting are input into the multimodal aspect sentiment classification task MASC to obtain the sentiment classification results of each aspect word.

6. The joint multimodal cyclic complementary attention aspect-level sentiment analysis device according to claim 5, characterized in that, The text representation extraction module uses the wordpiece algorithm to tokenize the input text, and the BERT pre-trained model is BERT-base.

7. The joint multimodal cyclic complementary attention aspect-level sentiment analysis device according to claim 5, characterized in that, The visual representation extraction module uses a 152-layer ResNet to extract image features and uses a multi-layer linear neural network to project the extraction results to match the text representation dimension.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the joint multimodal cyclic complementary attention aspect-level sentiment analysis method as described in any one of claims 1-4.

Images