Aspect-level multi-modal sentiment analysis method and system based on cross-modal inference fusion

Abstract

The application provides an aspect-level multi-modal sentiment analysis method and system based on cross-modal reasoning fusion, and belongs to the technical field of emotion recognition. The method comprises the following steps: obtaining multi-modal data of a target platform and performing aspect-level sentiment category labeling to obtain a labeled data set; an aspect-level multi-modal sentiment analysis model is established, which comprises a feature extraction module, a reasoning chain feature construction module, a cross-modal reasoning fusion module and a classification output module; the aspect-level multi-modal sentiment analysis model is trained through the labeled data set, and aspect-level sentiment category prediction is performed on multi-modal data to be analyzed through the trained aspect-level multi-modal sentiment analysis model. The application enhances the context consistency in the reasoning process by introducing multi-source information, effectively alleviates the semantic drift problem through cross-modal reasoning fusion, realizes the collaborative modeling of sentiment semantics and logical reasoning, and improves the accuracy of aspect-level sentiment classification.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition technology, and in particular to an aspect-level multimodal emotion analysis method and system based on cross-modal reasoning fusion. Background Technology

[0002] With the explosive growth of the internet and social media, user-generated content has become the primary carrier of information. This content typically exists in multiple forms, including text, images, and videos—that is, multimodal data. Automatically analyzing and understanding users' emotions, opinions, and attitudes from this massive amount of multimodal data is of crucial value for applications such as public opinion monitoring, market analysis, and intelligent recommendation, thus giving rise to sentiment analysis technology.

[0003] Early sentiment analysis primarily focused on unimodal text. However, in social media contexts, images often carry richer and more intuitive emotional information than text, sometimes even complementing, emphasizing, or satirizing it. Therefore, multimodal sentiment analysis has become a research hotspot in recent years. This technology aims to make more accurate sentiment judgments than single-modal analysis by integrating information from multiple modalities, including text and images.

[0004] To obtain more granular analysis results, aspect-level sentiment analysis has been proposed. Unlike simply judging the general sentiment of an entire text or image, aspect-level sentiment analysis aims to identify specific "aspects" (also called attributes or entities) mentioned in the text and determine the sentiment polarity (e.g., positive, negative, neutral) for that aspect. For example, for a comment like "This phone's screen is amazing, but the battery life is terrible," aspect-level sentiment analysis can identify that the sentiment for the "screen" aspect is positive, while the sentiment for the "battery" aspect is negative. Combining multimodal and aspect-level approaches forms aspect-level multimodal sentiment analysis, which enables sentiment analysis targeting specific aspect terms. Its mainstream technical approach typically follows a "feature extraction-feature fusion" paradigm.

[0005] However, existing aspect-level multimodal sentiment analysis methods rely on pre-trained language models to extract text semantics, and their ability to model the underlying logical relationships in the text remains insufficient. Therefore, there is an urgent need in this field for a new technical solution that introduces stronger reasoning capabilities to model the complex logical relationships between modalities, thereby achieving more accurate and robust aspect-level multimodal sentiment analysis. Summary of the Invention

[0006] This invention provides an aspect-level multimodal sentiment analysis method based on cross-modal reasoning fusion, which addresses the shortcomings of existing aspect-level multimodal sentiment analysis methods in modeling potential logical relationships in text, and achieves more accurate and robust aspect-level multimodal sentiment analysis.

[0007] In a first aspect, the present invention provides an aspect-level multimodal sentiment analysis method based on cross-modal reasoning fusion, comprising:

[0008] Acquire multimodal data from the target platform and perform aspect-level sentiment category annotation to obtain an annotated dataset;

[0009] An aspect-level multimodal sentiment analysis model is established, which includes a feature extraction module, an inference chain feature construction module, a cross-modal inference fusion module, and a classification output module.

[0010] The feature extraction module is used to extract text features, aspect entity features, and image features from the input multimodal data, respectively; the inference chain feature construction module is used to construct thought chain prompts based on the multimodal data and construct inference chain features in conjunction with a large language model; the cross-modal inference fusion module is used to fuse the text features, aspect entity features, and inference chain features to obtain cross-modal fused features; and the classification output module is used to predict aspect-level sentiment categories based on the cross-modal fused features.

[0011] Sentiment inference trains the aspect-level multimodal sentiment analysis model using the labeled dataset, and then uses the trained aspect-level multimodal sentiment analysis model to predict the aspect-level sentiment category of the multimodal data to be analyzed.

[0012] According to the present invention, an aspect-level multimodal sentiment analysis method based on cross-modal reasoning fusion is provided, wherein the multimodal data includes text data and image data of image-text matching;

[0013] The aspect-level sentiment category annotation includes annotating aspect words in the text data, where aspect words are aspect entities in the text data.

[0014] According to the present invention, an aspect-level multimodal sentiment analysis method based on cross-modal reasoning fusion is provided, wherein the feature extraction module is used for:

[0015] The text data is preprocessed, and the text features corresponding to the text data are extracted through a first pre-trained language model;

[0016] The aspect words are preprocessed, and the aspect entity features corresponding to the aspect words are extracted through the second pre-trained language model;

[0017] The image data is divided into multiple image blocks, and each of the multiple image blocks is input into a pre-trained visual model to obtain the image features corresponding to the image data.

[0018] According to the present invention, an aspect-level multimodal sentiment analysis method based on cross-modal reasoning fusion is provided, wherein the reasoning chain feature construction module is used for:

[0019] Based on the text data, a thought chain prompt is constructed using a structural template that includes preparatory knowledge, questions, answers, and reasons; wherein, the preparatory knowledge includes basic or prior knowledge related to aspect-level sentiment category prediction;

[0020] The thought chain prompts are input into a large language model multiple times for sentiment inference to obtain multiple candidate inference chains;

[0021] The optimal inference chain is selected from the multiple candidate inference chains using a self-consistent strategy;

[0022] The optimal inference chain is input into a third pre-trained language model for feature encoding to obtain the inference chain features.

[0023] According to the present invention, an aspect-level multimodal sentiment analysis method based on cross-modal reasoning fusion is provided, wherein the cross-modal reasoning fusion module is used for:

[0024] Based on an attention mechanism, feature fusion is performed on the entity features and image features of the aforementioned aspects to obtain the first fused feature;

[0025] The first fused feature is input into the Transformer model for feature enhancement to obtain the enhanced first fused feature.

[0026] Based on an attention mechanism, the enhanced first fused feature and the text feature are fused to obtain the second fused feature;

[0027] Based on the multi-head attention mechanism, the second fusion feature and the inference chain feature are fused bidirectionally to obtain cross-modal fusion features.

[0028] According to the present invention, an aspect-level multimodal sentiment analysis method based on cross-modal reasoning fusion is provided, wherein the method performs bidirectional feature fusion of the second fused feature and the reasoning chain feature based on a multi-head attention mechanism to obtain the cross-modal fused feature, including:

[0029] Using the second fusion feature as the query and the inference chain feature as the key and value, attention calculation and feature fusion are performed on the second fusion feature and the inference chain feature to obtain the third fusion feature;

[0030] Using the third fusion feature as the query and the second fusion feature as the key and value, attention calculation and feature fusion are performed on the third fusion feature and the second fusion feature to obtain cross-modal fusion features.

[0031] According to the present invention, an aspect-level multimodal sentiment analysis method based on cross-modal reasoning fusion is provided, wherein the classification output module is used for:

[0032] The zero-sample sentiment prediction results obtained based on the optimal inference chain are fused with the cross-modal fusion features to obtain the final fused features;

[0033] The final fused features are input into a classifier to predict the probability of sentiment classification, resulting in aspect-level sentiment category prediction.

[0034] Secondly, the present invention also provides an aspect-level multimodal sentiment analysis system based on cross-modal reasoning fusion, comprising:

[0035] The data annotation module is used to acquire multimodal data from the target platform and perform aspect-level sentiment category annotation to obtain an annotated dataset;

[0036] The model building module is used to build an aspect-level multimodal sentiment analysis model. This model includes a feature extraction module, an inference chain feature construction module, a cross-modal inference fusion module, and a sentiment classification module. Specifically, the feature extraction module extracts text features, aspect entity features, and image features from the input multimodal data. The inference chain feature construction module constructs thought chain prompts based on the multimodal data and combines them with a large language model to build inference chain features. The cross-modal inference fusion module fuses the text features, aspect entity features, and inference chain features to obtain cross-modal fused features. The classification output module predicts aspect-level sentiment categories based on the cross-modal fusion features.

[0037] The sentiment reasoning and sentiment classification module is used to train the aspect-level multimodal sentiment analysis model using the labeled dataset, and to predict the aspect-level sentiment category of the multimodal data to be analyzed using the trained aspect-level multimodal sentiment analysis model.

[0038] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the aspect-level multimodal sentiment analysis method based on cross-modal reasoning fusion as described above.

[0039] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aspect-level multimodal sentiment analysis method based on cross-modal reasoning fusion as described above.

[0040] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aspect-level multimodal sentiment analysis method based on cross-modal reasoning fusion as described above.

[0041] The beneficial effects of the technical solutions provided by some embodiments of the present invention include at least the following:

[0042] 1) This invention provides an aspect-level multimodal sentiment analysis method and system based on cross-modal reasoning fusion. By constructing an aspect-level multimodal sentiment analysis model, it achieves accurate sentiment polarity prediction for specific aspects. This aspect-level multimodal sentiment analysis model extracts features from different modalities through a feature extraction module, constructs thought chain prompts through a reasoning chain feature construction module, performs sentiment reasoning using a large language model, constructs reasoning chain features, and fuses features from different modalities and reasoning chain features through a cross-modal reasoning fusion module. By introducing multi-source information, it enhances contextual consistency in the reasoning process, effectively alleviates semantic shift problems, and achieves collaborative modeling of sentiment semantics and logical reasoning, thereby improving the accuracy of aspect-level sentiment classification and realizing efficient sentiment analysis of aspect-level multimodal content.

[0043] 2) The feature extraction module of this invention uses a pre-trained language model and a pre-trained visual model to extract text features, aspect entity features and image features of multimodal data, which effectively avoids data source limitations and category bias. This makes the aspect-level multimodal sentiment analysis model more applicable to the Chinese context and improves the application and generalization ability of the aspect-level multimodal sentiment analysis model in Chinese and other contexts.

[0044] 3) The reasoning chain feature construction module of the present invention uses a structural template of preparatory knowledge, questions, answers and reasons to construct a thinking chain prompt, which can maximize the reasoning ability of the large language model to perform multiple sentiment reasoning, and select the optimal reasoning chain through a self-consistent strategy. This optimal reasoning chain can effectively model the deep logical relationship within the text, providing strong logical support for aspect-level multimodal sentiment analysis.

[0045] 4) The cross-modal reasoning fusion module of the present invention performs cross-modal feature interaction and reasoning fusion based on the attention mechanism. It can dynamically filter cross-modal information that is not related to a specific aspect, accurately associate visual information with specific aspects in text data, effectively model the complex logical relationship between different modalities, and optimize the potential logical relationship in text data by combining the optimal reasoning chain, so as to achieve more accurate and robust aspect-level multimodal sentiment analysis.

[0046] 5) This invention integrates multimodal features obtained by fusing aspect entity features, image features, and text features with inference chain features in a two-way manner. On the one hand, it uses multimodal features to correct inference chain features, and on the other hand, it uses inference chain features to refocus multimodal features, thereby enhancing the consistency and interpretability of cross-modal reasoning.

[0047] 6) The classification output module of the present invention first fuses the zero-sample sentiment prediction results obtained based on text data with the cross-modal fusion features, and then performs classification output. By combining the single-modal prediction results with the fused multimodal features, the accuracy of sentiment polarity prediction for specific aspects is further improved. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0049] Figure 1 This is one of the flowcharts illustrating the aspect-level multimodal sentiment analysis method based on cross-modal reasoning fusion provided in an embodiment of the present invention.

[0050] Figure 2 A schematic diagram of data annotation provided in an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the structure of an aspect-level multimodal sentiment analysis model provided in an embodiment of the present invention;

[0052] Figure 4 The second flowchart illustrates the aspect-level multimodal sentiment analysis method based on cross-modal reasoning fusion provided in this embodiment of the invention.

[0053] Figure 5 An example diagram illustrating a thought chain prompt based on a structure of preliminary knowledge, questions, answers, and reasons, provided in an embodiment of the present invention.

[0054] Figure 6 A schematic diagram illustrating the principle of cross-modal feature fusion provided in an embodiment of the present invention;

[0055] Figure 7 This is a schematic diagram of the structure of the multi-head attention mechanism provided in an embodiment of the present invention;

[0056] Figure 8 This is an application diagram of aspect-level sentiment category prediction provided in an embodiment of the present invention;

[0057] Figure 9A schematic diagram of the structure of an aspect-level multimodal sentiment analysis system based on cross-modal reasoning fusion provided in an embodiment of the present invention;

[0058] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0060] Existing technologies still suffer from one or more of the following significant drawbacks when handling complex aspect-level multimodal sentiment analysis tasks:

[0061] 1. Data source limitations and biases: Most existing public datasets are in English and generally suffer from class imbalance, which limits the application and generalization ability of the model in other contexts such as Chinese, and leads to poor performance on minority classes.

[0062] 2. Insufficient modeling of underlying logical relationships: Although pre-trained language models can effectively extract the surface semantics of text, their ability to model the deep logical relationships within the text (such as transitions, causality, irony, etc.) remains limited.

[0063] 3. The challenge of filtering irrelevant information: Existing models struggle to accurately associate and filter visual information with specific textual "aspects." For example, irrelevant elements such as background in an image, acting as visual noise, are often incorrectly incorporated into the feature representation, interfering with the sentiment judgment of specific aspects.

[0064] Therefore, this invention proposes an aspect-level multimodal sentiment analysis method and system based on cross-modal reasoning fusion. This method can go beyond simple feature alignment and addition, introduce stronger reasoning capabilities to model complex logical relationships between modalities, and dynamically filter cross-modal information that is irrelevant to a specific aspect, thereby achieving more accurate and robust aspect-level multimodal sentiment analysis.

[0065] First, the technical terms of this invention will be explained and described:

[0066] BERT: short for Bidirectional Encoder Representations from Transformers, is a pre-trained language model that uses bidirectional encoder representations to extract contextual features and semantic representations of text.

[0067] ViT: short for Vison Transformer, is a pre-trained vision model based on the Transformer architecture, used to divide an input image into multiple image patches and extract fine-grained image features.

[0068] LLM stands for Large Language Model, which refers to a large language model used for natural language processing tasks such as zero-shot prediction, inference chain generation, and text understanding.

[0069] CoT stands for Chain-of-Thought, which refers to a chain-of-thought method that transforms tasks into step-by-step reasoning structures, enabling large language models to generate reasoning steps and produce interpretable results.

[0070] LIR stands for Locally ided Image Representation layer. It is used to extract local image features that are relevant to specific aspects, thus achieving local focus on image features.

[0071] GIR stands for Globally guided Image Representation layer. It is a deep learning layer that uses global contextual information to enhance local feature representations and is used for computer vision tasks.

[0072] Self-Consistency Strategy: Self-consistency strategy is a method for optimizing reasoning results in thought chains. Its basic principle is to sample the same input multiple times to generate different reasoning chains, and then statistically compare the outputs of these chains to select the one with the highest frequency or the most semantically reasonable result as the final prediction.

[0073] Example 1

[0074] Please see Figure 1 , Figure 1 One of the flowcharts for an aspect-level multimodal sentiment analysis method based on cross-modal reasoning fusion, provided as an embodiment of the present invention, includes:

[0075] S101. Obtain multimodal data from the target platform and perform aspect-level sentiment category annotation to obtain an annotated dataset;

[0076] S102. Establish an aspect-level multimodal sentiment analysis model, which includes a feature extraction module, an inference chain feature construction module, a cross-modal inference fusion module, and a classification output module.

[0077] S103. Train an aspect-level multimodal sentiment analysis model using a labeled dataset, and then use the trained aspect-level multimodal sentiment analysis model to predict the aspect-level sentiment category of the multimodal data to be analyzed.

[0078] This invention achieves accurate sentiment polarity prediction for specific aspects by constructing an aspect-level multimodal sentiment analysis model. Specifically, this aspect-level multimodal sentiment analysis model extracts features from different modalities through a feature extraction module, constructs thought chain hints through an inference chain feature construction module, performs sentiment inference using a large language model, constructs inference chain features, and fuses features from different modalities and inference chain features through a cross-modal inference fusion module. By introducing multi-source information, it enhances contextual consistency in the inference process, effectively alleviates semantic shift problems, and achieves collaborative modeling of sentiment semantics and logical reasoning, thereby improving the accuracy of aspect-level sentiment classification and realizing efficient sentiment analysis of aspect-level multimodal content.

[0079] In step S101 of this embodiment, a social media platform (such as Weibo) is used as the target platform. Multimodal data is obtained from the target platform, and preprocessed and labeled to obtain a labeled dataset. Specifically, this may include the following steps:

[0080] S101-1. Multimodal Data Acquisition. Web crawling technology can be used to collect multimodal data containing text and images published by users on target platforms, using keywords from multiple scenarios as search criteria. Text data and corresponding image data can be extracted by structured parsing of HTML pages.

[0081] S101-2. Preprocess the collected multimodal data. For example, remove illegal characters, web links, and other invalid symbols from the text, and perform text deduplication to ensure the validity and independence of the data. To improve the matching degree of images and text and the quality of the data, you can also filter out tweets without images and perform deduplication to obtain text data and image data that match the images and text.

[0082] S101-3, Aspect-level sentiment category annotation. A visual annotation platform can be built based on the Vue front-end framework, allowing annotators to upload and annotate multimodal data locally on the platform.

[0083] The aspect-level sentiment category annotation of this invention includes annotating aspect words in text data. Aspect words are aspect entities in the text data, such as personal names, place names, organization names, and geopolitical entity names. The sentiment category annotation of this invention can be sentiment polarity annotation, such as positive, negative, and neutral.

[0084] For example, according to the preset annotation specifications, the annotators complete the following tasks in sequence: determine whether the image and text match, identify whether the text contains annotable named entities, determine whether the text length is reasonable, and annotate the identified entities, while also annotating the sentiment polarity of the entity in the image and text, forming a complete aspect-level image-text sentiment pair.

[0085] In practical implementation, there are three main principles for labeling aspect-level multimodal sentiment data. First, when the sentiment tendencies of text data and image data are consistent in a specific aspect, the overall sentiment polarity of that aspect is consistent with the sentiment tendencies of the text. Second, when the sentiment tendencies of text data and image data are inconsistent in a specific aspect, a comprehensive judgment is made based on the contextual semantics and actual situation to determine the overall sentiment polarity of that aspect. Finally, when there is a sentiment shift in the text data in a specific aspect, the sentiment information of the image data in that aspect is used as the main basis to determine the overall sentiment polarity of that aspect. Figure 2 The image shown is a schematic diagram of data annotation.

[0086] The labeled dataset of this invention includes A multimodal data sample, each sample Contains a by Words Sentences composed of Images associated with this sentence A sequence of terms related to one aspect and its corresponding annotation results, among which It is a sentence A subsequence.

[0087] In S102 of this embodiment, an aspect-level multimodal sentiment analysis model is established, including a feature extraction module, an inference chain feature construction module, a cross-modal inference fusion module, and a classification output module. For example... Figure 3 The diagram shows the structure of an aspect-level multimodal sentiment analysis model.

[0088] like Figure 3 As shown, the feature extraction module extracts features from text data, aspect words in text data, and image data in multimodal data, respectively, to obtain text features, aspect entity features, and image features. For example, a pre-trained language model BERT can be used for text feature extraction and aspect entity feature extraction, and a pre-trained visual model ViT can be used for image feature extraction.

[0089] like Figure 3As shown, the reasoning chain feature construction module constructs thought chain prompts based on text data, inputs the thought chain prompts into the large language model multiple times for sentiment reasoning, obtains multiple reasoning chains, and selects the optimal reasoning chain based on a self-consistent strategy. The optimal reasoning chain is then encoded to construct reasoning chain features.

[0090] For example, prompt text can be designed to instruct the large language model to analyze text data in the following steps: identify keywords or phrases expressing emotion in the text; analyze the emotional polarity (positive / negative / neutral) of these keywords; consider whether the context modifies or reverses the emotion; and comprehensively judge the overall emotion. An example can be provided. Based on the emotion inference results output by the large language model, the inference chain can be extracted.

[0091] Multiple sentiment inferences are performed using a large language model to obtain multiple candidate inference chains. A self-consistent strategy is then used to filter these candidate chains, selecting the optimal one. Sentiment prediction results are then extracted from this optimal inference chain, and the text of the optimal inference chain is input into the BERT model for encoding, thereby obtaining the corresponding inference chain features.

[0092] Then, the cross-modal inference fusion module fuses the text features, aspect entity features, image features extracted by the feature extraction module, and the inference chain features obtained by the inference chain feature construction module to obtain cross-modal fused features.

[0093] For example, after aligning text features, aspect entity features, image features, and inference chain features, cross-modal feature fusion can be performed directly by feature concatenation. Alternatively, neural network models can be used for attention calculation and feature fusion. For instance, the Transformer model can be used for cross-modal feature fusion. The cross-modal attention mechanism in the Transformer model allows for deep interaction between features from different modalities, treating one modality as a query and the other as a key-value pair to achieve information retrieval and fusion between modalities.

[0094] Finally, the classification output module performs aspect-level sentiment category prediction based on cross-modal fusion features.

[0095] For example, cross-modal fused features can be fed into a softmax classifier to predict aspect-level sentiment categories, such as the polarity (positive / negative / neutral) of specific aspect words.

[0096] In S103 of this embodiment, the aspect-level multimodal sentiment analysis model constructed in S102 is trained using the labeled dataset in S101. The aspect-level sentiment category can then be predicted using the trained aspect-level multimodal sentiment analysis model.

[0097] Specifically, the aspect-level multimodal sentiment analysis model is trained in an end-to-end manner, using the cross-entropy loss function, which is as follows:

[0098]

[0099] in, The value of the loss function. For the input cross-modal fusion features, For real labels, For the sample size, For sample number, , For the first The predicted probability of a class.

[0100] Then, the multimodal data to be analyzed is acquired, preprocessed in a manner similar to S101-1 and S101-2, and then input into the trained aspect-level multimodal sentiment analysis model to obtain aspect-level sentiment category prediction results.

[0101] Example 2

[0102] Please see Figure 4 , Figure 4 A second flowchart illustrating an aspect-level multimodal sentiment analysis method based on cross-modal reasoning fusion, provided as an embodiment of the present invention, includes:

[0103] S201. Obtain the multimodal data to be analyzed;

[0104] S202. Preprocess the text data and extract the text features corresponding to the text data through the first pre-trained language model.

[0105] S203. Preprocess the aspect words and extract the aspect entity features corresponding to the aspect words through the second pre-trained language model;

[0106] S204. Divide the image data into multiple image blocks, and input the multiple image blocks into the pre-trained visual model to obtain the image features corresponding to the image data.

[0107] S205. Based on text data, construct mind chain prompts using a structural template of preparatory knowledge, questions, answers, and reasons;

[0108] S206. Input the thought chain prompts into the large language model multiple times for sentiment reasoning to obtain multiple candidate reasoning chains;

[0109] S207. Select the optimal inference chain from multiple candidate inference chains using a self-consistent strategy;

[0110] S208. Input the optimal inference chain into the third pre-trained language model for feature encoding to obtain the inference chain features;

[0111] S209. Based on the attention mechanism, feature fusion is performed on aspect entity features and image features to obtain the first fused feature;

[0112] S210. Input the first fused feature into the Transformer model for feature enhancement to obtain the enhanced first fused feature;

[0113] S211. Based on the attention mechanism, feature fusion is performed on the enhanced first fusion feature and the text feature to obtain the second fusion feature;

[0114] S212. Based on the multi-head attention mechanism, bidirectional feature fusion is performed on the second fusion feature and the inference chain feature to obtain cross-modal fusion features.

[0115] S213. Perform feature fusion on the zero-sample sentiment prediction results obtained based on the optimal inference chain and the cross-modal fusion features to obtain the final fused features;

[0116] S214. Input the final fused features into the classifier to predict the probability of sentiment classification, and obtain the aspect-level sentiment category prediction results.

[0117] This invention employs pre-trained language models and pre-trained visual models to extract textual features, aspect entity features, and image features from multimodal data. This effectively avoids data source limitations and category bias, making aspect-level multimodal sentiment analysis models more applicable to Chinese contexts and improving their application and generalization capabilities in Chinese and other contexts. Furthermore, by constructing thought chain prompts using a structural template including preparatory knowledge, questions, answers, and reasons, the invention maximizes the reasoning capabilities of the large language model for multiple sentiment inferences. A self-consistent strategy selects the optimal reasoning chain, which effectively models the deep-seated logical relationships within the text, providing strong logical support for aspect-level multimodal sentiment analysis.

[0118] Furthermore, this invention utilizes an attention mechanism for cross-modal feature interaction and inference fusion, dynamically filtering cross-modal information irrelevant to specific aspects. It precisely correlates visual information with specific aspects in text data, effectively modeling complex logical relationships between different modalities. By combining optimal inference chains to optimize potential logical relationships in text data, it achieves more accurate and robust aspect-level multimodal sentiment analysis. Finally, the zero-sample sentiment prediction results obtained from text data are fused with cross-modal fused features, and then classified and output. By combining single-modal prediction results with fused multimodal features, the accuracy of sentiment polarity prediction for specific aspects is further improved.

[0119] In step S201 of this embodiment, the multimodal data to be analyzed is preprocessed to obtain a dataset consisting of... A text sequence consisting of words Images associated with this text sequence and a sequence of aspect words The term "this aspect" refers to an entity.

[0120] In S202 of this embodiment, the first pre-trained language model can be the BERT model. The text sequence corresponding to the text data... Inputting these features into the BERT model yields textual features.

[0121] The text feature extraction formula is:

[0122]

[0123] in, For text features, Represents a text sequence Length, For dimensions. The mathematical expression representing the BERT model.

[0124] In S203 of this embodiment, the second pre-trained language model can also be a BERT model. The aspect word sequence... Inputting these features into the BERT model yields aspect entity features.

[0125] The formula for extracting entity features is:

[0126]

[0127] in, For aspect entity features, Sequence of aspect words The length.

[0128] In S204 of this embodiment, the pre-trained visual model can be the ViT model.

[0129] Specifically, the image Divided into sizes The image patch, the set of image patches is represented as These image patches are then independently input into a pre-trained ViT model to capture detailed image information. ViT achieves a more refined visual representation by processing the local features of each image patch. Furthermore, to ensure that image features and text features have the same dimensionality, a linear transformation layer is added to the output layer of the ViT model. Using this method, the feature representation of the image can be obtained, and the formula for extracting image features is:

[0130]

[0131] in For image features, It is the size of the hidden layer for image features. The mathematical expression representing the ViT model.

[0132] In S205 of this embodiment, a mind chain prompt is constructed based on the text data and aspect words, using a structural template of preliminary knowledge, questions, answers, and reasons.

[0133] The preparatory knowledge includes foundational knowledge related to aspect-level sentiment category prediction and may also include knowledge related to dataset annotations to enable reasonable predictions in the large language model. Preparatory knowledge may also include prior knowledge from the field of sentiment analysis to improve the large language model's understanding of sentiment context. The question section transforms the sentiment intensity prediction task into a question-and-answer structure in natural language, making it understandable to the large language model. The reasoning section aims to trigger the reasoning ability of the large language model, enabling it to progressively output reasoning or explanation steps. Figure 5 The image shown is an example of a mind chain prompt built based on a structure template of prerequisite knowledge, question, answer, and reason. Figure 5 The document also provides input and output examples.

[0134] In step S206 of this embodiment, the thought chain prompt is input into the large language model, and the reasoning chain is obtained based on the output of the large language model. To improve the rationality of the reasoning chain, the thought chain prompt can be repeatedly input into the large language model to form different candidate reasoning chains.

[0135] In S207 of this embodiment, a self-consistent strategy is used to statistically analyze and compare multiple candidate inference chains, and the candidate inference chain with the highest frequency of occurrence or the most reasonable semantics is selected as the optimal inference chain CoT.

[0136] In step S208 of this embodiment, the optimal inference chain CoT is input into the third pre-trained language model for encoding to obtain the inference chain features. The third pre-trained language model can be the BERT model. The formula for extracting inference chain features using the BERT model is as follows:

[0137]

[0138] in Features of the reasoning chain Indicates the length of the text sequence. For dimensions.

[0139] according to Figure 5The thought process chain shown indicates that the answers output by the large language model include sentiment intensity scores. Therefore, zero-shot sentiment prediction results can be obtained using number-perceptual sentiment estimation based on the answers output by the large language model. Specifically, a number is extracted from the first or last sentence of the output answer. If multiple numbers exist, the first number is selected as the zero-sample sentiment prediction result. For example, if the sentiment score is "1.0 or 1.5", then 1.0 is considered the predicted sentiment intensity. If no number is found in the output, the sentiment intensity can be considered to be 0.0, representing neutral sentiment; if it is less than 0, it is considered negative sentiment.

[0140] This invention constructs thought chain prompts through prompt engineering to induce a large language model to generate a reasoning process based on text data, thus making full use of the reasoning capabilities of the large language model to achieve zero-sample sentiment prediction results based on single-modality text data.

[0141] In S209 of this embodiment, the extracted aspect entity features and image features The input is fed into the locally guided image representation layer (LIR) for preliminary feature fusion to extract image fusion features related to aspect terms.

[0142] Specifically, the aspect entity features As a query, image features By using aspect terms as keys and values, and calculating the correlation between aspect terms and different regions of the image, the image patch most relevant to the aspect entity is selected, thereby generating the first fusion feature related to aspect terms. .

[0143] The expression for feature fusion using the locally guided image representation layer (LIR) is as follows:

[0144]

[0145]

[0146]

[0147] in, Representing the characteristics of entities in terms of aspects As a query, based on image features As a key-value fusion operation , , and It is a learnable weight matrix. , The number of attention heads in the multi-head attention mechanism of LIR. , For the first The dimensions of the body. The mathematical expression representing the locally guided image representation layer (LIR). Represents the concatenation function. Represents the attention calculation function. Representative layer normalization function.

[0148] In S210 of this embodiment, in order to further improve the expressive power of the first fusion feature, the first fusion feature output by LIR is... Input the Transformer model for feature enhancement.

[0149] The Transformer model can capture deep-level relationships between features of different modalities through a multi-head self-attention mechanism, enhancing aspect-related contextual information and thus better capturing global and local information related to aspect entities. The expression for feature enhancement using the Transformer model is:

[0150]

[0151] in, The first fusion feature Features enhanced by the Transformer model, Transformer ( ) is the mathematical expression for the Transformer model.

[0152] In S211 of this embodiment, the enhanced first fusion feature Text features The second fused feature is obtained by fusing the image representation layer GIR through global guidance. .

[0153] The expression for feature fusion using the globally guided image representation layer (GIR) is:

[0154]

[0155]

[0156]

[0157] in Represented by textual features For querying, based on the first fusion feature For key-value merging operations, , , and It is a learnable weight matrix. , For the multi-head attention mechanism of GIR, the number of heads of attention. For the first Size, , For the first The dimension of size. The mathematical expression representing the Global Guided Image Representation Layer (GIR).

[0158] This invention uses a globally guided image representation layer (GIR) to fuse deep features of text, aspect terms, and image features to generate semantically rich multimodal representations and achieve global interaction of multi-level features.

[0159] In S212 of this embodiment, bidirectional feature fusion is performed on the second fusion feature and the inference chain feature based on the multi-head attention mechanism to obtain cross-modal fusion features, including:

[0160] Using the second fusion feature as the query and the inference chain feature as the key and value, attention calculation and feature fusion are performed on the second fusion feature and the inference chain feature to obtain the third fusion feature;

[0161] Using the third fusion feature as the query and the second fusion feature as the key and value, attention calculation and feature fusion are performed on the third fusion feature and the second fusion feature to obtain cross-modal fusion features.

[0162] Specifically, with the second fusion feature As a query, based on the characteristics of the reasoning chain As keys and values, for the second fusion feature and inference chain features The expression for attention calculation and feature fusion is:

[0163]

[0164]

[0165]

[0166] in, Represented by the second fusion feature For querying, based on inference chain features For key-value merging operations, MultiHead( ) represents the multi-head attention function. , Let be the number of attention heads in this multi-head attention mechanism, where , , and It is a learnable weight matrix. For the first Size, , For the first The dimension of size. This is the third fusion feature.

[0167] The expression for attention calculation and feature fusion of the third fusion feature as the query and the second fusion feature as the key and value is as follows:

[0168]

[0169]

[0170]

[0171] in, This is a fusion operation that uses the third fusion feature as the query and the second fusion feature as the key and value. , , and It is a learnable weight matrix. , The number of attention heads in this multi-head attention mechanism. For the first Size, , For the first The dimension of size. This is a cross-modal fusion feature.

[0172] like Figure 6 The diagram shows the principle of cross-modal feature fusion corresponding to S210~S212. Here, Q, K, and V represent the query, key, and value vectors, respectively. First, the aspect entity features... The image features are initially fused with the local guided image representation layer (LIR) to obtain the first fused feature. ;Utilizing Transformer to fuse the first feature Enhancement is performed; the enhanced first fusion feature is then applied. Text features The second fused feature is obtained by further fusion in the globally guided image representation layer (GIR). Then, the second fusion feature... and inference chain features Bidirectional feature fusion is performed to obtain cross-modal fused features. In the bidirectional feature fusion process, the second fused feature is used... As a query, based on the characteristics of the reasoning chain Using keys and values ​​to suppress hallucinations, a corrected inference chain, i.e., the third fusion feature, is obtained using a multi-head attention mechanism. Then, using the third fusion feature As a query, using the second fusion feature Using key and value pairs, a multi-head attention mechanism is employed to refocus key evidence, forming a closed loop, thereby obtaining cross-modal fusion features. This enables enhanced cross-modal reasoning.

[0173] like Figure 7 The diagram illustrates the structure of a multi-head attention mechanism. Its principle is as follows: A linear layer performs multiple linear projections on the input query Q, key K, and value V. Each group of projected query Q, key K, and value V is input into a scaled dot-product attention layer for parallel scaling dot-product attention calculation, resulting in multiple outputs. These outputs are then concatenated and re-projected through a linear layer to obtain the final output with the same dimensions as single-head attention. Specifically, the scaled dot-product attention layer works by using matrix multiplication (MatMul) to calculate the dot product of query Q and all keys K to obtain the original attention score. This score is then scaled to stabilize the gradient, converted into probability weights using a normalized exponential function (Softmax), and finally weighted and summed across the value vectors to dynamically focus on the most relevant information.

[0174] This invention integrates multimodal features obtained by fusing aspect entity features, image features, and text features with inference chain features in a two-way manner. On the one hand, it uses multimodal features to correct inference chain features, and on the other hand, it uses inference chain features to refocus multimodal features, thereby enhancing the consistency and interpretability of cross-modal reasoning. At the same time, it integrates high-level reasoning information and cross-modal complementary information during the feature representation learning process, which can effectively improve the richness and accuracy of feature representation.

[0175] In S213 of this embodiment, cross-modal fusion features are marked by CLS. The corresponding classification embedding enhances the low-level cross-modal fusion features for inference. Categorical embedding as a high-level multimodal sentiment representation Single-mode zero-sample prediction results High-level multimodal sentiment representation The fusion process is performed to obtain the final fusion characteristics.

[0176] For example, feature fusion can be performed by feature concatenation to obtain the final fused features. :

[0177]

[0178] In S213 of this embodiment, the final fusion feature is... The input is fed into a linear layer, and then into a softmax function for classification, yielding an aspect-level sentiment classification probability distribution. :

[0179]

[0180] in, and These are the learnable weights and biases, respectively. Represents the emotional category.

[0181] like Figure 8 The diagram illustrates an application of aspect-level sentiment category prediction using multimodal data from a target platform. The image data is a photo of person A and person B, the text data is "Photo of person A with person B at a music festival in 1987 #person A#", and the aspect term is "person A".

[0182] First, the text data “A photo of person A and person B at a music festival in 1987” and the aspect term “person A” are processed by the pre-trained language model BERT to obtain text features and aspect entity features respectively. For a given image (e.g., a photo of person A and person B), the image data is pre-processed and input into the pre-trained visual model ViT to obtain image feature representation.

[0183] Secondly, in the inference chain feature construction stage, based on the input text data and aspect words, the system automatically constructs a thought chain prompt. For example, this prompt can be constructed as: "Prerequisite knowledge: The sentiment intensity score ranges from -1.0 to 1.0: -1.0 = negative, 0.0 = neutral, 1.0 = positive. Question: Given the sentence 'A photo of me with person B at a music festival in 1987 #$T$#', where the entity corresponding to position $T$ is person A, what is the most likely sentiment intensity score (range -1.0 to 1.0) for this entity? Please explain your reasoning. Answer: ". Subsequently, this prompt is input into a large language model. The large language model performs zero-shot inference, generating an inference chain. The system extracts the single-modal zero-shot sentiment prediction results and inputs them into the BERT model for encoding, generating inference chain features.

[0184] Next, in the cross-modal reasoning fusion module, the similarities and differences between the extracted aspect word features and image features are input into the locally guided image representation layer (LIR). The LIR uses the semantic information of the aspect words as guidance, causing the model to focus on the visual regions in the image related to "person A," generating aspect word-related image fusion features. These features are then enhanced with context by a Transformer encoder and input together with the text features into the globally guided image representation layer (GIR), generating a deeply fused multimodal representation. Finally, this multimodal representation is bidirectionally fused with the previously generated reasoning feature chain to obtain the final reasoning fusion feature. This feature contains both low-level perceptual information from the image and text and high-level logical reasoning information from the large language model.

[0185] Finally, the inference fusion features are transformed in dimension through one or more fully connected layers and then input into the Softmax activation function to predict and output the sentiment result for the aspect word "person A", such as "positive".

[0186] Example 3

[0187] Please see Figure 9 , Figure 9 A schematic diagram of an aspect-level multimodal sentiment analysis system based on cross-modal reasoning fusion, provided as an embodiment of the present invention, is shown. The system includes:

[0188] The data annotation module 910 is used to acquire multimodal data from the target platform and perform aspect-level sentiment category annotation to obtain an annotated dataset;

[0189] The model building module 920 is used to build an aspect-level multimodal sentiment analysis model. This model includes a feature extraction module, an inference chain feature construction module, a cross-modal inference fusion module, and a sentiment classification module. Specifically, the feature extraction module extracts text features, aspect entity features, and image features from the input multimodal data. The inference chain feature construction module constructs thought chain prompts based on the multimodal data and combines them with a large language model to build inference chain features. The cross-modal inference fusion module fuses text features, aspect entity features, and inference chain features to obtain cross-modal fused features. The classification output module predicts aspect-level sentiment categories based on the cross-modal fused features.

[0190] The sentiment classification module 930 is used to train an aspect-level multimodal sentiment analysis model using a labeled dataset, and then use the trained aspect-level multimodal sentiment analysis model to predict the aspect-level sentiment category of the multimodal data to be analyzed.

[0191] The aspect-level multimodal sentiment analysis system based on cross-modal reasoning fusion described above and the aspect-level multimodal sentiment analysis method based on cross-modal reasoning fusion described above can be referred to and correspond to each other.

[0192] Figure 10 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 10 As shown, the electronic device may include a processor 1010, a communications interface 1020, a memory 1030, and a communication bus 1040, wherein the processor 1010, the communications interface 1020, and the memory 1030 communicate with each other via the communication bus 1040. The processor 1010 can call logical instructions in the memory 1030 to execute the aspect-level multimodal sentiment analysis method based on cross-modal reasoning fusion provided in the above-described method embodiments.

[0193] Furthermore, the logical instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0194] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the aspect-level multimodal sentiment analysis method based on cross-modal reasoning fusion provided in the above-described method embodiments.

[0195] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the aspect-level multimodal sentiment analysis method based on cross-modal reasoning fusion provided in the above-described method embodiments.

[0196] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0197] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0198] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An aspect-level multimodal sentiment analysis method based on cross-modal reasoning fusion, characterized in that, include: Acquire multimodal data from the target platform and perform aspect-level sentiment category annotation to obtain an annotated dataset; An aspect-level multimodal sentiment analysis model is established, which includes a feature extraction module, an inference chain feature construction module, a cross-modal inference fusion module, and a classification output module. The feature extraction module is used to extract text features, aspect entity features, and image features from the input multimodal data, respectively. The inference chain feature construction module is used for: Based on text data, a thought chain prompt is constructed using a structural template that includes preparatory knowledge, questions, answers, and reasons; wherein, the preparatory knowledge includes basic or prior knowledge related to aspect-level sentiment category prediction; The thought chain prompts are input into a large language model multiple times for sentiment inference to obtain multiple candidate inference chains; The optimal inference chain is selected from the multiple candidate inference chains using a self-consistent strategy; The optimal inference chain is input into the third pre-trained language model for feature encoding to obtain the inference chain features; The cross-modal reasoning fusion module is used for: Based on an attention mechanism, feature fusion is performed on the entity features and image features of the aforementioned aspects to obtain the first fused feature; The first fused feature is input into the Transformer model for feature enhancement to obtain the enhanced first fused feature. Based on an attention mechanism, the enhanced first fused feature and the text feature are fused to obtain the second fused feature; Using the second fusion feature as the query and the inference chain feature as the key and value, attention calculation and feature fusion are performed on the second fusion feature and the inference chain feature to obtain the third fusion feature; Using the third fusion feature as the query and the second fusion feature as the key and value, attention calculation and feature fusion are performed on the third fusion feature and the second fusion feature to obtain cross-modal fusion features; The classification output module is used for: The zero-sample sentiment prediction results obtained based on the optimal inference chain are fused with the cross-modal fusion features to obtain the final fused features; The final fused features are input into a classifier to predict the probability of sentiment classification, resulting in an aspect-level sentiment category prediction. The aspect-level multimodal sentiment analysis model is trained using the labeled dataset, and then used to predict aspect-level sentiment categories in the multimodal data to be analyzed.

2. The aspect-level multimodal sentiment analysis method based on cross-modal reasoning fusion according to claim 1, characterized in that, The multimodal data includes text data and image data for image-text matching; The aspect-level sentiment category annotation includes annotating aspect words in the text data, where aspect words are aspect entities in the text data.

3. The aspect-level multimodal sentiment analysis method based on cross-modal reasoning fusion according to claim 2, characterized in that, The feature extraction module is used for: The text data is preprocessed, and the text features corresponding to the text data are extracted through a first pre-trained language model; The aspect words are preprocessed, and the aspect entity features corresponding to the aspect words are extracted through the second pre-trained language model; The image data is divided into multiple image blocks, and each of the multiple image blocks is input into a pre-trained visual model to obtain the image features corresponding to the image data.

4. A cross-modal reasoning fusion-based aspect-level multimodal sentiment analysis system, used to implement the cross-modal reasoning fusion-based aspect-level multimodal sentiment analysis method according to any one of claims 1 to 3, characterized in that, include: The data annotation module is used to acquire multimodal data from the target platform and perform aspect-level sentiment category annotation to obtain an annotated dataset; The model building module is used to build an aspect-level multimodal sentiment analysis model. This model includes a feature extraction module, an inference chain feature construction module, a cross-modal inference fusion module, and a classification output module. Specifically, the feature extraction module extracts text features, aspect entity features, and image features from the input multimodal data. The inference chain feature construction module constructs thought chain prompts based on the multimodal data and combines them with a large language model to build inference chain features. The cross-modal inference fusion module fuses the text features, aspect entity features, and inference chain features to obtain cross-modal fused features. The classification output module predicts aspect-level sentiment categories based on the cross-modal fusion features. The sentiment classification module is used to train the aspect-level multimodal sentiment analysis model using the labeled dataset, and to predict the aspect-level sentiment category of the multimodal data to be analyzed using the trained aspect-level multimodal sentiment analysis model.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the aspect-level multimodal sentiment analysis method based on cross-modal reasoning fusion as described in any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the aspect-level multimodal sentiment analysis method based on cross-modal reasoning fusion as described in any one of claims 1 to 3.

Images