Aspect-level sentiment classification method and system based on multi-modal alignment and reflection enhancement

Abstract

The invention provides an aspect-level sentiment classification method and system based on multi-modal alignment and reflection enhancement, and relates to the technical field of artificial intelligence, and the method comprises the steps: employing a pre-trained teacher model to enable an original image to be combined with a knowledge guide prompt to generate text description; splicing the description, the original text, the target aspect word and the structured reasoning prompt template into a multi-modal aligned enhanced input sequence, and inputting the multi-modal aligned enhanced input sequence into a pre-trained student model to obtain a predicted emotion tag; aiming at a sample with a prediction error, generating an reflection-correction inference chain by utilizing a teacher model, and constructing an reflection enhancement data set; and finally, combining the original data set and the reflection enhancement data set, and performing supervised fine tuning training on the student model. According to the method, the semantic alignment and self-error correction capability of the model is enhanced, so that the sentiment classification accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an aspect-level sentiment classification method and system based on multimodal alignment and reflective enhancement. Background Technology

[0002] Aspect-level sentiment analysis is a core fine-grained sentiment understanding task in natural language processing, aiming to identify the sentiment polarity of specific aspects in text. With the widespread use of social media platforms, user-generated content is increasingly exhibiting multimodal characteristics—a post typically consists of both text and images, which work together to express emotional intent. Against this backdrop, multimodal aspect-level sentiment classification has emerged, aiming to jointly infer the sentiment tendency of words related to specific aspects mentioned in text from image-text pairs. This task has significant application value in practical scenarios such as public opinion monitoring, brand reputation management, cross-modal recommendation systems, and intelligent customer service.

[0003] Although methods based on multimodal large language models have made some progress in multimodal aspect-level sentiment classification tasks in recent years, existing technologies still face two fundamental challenges that severely restrict the performance and robustness of models in real-world complex scenarios.

[0004] The first challenge is cross-modal semantic misalignment. There is often a lack of precise semantic correspondence between visual elements in an image and specific aspects mentioned in the text. The original image contains a lot of background noise that is irrelevant to the target aspect, and traditional methods usually directly input the whole image or general visual features into the model, making it difficult for the model to focus on key visual cues related to the aspect.

[0005] The second challenge is the lack of reflective reasoning ability. In real social media data, emotional expression often takes the form of implicit, ironic, or modal conflict.

[0006] In summary, current mainstream methods in multimodal aspect-level sentiment classification have significant shortcomings in two key dimensions: firstly, they fail to achieve fine-grained cross-modal alignment of aspect perception, resulting in low efficiency in utilizing visual information; secondly, they lack the ability to reflect on and correct their own reasoning processes, making it difficult to address the implicit sentiment and modal conflicts prevalent in the real world. Therefore, there is an urgent need for a new generation framework that can accurately align multimodal semantics and endow the model with self-correcting capabilities to overcome existing technological bottlenecks and propel multimodal sentiment analysis towards higher levels of robustness and interpretability. Summary of the Invention

[0007] To overcome the lack of accurate semantic alignment and the lack of reflective ability in the models of different modalities, this invention provides an aspect-level sentiment classification method and system based on multimodal alignment and reflective enhancement.

[0008] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: This invention provides an aspect-level sentiment classification method based on multimodal alignment and reflective enhancement, comprising: Obtain the original dataset, which includes original images, original text, target aspect words and their corresponding true sentiment labels; The original image and pre-set knowledge-guided prompts are input into a pre-trained teacher model to obtain a text description; The text description, original text, target aspect words, and preset structured reasoning prompt template are concatenated to obtain a multimodal aligned enhanced input sequence; The enhanced input sequence is input into a pre-trained student model to obtain predicted sentiment labels; Based on the consistency between the predicted sentiment labels and the actual sentiment labels, erroneous samples are identified; Input all the error samples into the pre-trained teacher model to obtain the reflection-correction inference chain corresponding to each error sample; The erroneous samples and their corresponding reflection-correction inference chains, predicted sentiment labels, and true sentiment labels are combined to form a reflection-enhanced dataset, which is then merged with the original dataset to obtain a complete dataset. Input the complete dataset into the pre-trained student model, construct a loss function and perform supervised fine-tuning training to obtain a fine-tuned student model; The data to be tested is input into the finely tuned student model to obtain the final predicted label.

[0009] Preferably, the original image and preset knowledge-guided prompts are input into a pre-trained teacher model to obtain a text description, including:

[0010] TLM stands for pre-trained teacher model. For the k-th preset knowledge guidance prompt, For the i-th original image, This is the text description of the i-th original image.

[0011] Preferably, the text description, original text, target aspect words, and preset structured reasoning prompt template are concatenated to obtain a multimodal aligned enhanced input sequence, including:

[0012] in, This serves as a structured reasoning prompt template. For the i-th target aspect term, For the i-th original text, Let be the i-th multimodal aligned enhanced input sequence.

[0013] Preferably, the enhanced input sequence is input into a pre-trained student model to obtain predicted sentiment labels, including:

[0014] SLM is a pre-trained student model. Let be the predicted sentiment label for the i-th augmented input sequence.

[0015] Preferably, based on the consistency between the predicted sentiment label and the actual sentiment label, erroneous samples are identified, including: Samples whose predicted sentiment labels do not match the actual sentiment labels are filtered out to obtain a set of erroneous samples:

[0016] in, This is an incorrect sample. For the sample with inconsistent label j, Let j be the predicted label for the sample with inconsistent label. Let be the true sentiment label of the j-th sample whose label is inconsistent.

[0017] Preferably, the erroneous samples are input into the pre-trained teacher model to obtain a reflection-correction inference chain, including: The expression for the reflection-correction inference chain is as follows:

[0018] in, To reflect on the data, To correct the data, This is the j-th reflection-correction inference chain.

[0019] Preferably, the erroneous samples and their corresponding reflection-correction inference chains, predicted sentiment labels, and true sentiment labels are combined to form a reflection-enhanced dataset, which is then merged with the original dataset to obtain a complete dataset, including: The expression for the reflection-enhanced dataset is as follows:

[0020] in, To reflect on and revise the reasoning chain.

[0021] Preferably, the expression for the complete dataset is:

[0022] in, For the original dataset, To enhance the dataset for reflection.

[0023] Preferably, the expression for the loss function is:

[0024] in, Indicates the first position in the sequence One token, This represents all context tokens preceding this token.

[0025] This invention also provides an aspect-level sentiment classification system based on multimodal alignment and reflective enhancement, comprising: The data acquisition module is used to acquire the raw dataset, which includes raw images, raw text, target aspect words and their corresponding real sentiment labels; The text description acquisition module is used to input the original image and preset knowledge guidance prompts into the pre-trained teacher model to obtain the text description; The enhanced input sequence acquisition module is used to concatenate the text description, original text, target aspect words and preset structured reasoning prompt template to obtain a multimodal aligned enhanced input sequence; The predicted sentiment label acquisition module is used to input the enhanced input sequence into a pre-trained student model to obtain the predicted sentiment label. The error sample identification module is used to identify error samples based on the consistency between the predicted sentiment label and the true sentiment label; The inference chain acquisition module is used to input all the error samples into the pre-trained teacher model to obtain the reflection-correction inference chain corresponding to each error sample; The complete dataset construction module is used to combine the erroneous samples and their corresponding reflection-correction inference chains, predicted sentiment labels and real sentiment labels into a reflection-enhanced dataset, and merge it with the original dataset to obtain a complete dataset. The model fine-tuning module is used to input the complete dataset into the pre-trained student model, construct a loss function for supervised fine-tuning training, and obtain a fine-tuned student model. The data detection module is used to input the data to be detected into the finely tuned student model to obtain the final predicted label.

[0026] Compared with the prior art, the beneficial effects of the technical solution of the present invention are: This invention first acquires an original dataset, which includes original images, original text, target aspect words, and their corresponding true sentiment labels. Then, the original images and preset knowledge-guided prompts are input into a pre-trained teacher model to obtain text descriptions. Next, the text descriptions, original text, target aspect words, and preset structured reasoning prompt templates are concatenated to obtain a multimodal aligned augmented input sequence. Subsequently, the augmented input sequence is input into a pre-trained student model to obtain predicted sentiment labels. Based on the consistency between the predicted sentiment labels and the true sentiment labels, erroneous samples are identified. All erroneous samples are input into the pre-trained teacher model to obtain a reflection-correction inference chain corresponding to each erroneous sample. The erroneous samples, their corresponding reflection-correction inference chains, predicted sentiment labels, and true sentiment labels are combined to form a reflection-enhanced dataset, which is then merged with the original dataset to obtain a complete dataset. The complete dataset is input into the pre-trained student model, and a loss function is constructed for supervised fine-tuning training to obtain a fine-tuned student model. The data to be detected is input into the fine-tuned student model to obtain the final predicted labels. This invention achieves more accurate sentiment classification of data by improving the model's data semantic alignment and self-correction capabilities. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating an aspect-level sentiment classification method based on multimodal alignment and reflection enhancement in Example 1. Figure 2 This is a schematic diagram of the structure of an aspect-level sentiment classification system based on multimodal alignment and reflection enhancement in Example 3. Detailed Implementation

[0028] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent. To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0029] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0030] Example 1 This embodiment provides a method, such as Figure 1 As shown, it includes: Obtain the original dataset, which includes original images, original text, target aspect words and their corresponding true sentiment labels; The original image and pre-set knowledge-guided prompts are input into a pre-trained teacher model to obtain a text description; The text description, original text, target aspect words, and preset structured reasoning prompt template are concatenated to obtain a multimodal aligned enhanced input sequence; The enhanced input sequence is input into a pre-trained student model to obtain predicted sentiment labels; Based on the consistency between the predicted sentiment labels and the actual sentiment labels, erroneous samples are identified; Input all the error samples into the pre-trained teacher model to obtain the reflection-correction inference chain corresponding to each error sample; The erroneous samples and their corresponding reflection-correction inference chains, predicted sentiment labels, and true sentiment labels are combined to form a reflection-enhanced dataset, which is then merged with the original dataset to obtain a complete dataset. Input the complete dataset into the pre-trained student model, construct a loss function and perform supervised fine-tuning training to obtain a fine-tuned student model; The data to be tested is input into the finely tuned student model to obtain the final predicted label.

[0031] In its implementation, this invention first acquires an original dataset, which includes original images, original text, target aspect words, and their corresponding true sentiment labels. Next, the original images and preset knowledge-guided prompts are input into a pre-trained teacher model to obtain text descriptions. Then, the text descriptions, original text, target aspect words, and preset structured reasoning prompt templates are concatenated to obtain a multimodal aligned augmented input sequence. This augmented input sequence is then input into a pre-trained student model to obtain predicted sentiment labels. Based on the consistency between the predicted sentiment labels and the true sentiment labels, erroneous samples are identified. All erroneous samples are input into the pre-trained teacher model to obtain a reflection-correction inference chain corresponding to each erroneous sample. The erroneous samples, their corresponding reflection-correction inference chains, predicted sentiment labels, and true sentiment labels are combined to form a reflection-enhanced dataset, which is then merged with the original dataset to obtain a complete dataset. This complete dataset is input into the pre-trained student model, and a loss function is constructed for supervised fine-tuning training to obtain a finely tuned student model. Finally, the data to be detected is input into the finely tuned student model to obtain the final predicted labels. This invention achieves more accurate sentiment classification of data by improving the model's data semantic alignment and self-correction capabilities.

[0032] Example 2 This embodiment provides an aspect-level sentiment classification method based on multimodal alignment and reflective enhancement, including: Obtain the original dataset, which includes original images, original text, target aspect words and their corresponding true sentiment labels; The original image and pre-set knowledge-guided prompts are input into a pre-trained teacher model to obtain a text description; The text description, original text, target aspect words, and preset structured reasoning prompt template are concatenated to obtain a multimodal aligned enhanced input sequence; The enhanced input sequence is input into a pre-trained student model to obtain predicted sentiment labels; Based on the consistency between the predicted sentiment labels and the actual sentiment labels, erroneous samples are identified; Input all the error samples into the pre-trained teacher model to obtain the reflection-correction inference chain corresponding to each error sample; The erroneous samples and their corresponding reflection-correction inference chains, predicted sentiment labels, and true sentiment labels are combined to form a reflection-enhanced dataset, which is then merged with the original dataset to obtain a complete dataset. Input the complete dataset into the pre-trained student model, construct a loss function and perform supervised fine-tuning training to obtain a fine-tuned student model; The data to be tested is input into the finely tuned student model to obtain the final predicted label.

[0033] It should be noted that, in this embodiment, the original image and preset knowledge guidance prompts are input into the pre-trained teacher model to obtain a text description, including:

[0034] TLM stands for pre-trained teacher model. For the k-th preset knowledge guidance prompt, For the i-th original image, This is the text description of the i-th original image.

[0035] The knowledge guidance prompts It is a set of artificially designed natural language instructions that explicitly require TLM to focus on visual attributes closely related to emotional judgment when generating image descriptions, including but not limited to facial expressions, semantic roles of key objects, overall color atmosphere, and the emotional tendency implied by the composition.

[0036] It should be noted that, in this embodiment, the text description, original text, target aspect words, and preset structured reasoning prompt template are concatenated to obtain a multimodal aligned enhanced input sequence, including:

[0037] in, This serves as a structured reasoning prompt template. For the i-th target aspect term, For the i-th original text, Let be the i-th multimodal aligned enhanced input sequence.

[0038] The prompt template It is a fixed natural language instruction used to specify the task objective, such as: Please determine the emotional polarity based on the following text and image information and specified aspects; It should be noted that, in this embodiment, the enhanced input sequence is input into a pre-trained student model to obtain predicted sentiment labels, including:

[0039] SLM is a pre-trained student model. Let be the predicted sentiment label for the i-th augmented input sequence.

[0040] It should be noted that, in this embodiment, erroneous samples are identified based on the consistency between the predicted sentiment label and the actual sentiment label, including: Samples whose predicted sentiment labels do not match the actual sentiment labels are filtered out to obtain a set of erroneous samples:

[0041] in, This is an incorrect sample. For the sample with inconsistent label j, Let j be the predicted label for the sample with inconsistent label. Let be the true sentiment label of the j-th sample whose label is inconsistent.

[0042] It should be noted that, in this embodiment, the erroneous sample is input into the pre-trained teacher model to obtain the reflection-correction inference chain, including: The expression for the reflection-correction inference chain is as follows:

[0043] in, To reflect on the data, To correct the data, This is the j-th reflection-correction inference chain.

[0044] It should be noted that, in this embodiment, the erroneous samples and their corresponding reflection-correction inference chains, predicted sentiment labels, and true sentiment labels are combined to form a reflection-enhanced dataset, which is then merged with the original dataset to obtain a complete dataset, including: The expression for the reflection-enhanced dataset is as follows:

[0045] in, To reflect on and revise the reasoning chain.

[0046] It should be noted that, in this embodiment, the expression for the complete dataset is:

[0047] in, For the original dataset, To enhance the dataset for reflection.

[0048] It should be noted that, in this embodiment, the expression for the loss function is:

[0049] in, Indicates the first position in the sequence One token, This represents all context tokens preceding this token.

[0050] In practical applications, comprehensive experimental validation was conducted on two widely used public benchmark datasets in the field of multimodal aspect-level sentiment classification—Twitter-2015 and Twitter-2017. The MARE framework proposed in this invention, combined with the Qwen2.5-VL-7B model, achieved macro-average F1 scores of 76.0% and 75.4% on the Twitter-2015 and Twitter-2017 datasets, respectively. This significantly surpasses existing state-of-the-art methods, including FITE, A2II, DQPSA, Chimera, ASFEN, and DEQA, fully demonstrating the effectiveness of this framework. Experimental data are shown in Table 1 below: Table 1

[0051] To further explore the contributions of each component in the aspect-level sentiment classification method based on multimodal alignment and reflective enhancement, a systematic ablation experiment was conducted. The experimental data are shown in Table 2 below: Table 2

[0052] Table 2 shows that the multimodal alignment module effectively enhances the model's understanding of visual cues by introducing visual descriptions of sentiment perception, especially on samples highly dependent on image information. For example, on the Twitter-2017 dataset, this module improved the F1 score of the Qwen2.5-VL-3B model for "negative" sentiment from 67.09% to 69.57%. More importantly, the reflection enhancement module brings breakthrough performance improvements when handling complex scenarios such as modal conflict, irony, or sentiment ambiguity. On the Twitter-2017 dataset, this module brought a 2.66% overall F1 improvement to the Qwen2.5-VL-3B model and a remarkable 6.44% improvement in its F1 score for the "negative" sentiment category. For the more capable Qwen2.5-VL-7B model, the RE module also contributed a 1.42% accuracy improvement and a 6.38% improvement in the F1 score for negative sentiment. These ablation studies consistently demonstrate that the alignment and reflection synergy mechanism designed in this invention can not only accurately integrate cross-modal information, but also endow the model with powerful self-diagnosis and correction capabilities, thus exhibiting excellent robustness and superior performance in real and complex social media sentiment analysis tasks.

[0053] Example 3 This embodiment provides an aspect-level sentiment classification system based on multimodal alignment and reflective enhancement, used to implement the aspect-level sentiment classification method based on multimodal alignment and reflective enhancement described in Embodiment 1 or 2, such as... Figure 2 As shown, it includes: The data acquisition module is used to acquire the raw dataset, which includes raw images, raw text, target aspect words and their corresponding real sentiment labels; The text description acquisition module is used to input the original image and preset knowledge guidance prompts into the pre-trained teacher model to obtain the text description; The enhanced input sequence acquisition module is used to concatenate the text description, original text, target aspect words and preset structured reasoning prompt template to obtain a multimodal aligned enhanced input sequence; The predicted sentiment label acquisition module is used to input the enhanced input sequence into a pre-trained student model to obtain the predicted sentiment label. The error sample identification module is used to identify error samples based on the consistency between the predicted sentiment label and the true sentiment label; The inference chain acquisition module is used to input all the error samples into the pre-trained teacher model to obtain the reflection-correction inference chain corresponding to each error sample; The complete dataset construction module is used to combine the erroneous samples and their corresponding reflection-correction inference chains, predicted sentiment labels and real sentiment labels into a reflection-enhanced dataset, and merge it with the original dataset to obtain a complete dataset. The model fine-tuning module is used to input the complete dataset into the pre-trained student model, construct a loss function for supervised fine-tuning training, and obtain a fine-tuned student model. The data detection module is used to input the data to be detected into the finely tuned student model to obtain the final predicted label.

[0054] The same or similar labels correspond to the same or similar parts; The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent. Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. An aspect-level sentiment classification method based on multimodal alignment and reflective enhancement, characterized in that, include: Obtain the original dataset, which includes original images, original text, target aspect words and their corresponding true sentiment labels; The original image and pre-set knowledge-guided prompts are input into a pre-trained teacher model to obtain a text description; The text description, original text, target aspect words, and preset structured reasoning prompt template are concatenated to obtain a multimodal aligned enhanced input sequence; The enhanced input sequence is input into a pre-trained student model to obtain predicted sentiment labels; Based on the consistency between the predicted sentiment labels and the actual sentiment labels, erroneous samples are identified; Input all the error samples into the pre-trained teacher model to obtain the reflection-correction inference chain corresponding to each error sample; The erroneous samples and their corresponding reflection-correction inference chains, predicted sentiment labels, and true sentiment labels are combined to form a reflection-enhanced dataset, which is then merged with the original dataset to obtain a complete dataset. Input the complete dataset into the pre-trained student model, construct a loss function and perform supervised fine-tuning training to obtain a fine-tuned student model; The data to be tested is input into the finely tuned student model to obtain the final predicted label.

2. The aspect-level sentiment classification method based on multimodal alignment and reflective enhancement according to claim 1, characterized in that, The original image and pre-defined knowledge-guided prompts are input into a pre-trained teacher model to obtain a text description, including: TLM stands for pre-trained teacher model. For the k-th preset knowledge guidance prompt, For the i-th original image, This is the text description of the i-th original image.

3. The aspect-level sentiment classification method based on multimodal alignment and reflection enhancement according to claim 2, characterized in that, The text description, original text, target aspect words, and preset structured reasoning prompt template are concatenated to obtain a multimodal aligned enhanced input sequence, including: in, This serves as a structured reasoning prompt template. For the i-th target aspect word, For the i-th original text, Let be the i-th multimodal aligned enhanced input sequence.

4. The aspect-level sentiment classification method based on multimodal alignment and reflection enhancement according to claim 3, characterized in that, The enhanced input sequence is fed into a pre-trained student model to obtain predicted sentiment labels, including: SLM is a pre-trained student model. Let be the predicted sentiment label for the i-th augmented input sequence.

5. The aspect-level sentiment classification method based on multimodal alignment and reflection enhancement according to claim 4, characterized in that, Based on the consistency between the predicted sentiment label and the actual sentiment label, erroneous samples are identified, including: Samples whose predicted sentiment labels do not match the actual sentiment labels are filtered out to obtain a set of erroneous samples: in, This is an incorrect sample. For the sample with inconsistent label j, Let j be the predicted label for the sample with inconsistent label. Let be the true sentiment label of the j-th sample whose label is inconsistent.

6. The aspect-level sentiment classification method based on multimodal alignment and reflection enhancement according to claim 5, characterized in that, The erroneous samples are input into the pre-trained teacher model to obtain a reflection-correction inference chain, including: The expression for the reflection-correction inference chain is as follows: in, To reflect on the data, To correct the data, This is the j-th reflection-correction inference chain.

7. The aspect-level sentiment classification method based on multimodal alignment and reflection enhancement according to claim 6, characterized in that, The erroneous samples and their corresponding reflective-correction inference chains, predicted sentiment labels, and true sentiment labels are combined to form a reflective augmentation dataset, which is then merged with the original dataset to obtain a complete dataset, including: The expression for the reflection-enhanced dataset is as follows: in, To reflect on and revise the reasoning chain.

8. The aspect-level sentiment classification method based on multimodal alignment and reflection enhancement according to claim 7, characterized in that, The expression for the complete dataset is: in, For the original dataset, To enhance the dataset for reflection.

9. The aspect-level sentiment classification method based on multimodal alignment and reflection enhancement according to claim 1, characterized in that, The expression for the loss function is: in, Indicates the first position in the sequence One token, This represents all context tokens preceding this token.

10. An aspect-level sentiment classification system based on multimodal alignment and reflective enhancement, used to implement the aspect-level sentiment classification method based on multimodal alignment and reflective enhancement described in claims 1-9, characterized in that, include: The data acquisition module is used to acquire the raw dataset, which includes raw images, raw text, target aspect words and their corresponding real sentiment labels; The text description acquisition module is used to input the original image and preset knowledge guidance prompts into the pre-trained teacher model to obtain the text description; The enhanced input sequence acquisition module is used to concatenate the text description, original text, target aspect words and preset structured reasoning prompt template to obtain a multimodal aligned enhanced input sequence; The predicted sentiment label acquisition module is used to input the enhanced input sequence into a pre-trained student model to obtain the predicted sentiment label. The error sample identification module is used to identify error samples based on the consistency between the predicted sentiment label and the true sentiment label; The inference chain acquisition module is used to input all the error samples into the pre-trained teacher model to obtain the reflection-correction inference chain corresponding to each error sample; The complete dataset construction module is used to combine the erroneous samples and their corresponding reflection-correction inference chains, predicted sentiment labels and real sentiment labels into a reflection-enhanced dataset, and merge it with the original dataset to obtain a complete dataset. The model fine-tuning module is used to input the complete dataset into the pre-trained student model, construct a loss function for supervised fine-tuning training, and obtain a fine-tuned student model. The data detection module is used to input the data to be detected into the finely tuned student model to obtain the final predicted label.

Images