Multi-modal sentiment analysis method and system based on thinking chain and background knowledge

Abstract

The invention relates to a multi-modal sentiment analysis method and system based on a thinking chain and background knowledge. The method comprises the following steps: constructing a sample set containing original texts, related images, aspect items and sentiment polarities thereof; constructing a multi-modal sentiment analysis model, and generating background knowledge information by utilizing a visual language large model and combining an input text and an image; the generated background knowledge is screened, optimized and fused; for each sample in the sample set, in combination with the text data and the emotion polarity, generating a thinking chain thinking process and adding the thinking chain thinking process into the sample set, and then finely adjusting the visual language large model through the sample set added with the thinking chain thinking process; finally, the fused background knowledge, the text data and the picture data are sequentially input into the trained visual language large model, aspects in the text data are extracted, and the emotion polarity corresponding to the aspects is predicted. The method and the system are beneficial to improving the accuracy of multi-modal emotion polarity analysis.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of natural language processing, and particularly relates to a multi-modal sentiment analysis method and system based on thought chains and background knowledge. BACKGROUND

[0002] Sentiment analysis, also known as opinion mining, is a major branch of natural language processing. It is a process of processing and analyzing text with emotional color and then judging the emotional polarity. Unlike text-level sentiment analysis, aspect-level sentiment analysis shifts the target of expressing emotion from the entire document or sentence to an entity or an aspect of an entity. For example, when a customer buys clothes, the customer needs to understand the price, quality, after-sales service, and other aspects of the clothes, so it is necessary to analyze the sentiment of these aspects to meet the needs of the customer.

[0003] Traditional sentiment analysis mainly uses text data for sentiment classification and sentiment recognition. Multi-modal aspect sentiment analysis is a technology that combines multiple modal data (such as text, images, audio, etc.) for sentiment analysis. With the widespread use of social media and multimedia content, more and more information appears in the form of multi-modal, such as the combination of text and images in social media posts, audio comments, and videos. By integrating multiple data sources, information from different perspectives can be complemented to enhance sentiment analysis of specific aspects. For example, text information expresses emotions more vaguely, while image information has a more explicit emotional tendency, so the expressed emotions of the user can be more accurately judged. Conversely, the same is true. Therefore, the extraction and fusion of sentiment features of different modalities are crucial for the final sentiment word extraction and sentiment polarity prediction tasks.

[0004] In recent years, researchers use DNN methods such as CNN, RNN and LSTM to obtain feature representation of text, and with the development of attention mechanism, DNN methods and attention mechanism are combined to capture semantic information; but it ignores the syntactic relationship, so it starts to apply graph convolutional network in aspect-level sentiment analysis to help the model further identify sentiment information. With the gradual development of multi-modal data as the main form of data, capturing information related to aspects from different modalities (such as text and images) for sentiment analysis has gradually become the main research problem in this field. In order to better explore the relationship between text modalities and image modalities, researchers interact with image information in single-modal text sentiment analysis methods, although some results have been achieved. However, the above work still has certain limitations. For example, simply interacting text information and image information will introduce visual noise unrelated to the aspect, so in order to reduce the introduction of visual noise, the fine-grained information of the image should be fully considered and aligned with the text information; secondly, most of the existing multi-modal aspect-level sentiment analysis uses graph attention network to model the syntactic dependency and semantic relationship of text, this method has achieved remarkable results, but only considers the fine-grained relationship of text, that is, the word level, and pays more attention to the local features of the text, without fully considering the global features of the text.

[0005] The existing multi-modal sentiment analysis is to model the text information and image information respectively, and then fuse to obtain the final feature representation for sentiment analysis. Or the two modal information is spliced before obtaining the feature representation, and input into the pre-trained model for modeling. The above two methods have certain problems. If the image information and the text information are fused too late, the feature information of the two modalities cannot be deeply interacted, and if they are fused too early, the information of the two modalities will not be sensitive to the information of their own modalities. These two ways may cause the spread of false sentiment information, affecting the final sentiment prediction accuracy.

[0006] In summary, the graph attention network has achieved certain results in fusing text and image information, but there are still deficiencies in controlling image information noise, global features of text information and fusion of text and image information, and the global features of text and the fine-grained information of image are not fully utilized. SUMMARY

[0007] The purpose of the present application is to provide a multi-modal sentiment analysis method and system based on thought chain and background knowledge, which can improve the accuracy of multi-modal sentiment polarity analysis.

[0008] In order to achieve the above purpose, the technical scheme adopted by the present application is: a multi-modal sentiment analysis method based on thought chain and background knowledge, comprising the following steps:

[0009] Step A: Collect text and image data, identify and label aspects and their sentiment polarity in the text, and thus form a sample set TS containing the original text, related images, aspect items, and their sentiment polarity. The aspect items and sentiment polarity of each sample are used as the true value to train the model to recognize and understand the emotional expression in text and images.

[0010] Step B: Construct a multimodal sentiment analysis model. The multimodal sentiment analysis model uses a large visual language model to combine input text and images to generate background knowledge information. The generated background knowledge is filtered and optimized using text similarity, cross-modal attention mechanism, BM25 score, and CXMI score to obtain and fuse the filtered background knowledge.

[0011] Step C: For each sample in the sample set TS, combine the text data and sentiment polarity to generate a thought chain thinking process, add the thought chain thinking process to the sample set TS, and then fine-tune the visual language model based on the sample set TS with the thought chain thinking process added;

[0012] Step D: Input the integrated background knowledge, text data and image data into the trained visual language model in sequence, extract aspects from the text data, and predict the sentiment polarity corresponding to the aspect.

[0013] Furthermore, the step B specifically includes the following steps:

[0014] Step B1: Use the large language model to generate background knowledge prompt templates covering multiple perspectives. This is used to guide the large visual language model LLaVA to focus on semantic diversity and cross-modal consistency when generating background knowledge, providing structured guidance for subsequent multi-perspective background knowledge generation.

[0015] Step B2: For each sample in the sample set TS, combine the text data and the image data, and use the LLaVA model under the guidance of the background knowledge prompt template generated in step B1 to generate multi-perspective background knowledge BG; encode the generated background knowledge, original text data and image data respectively to obtain the background knowledge feature representation E respectively. BG , text feature representation E T and image feature representation E V ;

[0016] Step B3: Use the background knowledge feature representation E of each sample in the sample set TS obtained in step B2 BG Calculate the background knowledge-text relationship weight and the background knowledge-image relationship matrix, introduce the BM25 score and CXMI score, calculate the correlation between background knowledge and text and image, and obtain the fused background knowledge-image relationship matrix;

[0017] Step B4: Based on the fusion background knowledge-picture relationship matrix obtained in step B3, the K most relevant background knowledge are screened out;

[0018] Step B5: Based on the K background knowledge obtained in step B4, a gating mechanism is used to fuse the K background knowledge features to obtain a gated fusion background knowledge vector The gated fusion background knowledge vector is converted into fusion background knowledge BG using a BERT2Text model final .

[0019] Further, the specific implementation method of step B1 is:

[0020] A background knowledge prompt template covering multiple perspectives is generated using ChatGPT, and the final determined prompt template and the required perspectives are obtained:

[0021] Artistry: Identify and explore the art movements or styles that influenced the creation of the image; analyze the consistency or deviation between the artist's chosen style and the prevailing artistic trends at the time; Text: [x];

[0022] Biographical: Delve into the personal background related to the image and text; explore the life of the artist, author, or other relevant figures, and discuss how their life experiences have influenced the creation and interpretation of the work; Text: [x];

[0023] Personality: Focus on the characters in the image and text, analyze their personality traits, relationships, and possible development trajectories; explore how the image and text elements jointly shape the character image; Text: [x];

[0024] Cultural: Explore how the image and text reflect or present aspects of a particular culture; discuss the cultural significance, traditions, or values implied; Text: [x];

[0025] Environmental: Analyze the environmental elements in the image and text, and explore the relationship between ecological factors, environmental changes, or human activities and the depicted scene; Text: [x];

[0026] Historical: Examine the historical background of the image and text, and explore the historical events, historical periods, or historical figures that influenced their creation or interpretation; Text: [x];

[0027] Literary: Conduct a literary analysis of the text, exploring its themes, symbolic meanings, and narrative techniques; analyze how the text and image complement or contrast each other; Text: [x];

[0028] ​Political: Analyze the political context of the image and text creation period; explore political events, movements, or ideologies that may have influenced the content and tone of the work; Text: [x];

[0029] Scientific: Study the scientific elements in the image, exploring scientific discoveries, advancements, or breakthroughs related to the theme presented in the text; Text: [x];

[0030] Social: Analyze the social aspects in the image and text, discussing how social norms, cultural customs, or social structures influenced the content; Text: [x];

[0031] Financial: Examine the financial background of the image and text, discussing relevant economic events, trends, or financial factors that may have impacted the depicted theme; Text: [x];

[0032] Psychological: Study the psychological or emotional states portrayed in the scene, analyzing how these states drive character actions or shape the overall atmosphere; delve into psychological theories or concepts that may influence the interpretation of the work; Text: [x];

[0033] Philosophical: Explore fundamental questions or philosophical ideas presented in the work, such as the nature of existence, ethics, or the meaning of life; analyze how these concepts are embodied or challenged in the image or narrative; Text: [x];

[0034] Religious: Identify religious symbols, rituals, or references that may be present in the work; discuss how these elements embody specific doctrines or spiritual themes and influence the meaning or context of the work; Text: [x];

[0035] Moral: Analyze the moral or ethical implications embedded in the scene, including the moral dilemmas or judgments presented by characters or plot; consider the relationship between these issues and broader societal values or personal beliefs; Text: [x];

[0036] Stylistic: Analyze the unique stylistic elements employed in the image, such as composition, color palette, lighting, and visual motifs; assess how these choices enhance the overall aesthetic and thematic expression of the work; Text: [x];

[0037] Market-oriented: Study the marketing or promotional strategies that may have been involved in the creation or presentation of the work; discuss how target audiences, brand positioning, or commercial considerations influenced the final representation of the image or narrative; Text: [x];

[0038] where [x] is the text data in the sample set TS.

[0039] Further, the step B2 specifically comprises the following steps:

[0040] Step B21: For the text data T and picture data I of each sample in the sample set TS, the corresponding background knowledge BG is generated using the background knowledge prompt template in step B1 using the LLaVA model;

[0041] Step B22: Add the background knowledge BG generated in step B21 to the sample set, and the sample set is represented as TS={T, V, Artistic, Biographical, Character, Cultural, Environmental, Historical, Literary, Political, Scientific, Social, Financial, Psychological, Philosophical, Religious, Moral, Stylistic, Marketing}, where T represents text data, V represents picture data, and the remaining 17 parameters respectively represent background knowledge of 17 different categories: Artistic represents Artistic, Biographical represents biography, Character represents character features, Cultural represents culture, Environmental represents environment, Historical represents history, Literary represents literature, Political represents politics, Scientific represents science, Social represents society, Financial represents finance, Psychological represents psychology, Philosophical represents philosophy, Religious represents religion, Moral represents morality, Stylistic represents style, and Marketing represents marketing;

[0042] Step B23: The background knowledge BG is composed of multiple text segments, and the encoder of the pre-trained model BERT is used to encode the background knowledge BG to obtain the encoded background knowledge feature representation E BG :

[0043]

[0044] Wherein, is the vector representation of the nth token in the background knowledge BG i encoded by BERT; wherein i∈{1,2,...,17} represents the i-th background knowledge category, and BERT( ) is the BERT encoder; is the vector representation of the nth token in the background knowledge BG 17 encoded by BERT; n1 represents the number of tokens in the background knowledge;

[0045] Step B24: encode the text data T using the encoder of the pre-trained model BERT to obtain the encoded text feature representation E T :

[0046]

[0047] wherein, is the vector representation of the n2th token in the text data T, output by the BERT encoder; n2 represents the number of tokens in the text data, i.e., the number of words contained after the text is encoded;

[0048] Step B25: encode the picture data using the encoder of the pre-trained model CLIP to obtain the encoded image feature representation E v :

[0049]

[0050] wherein, is the i-th feature vector of the encoded image data V, output by the CLIP encoder, CLIP() is the CLIP encoder, and n3 represents the number of image feature vectors, i.e., the length of the vector sequence output by the CLIP encoder after encoding the image.

[0051] Further, the step B3 specifically comprises the following steps:

[0052] Step B31: calculate the similarity between the background knowledge feature representation E BG of each sample in the sample set TS obtained in step B2 and the text feature representation E T of each sample in the sample set TS to obtain the background knowledge-text relationship weight W BGT :

[0053]

[0054] Step B32: calculate the BM25 score based on the aspect A and the background knowledge BG in the sample set TS; first calculate the inverse document frequency IDF(A), and then calculate the BM25 score BM25(A, BG):

[0055]

[0056] wherein, IDF(A) is the inverse document frequency of the aspect A; N represents the total number of documents in the entire document set; DOC_NUM(A) represents the number of documents containing the aspect A; f(A, BG) represents the word frequency of the aspect A in the background knowledge BG; |BG| is the length of the background knowledge BG, AVG_BG is the average length of the corpus background knowledge; k1 and b are adjustment parameters;

[0057] Step B33: Based on the background knowledge-text relationship weight W obtained in B31 BGT and the aspect BM25 score obtained in step B32, the fusion background knowledge-text weight R is obtained by weighted fusion T :

[0058] R T = λ1W BGT + λ2BM25(A,BG)

[0059] wherein: λ1 and λ2 are adjustment parameters;

[0060] Step B34: The background knowledge feature representation E BG of each sample in the sample set TS obtained in step B2 is cross-modal attention fused with the image feature representation of each sample in the sample set TS, to obtain the background knowledge-image relationship matrix B I :

[0061] Q = W Q E V , K = W K E BG , V = W V E BG

[0062]

[0063] wherein Q, K, V are respectively the query vector, key vector and value vector in the attention mechanism; W Q , W K , W V are respectively the learnable parameter matrix of Q, K, V; d represents the dimension size in the attention mechanism;

[0064] Step B35: Based on the background knowledge feature representation E BG of each sample in the sample set TS obtained in step B2, the information entropy is calculated after the image feature representation E V , and then the CXMI score CXMI(E BG ,E V ) is calculated using the information entropy:

[0065] CXMI(E BG ,E V ) = H(E BG ) + H(E V ) - H(E BG ,E V )

[0066] wherein H(E BG ) represents the information entropy of E BG , H(E V ) represents the information entropy of EV the information entropy of E, H(E BG , E V ) represents the joint information entropy of both;

[0067] Step B36: Based on the background knowledge-picture relationship matrix B I obtained in step B34 and the CXMI score obtained in step B35, a fused background knowledge-picture relationship matrix R I is obtained by weighted fusion:

[0068] R I = λ3B I + λ4CXMI(E BG , E V )

[0069] wherein λ3 and λ4 are adjustment parameters.

[0070] Further, the step B4 specifically comprises the following steps:

[0071] Step B41: Based on the fused background knowledge-text weight R T obtained in step B33 and the fused background knowledge-picture relationship matrix R I obtained in step B36, the total weight of the background knowledge is calculated and normalized:

[0072] W BG = αR T + βR I

[0073]

[0074] wherein: α and β are adjustment parameters; W BG is the total weight; W′ BG is the normalized total weight; 1≤i≤N, wherein N is the total number of background knowledge;

[0075] Step B42: Based on the normalized total weight obtained in step B41, the K background knowledge most matching the current sample are selected:

[0076] S BG = Selected(W′ BG , K)

[0077] wherein S BG is the selected background knowledge; Selected is a selector for selecting the top K background knowledge instances with the highest relevance to the current sample from all the background knowledge according to the total weight.

[0078] Further, the step B5 specifically comprises the following steps:

[0079] Step B51: Based on the K background knowledge obtained in step B4, a gating mechanism is introduced to calculate the weight distribution of each background knowledge to control the importance of different background knowledge:

[0080]

[0081] Gate = σ(W G S BG +b G )

[0082] where e is the natural logarithm, W G is a trainable weight matrix, b G is a bias term, σ(x) is a Sigmoid activation function, x is the input parameter of the function, and Gate is the gating weight, which ensures that the gating weight is in the range [0, 1];

[0083] Step B52: Adjust the importance of background knowledge using the gating mechanism, perform nonlinear weight adjustment, and obtain the gated fusion background knowledge vector

[0084]

[0085] where: ⊙ represents Hadamard product, i.e. element-wise multiplication of two vectors;

[0086] Step B53: Use the BERT2Text model to convert the gated fusion background knowledge vector into the fusion background knowledge BG final .

[0087] Further, the step C specifically includes the following steps:

[0088] Step C1: Use a pre-trained large model to generate a thinking process for each sample, specifically: for each sample in the sample set TS, given the text data, aspect and sentiment polarity, design a prompt word to let the large model output a thinking process that analyzes the sentiment polarity of the text data and aspect, so that the model not only outputs the sentiment judgment result, but also explains the reasoning basis and logical chain; By guiding the model to explicitly express intermediate judgments in multiple reasoning steps, the model's understanding ability of complex semantic relationships is enhanced; The thinking process is in the form of text;

[0089] Step C2: Use the thinking process obtained in step C1 to splice the thinking process to the sample set TS;

[0090] Step C3: Fine-tuning the pre-trained LLaVA model using the thought process calculated in Step C1, computing the KL divergence while also computing the cross-entropy loss between the true values and the predicted values, calculating the gradients of the parameters in the fine-tuned LLaVA model using the backpropagation algorithm, and updating the parameters using the stochastic gradient descent algorithm. The training of the model fine-tuning is terminated when the loss value is less than a set threshold or the maximum number of iterations is reached.

[0091] Further, the step C1 specifically comprises the following steps:

[0092] Step C11: Design a prompt word to let the pre-trained large model output the thought process of the sentiment polarity obtained from the text data and aspect analysis; the designed prompt word template is as follows:

[0093] First, explain the overall semantic context of the text and how it narrates or describes the target aspect;

[0094] Next, combined with specific language expressions, explain how the text directly or indirectly implies the given sentiment label;

[0095] Highlight the emotional clues closely related to the target aspect, and distinguish between explicit and implicit emotional signals;

[0096] If there are factors that may lead to misunderstanding, such as ambiguity or irony, please clearly explain how you rule out these factors and confirm the sentiment label;

[0097] Finally, provide a clear and detailed summary explaining why the text explicitly expresses the given sentiment label,

[0098] Please give your reasoning process step by step and logically, without adding any additional subheadings or titles,

[0099] Your output should only contain the analytical reasoning process, i.e. the thought process;

[0100] Text: [x]; Target aspect: [y]; Sentiment label: [z];

[0101] Where [x] is the input text data, [y] is the input target aspect, and [z] is the input sentiment label;

[0102] Step C12: Use the pre-trained large model to output the thought process Q of the sentiment polarity obtained from the text data and aspect analysis for each sample in the sample set TS using the prompt word template designed in Step C11;

[0103] In the step C2, the thinking process Q obtained in the step C1 is spliced into the sample set TS, and the sample set is represented as TS={T, V, A, Q}; wherein, T, V and A are text data, picture data and aspects in the sample set respectively, and Q is the spliced thinking process;

[0104] The step C3 specifically comprises the following steps:

[0105] Step C31: calculating the KL divergence loss L KL :

[0106]

[0107] Wherein, P(i) is the thinking process of the i-th sample predicted in the fine-tuning process, and Q(i) is the thinking process obtained in the step C1.

[0108] Step C32: calculating the cross-entropy loss L between the predicted value and the true value CE :

[0109]

[0110] Wherein, y i represents the true value of the i-th sample, and p i represents the predicted value of the i-th sample.

[0111] Step C33: calculating the final total loss L based on the losses obtained in the step C31 and the step C32 total :

[0112] L total = mu1L KL + mu2L CE

[0113] Wherein: mu1 and mu2 are hyperparameters, used for adjusting the influence of the KL divergence loss and the cross-entropy loss.

[0114] Step C34: calculating the gradient of each parameter in the deep network by using the back propagation algorithm for the total loss function obtained in the step C43, and updating the parameters by using the stochastic gradient descent algorithm.

[0115] The application also provides a multi-modal sentiment analysis system based on a thinking chain and background knowledge, which comprises a memory, a processor and computer program instructions stored in the memory and capable of being run by the processor, and when the processor runs the computer program instructions, the above method can be realized.

[0116] Compared with the prior art, the present application has the following beneficial effects: firstly, the present application is aimed at the modal alignment problem between different modalities, without introducing the information of the whole image, but using fine-grained image and fine-grained text for similarity calculation, and combining component tree to obtain different granularity of text, and obtaining multi-granularity text-visual fusion feature representation through a multi-layer graph attention network, so as to realize the alignment of image modalities and text modalities, and reduce the introduction of irrelevant noise and redundant information. Secondly, the present application introduces image labels and AMR graphs as external knowledge for images and texts respectively, so that the model can fully explore the semantic information of images and texts by using multi-source knowledge of images and texts, and obtain knowledge-enhanced feature representation. Finally, in order to further reduce the semantic gap between image modalities and text modalities, the present application introduces image-text contrast learning, fully utilizes multi-granularity text-visual fusion feature representation, knowledge-enhanced text feature representation and knowledge-enhanced image feature representation, narrows the distance between semantic information of different modalities in the same sample, and helps the model to learn the correlation and consistency between different modalities, so as to improve the aspect extraction and aspect sentiment polarity prediction precision.

[0117] Firstly, the present application constructs a sample set TS containing original text, related image, aspect item and its sentiment polarity, then constructs a multi-modal sentiment analysis model, generates background knowledge information by combining input text and image by using a visual language large model, and then screens and optimizes the generated background knowledge by using text similarity, cross-modal attention mechanism, BM25 score and CXMI score, and then fuses the background knowledge, and then generates a thinking chain thinking process by combining text data and sentiment polarity, and adds the thinking chain thinking process to the sample set TS, and finally fine-tunes the visual language large model by using the sample set TS added with the thinking chain thinking process, so that the accuracy of the visual language large model in performing multi-modal sentiment polarity analysis is greatly improved. BRIEF DESCRIPTION OF DRAWINGS

[0118] Figure 1 is the implementation flowchart of the multi-modal sentiment analysis method based on the thinking chain and the background knowledge provided by the embodiment of the present application;

[0119] Figure 2 is the implementation principle diagram of the multi-modal sentiment analysis method based on the thinking chain and the background knowledge provided by the embodiment of the present application. DETAILED DESCRIPTION

[0120] The present application will be further described below in combination with the drawings and embodiments.

[0121] It should be pointed out that the following detailed description is exemplary and is intended to provide further description of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0122] It is to be noted that the terms used herein are only intended to describe specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise, and it should also be understood that the terms "comprise" and / or "include" when used in this specification, indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0123] As shown in Figures 1-2 , the present embodiment provides a multi-modal sentiment analysis method and system based on thought chains and background knowledge, including the following steps:

[0124] Step A: Collecting text data and picture data, identifying and labeling aspects and their sentiment polarity in the text, thereby forming a sample set TS containing original text, related images, aspect items and their sentiment polarity, wherein the aspect items and sentiment polarity of each sample are used as true values to train the model to identify and understand the sentiment expression in the text and image;

[0125] Step B: Constructing a multi-modal sentiment analysis model based on large model generation, screening and fusion of multi-dimensional background knowledge, the multi-modal sentiment analysis model uses a visual language large model LLaVA to generate background knowledge information in combination with input text and images; and uses text similarity, cross-modal attention mechanism, BM25 score and CXMI score to screen and optimize the generated background knowledge, to obtain screened background knowledge and perform fusion;

[0126] Step C: For each sample in the sample set TS, generate a thought chain thinking process in combination with the text data and sentiment polarity, add the thought chain thinking process to the sample set TS, and fine-tune the LLaVA model through the sample set TS with the added thought chain thinking process;

[0127] Step D: Inputting the fused background knowledge, text data and picture data into the trained LLaVA model in sequence, extracting the aspects in the text data, and predicting the corresponding sentiment polarity of the aspects.

[0128] Figure 2 is the implementation principle diagram of the multi-modal sentiment analysis method based on thought chains and background knowledge provided by the present embodiment. As shown in Figure 2 , the specific implementation steps of step B are as follows.

[0129] Step B1: Use a large language model such as ChatGPT to generate a background knowledge prompt template covering multiple perspectives, which is used to guide the visual language large model LLaVA to focus on semantic diversity and cross-modal consistency when generating background knowledge, providing structured guidance for subsequent multi-perspective background knowledge generation.

[0130] In this embodiment, the specific implementation method of step B1 is:

[0131] Generate a background knowledge prompt template covering multiple perspectives using ChatGPT, and obtain the final determined prompt template and the required perspectives:

[0132] "Artistic":"Identify and discuss any artistic movements or stylesthat influenced the creation of the image.Explore how the artist's choice ofstyle aligns with or deviates from prevalent artistic trends of thetime.Sentence:[x]",

[0133] "Biographical":"Delve into the backgrounds of individuals associatedwith the image and text.Explore the biographies of artists,authors,or otherrelevant figures,and discuss how their life experiences shaped the creationand interpretation of the work.Sentence:[x]",

[0134] "Character":"Focus on characters within the image andsentence.Analyze their personalities,relationships,and potential characterdevelopment.Discuss how the visual and textual elements contribute tocharacter portrayal.Sentence:[x]",

[0135] "Cultural":"Explore how the image and sentence reflect or representaspects of a particular culture Discuss the cultural significance,traditions,or values implied by the elements in the image and sentence.Sentence:[x]",

[0136] "Environmental":"Examine the environmental elements within the imageand sentence,discussing ecological factors,environmental changes,or therelationship between human activities and the depicted setting.Sentence:[x]",

[0137] "Historical":"Investigate the historical context of the image andsentence,discussing events,periods,or historical figures that influenced thecreation or interpretation of the work.Sentence:[x]",

[0138] "Literary":"Conduct a literary analysis of the sentence,exploringthemes,symbolism,and narrative techniques.Discuss how the words complement orcontrast with the visual elements in the image.Sentence:[x]",

[0139] "Political": Examine the political during the time the image and text were created. Discuss any political events, movements, or ideologies that may have influenced the content and tone of the work. Sentence: [x]

[0140] "Scientific": Investigate the scientific elements within the image, delving into discoveries, advancements, or breakthroughs related to the subject matter mentioned in the sentence. Sentence: [x]

[0141] "Social": Explore the social aspects of the image and sentence, discussing societal norms, cultural practices, or the impact of social structures on the depicted content. Sentence: [x]

[0142] "Financial": Examine the financial context of the image and sentence, discussing economic events, trends, or the influence of financial factors on the depicted subject matter. Sentence: [x]

[0143] "Psychological": "Investigate the mental or emotional states depicted in the scene, analyzing how they drive the characters' actions or shape the overall atmosphere. Delve into underlying psychological theories or concepts that may influence the interpretation of the work. Sentence: [x]",

[0144] "Philosophical": "Explore the fundamental questions or philosophical ideas presented by the work, such as the nature of existence, ethics, or the meaning of life. Examine how these concepts are embodied or challenged within the image or narrative. Sentence: [x]",

[0145] "Religious": "Identify any religious symbolism, rituals, or references present in the work. Discuss how these elements reflect particular doctrines or spiritual themes and the ways they inform the message or context of the image. Sentence: [x]",

[0146] "Moral":"Analyze the moral or ethical implications depicted in the scene,including any dilemmas or judgments the characters or settingsuggest.Consider how these issues relate to broader societal values ​​or personal beliefs.Sentence:[x]",

[0147] "Stylistic":"Examine the distinctive stylistic elements used in the image, such as composition, color palette, lighting, and visual motifs. Assess how these choices contribute to the overall aesthetic and thematic impact of the work. Sentence: [x]",

[0148] "Marketing":"Investigate any marketing or promotional strategies tied to the creation or presentation of the work.Discuss how target audiences,branding,or commercial considerations may have shaped the final image ornarrative.Sentence:[x]"

[0149] The corresponding Chinese prompt template and the required perspectives are:

[0150] Artistic: Identify and discuss the artistic schools or styles that were influenced in the creation of this image; analyze the consistency or deviation between the artist's chosen style and the prevailing artistic trends of the time; Text: [x];

[0151] Biographical: Exploring the personal context surrounding the image and text; exploring the biography of the artist, author, or other relevant figure, and discussing how their life experiences influenced the creation and interpretation of the work; Text: [x];

[0152] Character: Focus on the characters in the image and text, analyze their personality traits, relationships, and possible development trajectories; explore how the image and textual elements work together to shape the character's image; Text: [x];

[0153] Cultural: Explore how the image and text reflect or represent aspects of a particular culture; discuss the cultural significance, traditions, or values implied; Text: [x];

[0154] Environmental: Analyze the environmental elements in the image and text, explore the relationship between ecological factors, environmental changes, or human activities and the depicted scene; Text: [x];

[0155] Historical: Examine the historical context of the image and text, explore the historical events, periods, or figures that influenced their creation or interpretation; Text: [x];

[0156] Literary: Conduct a literary analysis of the text, exploring its themes, symbolic meanings, and narrative techniques; analyze how the words and images complement or contrast with each other; Text: [x];

[0157] Political: Analyze the political context of the image and text's creation; explore political events, movements, or ideologies that may have influenced the content and tone of the work; Text: [x];

[0158] Scientific: Investigate the scientific elements in the image, explore scientific discoveries, advancements, or breakthroughs related to the themes presented in the text; Text: [x];

[0159] Social: Analyze the social aspects in the image and text, explore how social norms, cultural customs, or social structures influence the content; Text: [x];

[0160] Financial: Examine the financial context of the image and text, discuss relevant economic events, trends, or financial factors that may have influenced the depicted subject matter; Text: [x];

[0161] Psychological: Study the psychological or emotional states portrayed in the scene, analyze how these states drive character behavior or shape the overall atmosphere; delve into psychological theories or concepts that may influence the interpretation of the work; Text: [x];

[0162] Philosophical: Explore the fundamental questions or philosophical ideas presented in the work, such as the nature of existence, ethics, or the meaning of life; analyze how these concepts are embodied or questioned in the image or narrative; Text: [x];

[0163] Religious: Identify any religious symbolism, rituals, or references in the work; explore how these elements embody specific doctrines or spiritual themes and influence the meaning or context of the work; Text: [x];

[0164] Moral: Analyze the moral or ethical implications contained in the scene, including the moral dilemmas or judgments presented by characters or plot; consider the relationship between these issues and broader social values or personal beliefs; Text: [x];

[0165] Style: Analyze the unique stylistic elements employed in the image, such as composition, color palette, lighting treatment, and visual motifs; evaluate how these choices enhance the overall aesthetics and thematic expression of the work; Text: [x];

[0166] Marketability: Study the marketing or promotion strategies that may be involved in the creation or presentation of the work; discuss how the target audience, brand positioning, or commercial considerations influence the final presentation of the image or narrative; Text: [x];

[0167] where [x] is the text data in the sample set TS.

[0168] Step B2: For each sample in the sample set TS, generate multi-perspective background knowledge BG using the LLaVA model guided by the background knowledge prompt template generated in step B1, combining text data and picture data; encode the generated background knowledge, original text data, and picture data respectively to obtain background knowledge feature representation E BG , text feature representation E T , and image feature representation E V .

[0169] In this embodiment, step B2 specifically includes the following steps:

[0170] Step B21: For the text data T and picture data I of each sample in the sample set TS, use the LLaVA model to generate the corresponding background knowledge BG using the background knowledge prompt template in step B1.

[0171] Step B22: Add the background knowledge BG generated in step B21 to the sample set, and the sample set is represented as TS={T, V, Artistic, Biographical, Character, Cultural, Environmental, Historical, Literary, Political, Scientific, Social, Financial, Psychological, Philosophical, Religious, Moral, Stylistic, Marketing}, where T represents text data, V represents picture data, and the remaining 17 parameters respectively represent background knowledge of 17 different categories: Artistic represents Artistic, Biographical represents biography, Character represents character features, Cultural represents culture, Environmental represents environment, Historical represents history, Literary represents literature, Political represents politics, Scientific represents science, Social represents society, Financial represents finance, Psychological represents psychology, Philosophical represents philosophy, Religious represents religion, Moral represents morality, Stylistic represents style, and Marketing represents marketing.

[0172] Step B23: The background knowledge BG is composed of multiple text segments, and the encoder of the pre-trained model BERT is used to encode the background knowledge BG to obtain the encoded background knowledge feature representation E BG :

[0173]

[0174] wherein, is the background knowledge BG of the i-th category i vector sequence after BERT encoding; wherein i∈{1,2,...,17} represents the i-th background knowledge category, and BERT( ) is the BERT encoder; is the background knowledge BG of the 17th category 17 vector representation of the n1th token in the background knowledge, output by the BERT encoder; n1 represents the number of tokens in the background knowledge.

[0175] Step B24: Use the encoder of the pre-trained model BERT to encode the text data T to obtain the encoded text feature representation E T :

[0176]

[0177] wherein, is the vector representation of the n2th token in the text data T, output by the BERT encoder; n2 represents the number of tokens in the text data, i.e. the number of words contained after the text is encoded.

[0178] Step B25: encode the picture data using the encoder of the pre-trained model CLIP to obtain the encoded image feature representation E v :

[0179]

[0180] wherein, is the i-th feature vector of the image data V after encoding, output by the CLIP encoder, CLIP() is the CLIP encoder, and n3 represents the number of image feature vectors, i.e. the length of the vector sequence output by the CLIP encoder after encoding the image.

[0181] Step B3: utilize the background knowledge feature representation E BG calculate the background knowledge-text relationship weight and the background knowledge-picture relationship matrix, and introduce the BM25 score and the mutual information score (CXMI) to more comprehensively depict the correlation between the background knowledge and the text and the image, to obtain the fusion background knowledge-picture relationship matrix.

[0182] In this embodiment, the step B3 specifically comprises the following steps:

[0183] Step B31: utilize the background knowledge feature representation E BG and the text feature representation E T of each sample in the sample set TS obtained in step B2 to calculate the similarity, to obtain the background knowledge-text relationship weight W BGT :

[0184]

[0185] Step B32: calculate the BM25 score based on the aspect A and the background knowledge BG in the sample set TS; first calculate the inverse document frequency IDF(A), and then calculate the BM25 score BM25(A, BG):

[0186]

[0187] Wherein, IDF(A) is the inverse document frequency of aspect A; N represents the total number of documents in the entire document set; DOC_NUM(A) represents the number of documents containing aspect A, which is used to measure the universality of aspect A in the document set; f(A, BG) represents the word frequency of aspect A in the background knowledge BG; |BG| is the length of the background knowledge BG, AVG_BG is the average length of the corpus background knowledge; k1 and b are adjustment parameters.

[0188] Step B33: based on the background knowledge-text relationship weight W obtained in B31 BGT And the BM25 score of the aspect obtained in step B32, the fusion background knowledge-text weight R is obtained by weighted fusion T :

[0189] R T =λ1W BGT +λ2BM25(A,BG)

[0190] Wherein: λ1 and λ2 are adjustment parameters.

[0191] Step B34: the background knowledge feature representation E BG Of each sample in the sample set TS obtained in step B2 is cross-modal attention fusion with the picture feature representation of each sample in the sample set TS, to obtain the background knowledge-picture relationship matrix B I :

[0192] Q=W q E V ,K=W K E BG ,V=W V E BG

[0193]

[0194] Wherein, Q, K, V are respectively the query vector, key vector and value vector in the attention mechanism; W Q , W K , W V are respectively the learnable parameter matrix of Q, K, V; d represents the dimension size in the attention mechanism.

[0195] Step B35: based on the background knowledge feature representation E BG Of each sample in the sample set TS obtained in step B2 V , the information entropy is calculated and then the CXMI score CXMI(E BG ,E V ) is calculated:

[0196] CXMI(E BG ,EV )=H(E BG )+H(E V )-H(E BG ,E V )

[0197] Among them, H(E BG ) represents E BG Information entropy, H(E V ) represents E V Information entropy, H(E BG ,E V ) represents the joint information entropy of the two.

[0198] Step B36: Background knowledge-image relationship matrix B obtained based on B34 I The CXMI score obtained in step B35 is weighted fused to obtain the fusion background knowledge-image relationship matrix R I :

[0199] R I =λ3B I +λ4CXMI(E BG ,E V )

[0200] Among them, λ3 and λ4 are adjustment parameters.

[0201] Step B4: Based on the fused background knowledge-image relationship matrix obtained in step B3, the most relevant K background knowledge are screened out.

[0202] In this embodiment, step B4 specifically includes the following steps:

[0203] Step B41: Based on the fusion background knowledge obtained in step B33 - text weight R T and the fusion background knowledge-image relationship matrix R obtained in step B36 I , calculate the total weight of background knowledge and normalize it:

[0204] W BG =αR T +βR I

[0205]

[0206] Among them: α and β are adjustment parameters; W BG is the total weight; W B ' G is the normalized total weight; 1≤i≤N, where N is the total amount of background knowledge.

[0207] Step B42: Based on the normalized total weight obtained in step B41, select the K background knowledge that best matches the current sample:

[0208] S BG =Selected(W′ BG ,K)

[0209] Among them, S BG is the filtered background knowledge; Selected is a selector used to select the top K background knowledge instances with the highest relevance to the current sample from all background knowledge according to the total weight.

[0210] Step B5: Based on the K background knowledge obtained in step B4, a gating mechanism is used to fuse the K background knowledge features to obtain a gated fusion background knowledge vector Use the BERT2Text model to integrate the gated background knowledge vector Transformed into integrated background knowledge BG final .

[0211] In this embodiment, step B5 specifically includes the following steps:

[0212] Step B51: Based on the K pieces of background knowledge obtained in step B4, a gating mechanism is introduced to calculate the weight distribution of each piece of background knowledge to control the importance of different pieces of background knowledge:

[0213]

[0214] Gate=σ(W G S BG +b G )

[0215] Where, e is the natural logarithm, W G is the trainable weight matrix, b G is the bias term, σ(x) is the Sigmoid activation function, x is the input parameter of the function, ensuring that the gating weight value range is between [0,1], and Gate is the gating weight.

[0216] Step B52: Use the gating mechanism to adjust the importance of background knowledge, perform nonlinear weight adjustment, and obtain the gated fusion background knowledge vector

[0217]

[0218] Where: ⊙ represents the Hadamard product, which is the element-by-element multiplication of two vectors.

[0219] Step B53: Use the BERT2Text model to gate the fusion background knowledge vector Transforming into fusion background knowledge BG final .

[0220] As Figure 2 shown, the specific implementation steps of step C are as follows.

[0221] Step C1: Use a pre-trained large model (such as QWQ, etc.) to generate the thinking process of each sample, specifically: for each sample in the sample set TS, given the text data, aspect and sentiment polarity, design a prompt word to let the large model output the thinking process of the sentiment polarity obtained by analyzing the text data and aspect, so that the model can not only output the sentiment judgment result, but also explain the reasoning basis and logical chain; by guiding the model to explicitly express intermediate judgments in multiple reasoning steps, the understanding ability of the model for complex semantic relationships is enhanced, and the interpretability and robustness of the sentiment recognition task are improved; the thinking process is in the form of text.

[0222] In this embodiment, step C1 specifically includes the following steps:

[0223] Step C11: Design a prompt word to let the pre-trained large model output the thinking process of the sentiment polarity obtained by analyzing the text data and aspect, and the thinking process is a text output in the form of natural language, rather than hidden layer representation or vector form, specifically a coherent text or multiple coherent texts, simulating the thinking path of human emotion judgment; the designed prompt word template is as follows:

[0224] "First, explain the overall semantic context of the text and how itnarrates or describes the target aspect."

[0225] "Then, use specific linguistic expressions to illustrate how the textdirectly or indirectly suggests the provided sentiment label."

[0226] "Highlight particularly any emotional cues closely associated withthe target aspect in the text,distinguishing between explicit and subtlesentiment signals."

[0227] "If there are potential sources of misunderstanding (such as ambiguity or irony), clearly explain how you eliminated these factors and confirmed thesentiment label."

[0228] "Finally,provide a clear,detailed summary explaining why the textdistinctly conveys the given sentiment label."

[0229] "Provide your reasoning clearly and step by step, without any additional headings or titles."

[0230] "Your output should contain only your analytical reasoning process(thinking)."

[0231] "text:[x].target aspect:[y].sentiment label:[z]"

[0232] The corresponding Chinese prompt word template is as follows:

[0233] First, explain the overall semantic context of the text and how it narrates or describes the target aspect;

[0234] Next, using specific language expressions, explain how the text directly or indirectly implies the given emotional label;

[0235] Focus on the emotional cues that are closely related to the target aspect and distinguish between explicit and implicit emotional signals;

[0236] If there are factors that could lead to misunderstanding, such as ambiguity or irony, clearly explain how you eliminated these factors and confirmed the emotional label;

[0237] Finally, provide a clear and detailed summary explaining why the text clearly expresses the given sentiment label.

[0238] Please follow the step-by-step logical reasoning process, and do not add any additional headers or titles.

[0239] Your output should only contain the analytical reasoning process, i.e., the thought process.

[0240] Text: [x]; Target Aspect: [y]; Sentiment Label: [z].

[0241] Where [x] is the input text data, [y] is the input target aspect, and [z] is the input sentiment label.

[0242] Step C12: Use the pre-trained large model combined with the prompt word template designed in step C11 to output the thought process Q for each sample in the sample set TS, which includes text data and aspect analysis and sentiment polarity.

[0243] An example is given: the text of a certain sample is "The service attitude is very good, but the food taste is general", the target aspect is "service quality", and the expected sentiment polarity is "positive". The model generates the following thought process under the guidance of the prompt:

[0244] "First, analyze the description of'service quality' in the text. The phrase'service attitude is very good' is closely related to service quality, indicating that the customer has a positive evaluation of the service process. In contrast, 'food taste is general' is not related to service quality and is more related to 'food quality'. Therefore, in terms of'service quality', the sentiment tendency is positive."

[0245] Step C2: Use the thought process obtained in step C1 to splice the thought process into the sample set TS.

[0246] Where the sample set is represented as TS={T,V,A,Q}; where T, V, and A are the text data, image data, and aspects in the sample set, and Q is the spliced thought process.

[0247] Step C3: Fine-tune the pre-trained LLaVA model, calculate the KL divergence using the thought process obtained in step C1, calculate the cross-entropy loss of the true value and the predicted value, use the backpropagation algorithm to calculate the gradient of each parameter in the fine-tuned LLaVA model, and use the stochastic gradient descent algorithm to update the parameters. When the loss value is less than the set threshold or the maximum number of iterations is reached, terminate the training of the model fine-tuning.

[0248] In this embodiment, step C3 specifically includes the following steps:

[0249] Step C31: Calculate the KL divergence loss L KL :

[0250]

[0251] wherein P(i) is the thinking process of the i-th sample predicted in the fine-tuning process, and Q(i) is the thinking process obtained in step C1;

[0252] Step C32: Calculate the cross-entropy loss K of the predicted value and the true value CE

[0253]

[0254] wherein y i represents the true value of the i-th sample, and p i represents the predicted value of the i-th sample;

[0255] Step C33: Calculate the final total loss L based on the loss obtained in step C31 and step C32 total

[0256] L total = μ1L KL + μ2L CE

[0257] wherein μ1 and μ2 are hyperparameters for adjusting the influence of the KL divergence loss and the cross-entropy loss;

[0258] Step C34: Calculate the gradient of each parameter in the deep network using the back propagation algorithm based on the total loss function obtained in step C43, and update the parameters using the stochastic gradient descent algorithm.

[0259] In this embodiment, the performance of the multi-modal sentiment analysis model provided by the method is compared with that of other baseline models in the application of twitter2015 and twitter2017 data sets, and the results are shown in Table 1. As can be seen from Table 1, the performance of the model provided by the method is better than that of other models.

[0260] Table 1 Comparison of performance of the model of the method and the baseline model on twitter2015 and twitter2017 data sets

[0261]

[0262] The embodiment also provides a multi-modal sentiment analysis system based on thinking chain and background knowledge, which comprises a memory, a processor and computer program instructions stored in the memory and capable of being executed by the processor, and when the processor executes the computer program instructions, the above-mentioned method can be realized.

[0263] ​​Those skilled in the art will appreciate that embodiments of the application can be readily used as software, hardware, or a combination of software and hardware. In one

[0264] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or a combination thereof. These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 an apparatus to perform functions specified in the flowchart block or blocks.

[0265] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or a combination thereof. These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 an apparatus to perform functions specified in the flowchart block or blocks.

[0266] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or a combination thereof. These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 Figure 1 an apparatus to perform functions specified in the flowchart block or blocks.

[0267] The above descriptions are only preferred embodiments of the present application, and are not intended to limit the present application to other forms. Any person skilled in the art can make modifications or alterations to the above-described embodiments without departing from the technical scope of the present application. Any simple modifications, equivalent changes and alterations made to the above-described embodiments based on the technical essence of the present application shall fall within the scope of protection of the present application.

Claims

1. A multimodal sentiment analysis method based on thought chain and background knowledge, characterized in that: The following steps are involved: Step A: Collect text and image data, identify and label aspects and their sentiment polarity in the text, and thus form a sample set TS containing the original text, related images, aspect items, and their sentiment polarity. The aspect items and sentiment polarity of each sample are used as the true value to train the model to recognize and understand the emotional expression in text and images. Step B: Construct a multimodal sentiment analysis model. The multimodal sentiment analysis model uses a large visual language model to combine input text and images to generate background knowledge information. The generated background knowledge is filtered and optimized using text similarity, cross-modal attention mechanism, BM25 score, and CXMI score to obtain and fuse the filtered background knowledge. Step C: For each sample in the sample set TS, combine the text data and sentiment polarity to generate a thought chain thinking process, add the thought chain thinking process to the sample set TS, and then fine-tune the visual language model based on the sample set TS with the thought chain thinking process added; Step D: Input the integrated background knowledge, text data and image data into the trained visual language model in sequence, extract aspects from the text data, and predict the sentiment polarity corresponding to the aspect.

2. The multimodal sentiment analysis method based on thought chain and background knowledge according to claim 1 is characterized in that: The step B specifically comprises the following steps: Step B1: Use the large language model to generate background knowledge prompt templates covering multiple perspectives. This is used to guide the large visual language model LLaVA to focus on semantic diversity and cross-modal consistency when generating background knowledge, providing structured guidance for subsequent multi-perspective background knowledge generation. Step B2: For each sample in the sample set TS, combine the text data and the image data, and use the LLaVA model under the guidance of the background knowledge prompt template generated in step B1 to generate multi-perspective background knowledge BG; encode the generated background knowledge, original text data and image data respectively to obtain the background knowledge feature representation E respectively. BG , text feature representation E T and image feature representation E V ; Step B3: Use the background knowledge feature representation E of each sample in the sample set TS obtained in step B2 BG Calculate the background knowledge-text relationship weight and the background knowledge-image relationship matrix, introduce the BM25 score and CXMI score, calculate the correlation between background knowledge and text and image, and obtain the fused background knowledge-image relationship matrix; Step B4: Based on the fused background knowledge-image relationship matrix obtained in step B3, select the most relevant K pieces of background knowledge; Step B5: Based on the K background knowledge obtained in step B4, a gating mechanism is used to fuse the K background knowledge features to obtain a gated fusion background knowledge vector Use the BERT2Text model to integrate the gated background knowledge vector Transformed into integrated background knowledge BG final .

3. The multimodal sentiment analysis method based on thought chain and background knowledge according to claim 2 is characterized in that: The specific implementation method of step B1 is: Use ChatGPT to generate background knowledge prompt templates covering multiple perspectives, and obtain the final prompt template and the required perspectives: Artistic: Identify and discuss the artistic schools or styles that were influenced in the creation of this image; analyze the consistency or deviation between the artist's chosen style and the prevailing artistic trends of the time; Text: [x]; Biographical: Exploring the personal context surrounding the image and text; exploring the biography of the artist, author, or other relevant figure, and discussing how their life experiences influenced the creation and interpretation of the work; Text: [x]; Character: Focus on the characters in the image and text, analyze their personality traits, relationships, and possible development trajectories; explore how image and text elements work together to shape the character image; Text: [x]; Culturality: Explores how images and texts reflect or represent aspects of a particular culture; discusses the cultural meanings, traditions, or values ​​implied; Text: [x]; Environmental: Analyze environmental elements in images and texts to explore the relationship between ecological factors, environmental changes, or human activities and the depicted scenes; Text: [x]; Historicity: Examining the historical context of images and texts, exploring historical events, periods, or figures that influenced their creation or interpretation; Text: [x]; Literary: Conduct literary analysis of the text, exploring its themes, symbolism, and narrative techniques; analyze how words and images complement or contrast with each other; Text: [x]; Political: Analyze the political context of the period in which the image and text were created; Explore political events, movements, or ideologies that may have influenced the content and tone of the work; Text: [x]; Scientific: Examine the scientific elements in the image and explore scientific discoveries, advances, or breakthroughs related to the topic raised in the text; Text: [x]; Sociality: Analyze the social aspects of images and texts, exploring the influence of social norms, cultural practices, or social structures on the content; Text: [x]; Financial: Examine the financial context of the image and text, and discuss how relevant economic events, trends, or financial factors may influence the depicted subject matter; Text: [x]; Psychological: Study the psychological or emotional states expressed in the scene and analyze how these drive character behavior or shape the overall atmosphere; delve into psychological theories or concepts that may influence the interpretation of the work; Text: [x]; Philosophical: Explores fundamental questions or philosophical ideas presented in the work, such as the nature of existence, ethics, or the meaning of life; analyzes how these ideas are reflected or questioned in the image or narrative; Text: [x]; Religiousness: Identify any religious symbols, rituals, or references that may be present in the work; discuss how these elements embody specific doctrines or spiritual themes and influence the meaning or context of the work; Text: [x]; Morality: Analyze the moral or ethical implications of the scene, including the moral dilemmas or judgments presented by the characters or plot; consider how these issues relate to broader societal values ​​or personal beliefs; Text: [x]; Stylistics: Analyze the unique stylistic elements used in the image, such as composition, color matching, lighting and shadow treatment, and visual motifs; Evaluate how these choices enhance the overall aesthetic and thematic expression of the work; text: [x]; Marketability: Research into the marketing or promotional strategies that may have been involved in the creation or presentation of the work; discuss how target audience, brand positioning, or commercial considerations may have influenced the final presentation of the image or narrative; Text: [x]; Where [x] is the text data in the sample set TS.

4. The multimodal sentiment analysis method based on thought chain and background knowledge according to claim 2 is characterized in that: The step B2 specifically includes the following steps: Step B21: For the text data T and image data I of each sample in the sample set TS, use the LLaVA model to generate the corresponding background knowledge BG using the background knowledge prompt template in step B1; Step B22: Add the background knowledge BG generated in step B21 to the sample set. The sample set is represented as TS = {T, V, Artistic, Biographical, Character, Cultural, Environmental, Historical, Literary, Political, Scientific, Social, Financial, Psychological, Philosophical, Religious, Moral, Stylistic, Marketing}, where T represents text data, V represents image data, and the remaining 17 parameters represent 17 different categories of background knowledge: Artistic represents Artistic, Biographical represents biography, Character represents character characteristics, Cultural represents culture, Environmental represents environment, Historical represents history, Literary represents literature, Political represents politics, Scientific represents science, Social represents society, Financial represents finance, Psychological represents psychology, Philosophical represents philosophy, Religious represents religion, Moral represents morality, Stylistic represents style, and Marketing represents marketing; Step B23: The background knowledge BG consists of multiple text fragments. The background knowledge BG is encoded using the encoder of the pre-trained model BERT to obtain the encoded background knowledge feature representation E BG : ………… in, is the background knowledge BG of the i-th category i The vector sequence after BERT encoding; where i∈{1,2,...,17} represents the i-th background knowledge category, and BERT() is the BERT encoder; Background knowledge for the 17th category 17 The vector representation of the n1th token in is output by the BERT encoder; n1 represents the number of tokens in the background knowledge; Step B24: Use the encoder of the pre-trained model BERT to encode the text data T and obtain the encoded text feature representation E T : in, is the vector representation of the n2th token in the text data T, output by the BERT encoder; n2 represents the number of tokens in the text data, that is, the number of words contained in the text after being encoded; Step B25: Use the encoder of the pre-trained model CLIP to encode the image data and obtain the encoded image feature representation E v : in, is the i-th feature vector after the image data V is encoded, output by the CLIP encoder, CLIP() is the CLIP encoder, and n3 represents the number of image feature vectors, that is, the length of the vector sequence output by the CLIP encoder after encoding the image.

5. The multimodal sentiment analysis method based on thought chain and background knowledge according to claim 2 is characterized in that: The step B3 specifically includes the following steps: Step B31: Represent the background knowledge feature E of each sample in the sample set TS obtained in step B2 BG And the text feature representation E of each sample in the sample set TS T Calculate the similarity and obtain the background knowledge-text relationship weight W BGT : Step B32: Calculate the BM25 score based on the aspect A and background knowledge BG in the sample set TS; first calculate the inverse document frequency IDF(A), and then calculate the BM25 score BM25(A, BG): Where, IDF(A) is the inverse document frequency of aspect A; N is the total number of documents in the entire document set; DOC_NUM(A) is the number of documents containing aspect A; f(A,BG) is the word frequency of aspect A in the background knowledge BG; |BG| is the length of the background knowledge BG, AVG_BG is the average length of the background knowledge in the corpus; k1 and b are adjustment parameters; Step B33: Background knowledge-text relationship weight W obtained based on B31 BGT The BM25 scores of the aspects obtained in step B32 are weighted fused to obtain the fusion background knowledge-text weight R T : R T =λ1W BGT +λ2BM25(A,BG) Among them: λ1 and λ2 are adjustment parameters; Step B34: Represent the background knowledge feature E of each sample in the sample set TS obtained in step B2 BG Perform cross-modal attention fusion with the image feature representation of each sample in the sample set TS to obtain the background knowledge-image relationship matrix B I : Q=W Q E V ,K=W K E BG ,V=W V E BG Among them, Q, K, and V are the query vector, key vector, and value vector in the attention mechanism respectively; W Q 、W K 、W V are the learnable parameter matrices of Q, K, and V respectively; d represents the dimension size in the attention mechanism; Step B35: Based on the background knowledge feature representation E of each sample in the sample set TS obtained in step B2 BG and image feature representation E V , calculate the information entropy and use it to calculate the CXMI score CXMI (E BG ,E V ): CXMI(E BG ,E V )=H(E BG )+H(E V )-H(E BG ,E V ) Among them, H(E BH ) represents E BG Information entropy, H(E V ) represents E V Information entropy, H(E BG ,E V ) represents the joint information entropy of the two; Step B36: Background knowledge-image relationship matrix B obtained based on B34 I The CXMI score obtained in step B35 is weighted fused to obtain the fusion background knowledge-image relationship matrix R I : R I =λ3B i +λ4CXMI(E BG ,E V ) Among them, λ3 and λ4 are adjustment parameters.

6. The multimodal sentiment analysis method based on thought chain and background knowledge according to claim 5 is characterized in that: The step B4 specifically includes the following steps: Step B41: Based on the fusion background knowledge obtained in step B33 - text weight R T and the fusion background knowledge-image relationship matrix R obtained in step B36 I , calculate the total weight of background knowledge and normalize it: W BG =αR T +βR I Among them: α and β are adjustment parameters; W BG is the total weight; W′ BG is the normalized total weight; 1≤i≤N, where N is the total amount of background knowledge; Step B42: Based on the normalized total weight obtained in step B41, select the K background knowledge that best matches the current sample: S BG =Selected(W′ Bg ,K) Among them, S BG is the filtered background knowledge; Selected is a selector used to select the top K background knowledge instances with the highest relevance to the current sample from all background knowledge according to the total weight.

7. The multimodal sentiment analysis method based on thought chain and background knowledge according to claim 2 is characterized in that: The step B5 specifically includes the following steps: Step B51: Based on the K pieces of background knowledge obtained in step B4, a gating mechanism is introduced to calculate the weight distribution of each piece of background knowledge to control the importance of different pieces of background knowledge: Gate=σ(W G S BG +b G ) Where, e is the natural logarithm, W G is the trainable weight matrix, b G is the bias term, σ(x) is the Sigmoid activation function, x is the input parameter of the function, ensuring that the gating weight value range is between [0,1], and Gate is the gating weight; Step B52: Use the gating mechanism to adjust the importance of background knowledge, perform nonlinear weight adjustment, and obtain the gated fusion background knowledge vector Where: ⊙ represents the Hadamard product, which is the element-by-element multiplication of two vectors; Step B53: Use the BERT2Text model to gate the fusion background knowledge vector Transformed into integrated background knowledge BG final .

8. The multimodal sentiment analysis method based on thought chain and background knowledge according to claim 1 is characterized in that: The step C specifically comprises the following steps: Step C1: Use the pre-trained large model to generate the thought chain thinking process for each sample. Specifically, for each sample in the sample set TS, given the text data, aspect, and sentiment polarity, design prompt words to let the large model output the thought process of sentiment polarity obtained by analyzing the text data and aspect. This enables the model to not only output the sentiment judgment result, but also explain its reasoning basis and logical chain. By guiding the model to explicitly express intermediate judgments in multiple reasoning steps, the model's ability to understand complex semantic relationships is enhanced. The thought process is expressed in text form. Step C2: Use the thought process obtained in step C1 to splice the thought process into the sample set TS; Step C3: Fine-tune the pre-trained LLaVA model, use the thinking process obtained in step C1 to calculate the KL divergence, and calculate the cross entropy loss of the true value and the predicted value. Use the backpropagation algorithm to calculate the gradient of each parameter in the fine-tuned LLaVA model, and use the stochastic gradient descent algorithm to update the parameters. When the loss value is less than the set threshold or the maximum number of iterations is reached, terminate the model fine-tuning training.

9. The multimodal sentiment analysis method based on thought chain and background knowledge according to claim 8 is characterized in that: The step C1 specifically includes the following steps: Step C11: Design prompt words to let the pre-trained large model output the thought process of sentiment polarity obtained from text data and aspect analysis. The designed prompt word template is as follows: First, explain the overall semantic context of the text and how it narrates or describes the target aspect; Next, using specific language expressions, explain how the text directly or indirectly implies the given emotional label; Focus on the emotional cues that are closely related to the target aspect and distinguish between explicit and implicit emotional signals; If there are factors that could lead to misunderstanding, such as ambiguity or irony, clearly explain how you eliminated these factors and confirmed the emotional label; Finally, provide a clear and detailed summary explaining why the text clearly expresses the given sentiment label. Please provide your reasoning step by step and in a clear and logical manner. Do not add any additional subheadings or titles. Your output should only contain analytical reasoning, i.e., thinking processes; Text: [x]; Target aspect: [y]; Sentiment label: [z]; Where [x] is the input text data, [y] is the input target aspect, and [z] is the input sentiment label; Step C12: Using the pre-trained large model combined with the prompt word template designed in step C11, output the sentiment polarity thinking process Q obtained by analyzing the text data and aspects for each sample in the sample set TS; In step C2, the thought process Q obtained in step C1 is used to splice the thought process Q into the sample set TS. The sample set is represented as TS = {T, V, A, Q}; where T, V, and A are the text data, image data, and aspects in the sample set, respectively, and Q is the spliced-in thought process. The step C3 specifically includes the following steps: Step C31: Calculate KL divergence loss L KL : Where P(i) is the thought process of the i-th sample predicted during fine-tuning, and Q(i) is the thought process obtained in step C1; Step C32: Calculate the cross entropy loss L between the predicted value and the true value CE : Among them, y i represents the true value of the i-th sample, p i Represents the predicted value of the i-th sample; Step C33: Calculate the final total loss L based on the losses obtained in steps C31 and C32 total : L total =μ1L KL +μ2L CE Among them: μ1 and μ2 are hyperparameters used to adjust the effects of KL divergence loss and cross entropy loss; Step C34: For the total loss function obtained in step C43, the back propagation algorithm is used to calculate the gradient of each parameter in the deep network, and the stochastic gradient descent algorithm is used to update the parameters.

10. A multimodal sentiment analysis system based on thought chain and background knowledge, characterized in that: The method comprises a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method according to any one of claims 1 to 9 can be implemented.

Images