Multi-modal sentiment analysis method and system based on thinking chain preference

Abstract

The invention discloses a multi-modal sentiment analysis method and system based on thinking chain preference, and relates to the field of multi-modal sentiment analysis, and the method comprises the following steps: inputting analysis data containing an input text, an image and a to-be-analyzed target into a multi-modal language model for pre-training, generating a plurality of thinking chains and sentiment polarity prediction results thereof, dividing the thinking chains into sets with consistent and inconsistent results based on the consistency of the prediction result and the real label, and calculating the semantic similarity score of each thinking chain and the input text and image; the similarity scores are subjected to weighted fusion, preference sample pairs are determined, input information is combined with selected and rejected thinking chains, conditional probabilities of the input information and the selected and rejected thinking chains are extracted, an optimization function is set according to the two conditional probabilities, and the tendency of generating the selected thinking chains is enhanced, so that model parameters are updated, and the purposes of reducing illusion reasoning and improving the thinking efficiency are achieved. And accurate emotion polarity prediction is realized.

Description

Technical Field

[0001] This invention relates to the field of multimodal sentiment analysis, specifically to a multimodal sentiment analysis method and system based on thought chain preferences. Background Technology

[0002] In the field of multimodal sentiment analysis, researchers have been working to combine textual and image information to improve the accuracy of sentiment recognition. Existing techniques typically employ single-modal or simple feature fusion methods for sentiment analysis. Some models rely solely on the sentiment features of the text or simply concatenate text and image information. While these methods can make sentiment judgments to some extent, they often overlook the complex interactions between the two modalities and the dynamic generation of thought chains during reasoning. This leads to insufficient accuracy and reliability of sentiment judgment results, causing the models to perform poorly in complex situations. Existing multimodal fine-grained sentiment analysis methods generally lack a complete reasoning logic from input information to sentiment judgment. Multimodal fine-grained sentiment analysis methods based on conditional relation detection predict sentiment polarity by modeling the modal information interaction between text, images, and targets. Although this method can integrate multimodal features to a certain extent, it relies solely on the conditional relations between features for direct prediction during the reasoning process. It lacks a complete logical derivation path from input content to sentiment analysis, and the decision-making process of the model is difficult to interpret. In addition, multimodal fine-grained sentiment analysis methods based on contextual knowledge attempt to introduce external knowledge, such as common sense background knowledge, to assist multimodal large models in sentiment analysis. However, while relying on external knowledge can enhance semantic understanding, it is prone to hallucinatory reasoning, which affects the accuracy and reliability of sentiment analysis.

[0003] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal sentiment analysis method and system based on thought chain preferences to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A multimodal sentiment analysis method based on thought chain preferences includes the following steps: The analysis data set containing input text, input image, target to be analyzed and its prompt information is input into a multimodal language model for pre-training. The model is then guided to generate multiple thought chains and corresponding sentiment polarity prediction results. Each thought chain contains intermediate reasoning text from input information to sentiment polarity judgment. Based on the consistency between the predicted sentiment polarity and the actual sentiment polarity label, all generated thought chains are divided into a consistent set and an inconsistent set. For each thought chain in the set, the text semantic similarity score between its thought chain text and the input text, as well as the image semantic similarity score between its thought chain text and the input image are calculated. The text and image semantic similarity scores corresponding to each thought chain are weighted and fused to obtain the comprehensive semantic similarity score of the thought chain. Based on the comprehensive semantic similarity score, the preferred sample pair consisting of the selected thought chain and the rejected thought chain is determined in the result consistency set and the result inconsistency set. The input text, image, target to be analyzed and its prompt information are combined with the selected thought chain and the rejected thought chain, respectively, and input into the multimodal language model to extract the conditional probability of generating the selected thought chain and the conditional probability of generating the rejected thought chain. Based on two conditional probabilities, the optimization function is set through direct preference optimization to strengthen the model's tendency to generate the selected thought chain, thereby updating the model parameters and outputting accurate sentiment polarity prediction results.

[0006] Furthermore, the target to be analyzed specifically refers to the specific object or entity that is required to determine the sentiment tendency in the sentiment polarity analysis task, specifically including the identification objects in the input text such as products, people, events, or images; The prompts for the target to be analyzed are used to guide the multimodal language model to generate a specific output text. The specific method for pre-training a multimodal language model is as follows: collect several sets of text and images, determine the analysis target based on each set of text and images, collect several thought texts related to the analysis target as thought chains, and label the corresponding sentiment polarity of each thought chain through manual annotation. Use the text and images, the analysis target and its prompts, as well as the corresponding thought chains and sentiment polarity annotations as pre-training data. Under the initially set model parameters, use the text, images, the analysis target and its prompts as inputs, and the corresponding thought chains and sentiment polarities as labels to pre-train the multimodal language model.

[0007] Furthermore, the model parameters include weight parameters, bias parameters, and model hyperparameters among different network structures in the multimodal language model; The emotional polarity is used to identify the category of emotion, specifically including three categories: positive, negative, and neutral. The logic behind dividing all generated thought chains into sets with consistent results and sets with inconsistent results is as follows: Based on each thought chain output by the multimodal language model and the sentiment polarity prediction results of each thought chain, the generated thought chains are divided into two sets according to the consistency of the results. The specific logic is as follows: In the formula, Represents the set of consistent results. For the set of inconsistent results, The text representing the i-th thought chain. This represents the predicted sentiment polarity for the i-th thought chain. Let i represent the true sentiment polarity label of the i-th thought chain, where i is the index of the thought chain. The consistent result set gathers several thought chains whose sentiment polarity prediction results are consistent with the true sentiment polarity label and their corresponding sentiment polarity prediction results. The inconsistent result set gathers several thought chains whose sentiment polarity prediction results are inconsistent with the true sentiment polarity label and their corresponding sentiment polarity prediction results.

[0008] Furthermore, the logic underlying the calculation of the semantic similarity score between the thought chain text and the input text is as follows: For each thought chain and its corresponding sentiment polarity prediction result in the consistent and inconsistent result sets, the thought chain text is encoded using the CLIP model's text encoder to obtain the thought chain's representation vector, specifically as follows: In the formula, Let i be the text representation vector of the i-th thought chain in the consistent result set. This represents the text encoding operation in the CLIP model; Simultaneously, the input text is encoded using the CLIP model's text encoder to obtain the representation vector of the input text modality, specifically as follows: In the formula, The representation vector of the input text. Indicates the input text; The formula used to calculate the semantic similarity score between the text of each thought chain and the input text, based on the combined text representation vectors of each thought chain and the input text, is as follows: In the formula, Let be the text semantic similarity score between the text of the i-th thought chain and the input text in the consistent result set. and Let represent the text of the i-th thought chain and the magnitude of the input text representation vector, respectively.

[0009] Furthermore, the logic underlying the calculation of the semantic similarity score between the thought chain text and the input image is as follows: the input image is encoded using an image encoder based on the CLIP model to obtain the representation vector of the input image, specifically represented as: In the formula, Let be the representation vector of the input image. For image encoding operations in the CLIP model, Indicates the input image; The formula used to calculate the semantic similarity score between the text representation vector of each thought chain and the representation vector of the input image is as follows: In the formula, The semantic similarity score between the text of the i-th thought chain and the input image in the consistent result set. This represents the magnitude of the input image representation vector; The same method is used to obtain the text semantic similarity score and image semantic similarity score for each thought chain in the inconsistent results set.

[0010] Furthermore, the formula used to calculate the comprehensive semantic similarity score of the thought chain is as follows: In the formula, For the i-th thought chain in the consistent result set, the comprehensive semantic similarity score is... These are the weighting coefficients; The logic underlying the determination of the preference sample set composed of the selected and rejected thought chains, based on the comprehensive semantic similarity score of each thought chain in the consistent and inconsistent result sets, is as follows: For the consistent results set, the thought chain with the highest comprehensive semantic similarity score is selected as the selected thought chain. For the set of inconsistent results, the thought chain with the lowest comprehensive semantic similarity score is selected as the rejected thought chain, thus forming a preference sample pair consisting of the selected thought chain and the rejected thought chain.

[0011] Furthermore, the specific method for extracting the conditional probability of generating the selected thought chain and the conditional probability of generating the rejected thought chain is as follows: construct a feature input vector, specifically including the input text, image, target to be analyzed, the selected thought chain, and the sentiment polarity prediction result corresponding to the selected thought chain; input the constructed feature input vector into a multimodal language model for inference, and extract the conditional probability of the selected thought chain from the model output through the softmax network layer in the multimodal language model; The conditional probability of a thought chain where the output is rejected is similar.

[0012] Furthermore, the logic behind setting the optimization function through direct preference optimization is as follows: The specific optimization objective of direct preference optimization is to maximize the difference between the probability of the multimodal language model generating the selected thought chain and the probability of generating the rejected thought chain, thereby maximizing the tendency of the multimodal language model to generate the selected thought chain and minimizing the tendency to generate the rejected thought chain. The optimization function is set according to the optimization objective of direct preference optimization, and the optimization function is specifically expressed as follows: In the formula, For the first The optimization function values ​​for each set of model parameters For balance coefficient, This represents the Sigmoid function. Indicates the first Given a set of model parameters, the multimodal language model outputs the conditional probability of the selected thought chain. Indicates the first Given a set of model parameters, the multimodal language model outputs the conditional probability of the rejected thought chain. and These are the reference conditional probabilities of the thought chains being selected and rejected, respectively. This is an index for the model parameter set, which specifically refers to a data combination including weight parameters, bias parameters, and model hyperparameters among different network structures of a multimodal language model. Based on the optimization function value, the model parameter set corresponding to the minimum optimization function value is determined by the optimization algorithm, and this set is used as the optimal model parameter set to update the multimodal language model.

[0013] This invention also provides a multimodal sentiment analysis system based on thought chain preferences, wherein the multimodal sentiment analysis system based on thought chain preferences is used to execute the above-described multimodal sentiment analysis method based on thought chain preferences, comprising: The thought construction training module is used to input the analysis data set containing input text, input image, target to be analyzed and its prompt information into the multimodal language model for pre-training, and guide the model to generate multiple thought chains and corresponding sentiment polarity prediction results. Each thought chain contains intermediate reasoning text from input information to sentiment polarity judgment. The similarity analysis module is used to divide all generated thought chains into a set of consistent results and a set of inconsistent results based on the consistency between the sentiment polarity prediction results and the real sentiment polarity labels. For each thought chain in the set, the text semantic similarity score between its thought chain text and the input text, as well as the image semantic similarity score between its thought chain text and the input image are calculated. The selection bias module is used to weight and fuse the semantic similarity scores of the text and images corresponding to each thought chain to obtain the comprehensive semantic similarity score of the thought chain. Based on the comprehensive semantic similarity score, the preferred sample pair consisting of the selected thought chain and the rejected thought chain is determined in the result consistency set and the result inconsistency set. The probability distribution calculation module is used to combine the input text, image, target to be analyzed and its prompt information with the selected thought chain and the rejected thought chain respectively, input them into the multimodal language model, and extract the conditional probability of generating the selected thought chain and the conditional probability of generating the rejected thought chain respectively. The optimization and update module is used to optimize the model based on two conditional probabilities by setting the optimization function through direct preference optimization. This strengthens the model's tendency to generate the selected thought chain, thereby updating the model parameters and outputting accurate sentiment polarity prediction results.

[0014] Compared with the prior art, the beneficial effects of the present invention are: This invention guides a multimodal large model to generate a thought chain during the reasoning process through prompt words, demonstrating a complete logical process from input to emotion judgment; Furthermore, this invention constructs preference sample pairs by evaluating the semantic similarity and consistency of the generated multiple thought chains, selecting the "selected thought chain" with correct reasoning logic and the "rejected thought chain" with illusory reasoning. To address the hallucination reasoning problem that often occurs when generating thought chains in multimodal large models, this invention designs a thought chain preference optimization algorithm based on the thought chain pair of "selected thought chain - rejected thought chain". This algorithm reduces hallucination reasoning by maximizing the model's preference probability for the "selected thought chain" and minimizing the preference probability for the "rejected thought chain". Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall method flow of the present invention; Figure 2A bar chart showing the distribution of semantic similarity between text and image in each thought chain; Figure 3 This is a schematic diagram of the overall system structure of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0017] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0018] Example: Please see Figures 1-2 The present invention provides a technical solution: A multimodal sentiment analysis method based on thought chain preferences includes the following steps: Step 1: Input the analysis data set containing input text, input image, target to be analyzed and its prompt information into the multimodal language model for pre-training, guide the model to generate multiple thought chains and corresponding sentiment polarity prediction results. Each thought chain contains intermediate reasoning text from input information to sentiment polarity judgment.

[0019] The target to be analyzed specifically refers to the specific object or entity that is required to determine the sentiment tendency in the sentiment polarity analysis task, including the identification objects in the input text such as products, people, events or images. The prompts for the target to be analyzed are used to guide the multimodal language model to generate a specific output text; the prompts are a text or structured instructions that guide a large language model or a large multimodal model to generate a specific output; specifically used for task definition, reasoning heuristics, etc. The specific method for pre-training a multimodal language model is as follows: collect several sets of text and images, determine the analysis target based on each set of text and images, collect several related thought texts for the analysis target as thought chains, and the related thought texts specifically refer to the logical inference texts for obtaining the emotional polarity of the analysis target, which describe the specific analytical reasoning process and logic for obtaining the emotional polarity of the analysis target. Existing multimodal fine-grained sentiment analysis methods typically predict sentiment polarity labels directly after fusing text, image, and target information. While these methods achieve multimodal feature interaction, the model's reasoning process is entirely implicit, failing to demonstrate the logical chain from input content to sentiment judgment, resulting in a lack of interpretability. Therefore, this embodiment introduces a thought chain generation mechanism into the multimodal sentiment analysis framework. This allows the model to explicitly demonstrate the complete reasoning logic from input information to sentiment judgment while outputting sentiment polarity. Prompt words guide the multimodal language model to perform joint reasoning on the input text, image, and target to generate multiple thought chains.

[0020] The corresponding sentiment polarity of each thought chain is labeled by manual annotation. The text, images, target to be analyzed and its prompts, as well as the corresponding thought chains and sentiment polarity labels are used as pre-training data. Under the initially set model parameters, the multimodal language model is pre-trained with text, images, target to be analyzed and its prompts as inputs and corresponding thought chains and sentiment polarities as labels.

[0021] The multimodal language model is a large model that extends the visual encoder on the basis of the large language model, supporting the generation and understanding of multiple input modalities (such as text, images, etc.). The large language model is an autoregressive deep neural network model trained on a large-scale corpus, which has the ability to understand, generate and reason about language. Typical large language models include the GPT series, DeepSeek, etc. The thought chain includes intermediate reasoning text from input information to sentiment polarity judgment. Specifically, it refers to a series of intermediate reasoning steps, logical chains, or explanatory texts explicitly generated by the multimodal language model to solve problems or make inferences before generating sentiment polarity prediction results. The thought chain reflects the step-by-step reasoning process inside the model, which helps to improve the interpretability and reasoning accuracy of the model.

[0022] Step 2: Based on the consistency between the predicted sentiment polarity and the actual sentiment polarity label, all generated thought chains are divided into a consistent set and an inconsistent set. For each thought chain in the set, the textual semantic similarity score between the thought chain text and the input text, and the image semantic similarity score between the thought chain text and the input image are calculated.

[0023] The model parameters include weight parameters, bias parameters, and model hyperparameters among different network structures in the multimodal language model; The weight parameters between different network structures are the weight values ​​of each layer connection in the neural network. The weight parameters determine how the input data is transformed and transmitted through the network. The output of each neuron is the weighted sum of its inputs. By adjusting these weights to minimize the prediction error, the model can better fit the training data. Bias parameters are additional parameters for each neuron. They are added to the output of each neuron and are used to adjust the output of the model. Bias allows the model to fit the data more flexibly, especially when the input is zero. Bias allows the neurons to still be activated. In each layer of the neural network of a multimodal language model, there are corresponding bias parameters, which together with the weights affect the final output. Hyperparameters directly affect the learning process and final performance of a model, including learning rate, batch size, number of hidden layers, number of neurons per layer, and regularization term; The emotional polarity is used to identify the category of emotion, specifically including three categories: positive, negative, and neutral; it is used to guide the training and evaluation of the model's emotion classification.

[0024] The logic behind dividing all generated thought chains into sets with consistent results and sets with inconsistent results is as follows: Based on each thought chain output by the multimodal language model and the sentiment polarity prediction results of each thought chain, the generated thought chains are divided into two sets according to the consistency of the results. The specific logic is as follows: In the formula, Represents the set of consistent results. For the set of inconsistent results, The text representing the i-th thought chain. This represents the predicted sentiment polarity for the i-th thought chain. Let i represent the true sentiment polarity label of the i-th thought chain, where i is the index of the thought chain. The consistent result set gathers several thought chains whose sentiment polarity prediction results are consistent with the true sentiment polarity label and their corresponding sentiment polarity prediction results. The inconsistent result set gathers several thought chains whose sentiment polarity prediction results are inconsistent with the true sentiment polarity label and their corresponding sentiment polarity prediction results.

[0025] The logic underlying the calculation of the semantic similarity score between the thought chain text and the input text is as follows: For each thought chain and its corresponding sentiment polarity prediction result in the consistent and inconsistent result sets, the thought chain text is encoded using the CLIP model's text encoder to obtain the thought chain's representation vector, specifically as follows: In the formula, Let i be the text representation vector of the i-th thought chain in the consistent result set. This represents the text encoding operation in the CLIP model; It should be noted that the CLIP model is a multimodal representation learning model proposed by OpenAI. It trains both the text encoder and the image encoder simultaneously through contrastive learning, enabling text and images to be mapped to a unified representation vector space. The specific steps for encoding the thought chain text using the CLIP model's text encoder include: before using the CLIP model, it is necessary to load the pre-trained CLIP model and its text encoder; pre-train the CLIP model and its text encoder using a deep learning framework such as PyTorch; encode the thought chain text using the text processing functions provided by CLIP; and output the thought chain representation vector through the CLIP model's text encoder based on the encoded thought chain text. Simultaneously, the input text is encoded using the CLIP model's text encoder to obtain the representation vector of the input text modality, specifically as follows: In the formula, The representation vector of the input text. Indicates the input text; The specific method for obtaining the representation vector of the input text modality is the same as the method for obtaining the representation vector of the thought chain, and will not be repeated here.

[0026] The formula used to calculate the semantic similarity score between the text of each thought chain and the input text, based on the combined text representation vectors of each thought chain and the input text, is as follows: In the formula, Let be the text semantic similarity score between the text of the i-th thought chain and the input text in the consistent result set. and Let represent the text of the i-th thought chain and the magnitude of the input text representation vector, respectively.

[0027] It should be noted that this formula is based on the cosine similarity calculation formula. Cosine similarity is the cosine value of the cosine angle between two non-zero vectors, ranging from -1 to 1, where 1 represents complete similarity, 0 represents no similarity, and -1 represents complete opposites. Among the consistent results set, the semantic similarity score between the text of the i-th thought chain and the input text is... The larger the value, the more similar the semantics of the thought chain text is to the input text; This represents the dot product between the thought chain text representation vector and the input text representation vector. The dot product is used to measure the similarity between two vectors. If the two vectors are semantically similar, their dot product value will be larger; if they are semantically dissimilar, their dot product value will be smaller. When calculating similarity, the simple dot product can be affected by the size of the vectors; therefore, the dot product result is divided by the magnitudes of the two vectors. This allows us to obtain a standardized similarity score, eliminating the impact of differences in vector size.

[0028] The logic behind calculating the semantic similarity score between the thought chain text and the input image is as follows: The input image is encoded using an image encoder based on the CLIP model to obtain its representation vector, specifically: In the formula, Let be the representation vector of the input image. For image encoding operations in the CLIP model, Indicates the input image; The formula used to calculate the semantic similarity score between the text representation vector of each thought chain and the representation vector of the input image is as follows: In the formula, The semantic similarity score between the text of the i-th thought chain and the input image in the consistent result set. This represents the magnitude of the input image representation vector; the calculation logic for image semantic similarity score is the same as that for text semantic similarity score, and will not be elaborated here. The same method is used to obtain the text semantic similarity score and image semantic similarity score for each thought chain in the inconsistent results set.

[0029] Step 3: Weight and fuse the text and image semantic similarity scores corresponding to each thought chain to obtain the comprehensive semantic similarity score of the thought chain. Based on the comprehensive semantic similarity score, determine the preference sample pair consisting of the selected thought chain and the rejected thought chain in the result consistency set and the result inconsistency set.

[0030] The formula used to calculate the comprehensive semantic similarity score of the thought chain is as follows: In the formula, For the i-th thought chain in the consistent result set, the comprehensive semantic similarity score is... This is the weighting coefficient, typically set to 0.6. The specific value is determined by the fact that text may provide richer contextual information, while images supplement visually intuitive content; therefore, the weighting coefficient is adjusted accordingly. The setting is 0.6, which means that the text similarity score accounts for 60% of the overall score, while the image similarity score accounts for 40% of the weight. The logic underlying the determination of the preference sample set composed of the selected and rejected thought chains, based on the comprehensive semantic similarity score of each thought chain in the consistent and inconsistent result sets, is as follows: For the consistent results set, the thought chain with the highest comprehensive semantic similarity score is selected as the selected thought chain. For the set of inconsistent results, the thought chain with the lowest comprehensive semantic similarity score is selected as the rejected thought chain, thus forming a preference sample pair consisting of the selected thought chain and the rejected thought chain.

[0031] It should be noted that selecting the thought chain with the highest comprehensive semantic similarity score as the selected thought chain helps to strengthen the semantic consistency of thought chains in the result consistency set; it ensures that the selected thought chain is highly semantically consistent with the input text or image, increasing the credibility of the results; by selecting the thought chain with the lowest comprehensive semantic similarity score as the rejected thought chain, it can effectively distinguish thought chains that are inconsistent with the input, helping the model to identify those thought chains that may produce misleading or irrelevant information, thereby improving the quality of the output results.

[0032] Step 4: Combine the input text, image, target to be analyzed and its prompt information with the selected thought chain and the rejected thought chain respectively, input them into the multimodal language model, and extract the conditional probability of generating the selected thought chain and the conditional probability of generating the rejected thought chain respectively.

[0033] The specific method for extracting the conditional probability of generating the selected thought chain and the conditional probability of generating the rejected thought chain is as follows: Construct a feature input vector, which specifically includes the input text, image, target to be analyzed, the selected thought chain, and the sentiment polarity prediction result corresponding to the selected thought chain; input the constructed feature input vector into a multimodal language model for inference, and extract the conditional probability of the selected thought chain from the model output through the softmax network layer in the multimodal language model; It should be noted that in the multimodal language model, forward inference is performed by combining the input text, image, target to be analyzed and its prompt information with the selected and rejected thought chains as feature vectors, and then calculating them through its internal network structure to generate conditional probabilities for the selected and rejected thought chains. The Softmax layer transforms the model's output into a probability distribution to represent the relative probability of each possible output. The conditional probability of a thought chain where the output is rejected is similar.

[0034] Step 5: Based on the two conditional probabilities, the optimization function is set through direct preference optimization to strengthen the model's tendency to generate the selected thought chain, thereby updating the model parameters to output accurate sentiment polarity prediction results.

[0035] The logic behind setting the optimization function through direct preference optimization is as follows: The specific optimization objective of direct preference optimization is to maximize the difference between the probability of the multimodal language model generating the selected thought chain and the probability of generating the rejected thought chain, thereby maximizing the tendency of the multimodal language model to generate the selected thought chain and minimizing the tendency to generate the rejected thought chain. Preference optimization is a class of training methods that optimize model generation behavior using preference signals; its goal is to make the output of large language models more consistent with human preferences or task objectives. These algorithms achieve preference alignment and performance improvement of the generative model by maximizing the probability of the model's "selected output" and minimizing the probability of its "rejected output"; common forms include direct preference optimization.

[0036] The optimization function is set according to the optimization objective of direct preference optimization, and the optimization function is specifically expressed as follows: In the formula, For the first The optimization function values ​​for each set of model parameters For balance coefficient, This represents the Sigmoid function. Indicates the first Given a set of model parameters, the multimodal language model outputs the conditional probability of the selected thought chain. Indicates the first Given a set of model parameters, the multimodal language model outputs the conditional probability of the rejected thought chain. and These are the reference conditional probabilities of the thought chains being selected and rejected, respectively. This is an index for the model parameter set, which specifically refers to a data combination including weight parameters, bias parameters, and model hyperparameters among different network structures of a multimodal language model. It should be noted that the optimization objective of this formula is to maximize the contrast signal between the conditional probability of the selected thought chain and the conditional probability of the rejected thought chain, thereby strengthening the model's ability to distinguish between preferred sample pairs. Indicates the current model parameter group The log ratio of the conditional probability of the selected thought chain to the reference conditional probability is used. If this value is large, it means that the model is currently more inclined to select the selected thought chain. The difference between these two log ratios is used by the optimization function to measure the model's ability to distinguish between preferred sample pairs, that is, it is more inclined to strengthen the probability of the selected chain and weaken the probability of the rejected chain. The Sigmoid function maps the output of the optimization objective to... The range of 1 to 1 facilitates the calculation of the loss value. The Sigmoid function can smoothly handle the differences in the conditional probability ratio, avoiding the gradient being too large or too small during the optimization process. The reference conditional probabilities for the selected and rejected thought chains can be set based on expert experience and model accuracy requirements, and are generally set to... .

[0037] Balance coefficient This is used to adjust the degree of influence of the difference between the two logarithmic ratios in the formula on the overall optimization objective, and is generally set between 0.1 and 0.5.

[0038] Based on the optimization function value, the model parameter set corresponding to the minimum optimization function value is determined by the optimization algorithm, and this set is used as the optimal model parameter set to update the multimodal language model. Specifically, a genetic algorithm is used as the optimization algorithm, treating the model parameter set as individuals, the parameters in the model parameter set as genes, and the optimization function value as the fitness value. An initial population is constructed based on the individuals formed by the model parameter set. Iterative operations of selection, crossover, and mutation are performed on the initial population to determine the individual with the smallest fitness value, which is then used as the optimal model parameter set to update the multimodal language model. Genetic algorithms are a conventional existing technology in optimization algorithms, and the specific optimization process will not be elaborated here.

[0039] Please see Figure 3 The present invention also provides a multimodal sentiment analysis system based on thought chain preferences, wherein the multimodal sentiment analysis system based on thought chain preferences is used to execute the above-described multimodal sentiment analysis method based on thought chain preferences, comprising: The thought construction training module is used to input the analysis data set containing input text, input image, target to be analyzed and its prompt information into the multimodal language model for pre-training, and guide the model to generate multiple thought chains and corresponding sentiment polarity prediction results. Each thought chain contains intermediate reasoning text from input information to sentiment polarity judgment. The similarity analysis module is used to divide all generated thought chains into a set of consistent results and a set of inconsistent results based on the consistency between the sentiment polarity prediction results and the real sentiment polarity labels. For each thought chain in the set, the text semantic similarity score between its thought chain text and the input text, as well as the image semantic similarity score between its thought chain text and the input image are calculated. The selection bias module is used to weight and fuse the semantic similarity scores of the text and images corresponding to each thought chain to obtain the comprehensive semantic similarity score of the thought chain. Based on the comprehensive semantic similarity score, the preferred sample pair consisting of the selected thought chain and the rejected thought chain is determined in the result consistency set and the result inconsistency set. The probability distribution calculation module is used to combine the input text, image, target to be analyzed and its prompt information with the selected thought chain and the rejected thought chain respectively, input them into the multimodal language model, and extract the conditional probability of generating the selected thought chain and the conditional probability of generating the rejected thought chain respectively. The optimization and update module is used to optimize the model based on two conditional probabilities by setting the optimization function through direct preference optimization. This strengthens the model's tendency to generate the selected thought chain, thereby updating the model parameters and outputting accurate sentiment polarity prediction results.

[0040] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0041] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0042] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0043] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A multimodal sentiment analysis method based on thought chain preferences, characterized in that, The specific steps include: The analysis data set containing input text, input image, target to be analyzed and its prompt information is input into a multimodal language model for pre-training. The model is then guided to generate multiple thought chains and corresponding sentiment polarity prediction results. Each thought chain contains intermediate reasoning text from input information to sentiment polarity judgment. Based on the consistency between the predicted sentiment polarity and the actual sentiment polarity label, all generated thought chains are divided into a consistent set and an inconsistent set. For each thought chain in the set, the text semantic similarity score between its thought chain text and the input text, as well as the image semantic similarity score between its thought chain text and the input image are calculated. The text and image semantic similarity scores corresponding to each thought chain are weighted and fused to obtain the comprehensive semantic similarity score of the thought chain. Based on the comprehensive semantic similarity score, the preferred sample pair consisting of the selected thought chain and the rejected thought chain is determined in the result consistency set and the result inconsistency set. The input text, image, target to be analyzed and its prompt information are combined with the selected thought chain and the rejected thought chain, respectively, and input into the multimodal language model to extract the conditional probability of generating the selected thought chain and the conditional probability of generating the rejected thought chain. Based on two conditional probabilities, the optimization function is set through direct preference optimization to strengthen the model's tendency to generate the selected thought chain, thereby updating the model parameters and outputting accurate sentiment polarity prediction results.

2. The multimodal sentiment analysis method based on thought chain preferences according to claim 1, characterized in that: The target to be analyzed specifically refers to the specific object or entity that is required to determine the sentiment tendency in the sentiment polarity analysis task, including the identification objects in the input text such as products, people, events or images. The prompts for the target to be analyzed are used to guide the multimodal language model to generate a specific output text. The specific method for pre-training a multimodal language model is as follows: collect several sets of text and images, determine the analysis target based on each set of text and images, collect several thought texts related to the analysis target as thought chains, and label the corresponding sentiment polarity of each thought chain through manual annotation. Use the text and images, the analysis target and its prompts, as well as the corresponding thought chains and sentiment polarity annotations as pre-training data. Under the initially set model parameters, use the text, images, the analysis target and its prompts as inputs, and the corresponding thought chains and sentiment polarities as labels to pre-train the multimodal language model.

3. The multimodal sentiment analysis method based on thought chain preferences according to claim 2, characterized in that: The model parameters include weight parameters, bias parameters, and model hyperparameters among different network structures in the multimodal language model; The emotional polarity is used to identify the category of emotion, specifically including three categories: positive, negative, and neutral. The logic behind dividing all generated thought chains into sets with consistent results and sets with inconsistent results is as follows: Based on each thought chain output by the multimodal language model and the sentiment polarity prediction results of each thought chain, the generated thought chains are divided into two sets according to the consistency of the results. The specific logic is as follows: In the formula, Represents the set of consistent results. For the set of inconsistent results, The text representing the i-th thought chain. This represents the predicted sentiment polarity for the i-th thought chain. Let i represent the true sentiment polarity label of the i-th thought chain, where i is the index of the thought chain. The consistent result set gathers several thought chains whose sentiment polarity prediction results are consistent with the true sentiment polarity label and their corresponding sentiment polarity prediction results. The inconsistent result set gathers several thought chains whose sentiment polarity prediction results are inconsistent with the true sentiment polarity label and their corresponding sentiment polarity prediction results.

4. The multimodal sentiment analysis method based on thought chain preferences according to claim 3, characterized in that: The logic underlying the calculation of the semantic similarity score between the thought chain text and the input text is as follows: For each thought chain and its corresponding sentiment polarity prediction result in the consistent and inconsistent result sets, the thought chain text is encoded using the CLIP model's text encoder to obtain the thought chain's representation vector, specifically as follows: In the formula, Let i be the text representation vector of the i-th thought chain in the consistent result set. This represents the text encoding operation in the CLIP model; Simultaneously, the input text is encoded using the CLIP model's text encoder to obtain the representation vector of the input text modality, specifically as follows: In the formula, The representation vector of the input text. Indicates the input text; The formula used to calculate the semantic similarity score between the text of each thought chain and the input text, based on the combined text representation vectors of each thought chain and the input text, is as follows: In the formula, Let be the text semantic similarity score between the text of the i-th thought chain and the input text in the consistent result set. and Let represent the text of the i-th thought chain and the magnitude of the input text representation vector, respectively.

5. The multimodal sentiment analysis method based on thought chain preferences according to claim 4, characterized in that: The logic behind calculating the semantic similarity score between the thought chain text and the input image is as follows: The input image is encoded using an image encoder based on the CLIP model to obtain its representation vector, specifically: In the formula, Let be the representation vector of the input image. For image encoding operations in the CLIP model, Indicates the input image; The formula used to calculate the semantic similarity score between the text representation vector of each thought chain and the representation vector of the input image is as follows: In the formula, The semantic similarity score between the text of the i-th thought chain and the input image in the consistent result set. This represents the magnitude of the input image representation vector; The same method is used to obtain the text semantic similarity score and image semantic similarity score for each thought chain in the inconsistent results set.

6. The multimodal sentiment analysis method based on thought chain preferences according to claim 5, characterized in that: The formula used to calculate the comprehensive semantic similarity score of the thought chain is as follows: In the formula, For the i-th thought chain in the consistent result set, the comprehensive semantic similarity score is... These are the weighting coefficients; The logic underlying the determination of the preference sample set composed of the selected and rejected thought chains, based on the comprehensive semantic similarity score of each thought chain in the consistent and inconsistent result sets, is as follows: For the consistent results set, the thought chain with the highest comprehensive semantic similarity score is selected as the selected thought chain. For the set of inconsistent results, the thought chain with the lowest comprehensive semantic similarity score is selected as the rejected thought chain, thus forming a preference sample pair consisting of the selected thought chain and the rejected thought chain.

7. The multimodal sentiment analysis method based on thought chain preferences according to claim 6, characterized in that: The specific method for extracting the conditional probability of generating the selected thought chain and the conditional probability of generating the rejected thought chain is as follows: Construct a feature input vector, which specifically includes the input text, image, target to be analyzed, the selected thought chain, and the sentiment polarity prediction result corresponding to the selected thought chain; input the constructed feature input vector into a multimodal language model for inference, and extract the conditional probability of the selected thought chain from the model output through the softmax network layer in the multimodal language model; The conditional probability of the thought chain that is rejected is similar.

8. The multimodal sentiment analysis method based on thought chain preferences according to claim 7, characterized in that: The logic behind setting the optimization function through direct preference optimization is as follows: The specific optimization objective of direct preference optimization is to maximize the difference between the probability of the multimodal language model generating the selected thought chain and the probability of generating the rejected thought chain, thereby maximizing the tendency of the multimodal language model to generate the selected thought chain and minimizing the tendency to generate the rejected thought chain. The optimization function is set according to the optimization objective of direct preference optimization, and the optimization function is specifically expressed as follows: In the formula, For the first The optimization function values ​​for each set of model parameters For balance coefficient, This represents the Sigmoid function. Indicates the first Given a set of model parameters, the multimodal language model outputs the conditional probability of the selected thought chain. Indicates the first Given a set of model parameters, the multimodal language model outputs the conditional probability of the rejected thought chain. and These are the reference conditional probabilities of the thought chains being selected and rejected, respectively. This is an index for the model parameter set, which specifically refers to a data combination including weight parameters, bias parameters, and model hyperparameters among different network structures of a multimodal language model. Based on the optimization function value, the model parameter set corresponding to the minimum optimization function value is determined by the optimization algorithm, and this set is used as the optimal model parameter set to update the multimodal language model.

9. A multimodal sentiment analysis system based on thought chain preferences, characterized in that: The aforementioned multimodal sentiment analysis system based on thought chain preferences is used to execute the multimodal sentiment analysis method based on thought chain preferences as described in any one of claims 1-8, comprising: The thought construction training module is used to input the analysis data set containing input text, input image, target to be analyzed and its prompt information into the multimodal language model for pre-training, and guide the model to generate multiple thought chains and corresponding sentiment polarity prediction results. Each thought chain contains intermediate reasoning text from input information to sentiment polarity judgment. The similarity analysis module is used to divide all generated thought chains into a set of consistent results and a set of inconsistent results based on the consistency between the sentiment polarity prediction results and the real sentiment polarity labels. For each thought chain in the set, the text semantic similarity score between its thought chain text and the input text, as well as the image semantic similarity score between its thought chain text and the input image are calculated. The selection bias module is used to weight and fuse the semantic similarity scores of the text and images corresponding to each thought chain to obtain the comprehensive semantic similarity score of the thought chain. Based on the comprehensive semantic similarity score, the preferred sample pair consisting of the selected thought chain and the rejected thought chain is determined in the result consistency set and the result inconsistency set. The probability distribution calculation module is used to combine the input text, image, target to be analyzed and its prompt information with the selected thought chain and the rejected thought chain respectively, input them into the multimodal language model, and extract the conditional probability of generating the selected thought chain and the conditional probability of generating the rejected thought chain respectively. The optimization and update module is used to optimize the model based on two conditional probabilities by setting the optimization function through direct preference optimization. This strengthens the model's tendency to generate the selected thought chain, thereby updating the model parameters and outputting accurate sentiment polarity prediction results.

Images