A multi-task image sentiment analysis method based on sentiment cue generation

Abstract

The application relates to the technical field of artificial intelligence, and provides a multi-task image sentiment analysis method based on sentiment prompt generation, which comprises the following steps: obtaining a unified feature representation, decoupling the unified feature representation, obtaining a first task feature representation, a second task feature representation and a third task feature representation; obtaining a sentiment caption corresponding to the first task feature representation; converting a sentiment caption corresponding to a sentiment that is not contained in a sentiment image into a sentiment prompt, minimizing a first cosine similarity between the sentiment prompt and the second task feature representation, and minimizing a second cosine similarity between the sentiment prompt and the third task feature representation; aggregating regional feature representations in the third task feature representation to generate a background representation, minimizing a third cosine similarity between the background representation and the second task feature representation, and obtaining a model; and inputting the sentiment image into the model to obtain a sentiment analysis result. The method improves the accuracy of an image sentiment analysis model in understanding image sentiment.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multi-task image sentiment analysis method based on sentiment cues generated. Background Technology

[0002] Computer vision tasks, including image sentiment classification, image sentiment captioning, and weakly supervised sentiment intensity prediction, aim to understand the emotions in emotional images. Cue learning has achieved state-of-the-art performance in various vision tasks such as classification, segmentation, and object detection. Some multi-task image sentiment analysis methods use cue learning to replace complexly designed modules with concise, flexible, and learnable cues to learn task-uniform and task-specific representations and facilitate cross-task interactions at every layer of the model.

[0003] However, building multi-task image sentiment analysis models using cue learning still faces several difficulties and challenges. First, unlike other computer vision research areas that heavily rely on objective semantics, image sentiment analysis requires models to have a highly abstract understanding of images; therefore, pre-trained models may not be able to capture sentiment-related information well. Furthermore, because human emotion involves a subjective cognitive process, the performance of general visual cues may be limited by the significant emotional gap between low-level pixels and high-level emotions. These factors restrict the application of cue learning in visual task models. Summary of the Invention

[0004] This invention aims to address the lack of emotional language cues in multi-task image sentiment analysis. To this end, this invention enhances the understanding of emotions in emotional images across multiple visual tasks by generating emotional cues from emotional captions of emotional images, thereby improving the accuracy of image sentiment analysis models in understanding and interpreting emotions in images. This provides a multi-task image sentiment analysis method based on emotional cues generation.

[0005] This invention provides a multi-task image sentiment analysis method based on sentiment cues, comprising the following steps: S100: Extracts features from emotional images using multiple learnable visual task cues. A unified feature representation is obtained, and the unified feature representation is decoupled to obtain a first task feature representation for the image sentiment captioning generation task, a second task feature representation for the image sentiment classification task, and a third task feature representation for the weakly supervised sentiment intensity prediction task. S200: Convert the first task feature representation into word probabilities, sample the word probabilities through greedy beam search, and generate sentiment captions; S300: Convert the emotion captions that do not contain the corresponding emotion in the emotion image into emotion cues, calculate the first cosine similarity between the emotion cues and the feature representation of the second task, and calculate the second cosine similarity between the emotion cues and the feature representation of the third task. S400: Aggregate the region feature representations in the third task feature representation to generate the background representation, and calculate the third cosine similarity between the background representation and the second task feature representation; S500: Use the second cosine similarity to optimize the weakly supervised sentiment intensity prediction task, and use the first and third cosine similarities to optimize the image sentiment classification task to obtain an image sentiment analysis model. S600: Input the emotional image into the image sentiment analysis model to obtain the sentiment analysis results.

[0006] Furthermore, according to the multi-task image sentiment analysis method based on sentiment cues provided by the present invention, step S100, which involves extracting a unified feature representation, includes: S110: Input multiple learnable visual task cues and emotional images into the visual encoder; S120: The emotional image is scaled up using a bilinear interpolation algorithm to obtain a standard image, and the standard image is cropped to obtain multiple image patches; S130: Set position encoding for multiple image patches, extract features from multiple image patches with position encoding, and obtain features of multiple image patches; S140: Obtain a unified feature representation based on feature learning from multiple image patches.

[0007] Furthermore, in the multi-task image sentiment analysis method based on sentiment cues provided by the present invention, in step S100, the task of decoupling the unified feature representation is performed by the last layer of the visual encoder.

[0008] Furthermore, according to the multi-task image sentiment analysis method based on sentiment cues provided by the present invention, step S200 includes: S210: Input sentiment tags, and use the sentiment tags as the condition for predicting sentiment captions; S220: Aggregate all feature representations in the first task feature representation to obtain the first global feature representation; S230: Using a text decoder, the first global feature representation is converted into word probabilities based on the sentiment tags.

[0009] Furthermore, in the multi-task image sentiment analysis method based on sentiment cues provided by the present invention, step S300, the step of obtaining sentiment cues includes: S310: Convert emotional captions not included in the emotional image into probabilities of false emotional words using a text decoder; S320: Obtain false sentiment subtitles by probabilistically sampling false sentiment words through greedy beam search; S330: Generates emotional cues by converting fake emotional captions using a text encoder.

[0010] Furthermore, according to the multi-task image sentiment analysis method based on sentiment cues provided by the present invention, step S300, the step of obtaining the first cosine similarity includes: S311: Aggregate all feature representations in the second task feature representation to obtain the second global feature representation; S312: Using two fully connected layers of the visual encoder, the second global feature representation and sentiment cues are mapped to a unified feature space to obtain the first cosine similarity.

[0011] Furthermore, in the multi-task image sentiment analysis method based on sentiment cues provided by the present invention, step S300, the step of obtaining the second cosine similarity, includes: S321: Use a class activation function to map the regional feature representation in the third task feature representation to obtain the sentiment intensity map; S322: Flatten the sentiment intensity map into a sentiment intensity vector; S323: Use the sentiment intensity vector to guide the aggregation of the region feature representations to obtain sentiment intensity feature representations; S324: Using two fully connected layers, the sentiment intensity feature representation and sentiment cues are mapped to a unified feature space to obtain the second cosine similarity.

[0012] Furthermore, according to the multi-task image sentiment analysis method based on sentiment cues provided by the present invention, the background representation in step S400 is generated guided by the sentiment intensity vector.

[0013] Furthermore, according to the multi-task image sentiment analysis method based on sentiment cues provided by the present invention, step S500 further includes: The weakly supervised sentiment intensity prediction task is optimized by minimizing the second cosine similarity. The image sentiment classification task is optimized by minimizing the first cosine similarity and the third cosine similarity.

[0014] Furthermore, according to the multi-task image sentiment analysis method based on sentiment cues provided by the present invention, the expression for minimizing the first cosine similarity is: It minimizes the first cosine similarity. It is the second global feature representation. It's an emotional cue. It is the first fully connected layer. It is the second fully connected layer.

[0015] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: This invention provides a multi-task image sentiment analysis method based on sentiment cues. Multiple learnable task visual cues are input into a visual encoder along with sentiment images. Features are extracted from the sentiment images to obtain unified feature representations for multiple visual tasks. Task feature representations for each visual task are then decoupled and generated. The first global feature representation corresponding to the image sentiment caption generation task is input into a text decoder to generate diverse sentiment captions containing different emotions. Sentiment captions for emotions not present in the sentiment images are input into the text encoder to generate sentiment cues. By minimizing the first cosine similarity between the sentiment cues and the second task feature representation of the image sentiment classification task, and minimizing the second cosine similarity between the sentiment cues and the third task feature representation of the weakly supervised sentiment intensity prediction task, the image sentiment classification task and the weakly supervised sentiment intensity prediction task are optimized in opposite directions, keeping interfering factors away from the target areas of these two visual tasks. Simultaneously, this invention also uses the background representation of the weakly supervised sentiment intensity prediction task to optimize the image sentiment classification task. By minimizing the third cosine similarity between the background representation and the feature representation of the image sentiment classification task, the influence of the background region on the target area of ​​the image sentiment classification task is suppressed. This invention utilizes learnable visual task cues to enhance the learning capabilities of image sentiment caption generation, image sentiment classification, and weakly supervised sentiment intensity prediction tasks. By using sentiment cues, it ensures that images lacking sentiment are located away from the target regions in the sentiment images identified by the image sentiment classification and weakly supervised sentiment intensity prediction tasks. This enhances the accuracy of the image sentiment classification task in classifying sentiment within images and the accuracy of the weakly supervised sentiment intensity prediction task in understanding the intensity of sentiment within images. Furthermore, by combining learnable visual task cues, sentiment cues, and background representation, this invention improves the accuracy of multi-task image sentiment analysis methods, thereby enhancing the image sentiment analysis model's understanding of image sentiment and leading to better generation of image sentiment interpretations.

[0016] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a multi-task image sentiment analysis method based on sentiment cues provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but cannot be used to limit the scope of this invention.

[0020] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples, without contradiction. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0021] The following is combined with Figure 1 This invention describes a multi-task image sentiment analysis method based on sentiment cues generated by the present invention.

[0022] This invention provides a multi-task image sentiment analysis method based on sentiment cues, comprising the following steps: S100: Extracts features from emotional images using multiple learnable visual task cues. A unified feature representation is obtained, and the unified feature representation is decoupled to obtain a first task feature representation for the image sentiment captioning generation task, a second task feature representation for the image sentiment classification task, and a third task feature representation for the weakly supervised sentiment intensity prediction task.

[0023] The step of extracting unified feature representation in step S100 includes: S110: Input multiple learnable visual task cues and emotional images into the visual encoder; S120: The emotional image is scaled up using a bilinear interpolation algorithm to obtain a standard image, and the standard image is cropped to obtain multiple image patches; S130: Set position encoding for multiple image patches, extract features from multiple image patches with position encoding, and obtain features of multiple image patches; In one embodiment, the input sentiment image is scaled down to a standard image of 256×256 pixels using a bilinear interpolation algorithm, and the standard image is cropped into 196 image patches. The number of image patches can be adjusted according to the needs of the image sentiment analysis task. A class token is input into the visual encoder, and positional encoding is set for multiple image patches and the class token.

[0024] S140: Obtain a unified feature representation based on feature learning from multiple image patches.

[0025] An image sentiment analysis model is constructed based on a visual encoder. The image sentiment analysis model includes multiple visual tasks, namely, image sentiment caption generation, image sentiment classification, and weakly supervised sentiment intensity prediction.

[0026] In one embodiment, the visual encoder is a self-attention-based Vision Transformer network. Through multiple learnable visual task cues, the Vision Transformer network extracts features from the input emotion image to obtain a unified feature representation, and uses the unified feature representation extracted by the last layer of the Vision Transformer.

[0027] The expression for a visual encoder to extract a unified feature representation for multiple visual tasks is as follows: , , , in, It is the first layer of the Vision Transformer network. It is the first Vision Transformer network. layer, It serves as the input to the visual encoder, including emotion images, class tokens, location encodings, and multiple learnable visual task cues. It is a token-like identifier for Layer 1 of the Vision Transformer network. It is the first -1 level class token, It is the first Layer-level class tokens; These are multiple learnable visual task cues in Layer 1. It is the first Multiple learnable visual task cues at layer -1, No. Multiple learnable visual task cues for the layers, These are image patch features from layer 1. It is the first Image patch features at layer -1 It is the first Image patch features of the layer, It is a join operation. This refers to the number of layers in the Vision Transformer network; you can set it to 12 or 24 layers.

[0028] Based on learnable visual task cues, the final layer of the Vision Transformer network is decoupled from the unified feature representation to obtain the first task feature representation for image sentiment caption generation, the second task feature representation for image sentiment classification, and the third task feature representation for weakly supervised sentiment intensity prediction, as shown in the following expressions: , Among them, It is the first The visual task in the first The complete feature representation of the layer. It is the first The visual task in the first Layer regional feature representation, It is the first -1st floor Learnable visual task cues corresponding to each visual task.

[0029] Each visual task corresponds to a learnable visual task cue.

[0030] By aggregating all feature representations of the second task feature representation, a second global feature representation is obtained; by aggregating all feature representations of the third task feature representation, a third global feature representation is obtained. This can be further understood as the regional feature representation being a part of the global feature representation, and the third task feature representation includes the regional feature representation of the weakly supervised sentiment intensity prediction task.

[0031] For the image sentiment captioning generation task, image sentiment classification task, and weakly supervised sentiment intensity prediction task, corresponding learnable visual task cues are set. The unified feature representation includes multiple learnable visual task cues and multiple image patch features with positional encoding. The specific feature representation includes one learnable visual task cues and multiple image patch features with positional encoding, which are the task feature representations corresponding to each visual task. The multiple learnable visual task cues, the class tokens with positional encoding, and the multiple image patches with positional encoding learn the unified feature representation and the specific feature representation, respectively. S200: Convert the first task feature representation into word probabilities, sample the word probabilities through greedy beam search, and generate sentiment captions;

[0032] Step S200 includes: S210: Input sentiment tags, and use the sentiment tags as the condition for predicting sentiment captions; S220: Aggregate all feature representations in the first task feature representation to obtain the first global feature representation; S230: Using a text decoder, the first global feature representation is converted into word probabilities based on the sentiment tags.

[0033] In one embodiment, input sentiment tags Emotional tags It refers to the linguistic interpretation of generated emotions by emotional images, and the emotion tags. The setting depends on what emotion the user wants to output for the input emotional image.

[0034] The first global feature representation of the image sentiment captioning generation task is aggregated using token-based aggregation, and the first global feature representation and sentiment labels are then combined. The words are input together into the text decoder to obtain word probabilities, as shown in the following expression: , in It is a text decoder. It is the probability of a word. It is the first global feature representation.

[0035] The present invention provides a multi-task image sentiment analysis method based on sentiment cues, which further includes optimizing the image sentiment caption generation task using the Kullback-Leibler divergence loss function.

[0036] In one embodiment, a greedy beam search sampling method is used to obtain emotional captions. Emotional subtitles It is in text format, and the training objective is to predict sentiment labels using an autoregressive approach. For conditional sentiment subtitles, cross-entropy loss is used as the subtitle loss. This is used to optimize the image emotion captioning generation task, i.e., the corresponding learnable visual task cues. ,

[0037] in, It is the number of emotional images. It is the number of emotion categories. These are emotional subtitles in numerical format mapped from the dictionary database. It represents the predicted word probability for the image sentiment captioning generation task.

[0038] in, This can be further interpreted as referring to the first The first image sentiment interpretation The mapping of emotional captions in text format to their corresponding dictionary entries; Image sentiment analysis models The probability of the predicted word.

[0039] A dictionary is included in the dataset of the image sentiment analysis model. S300: Convert the emotion captions that do not contain the corresponding emotion in the emotion image into emotion cues, calculate the first cosine similarity between the emotion cues and the feature representation of the second task, and calculate the second cosine similarity between the emotion cues and the feature representation of the third task.

[0040] Step S300, the steps for obtaining the emotion cues, include: S310: Convert emotional captions not included in the emotional image into probabilities of false emotional words using a text decoder;

[0041] In one embodiment, defining an image as not containing emotion is... The text decoder conversion did not include emotion. The probability of the corresponding false sentiment words is obtained, as shown in the following expression: in It represents the probability that the image does not contain any false sentiment words corresponding to the sentiment. S320: Obtain false sentiment subtitles by probabilistically sampling false sentiment words through greedy beam search;

[0042] S330: Generates emotional cues by converting fake emotional captions using a text encoder.

[0043] In one embodiment, false sentiment words are probabilistically sampled using a greedy beam search to obtain false sentiment captions. , to use fake emotional subtitles The input is fed into a text encoder to generate sentiment cues, expressed as follows: , in It is a text encoder. It's an emotional cue. These are fake emotional subtitles.

[0044] Step S300, the step of obtaining the first cosine similarity, includes: S311: Aggregate all feature representations in the second task feature representation to obtain the second global feature representation; S312: Using two fully connected layers of the visual encoder, the second global feature representation and sentiment cues are mapped to a unified feature space to obtain the first cosine similarity.

[0045] In step S300, the step of obtaining the second cosine similarity includes: S321: Use a class activation mapping function to map the regional feature representations in the third task feature representation to obtain the sentiment intensity map; S322: Flatten the sentiment intensity map into a sentiment intensity vector; S323: Use the sentiment intensity vector to guide the aggregation of the region feature representations to obtain sentiment intensity feature representations;

[0046] In one embodiment, an initial sentiment intensity map is obtained from the region feature representation of the weakly supervised sentiment intensity prediction task through class activation mapping. The emotion intensity map is then flattened into an emotion intensity vector. Since fully connected layers cannot directly align the region feature representation and the sentiment cues, the sentiment intensity vector is used to guide the aggregation of the region feature representations to obtain the sentiment intensity feature representation, as shown in the following expression: in, This represents the intensity of emotion. It is an intensity vector The first in vector elements, It is an intensity vector The first in vector elements, It is the first Image patch features.

[0047] S324: Using two fully connected layers, the sentiment intensity feature representation and sentiment cues are mapped to a unified feature space to obtain the second cosine similarity.

[0048] In one embodiment, two fully connected layers are used, namely the first fully connected layer. Second fully connected layer The emotional intensity features are represented by... Mapped to a unified feature space, the emotion intensity feature representation and the emotion cue are calculated. The first cosine similarity between them. S400: Aggregate the region feature representations in the third task feature representation to generate the background representation, and calculate the third cosine similarity between the background representation and the second task feature representation;

[0049] The background representation in step S400 is generated under the guidance of the emotion intensity vector.

[0050] In one embodiment, the emotion intensity vector generated in step S300 above is further used. Aggregate sentiment intensity map with regional features The background representation is expressed as follows: in, As background information, It is an intensity vector The first in vector elements, It is an intensity vector The first in vector elements, It is the first Image patch features. S500: Use the second cosine similarity to optimize the weakly supervised sentiment intensity prediction task, and use the first and third cosine similarities to optimize the image sentiment classification task to obtain an image sentiment analysis model.

[0051] Step S500 further includes: The weakly supervised sentiment intensity prediction task is optimized by minimizing the second cosine similarity. The image sentiment classification task is optimized by minimizing the first cosine similarity and the third cosine similarity.

[0052] In one embodiment, the weakly supervised sentiment intensity prediction task is optimized by minimizing the second cosine similarity between the sentiment cue and the region feature representation of the weakly supervised sentiment intensity prediction task. This is achieved using two fully connected layers, namely the first fully connected layer... Second fully connected layer The region feature representation and the sentiment cues are mapped to a unified feature space. Then, the second cosine similarity between the region feature representation and the sentiment cues is minimized, as shown in the following expression: It is the minimized second cosine similarity. This is the representation of the emotional intensity feature. It is the first fully connected layer. It is the second fully connected layer. The emotion cue is optimized by using the region feature representation in the third feature representation to minimize the second cosine similarity with the emotion cue, thereby optimizing the parameters of the learnable visual task cue corresponding to the weakly supervised image region intensity prediction task.

[0053] In one embodiment, the image sentiment classification task in the visual task aligns the task feature representation with the sentiment cues through a fully connected layer, that is, aligns the second global feature representation with the sentiment cues, and learns based on the sentiment cues.

[0054] The approach used is to employ two fully connected layers, namely the first fully connected layer. Second fully connected layer The second global feature representation With the aforementioned emotional cues Mapping to a unified feature space and minimizing the first cosine similarity between the second global feature representation and the sentiment cue, the expression is as follows: It minimizes the first cosine similarity, making the sentiment cues (i.e., the image does not contain sentiment) far from the target region of the image sentiment classification task. It optimizes the image sentiment classification task from the opposite direction, so that the sentiment prior is correctly and fully integrated into the image sentiment analysis model. The sentiment prior is the image sentiment label that is set by the user in the image sentiment analysis model.

[0055] This invention further utilizes the task loss function for image sentiment classification, optimizing the image sentiment classification task through the task loss function and the first cosine similarity, specifically by minimizing the first cosine similarity. The parameters of the learnable visual task cues are optimized using Kullback-Leibler divergence. The task loss function for image sentiment classification is calculated using Kullback-Leibler divergence. The present invention will and When applied to image sentiment classification tasks, this approach optimizes the task from two different perspectives: optimizing the parameters of the corresponding learnable visual task cues.

[0056] Among them, the task loss function The expression is as follows: in, It is the number of emotional images. It is the number of emotion categories. It is the first The emotional image in the first A true distribution of emotions, It is the first The emotional image in the first The predicted distribution of an image sentiment analysis model.

[0057] The image sentiment analysis model includes image sentiment classification task parameters that have been trained using a large number of labeled sentiment images. Unlabeled sentiment images are input into the image sentiment analysis model to generate an image sentiment classification task loss function.

[0058] This invention also uses third cosine similarity to optimize image sentiment classification tasks, employing two fully connected layers, namely the first fully connected layer. With the second fully connected layer The emotional cues are combined with the background representation. Mapping to a unified feature space, the third cosine similarity between the background representation and the second feature representation corresponding to the image sentiment classification task is calculated, and the minimized third cosine similarity is obtained. The expression is as follows: To mitigate the influence of background regions on the image sentiment classification task, we minimize the third cosine similarity between the region feature representations of the weakly supervised sentiment intensity prediction task and the feature representations of the sentiment image classification task. Background regions are image patches whose features are irrelevant to the sentiment representation; the feature representation of these irrelevant regions constitutes the background representation.

[0059] S600: Input the emotional image into the image sentiment analysis model to obtain the sentiment analysis results. During the training of the image sentiment analysis model, the following overall objective function is used to optimize the training of the model, expressed as follows: in, , , and These are single objective functions, i.e. , , , and The trade-offs between them, setting =0.5, = = =0.2, the image sentiment analysis model has the best effect in understanding the emotions in the emotional images.

[0060] Using the above methods, input emotional images to obtain sentiment analysis results.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-task image sentiment analysis method based on sentiment cues, characterized in that, Includes the following steps: S100: Extracts features from emotional images using multiple learnable visual task cues. A unified feature representation is obtained, and the unified feature representation is decoupled to obtain a first task feature representation for the image sentiment captioning generation task, a second task feature representation for the image sentiment classification task, and a third task feature representation for the weakly supervised sentiment intensity prediction task. S200: Convert the first task feature representation into word probabilities, sample the word probabilities through greedy beam search, and generate sentiment captions; S300: Convert the emotional captions that do not contain the corresponding emotions in the emotional image into emotional cues, calculate the first cosine similarity between the emotional cues and the feature representation of the second task, and calculate the second cosine similarity between the emotional cues and the feature representation of the third task. S400: Aggregate the region feature representations in the third task feature representation to generate the background representation, and calculate the third cosine similarity between the background representation and the second task feature representation; S500: Use the second cosine similarity to optimize the weakly supervised sentiment intensity prediction task, and use the first and third cosine similarities to optimize the image sentiment classification task to obtain an image sentiment analysis model. S600: Input the emotional image into the image sentiment analysis model to obtain the sentiment analysis results.

2. The multi-task image sentiment analysis method based on sentiment cues as described in claim 1, characterized in that, The steps in step S100 to extract the unified feature representation include: S110: Input multiple learnable visual task cues and emotional images into the visual encoder; S120: The emotional image is scaled up using a bilinear interpolation algorithm to obtain a standard image, and the standard image is cropped to obtain multiple image patches; S130: Set position encoding for multiple image patches, extract features from multiple image patches with position encoding, and obtain features of multiple image patches; S140: Obtain a unified feature representation based on feature learning from multiple image patches.

3. The multi-task image sentiment analysis method based on sentiment cues as described in claim 1, characterized in that, In step S100, the task of decoupling the unified feature representation is performed by the last layer of the visual encoder.

4. The multi-task image sentiment analysis method based on sentiment cues as described in claim 1, characterized in that, Step S200 includes: S210: Input sentiment tags, and use the sentiment tags as the condition for predicting sentiment captions; S220: Aggregate all feature representations in the first task feature representation to obtain the first global feature representation; S230: Using a text decoder, the first global feature representation is converted into word probabilities based on the sentiment tags.

5. The multi-task image sentiment analysis method based on sentiment cues as described in claim 1, characterized in that, In step S300, the steps for obtaining the emotion cues include: S310: Convert emotional captions not included in the emotional image into probabilities of false emotional words using a text decoder; S320: Obtain false sentiment subtitles by probabilistically sampling false sentiment words through greedy beam search; S330: Generates emotional cues by converting fake emotional captions using a text encoder.

6. The multi-task image sentiment analysis method based on sentiment cues as described in claim 1, characterized in that, In step S300, the step of obtaining the first cosine similarity includes: S311: Aggregate all feature representations in the second task feature representation to obtain the second global feature representation; S312: Using two fully connected layers of the visual encoder, the second global feature representation and sentiment cues are mapped to a unified feature space to obtain the first cosine similarity.

7. The multi-task image sentiment analysis method based on sentiment cues as described in claim 1, characterized in that, In step S300, the step of obtaining the second cosine similarity includes: S321: Use a class activation function to map the regional feature representation in the third task feature representation to obtain the sentiment intensity map; S322: Flatten the sentiment intensity map into a sentiment intensity vector; S323: Use the sentiment intensity vector to guide the aggregation of the region feature representations to obtain sentiment intensity feature representations; S324: Using two fully connected layers, the sentiment intensity feature representation and sentiment cues are mapped to a unified feature space to obtain the second cosine similarity.

8. The multi-task image sentiment analysis method based on sentiment cues as described in claim 7, characterized in that, The background representation in step S400 is generated under the guidance of the emotion intensity vector.

9. A multi-task image sentiment analysis method based on sentiment cues as described in claim 1, characterized in that, Step S500 further includes: The weakly supervised sentiment intensity prediction task is optimized by minimizing the second cosine similarity. The image sentiment classification task is optimized by minimizing the first cosine similarity and the third cosine similarity.

10. A multi-task image sentiment analysis method based on sentiment cues as described in claim 9, characterized in that, The expression for minimizing the first cosine similarity is: It minimizes the first cosine similarity. It is the second global feature representation. It's an emotional cue. It is the first fully connected layer. It is the second fully connected layer.

Images