An image filter generation method based on sentiment analysis

Abstract

The application provides an image filter generation method based on emotion analysis, introduces prior knowledge of visual emotion analysis, adopts a pre-trained BERT model and a VAD emotion dictionary, effectively extracts emotions in the text, and improves the perception ability of the model to the emotions of the text. Through learnable grouping coding, the content image coding and the text coding features are fused into a cross-modal sequence, the model is given cross-modal perception ability, and the ability of the image to express the text emotion under the control of the text is improved. The emotion vector loss function and the emotion classification loss function are designed, the objective and subjective emotions are comprehensively displayed, and the model accurately reflects the text emotion in the generated image. Significant performance improvement is achieved in the aspects of emotion display degree, content retention degree, image generation quality and condition consistency.

Description

Technical Field

[0001] This invention relates to the field of Internet technology, and in particular to an image filter generation method based on sentiment analysis. Background Technology

[0002] With the rapid development of artificial intelligence technology, AI-generated content (AIGC), as an innovative technological application, has gradually attracted widespread attention and research. This invention relates to a content generation method based on artificial intelligence technology, which can automatically generate various forms of content, such as text, images, and audio. This content is generated by computer programs without the need for manual human creation. This method achieves automatic content generation by utilizing large-scale datasets and deep learning models, such as neural networks and language models. Since traditional image creation requires high levels of drawing and artistic skills from designers and is time-consuming and resource-intensive, this method can create artistic images more efficiently, meeting the goal of nationwide creative expression and gaining widespread attention and popularity.

[0003] Sentiment analysis, also known as opinion mining, is a natural language processing technique used to identify and analyze the emotional tendencies or sentiments expressed in text. This technology aims to automatically identify positive, negative, or neutral sentiments from text data, helping to understand people's feelings and emotional attitudes in various texts such as social media, product reviews, and news articles. In the field of artificial intelligence, researchers hope to imbue machines with emotions. The concept of affective computing was first proposed by the MIT Media Lab in 1997. In subsequent research, affective computing has become an important area of ​​natural language processing, aiming to enable computers to automatically identify and understand the emotional information contained in text, giving machines the ability to recognize, understand, and express emotions.

[0004] Sentiment analysis can be applied to various fields, such as social media monitoring, brand reputation management, marketing, public opinion analysis, product promotion, and customer service. It can help businesses understand user satisfaction with their products or services, identify potential user needs and pain points, and improve customer satisfaction and loyalty. Implementing sentiment analysis typically involves using machine learning and deep learning techniques to build sentiment classification models, classify text data, and determine its sentiment orientation. Common sentiment analysis methods include rule-based methods, sentiment dictionary methods, and machine learning methods, with deep learning methods performing exceptionally well when handling large-scale text data. However, sentiment analysis also faces challenges such as semantic understanding, textual ambiguity, and cultural differences, and therefore continues to evolve and improve.

[0005] In today's society, people share their opinions and experiences on events or activities on social networks, with text becoming a medium for directly expressing opinions and establishing emotional connections with other users. For example, phrases like "I didn't do well on my exam today, I'm in a terrible mood" or "I'm about to take the train for a trip, it's such an exciting journey!" often contain emotional vocabulary such as opinions and intentions, and users may want to achieve a corresponding emotional style transfer through emotionally charged statements. This poses a significant challenge to previous models, as they cannot extract emotional information from the text, making it difficult to understand the emotions conveyed and generate styles and textures that correspond to the emotions expressed in the text. Therefore, this invention proposes an image filter generation method based on sentiment analysis. This method allows users to input emotional text and content images. The model extracts the emotions from the text and performs corresponding sentiment transfer on the content images, ultimately generating new images that reflect the emotions while preserving the content.

[0006] like Figure 1 As shown in the paper "Image Style Transfer Based on Single Text Conditions," one of the existing technologies introduces the CLIP cross-modal large model to understand the sentiment in the text and project it onto the generated image.

[0007] First, to address the issue of text-image correspondence, this invention introduces a CLIP cross-modal large model and designs a CLIP loss. By calculating the CLIP distance between texts, the style transfer of the final generated image is guided. Second, to better project the sentiment in the text into the generated image, this invention employs a method of randomly segmenting the generated image, taking multiple small image patches for CLIP loss calculation. Finally, this invention proposes a fast style transfer method, making the model applicable to images with any content.

[0008] like Figure 2 As shown, the second existing technology, "Language-Driven Art Style Transfer," designs a discriminator and a contrastive learning loss, which effectively preserves the content of the generated graph. Figure 1 To:

[0009] First, this invention designs a structural reconstruction module to ensure that the content of the generated image is consistent with the content image. Second, to establish a visual semantic connection between the text and the content image, this invention introduces a block-by-block style discriminator. By randomly segmenting the generated image and inputting it into the style discriminator, it is determined whether the text and content image match. Finally, this invention proposes a contrastive learning module. For different style instructions and content images, a relative matching relationship is defined by calculating the cosine similarity between reference style images. In this process, even if paired style instructions are different, style features still need to be aligned to achieve a certain degree of matching.

[0010] During their research, the inventors discovered that, regarding existing technologies for "image style transfer based on single text conditions" and "language-driven art style transfer":

[0011] 1. Utilize the prior knowledge of CLIP cross-modal large model to learn the sentiment in the text and project the sentiment into the generated image;

[0012] 2. By using discriminators and contrastive learning methods, the content structure of the content image and the style texture of the style image are learned and well represented in the generated image;

[0013] The aforementioned technical problems result in the following drawbacks in the existing technology:

[0014] 1. The model's generated graphs focus more on the text content, resulting in poor preservation of the content of the user input image;

[0015] 2. The CLIP model cannot fully reflect the emotion of the text, and the model lacks the extraction and understanding of the emotion of the text, which results in the emotion in the text not being correctly reflected in the generated image. Summary of the Invention

[0016] To address the aforementioned technical problems, this invention provides an image filter generation method based on sentiment analysis. This method incorporates prior knowledge of visual sentiment analysis, utilizes a pre-trained BERT model to encode text into word vectors, and further obtains sentiment embeddings from the VAD sentiment dictionary to learn the inherent sentiment attributes of each word, thereby enhancing the model's ability to perceive emotions in text. Furthermore, this invention designs an image filter model with a grouped Transformer architecture, fusing text encoding and image encoding. To comprehensively visualize both objective and subjective emotions, this invention also designs a novel loss function. Through these innovations, the model more accurately represents the emotions in the text in the generated images.

[0017] This invention provides an image filter generation method based on sentiment analysis. During model training, the method includes:

[0018] Step 1: Randomly select a text description from the ArtEmis dataset as the anchor text. Then, extract the style map corresponding to the anchor text.

[0019] Step 2: Classify the anchor texts based on their position in the Mikels emotion wheel. Select one text from the descriptions of texts in the same region as the anchor text as a positive example, one text from the descriptions of texts in adjacent regions as a relevant example, and one text from the descriptions of texts in opposite regions as a negative example. Then, extract the style maps corresponding to the positive example, relevant example, and negative example texts.

[0020] Step 3: Encode the anchor text into word vectors using pre-trained BERT, and further obtain sentiment vectors from the VAD dictionary to reveal the inherent sentiment attributes of each word. Then, project the word vectors and sentiment vectors to form text word encoding based on the anchor text.

[0021] Step 4: Input the content image into the image encoder to obtain image block encoding;

[0022] Step 5: Concatenate the image block vector and the text word encoding based on the anchor text into a sequence, and add modality type encoding to the encoding of the two different modalities respectively to obtain the feature fusion cross-modal encoding sequence based on the anchor text.

[0023] Step 6: Input the cross-modal encoding sequence based on anchor text into the grouped Transformer for feature fusion, and fuse the features of the cross-modal sequence to obtain the feature-fused cross-modal sequence based on anchor text.

[0024] Step 7: Decompose the cross-modal sequence of anchor text-based feature fusion, encode the image patches in it and input them into the decoder to obtain the generated image based on anchor text;

[0025] Step 8: Replace the anchor text in Step 3 with positive example text, relevant text, and negative example text respectively. Repeat Step 5, Step 6, and Step 7 to obtain the generated graph based on positive example text, relevant text, and negative example text.

[0026] Step 9: Input all generated images into the sentiment extractor to extract the sentiment vectors of the corresponding images. Calculate the sentiment vector loss function through comparative learning. Then input the generated images into the sentiment classifier and use Kullback-Leibler (KL) loss to calculate the sentiment classification loss function.

[0027] Step 10: Calculate the content loss by combining the generated image with the content image, calculate the style loss by combining the generated image with the style image, and calculate the conditional loss by combining the generated image with the input text. Sum the above losses in a weighted manner to obtain the image visualization loss function.

[0028] Step 11: Sum the sentiment vector loss function, sentiment classification loss function, and image visualization loss function in a weighted manner to obtain the overall loss function. Calculate the gradient using the back propagation (BP) algorithm and use the Adam optimizer to optimize the overall loss function and update the weights of each layer of the model.

[0029] Furthermore, in the non-training case, when generating image filters based on sentiment analysis, steps two, eight, nine, ten, and eleven are removed, and steps one, three, five, six, and seven are replaced, as follows:

[0030] Step 1: Use the emotional text entered by the user as input;

[0031] Step 3: Encode the user-input text into word vectors using pre-trained BERT, and further obtain sentiment vectors from the VAD dictionary to reveal the inherent sentiment attributes of each word. Then, project the word vectors and sentiment vectors to form the text word encoding.

[0032] Step 5: Concatenate the image patch vector and text word encoding into a sequence, and add modality type encoding to the encodings of the two different modalities respectively to obtain the feature fusion cross-modal encoding sequence;

[0033] Step 6: Input the cross-modal encoded sequence into the grouped Transformer for feature fusion, fuse the features of the cross-modal sequence, and obtain the feature-fused cross-modal sequence;

[0034] Step 7: Split the cross-modal sequence of feature fusion, encode the image blocks in it and input them into the decoder to obtain the final generated image.

[0035] Furthermore, in step two, the input to the model is represented as follows:

[0036] Input = {T seed ,T pos ,T rel ,T neg}

[0037] Among them, T seed ,T pos ,T rel ,T neg These represent anchor text, positive example text, related text, and negative example text, respectively.

[0038] Furthermore, in step six, the grouped Transformer contains several computational units, and the computation process of each computational unit is as follows:

[0039]

[0040]

[0041] Among them, f tex and f img These are text encoding and image encoding, respectively. and Modal type encoding is used. Furthermore, the initial input to the grouped transformer is represented as:

[0042]

[0043] The grouped transformer consists of L standard transformer modules, each containing a multi-head attention layer (MSA), an MLP layer, and two residual connections. LN represents layer normalization.

[0044]

[0045]

[0046] Furthermore, in step nine, the sentiment extractor extracts multi-level features from the image's VGG network, performs Gram matrix operations on these features, and thereby defines the sentiment vector, as follows:

[0047]

[0048] in, This represents the i-th upper triangular element of the j-th eigenvalue in the Gram matrix.

[0049] Therefore, the distance between emotion vectors can be expressed by the formula:

[0050] F sw (V i V j )=||V i -V j || / F dis (V i V j )

[0051] The distance between emotions is defined as the minimum number of steps F required to travel from one emotional region to another. dis It is defined as the minimum number of steps required to move from one emotional zone to another.

[0052] Therefore, the sentiment vector loss function is:

[0053] L sw =max((F sw (V seed V pos )-F sw (V seed V rel )+a),0)+max((F sw (V seed V rel )-F sw(V seed V neg )+b),0)

[0054] Where the hyperparameters a = 0.02 and b = 0.01. V seed V pos V rel V neg These are the sentiment vectors of the images corresponding to the anchor text, the images corresponding to the positive example text, the images corresponding to the relevant text, and the images corresponding to the negative example text, respectively.

[0055] Furthermore, in step nine, a pre-trained sentiment classifier σ is used, and the sentiment classification loss function is calculated using the Kullback-Leibler (KL) loss, as shown below:

[0056] F ed =∑d i ln(d i / σ(I out ) i )

[0057] Among them, σ(I out ) i and d i These represent the estimated class distribution and the actual class distribution of the generated graph, respectively. This helps image sentiment filter models more accurately reflect the emotions in the text into the image.

[0058] Furthermore, in step ten, a content loss mechanism is designed to maintain content consistency between the generated graph and the user-provided content graph, as shown below:

[0059]

[0060] in, and These are the features extracted at level i for the generated graph and the content graph, respectively.

[0061] In step ten, a style loss is used to reduce the style differences between extracted features through a pre-trained VGG network, as shown below:

[0062]

[0063] Where u and σ are the mean function and variance function, respectively. and These are the features extracted from the generated image and the style image at level i, respectively.

[0064] In step ten, a multi-level conditional and unconditional discriminator is designed to align the generated image with the user-provided text and to distinguish whether the sentiment of the generated image is the same as the sentiment provided in the text, as shown below:

[0065] L GAN =logD(I acr )+log(1-D(G(I pat ,T tex )))+logD(I acr ,T tex )+log(1-D(G(I pat ,T tex ),T tex ))

[0066] Where D is the discriminator, G is the image emotion filter model, and I... pat I acr T tex These are the content image, the style image, and the input text, respectively.

[0067] Furthermore, in step eleven, this embodiment trains on four NVIDIA TITAN RTX GPUs with a batch size of 24 and performs 80,000 training iterations using the Adam optimizer and a learning rate of 0.0005.

[0068] This invention provides an image filter generation method based on sentiment analysis. It introduces prior knowledge of visual sentiment analysis, using a pre-trained BERT model and a VAD sentiment dictionary to effectively extract sentiment from text, enhancing the model's ability to perceive sentiment within the text. By employing learnable group coding, it fuses features from content image coding and text coding into cross-modal sequences, endowing the model with cross-modal perception capabilities and improving the ability to make images reflect the sentiment of the text under textual control. Through the design of sentiment vector loss functions and sentiment classification loss functions, it comprehensively visualizes objective and subjective sentiment, enabling the model to more clearly represent the sentiment in the generated image. This achieves performance improvements in sentiment representation, content preservation, image generation quality, and conditional consistency. Attached Figure Description

[0069] Figure 1 This is a schematic diagram of "image style transfer based on a single text condition";

[0070] Figure 2 This is a schematic diagram of "language-driven artistic style transfer";

[0071] Figure 3 This is a flowchart of Example 1;

[0072] Figure 4 The flowchart illustrates an image filter generation method based on sentiment analysis provided by this invention. Detailed Implementation

[0073] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention. The abbreviations and key terms appearing in this embodiment are defined as follows:

[0074] AIGC: Artificial Intelligence Generated Content;

[0075] BP: Back Propagation;

[0076] CLIP: Contrastive Language-Image Pre-Training utilizes contrastive learning to pre-train large language-image models;

[0077] VAD: Valence-Arousal-Dominance uses these three attributes to evaluate the sentimental properties of a word;

[0078] ViT:Vision Transformer is a model proposed by the Google team in 2020 that applies the Transformer to image classification.

[0079] ReLU: Rectified Linear Unit, is an activation function.

[0080] BERT: Bidirectional Encoder Representation from Transformers is a pre-trained model based on the Transformer bidirectional encoder representation.

[0081] Mikels' emotion wheel: a method for labeling the distribution of emotions;

[0082] Gram matrix: The Gram matrix can be used to calculate the eccentric covariance matrix between image feature layers;

[0083] VGG: A convolutional neural network architecture;

[0084] KL Loss: KL (Kullback-Leibler) divergence can be used to measure the similarity between two probability distributions. The more similar the two probability distributions are, the smaller the KL divergence.

[0085] Adam: A method for stochastic optimization.

[0086] Example 1

[0087] Reference Figure 3 , 4 As shown, Figure 3 Figure 4 illustrates an image filter generation method based on sentiment analysis provided by the present invention. Specifically, during model training, the method includes:

[0088] Step 1: Randomly select a text description from the ArtEmis dataset as the anchor text. Then, extract the style map corresponding to the anchor text.

[0089] Step 2: Classify the anchor texts based on their position in the Mikels emotion wheel. Select one text from the descriptions of texts in the same region as the anchor text as a positive example, one text from the descriptions of texts in adjacent regions as a relevant example, and one text from the descriptions of texts in opposite regions as a negative example. Then, extract the style maps corresponding to the positive example, relevant example, and negative example texts.

[0090] Furthermore, in step two, the input to the model is represented as follows:

[0091] Input = {T seed ,T pos ,T rel ,T neg}

[0092] Among them, T seed ,T pos ,T rel ,T neg These represent anchor text, positive example text, related text, and negative example text, respectively.

[0093] Step 3: Encode the anchor text into word vectors using pre-trained BERT, and further obtain sentiment vectors from the VAD dictionary to reveal the inherent sentiment attributes of each word. Then, project the word vectors and sentiment vectors to form text word encoding based on the anchor text.

[0094] Step 4: Input the content image into the image encoder to obtain image block encoding;

[0095] Step 5: Concatenate the image block vector and the text word encoding based on the anchor text into a sequence, and add modality type encoding to the encoding of the two different modalities respectively to obtain the feature fusion cross-modal encoding sequence based on the anchor text.

[0096] Step 6: Input the cross-modal encoding sequence based on anchor text into the grouped Transformer for feature fusion, and fuse the features of the cross-modal sequence to obtain the feature-fused cross-modal sequence based on anchor text.

[0097] Furthermore, in step six, the grouped Transformer contains several computational units, and the computation process of each computational unit is as follows:

[0098]

[0099]

[0100] Among them, f tex and f img These are text encoding and image encoding, respectively. and Modal type encoding is used. Furthermore, the initial input to the grouped transformer is represented as:

[0101]

[0102] The grouped transformer consists of L standard transformer modules, each containing a multi-head attention layer (MSA), an MLP layer, and two residual connections. LN represents layer normalization.

[0103]

[0104]

[0105] Step 7: Decompose the cross-modal sequence of anchor text-based feature fusion, encode the image patches in it and input them into the decoder to obtain the generated image based on anchor text;

[0106] Step 8: Replace the anchor text in Step 3 with positive example text, relevant text, and negative example text respectively. Repeat Step 5, Step 6, and Step 7 to obtain the generated graph based on positive example text, relevant text, and negative example text.

[0107] Step 9: Input all generated images into the sentiment extractor to extract the sentiment vectors of the corresponding images. Calculate the sentiment vector loss function through comparative learning. Then input the generated images into the sentiment classifier and use Kullback-Leibler (KL) loss to calculate the sentiment classification loss function.

[0108] Furthermore, in step nine, the sentiment extractor extracts multi-level features from the image's VGG network, performs Gram matrix operations on these features, and thereby defines the sentiment vector, as follows:

[0109]

[0110] in, This represents the i-th upper triangular element of the j-th eigenvalue in the Gram matrix.

[0111] Therefore, the distance between emotion vectors can be expressed by the formula:

[0112] F sw (V i V j )=||V i -V j || / F dis (V i V j )

[0113] The distance between emotions is defined as the minimum number of steps F required to travel from one emotional region to another. dis It is defined as the minimum number of steps required to move from one emotional zone to another.

[0114] Therefore, the sentiment vector loss function is:

[0115] L sw =max((F sw (V seed V pos )-F sw (V seed V rel )+a),0)+max((F sw (V seed V rel )-F sw (V seed V neg )+b),0)

[0116] Where the hyperparameters a = 0.02 and b = 0.01. V seed V pos V rel V negThese are the sentiment vectors of the images corresponding to the anchor text, the images corresponding to the positive example text, the images corresponding to the relevant text, and the images corresponding to the negative example text, respectively.

[0117] Furthermore, in step nine, a pre-trained sentiment classifier σ is used, and the sentiment classification loss function is calculated using the Kullback-Leibler (KL) loss, as shown below:

[0118] F ed =∑d i ln(d i / σ(I out ) i )

[0119] Among them, σ(I out ) i and d i These represent the estimated class distribution and the actual class distribution of the generated graph, respectively. This helps image sentiment filter models more accurately reflect the emotions in the text into the image.

[0120] Step 10: Calculate the content loss by combining the generated image with the content image, calculate the style loss by combining the generated image with the style image, and calculate the conditional loss by combining the generated image with the input text. Sum the above losses in a weighted manner to obtain the image visualization loss function.

[0121] Furthermore, in step ten, a content loss mechanism is designed to maintain content consistency between the generated graph and the user-provided content graph, as shown below:

[0122]

[0123] in, and These are the features extracted at level i for the generated graph and the content graph, respectively.

[0124] In step ten, a style loss is used to reduce the style differences between extracted features through a pre-trained VGG network, as shown below:

[0125]

[0126] Where u and σ are the mean function and variance function, respectively. and These are the features extracted from the generated image and the style image at level i, respectively.

[0127] In step ten, a multi-level conditional and unconditional discriminator is designed to align the generated image with the user-provided text and to distinguish whether the sentiment of the generated image is the same as the sentiment provided in the text, as shown below:

[0128] L GAN =logD(I acr )+log(1-D(G(I pat ,T tex )))+logD(I acr ,T tex )+log(1-D(G(I pat ,T tex ),T tex ))

[0129] Where D is the discriminator, G is the image emotion filter model, and I... pat I acr T tex These are the content image, the style image, and the input text, respectively.

[0130] Step 11: Sum the sentiment vector loss function, sentiment classification loss function, and image visualization loss function in a weighted manner to obtain the overall loss function. Calculate the gradient using the back propagation (BP) algorithm and use the Adam optimizer to optimize the overall loss function and update the weights of each layer of the model.

[0131] Furthermore, in step eleven, this embodiment trains on four NVIDIA TITAN RTX GPUs with a batch size of 24 and performs 80,000 training iterations using the Adam optimizer and a learning rate of 0.0005.

[0132] Furthermore, in the non-training case, when generating image filters based on sentiment analysis, steps two, eight, nine, ten, and eleven are removed, and steps one, three, five, six, and seven are replaced, as follows:

[0133] Step 1: Use the emotional text entered by the user as input;

[0134] Step 3: Encode the user-input text into word vectors using pre-trained BERT, and further obtain sentiment vectors from the VAD dictionary to reveal the inherent sentiment attributes of each word. Then, project the word vectors and sentiment vectors to form the text word encoding.

[0135] Step 5: Concatenate the image patch vector and text word encoding into a sequence, and add modality type encoding to the encodings of the two different modalities respectively to obtain the feature fusion cross-modal encoding sequence;

[0136] Step 6: Input the cross-modal encoded sequence into the grouped Transformer for feature fusion, fuse the features of the cross-modal sequence, and obtain the feature-fused cross-modal sequence;

[0137] Step 7: Split the cross-modal sequence of feature fusion, encode the image blocks in it and input them into the decoder to obtain the final generated image.

[0138] A preferred embodiment, such as Figure 3 As shown, firstly, anchor text, positive example text, relevant text, and negative example text are selected from the ArtEmis dataset, and corresponding style images are extracted. Then, the anchor text is encoded into word vectors using a pre-trained BERT, and sentiment vectors are obtained from the VAD dictionary to reveal the inherent sentiment attributes of each word. The word vectors and sentiment vectors are projected to form text word encoding based on the anchor text. Similarly, the content image is input into the image encoder to obtain image patch encoding. Next, the image patch encoding and text word encoding are concatenated into a sequence, and modality type encoding is added to the encodings of the two different modalities. These encodings are input into a grouped Transformer to obtain the corresponding results. The image patch encoding is input into the decoder to generate a generated image based on the anchor text. Finally, the positive example text, relevant text, and negative example text are input into the model to obtain the generated image. The sentiment vector loss function and sentiment classification loss function are calculated through contrastive learning, and then weighted and summed with the content loss function, style loss function, and conditional generative adversarial network loss to obtain the overall loss function. The gradient is calculated using the backpropagation algorithm, and the Adam optimizer is used to optimize the overall loss function to update the weights of each layer of the model. In non-training scenarios, there is no need to extract positive example text, related text, negative example text, and corresponding style images, nor is it necessary to calculate the loss function. The generated image can be obtained by directly inputting the data into the decoder.

[0139] This invention incorporates prior knowledge of visual sentiment analysis, employing a pre-trained BERT model and a VAD sentiment dictionary to effectively extract sentiment from text and enhance the model's ability to perceive textual emotions. Through learnable grouping encoding, content image encoding and text encoding features are fused into a cross-modal sequence, endowing the model with cross-modal perception capabilities and improving the ability of images to express textual emotions under textual control. A sentiment vector loss function and a sentiment classification loss function are designed to comprehensively represent objective and subjective emotions, enabling the model to accurately reflect textual emotions in the generated images. Significant performance improvements are achieved in terms of sentiment representation, content preservation, image generation quality, and conditional consistency.

[0140] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0141] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An image filter generation method based on sentiment analysis, characterized by The method adopts a grouped Transformer architecture, and designs a sentiment vector loss and a sentiment classification loss to better learn the sentiment information in the text, and the method comprises the following steps: Step one, randomly select a text description as an anchor text from the data set, and take out the style graph corresponding to the anchor text; Step two, classify the anchor text according to its position in the Mikels emotion wheel, and select a text in the same region as the anchor text as a positive example text, select a text in the adjacent region as a related text, and select a text in the opposite region as a negative example text, and take out the style graph corresponding to the positive example text, the related text and the negative example text; Step three, encode the anchor text into a word vector using a pre-trained BERT, and further obtain a sentiment vector in a VAD dictionary to reveal the inherent sentiment attribute of each word, and project the word vector and the sentiment vector to form a text word encoding based on the anchor text; Step four, input the content image into the image encoder to obtain the image block encoding; Step five, splice the image block encoding and the text word encoding based on the anchor text into a sequence, and add a modal type encoding to the encoding of the two different modalities respectively to obtain a feature fusion cross-modal encoding sequence based on the anchor text; Step six, input the cross-modal encoding sequence based on the anchor text into the grouped Transformer for feature fusion to obtain a feature fusion cross-modal sequence based on the anchor text; Step seven, split the feature fusion cross-modal sequence based on the anchor text, and input the image block encoding into the decoder to obtain a generated image based on the anchor text; Step eight, replace the anchor text in step three with the positive example text, the related text and the negative example text respectively, and repeat steps five, six and seven to obtain generated images based on the positive example text, the related text and the negative example text; Step nine, input all the generated images into the sentiment extractor to extract the sentiment vector of the corresponding image, calculate the sentiment vector loss function through contrastive learning, input the generated image into the sentiment classifier, and calculate the sentiment classification loss function using the Kullback-Leibler (KL) loss; Step ten, calculate the content loss of the generated image and the content image, the style loss of the generated image and the style image, and the conditional loss of the generated image and the input text, and then weight and sum the above losses to obtain an image visualization loss function; Step eleven, weight and sum the sentiment vector loss function, the sentiment classification loss function and the image visualization loss function to obtain an overall loss function, calculate the gradient using the Back Propagation (BP) algorithm, and update the weights of each layer of the model using the Adam optimizer to optimize the overall loss function.

2. The method of claim 1, wherein, In step six, a plurality of calculation units are used in the grouped Transformer, and the calculation process of each calculation unit is as follows: For image patch encoding ( ) and text word encoding ( ), a multi-layer perceptron and are used respectively. The role of the multi-layer perceptrons is to project the image patch encodings and the text word encodings into a shared latent space, which is represented as: wherein, and is a modality type encoding to distinguish different modalities, and are projected and modality encoded feature vectors, thus, the initial input of the grouping transformer is represented as: wherein the group transformer comprises L standard transformer modules, each module comprising a multi-head attention layer (MSA), an MLP layer and two residual connections, LN denotes layer normalization, and for the i-th module its computation process is as follows:​ wherein, is an input to a th module, is an intermediate variable, is an output from a th module; Further, in step nine, the emotion extractor extracts multi-level features of the image VGG network, performs Gram matrix operation on the features, and thereby defines an emotion vector, as follows: wherein, represents the upper triangular element of the Gram matrix at the is the number of upper triangular elements in each Gram matrix, is the total number of feature levels;​​ Thus, the weighted distance between two arbitrary emotion vectors and can be formulated as: where w is a weighting factor. wherein the distance between the emotions is defined as the minimum number of steps from one emotional region to another emotional region ; Therefore, the emotion vector loss function is: wherein the hyperparameters = 0.02, = 0.01 are hyperparameters that control the boundaries between the emotion vectors, , , , are the emotion vectors of the anchor text corresponding image, the positive example text corresponding image, the related text corresponding image, and the negative example text corresponding image, respectively.

3. The method of claim 1, wherein, In step nine, a pre-trained sentiment classifier is used and the sentiment classification loss function is computed using the Kullback-Leibler (KL) divergence as follows: wherein, and respectively represent the class distribution of the generated image estimation and the actual class distribution, which helps the image sentiment filter model to more accurately reflect the sentiment in the text to the image.

4. The method of claim 1, wherein, In step ten, a content loss is designed to keep the generated image consistent with the content image provided by the user in terms of content, as follows: wherein, and are features extracted at the first level for the generated graph and the content graph, respectively; In step ten, the pre-trained VGG network is used to reduce the style difference between the extracted features by using a style loss as follows: where u and are mean and variance functions, respectively, and are features extracted from the generated image and style image at the first stage, respectively. In step ten, a multi-stage conditional and unconditional discriminator corresponding discriminator loss is designed The purpose is to align the generated image with the text provided by the user, and distinguish whether the generated image emotion is the same as the emotion provided in the text, as follows: Wherein, D is a discriminator, the generator G is an image emotion filter model, , , are content images, style images and input text respectively.

Images