Image sentiment synthesis method based on diffusion model

Abstract

The application provides an image emotion synthesis method based on a diffusion model, and belongs to the technical field of image emotion synthesis. The image emotion synthesis method based on the diffusion model comprises the following steps: S1, embedding a to-be-generated image in a hidden space; S2, performing emotion injection on an input prompt description; S3, for the clean image embedding in S1, the emotion label in S2 is optimized by minimizing the emotion loss through reverse gradient optimization; S4, controlling the emotion injection timing according to the image-text similarity; and S5, optimizing the emotion label by using a multi-emotion cross-entropy loss function. The image emotion synthesis framework optimizes the emotion label by minimizing the cross-entropy emotion loss, and introduces the labels at the best time to guide the image emotion synthesis. The image emotion synthesis framework is superior to the current emotion image generation method in the accuracy of the emotion generation task and the image quality, and also performs well in the emotion editing task.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image sentiment synthesis, in particular to an image sentiment synthesis method based on a diffusion model. BACKGROUND

[0002] At present, image sentiment synthesis is a new technology aiming to generate or edit image content according to text prompts to evoke specific emotions of users. Existing methods usually treat sentiment generation and editing as independent tasks. Due to the subjectivity and abstraction of emotions, these methods usually associate target emotions with specific attributes (such as color and style), use large visual language models or fine-tune models on sentiment datasets to modify these attributes. The core idea of these methods usually strongly couples emotions with certain fixed elements, thus limiting the performance of sentiment editing and generation tasks in many aspects such as semantic diversity.

[0003] Early sentiment image editing methods mainly transfer emotions by editing low-level features such as color and texture. For example, some methods select theme candidates by establishing a relationship between color and emotion vocabulary, while other methods combine emotion classifiers for color constraints and use colorization networks to enhance emotional expression. In addition, higher-level style information is also important for emotional expression. Some studies propose a multi-emotion semantic space to realize multi-emotion transfer; some studies extract emotional cues from text to guide color conversion.

[0004] Generation models (such as diffusion models) have been successfully applied to complex tasks such as image and video generation, and have been first used for sentiment generation. Researchers have realized the mapping between emotions and semantic spaces through networks, allowing abstract emotions to be translated into specific image attributes. In addition, sentiment editing methods based on large models have also attracted widespread attention. For example, some methods use GPT-4 to extract an emotion factor tree to determine the most relevant emotional attributes, and then modify them through specific models;

[0005] The prior art in the above has the following defects: image sentiment synthesis aims to generate media content consistent with specific emotions, and its methods mainly include two tasks of image generation and image editing. At present, there is still a lack of a flexible framework that can handle both sentiment generation and editing tasks with high emotional precision and high quality. SUMMARY

[0006] In order to make up for the above shortcomings, the present application provides an image sentiment synthesis method based on a diffusion model, aiming to improve the problem of lacking a flexible framework that can handle both sentiment generation and editing tasks with high emotional precision and high quality.

[0007] The embodiment of the present application provides an image sentiment synthesis method based on a diffusion model, including the following steps:

[0008] S1, for the image embedding to be generated under the hidden space, using single-step inference at each time step, using noise map inversion approximation to obtain the estimated clean image embedding;

[0009] S2, the input prompt description is injected with emotion, additional emotional labels are added to the original text, and the input is input into the diffusion model in the emotion guiding process, and new emotional label weights are updated at each inference step;

[0010] S3, for the clean image embedding in S1, the pre-trained emotion classifier is used for emotion classification, the cross-entropy loss is used to calculate the gap with the target emotion, and the emotional loss is minimized by optimizing the emotional label in S2 through the backpropagation gradient.

[0011] S4, the emotion injection timing is controlled according to the image-text similarity, for the clean image embedding in S1, the image-text semantic similarity score is obtained at each step, when the semantic similarity between the synthesized image and the target reaches or exceeds the set threshold, the emotional label is introduced, the time of emotional expression is ensured, and the diversity of semantics is ensured.

[0012] S5, for the emotional loss, the multi-emotion cross-entropy loss function is used to optimize the emotional label.

[0013] In a preferred mode of the present application, the single-step inference of the noise map in S1 utilizes the deterministic sampling characteristics of the DDI M model, allowing high-quality images to be quickly generated from a small number of sampling steps; without adding additional random noise, the original image's latent representation is recovered by predicting noise and gradually reducing noise.

[0014] In a preferred mode of the present application, the emotional label added after the text embedding in S2 is connected to the original text prompt using an additional text through the same text encoder.

[0015] The predicted noise is not directly backpropagated, avoiding the conflict between the gradient of the emotion classifier and the gradient generated by the text prompt, more stably integrating emotional information, and maintaining the consistency of the text content.

[0016] In a preferred mode of the present application, in S3, the emotional image generated at each time step is input into the emotion classifier to calculate its emotional label, and the similarity cross-entropy loss with the target emotion is calculated, and the gradient is optimized in the reverse direction, and the weight of the added emotional label is optimized.

[0017] In a preferred mode of the present application, S4 uses cosine similarity to calculate the text and image features during the image generation process, and uses a temperature factor to scale the results to obtain the image-text semantic similarity.

[0018] better control. Because, the introduction of emotion guidance too early will cause the deviation of image content, the complexity and uncontrollability of emotion will conflict with the latent space and image guidance, resulting in the loss of content of text prompt; the introduction of emotion too late, in the later stage of image generation, will make it difficult to achieve the expected emotional effect of the image.

[0019] In a preferred mode of the present application, the CLIP is used to obtain the semantic similarity score of text and image, and the score is compared with the set threshold.

[0020] In a preferred mode of the present application, the image and text encoder of CLIP is used And The scaled CLIP score between the generated content and the text prompt is measured at each time step t (Temperature factor τ = 0.07):

[0021] Where τ represents the temperature factor, P represents the text prompt, The denoising embedding obtained from .

[0022] In a preferred mode of the present application, due to the diversity and complexity of emotion in S5, the target emotion will be affected by the original emotion under the text description and the similar emotion in the emotion wheel, and the new emotion loss adopts contrast learning respectively using cross entropy loss to suppress it.

[0023] In a preferred mode of the present application, in order to make the synthesis process approach the target emotion, the following target emotion loss is proposed:

[0024]

[0025] Where, The loss function based on the target emotion, The inherent emotion loss, The similar emotion loss, λ1 and λ2 are hyperparameters.

[0026] In a preferred mode of the present application, the emotion synthesis framework based on the classifier simultaneously completes the emotion image generation and editing tasks in the accuracy of emotion awakening and image quality.

[0027] Beneficial effects:

[0028] 1. Through the method of emotional guidance, the timing of emotional guidance, and the accurate injection of emotion, the three aspects of emotional image generation and emotional editing tasks are efficiently completed.

[0029] 2、Specifically, the prior knowledge of the pre-trained sentiment classifier guides the synthesis, and the gradient back propagation is used to optimize the pre-defined sentiment label to ensure the stability of the synthesis.

[0030] 3、The sentiment guidance is too early or too late, which will destroy the image content or fail to add the target emotion; therefore, the semantic similarity is used as a supervision signal to determine the best time to apply the sentiment guidance.

[0031] 4、A multi-sentiment cross-entropy loss is proposed to reduce the interference of inherent emotions and similar emotions, thereby improving the accuracy of sentiment synthesis; the image sentiment synthesis framework optimizes the sentiment label by minimizing the cross-entropy sentiment loss, and introduces these labels at the best time to guide the image sentiment synthesis. The image sentiment synthesis framework is superior to the current sentiment image generation method in terms of accuracy of sentiment generation task and image quality, and also performs well in sentiment editing task. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0033] Figure 1 The overall structure of the sentiment synthesis framework proposed by the present application is shown in the figure;

[0034] Figure 2 The comparison chart of the semantic similarity control effect proposed by the present application is shown in the figure;

[0035] Figure 3 The comparison chart of the sentiment suppression method proposed by the present application is shown in the figure;

[0036] Figure 4 The actual sentiment image effect chart generated by the present application is shown in the figure;

[0037] Figure 5 The actual effect chart of the sentiment image editing by the present application is shown in the figure. DETAILED DESCRIPTION

[0038] In the present application, unless otherwise explicitly specified and limited, the first feature is "on" or "under" the second feature, which can include that the first and second features are in direct contact, or that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, the first feature "on", "above" and "over" the second feature includes that the first feature is directly above and obliquely above the second feature, or only indicates that the first feature is higher in horizontal height than the second feature. The first feature "under", "below" and "under" the second feature includes that the first feature is directly below and obliquely below the second feature, or only indicates that the first feature is lower in horizontal height than the second feature.

[0039] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

[0040] Please refer to Figures 1-5 The present application provides an image emotion synthesis method based on a diffusion model, as shown in Figure 1 The present application provides an image emotion synthesis method based on a diffusion model, as shown in

[0041] S1, for the hidden space to be generated image embedding, using single step inference at each time step, inverting the approximation of the noise map, obtaining the estimated clean image embedding;

[0042] S2, the input prompt description is injected with emotion, and additional emotion labels are added to the original text. In the emotion guiding process, input the diffusion model, and update the new emotion label weight at each inference step;

[0043] S3, for the clean image embedding in S1, use the pre-trained emotion classifier to perform emotion classification, use cross entropy loss to calculate the gap with the target emotion, and optimize the emotion label in S2 by minimizing the emotion loss through backpropagation;

[0044] S4, according to the image-text similarity, the emotion injection time is controlled, for the clean image embedding in S1, the image-text semantic similarity score is obtained at each step, when the semantic similarity between the synthesized image and the target reaches or exceeds the set threshold, the emotion label is introduced, the time of emotion expression is guaranteed, and the diversity of semantics is ensured;

[0045] S5, for the emotion loss, use multi-emotion cross entropy loss function to optimize the emotion label.

[0046] Allowing to quickly generate high-quality images from a small number of sampling steps; without adding additional random noise, the original image latent representation is recovered by predicting noise and gradually reducing noise.

[0047] For emotion image generation, given an initial random noise image The corresponding text prompt P is input at the same time, the text prompt P is input to a text encoder to obtain a starting text embedding The text embedding and initial random noise are input into the diffusion model before emotion guidance, to predict the next noise, where z0represents the initial latent embedding obtained by VAE encoder for a given image I0, represents the gradually reduced noise intensity:

[0048]

[0049] The image embedding under the next noise t-1 can be calculated by the predicted noise,

[0050]

[0051] where the network parameters θ are trained by minimizing the Mean Squreed Error (MSE) between the predicted noise and the real noise, where represents sampling a sample pair z0,t from a standard normal distribution:

[0052]

[0053] The model is trained to recover the initial embedding distribution z T from the noise samples z T in an iterative manner, where z T is sampled from the distribution z T . In order to introduce conditional information c in the generation process, a classifier-free guidance method is used, which uses a pre-trained classifier p(c∣x t ) to guide the noise ∈ θ in the inference process, and controls the guidance intensity of the condition c by a scalar ω greater than 0

[0054]

[0055] In order to support more complex conditions, the emotion guidance uses a method of replacing the explicit classifier with an implicit classifier: during the training process, the condition is not used, but the same network is used to calculate and This method is called "Classifier-Free Guidance (CFG)", also controlled by a scalar ω:

[0056]

[0057] Image synthesis emotion guidance: an additional pre-trained emotion classifier is used as guidance to utilize its learned emotion prior. The original text prompt P is passed through a text encoder Then connect with sentiment label S to obtain sentiment text embedding Here The connection operation represents the embedding, and then, in c emo The de-noising embedding is calculated under the condition In order to make the pre-trained sentiment classifier on clean images Be able to adapt to noisy data well, DDIM single step inference method is adopted to obtain estimated clean embedding from each time step t Specific as follows:

[0058]

[0059] Among them Then optimize the sentiment label S through the gradient of back propagation, minimize the sentiment loss between the target sentiment y target And the predicted sentiment As follows:

[0060]

[0061] This method involves using a pre-trained sentiment classifier to change the weight of the sentiment cue word in the text input at each de-noising step, and inferring the current noisy image. The model can learn how to synthesize sentiment in the presence of noise, thereby improving performance and accuracy on noisy data.

[0062] Design optimization additional sentiment label S ' Provides two main advantages:

[0063] (1) Directly use the prior knowledge of the most advanced sentiment classifier, eliminate the need to create large-scale sentiment manipulation data sets. Given the subjectivity of sentiment and the huge effort required to create such data sets.

[0064] (2) S ' And P together as framework input, allows ∈ θ Consider the semantic and sentiment information in the prompt when estimating noise. This prevents gradient inconsistency, avoids ignoring text information, and ensures stable synthesis.

[0065] Finally, since the sentiment label S only affects the input layer of the model, there is no need for further complex operations on the intermediate embedding. The framework can be easily applied to image generation and editing tasks by simply adjusting the input (P, S) and noise sampling z T .

[0066] ​In the application of generation: image sentiment generation can be divided into with and without text description. Both cases start from random noise. In the application of editing: for sentiment editing tasks, images input text prompts P according to their content, and initial noise z T By DDIMinversion on the initial image I.

[0067] The cosine similarity is used to calculate the text and image features, and the temperature factor is used to scale the results to obtain the text-image semantic similarity;

[0068] Semantic similarity supervision: introducing target sentiment guidance at the optimal stage can suppress inherent sentiment expression while maintaining semantic accuracy, ensuring the expression of the desired sentiment. To achieve this without increasing the complexity of denoising, the image and text encoders of CLIP are used And The scaled CLIP score between the generated content and the text prompt is measured at each time step t (Temperature factor τ = 0.07):

[0069]

[0070] Where is the denoising embedding obtained from In the reverse process, once Exceeds the preset threshold η, which indicates that the image content has been initially formed, then optimize and introduce sentiment guidance S:

[0071]

[0072] Is an empty placeholder. This mechanism enables the invention to introduce target sentiment at the appropriate stage, ensuring the conveyance of target sentiment while maintaining the diversity and accuracy of semantics, such as Figure 2 Demonstrates the contrast effect of whether to perform semantic supervision, and the appropriate timing of sentiment guidance can ensure the quality and accuracy of the image.

[0073] Due to the diversity and complexity of emotions, the target emotion will be affected by the original emotion under the text description and the similar emotion in the emotion wheel, and the new emotion loss adopts a contrast learning respectively using cross-entropy loss to suppress it;

[0074] Inherent and similar emotion suppression: the target emotion should be emphasized during the generation process. There are two undesirable emotions that will affect the expression of the target emotion:

[0075] a) Inherent emotion: derived from the source image or text prompt. To suppress inherent emotions, the invention applies a semantic similarity supervision mechanism to obtain Time step t before crossing threshold η ' Embedding of Then use the emotion classifier to predict the intrinsic emotion y from inh and build the loss term:

[0076]

[0077] where l cc is the cross-entropy loss, is the intrinsic emotion loss.

[0078] b) Similar emotions: Similar emotions are related to the target emotion. Unlike other classification tasks, in the emotion classification task, the classes have a relative similarity, rather than a completely independent and mutually exclusive feature. In Mikel's emotion wheel psychological model, adjacent. Similar emotions can cause the guiding emotion to shift towards them during the synthesis process. Similar emotions y target related to the target emotion y sim can be obtained by querying the emotion wheel, and cross-entropy loss can be used to suppress them:

[0079]

[0080] is the similar emotion loss.

[0081] In summary, in order to suppress these two undesirable emotions and force the synthesis process to move towards the target emotion, the following target emotion loss is proposed:

[0082]

[0083] where, is the loss function based on the target emotion, and λ1 and λ2 are hyperparameters used to control the suppression intensity. This loss term can suppress the intrinsic emotion and similar emotion in the synthesized image Figure 3 .

[0084] The target emotion loss combines the suppression of intrinsic emotions and similar emotions, ensuring that the emotional expression during the synthesis process can more accurately conform to the target emotion. In this way, the model can more effectively guide the emotional expression when denoising and synthesizing images, reducing the impact of undesirable emotions, and thus improving the emotional accuracy and quality of the synthesized images.

[0085] In summary, the emotion synthesis framework based on the classifier designed in the present application can simultaneously complete the tasks of generating and editing emotional images, and is superior to the current methods in terms of the accuracy of emotional awakening and image quality.

[0086] Table 1

[0087]

[0088] Table 2

[0089]

[0090] Table 1 shows the comparison of the present application with the existing sota task in the emotion generation task in 6 evaluation indexes. The "↑" symbol in the table indicates that the higher the index, the better the effect, and "↓" indicates the lower the better.

[0091] Table 2 shows the comparison of the present application with the existing sota task in the emotion editing task in 4 evaluation indexes. The "↑" symbol in the table indicates that the higher the index, the better the effect.

[0092] For the generation task, FID, LPIPS, Sem-C, Sem-D, and sentiment accuracy (ACC1 and ACC2) are followed as evaluation indexes. FID and LPIPS measure the fidelity and diversity of images, respectively, while Sem-C and Sem-D evaluate the semantic clarity and content richness of images. Two classifiers are used to measure sentiment accuracy. For the editing task, in addition to the LPIPS, ACC1, ACC2 indexes used in the generation task, the CLIP image score is also used to measure the similarity between the input image and the edited image. The table shows the comparison of the present application with other sota methods in the emotion image generation, image emotion generation under specific prompt words, and emotion image editing tasks, including traditional methods and diffusion model-based methods. The results show that the emotion image synthesis effect of the present application in multiple tasks surpasses all published results.

[0093] The present application is the first framework to simultaneously complete emotion generation and emotion editing. Figure 4 and Figure 5 respectively show the effect of the present application in emotion image generation and editing in different emotions, Figure 4 in which the upper name of each group of images represents the target emotion and the corresponding text prompt description, Figure 5 in which the left of the input image represents the target emotion. It can be seen that, through comparison with different methods, the present application has better generation and editing effect than other image generation methods and editing methods, and has obvious improvement in image quality and generation accuracy, image content diversity.

[0094] The emotion image generation and emotion editing task is efficiently completed through the method of emotion guidance, the timing of emotion guidance, and the accurate injection of emotion.

[0095] Specifically, the prior knowledge of the pre-trained emotion classifier is used to guide the synthesis, and the gradient backpropagation is used to optimize the pre-defined emotion label to ensure the stability of the synthesis.

[0096] Early or late affective guidance can destroy the image content or fail to add the target affect; for this reason, semantic similarity is used as a supervisory signal to determine the best time to apply affective guidance.

[0097] A multi-affect cross-entropy loss is proposed to reduce the interference of inherent and similar affects, thereby improving the accuracy of affect synthesis; the image affect synthesis framework optimizes affect labels by minimizing the cross-entropy affect loss and introduces these labels at the best time to guide image affect synthesis. The image affect synthesis framework outperforms current affect image generation methods in terms of accuracy on affect generation tasks and image quality, and also performs well on affect editing tasks.

[0098] A new framework is designed to address the shortcomings of existing affect image synthesis tasks, which cannot accurately evoke specific affects, lack semantic diversity, and separate affect generation and editing tasks. A simple and effective image affect synthesis framework is proposed that can significantly change image content and does not rely on specialized datasets; it does not limit semantic diversity; and it addresses generation and editing tasks through three key aspects: (1) how to generate affects (2) when to introduce affects (3) which affect to guide the synthesis process. In addition to image and text prompt conditions, affect labels are introduced to ensure stable affect synthesis. The timing of affect introduction is controlled using image-text semantic similarity during the denoising process, and a affect loss function is constructed to optimize the synthesis process by suppressing inherent and similar affects, thereby achieving high affect accuracy. This method outperforms the current state-of-the-art methods in terms of affect accuracy, semantic diversity, and image quality on generation and editing tasks, and does not require specialized datasets or visual large models.

[0099] The above only describes the embodiments of the present application and does not limit the protection scope of the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numbers and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

Claims

1. A method for image emotion synthesis based on a diffusion model, characterized in that, Includes the following steps: S1. For the image embedding to be generated in the hidden space, single-step inference is used at each time step to invert the noise map and obtain the estimated clean image embedding. S2. Inject sentiment into the input prompt description, add additional sentiment labels to the original text, input it into the diffusion model during the sentiment guidance process, and update the new sentiment label weights at each inference step. S3. For the clean image embedding in S1, perform sentiment classification using a pre-trained sentiment classifier, use cross-entropy loss to calculate the difference between the sentiment and the target sentiment, and optimize the sentiment label in S2 by minimizing the sentiment loss through inverse gradient optimization. S4. Control the timing of emotion injection based on image-text similarity. For the clean image embedding described in S1, obtain the image-text semantic similarity score at each step. When the semantic similarity between the synthesized image and the target reaches or exceeds the set threshold, introduce emotion tags to ensure the time of emotion expression and ensure semantic diversity. S5. For sentiment loss, the sentiment label is optimized using the multi-sentiment cross-entropy loss function.

2. The image emotion synthesis method based on a diffusion model according to claim 1, characterized in that, The single-step inference of the noise map in S1 utilizes the deterministic sampling characteristics of the DDIM model, allowing for the rapid generation of high-quality images from a small number of sampling steps; without adding additional random noise, the latent representation of the original image is recovered by predicting the noise and gradually reducing it.

3. The image emotion synthesis method based on a diffusion model according to claim 1, characterized in that, In S2, sentiment tags are appended after the text embedding. The sentiment tags are connected to the original text prompt using additional text processed by the same text encoder.

4. The image emotion synthesis method based on a diffusion model according to claim 1, characterized in that, In S3, for the sentiment image generated at each time step, after single-step inference, it is input into the sentiment classifier to calculate its sentiment label, and the cross-entropy loss of similarity with the target sentiment is calculated. The gradient is optimized in reverse, and the weights of the added sentiment labels are optimized.

5. The image emotion synthesis method based on a diffusion model according to claim 1, characterized in that, In the process of image generation, S4 uses cosine similarity to calculate text and image features, and uses a temperature factor to scale the results to obtain the semantic similarity between the text and the image.

6. The image emotion synthesis method based on a diffusion model according to claim 1, characterized in that, CLIP is used to obtain the semantic similarity score between images and text, and the score is compared with a set threshold.

7. The image emotion synthesis method based on a diffusion model according to claim 6, characterized in that, Image and text encoders using CLIP and Measure the scaling CLIP score between the generated content and the text prompt at each time step t. Where τ represents the temperature factor and P represents the text prompt. Indicates from The resulting denoised embedding.

8. The image emotion synthesis method based on a diffusion model according to claim 1, characterized in that, In S5, due to the diversity and complexity of emotions, the target emotion is affected by the original emotion under the text description and similar emotions in the emotion wheel. The new emotion loss is suppressed by using cross-entropy loss through contrastive learning.

9. The image emotion synthesis method based on a diffusion model according to claim 8, characterized in that, To make the synthesis process more closely resemble the target emotion, the following target emotion loss is proposed: in, It is a loss function based on the target sentiment. It is a loss of inherent emotions. It is a similarity emotion loss, where λ1 and λ2 are hyperparameters.

10. The image emotion synthesis method based on a diffusion model according to claim 1, characterized in that, A classifier-based emotion synthesis framework simultaneously accomplishes the tasks of emotion image generation and editing, achieving both accuracy in emotion arousal and image quality.

Images