A subject consistency-oriented emotion controllable image generation method, device, equipment and medium

Abstract

This application discloses a method, apparatus, device, and medium for generating emotion-controlled images with subject consistency. The method separates stable subject content from variable emotional states using basic description and emotion description. It constructs an emotion consistency loss using the feature differences between predicted emotion features determined by the basic description and emotion category and the emotion features corresponding to the emotion description. It constructs an emotion feature contrast loss using enhanced emotion features and an image supervision loss using generated images. The emotion-controlled generation model is trained using the emotion consistency loss, emotion feature contrast loss, and image supervision loss. This enables the emotion-controlled generation model to learn the relationship between the image subject, the real state, and the emotional semantics. As a result, the emotion-controlled generation model maintains the consistency of the target subject under different emotional conditions while forming emotional states with clear differences, thereby improving the subject consistency of the generated content, the accuracy of emotion control, and the authenticity and rationality of the emotional content.

Description

Technical Field

[0001] This application relates to the field of image generation technology, and in particular to a method, apparatus, device and medium for generating emotion-controlled images with subject consistency. Background Technology

[0002] With the rapid development of generative models, significant progress has been made in techniques for generating images with specific semantic content based on text, tags, and other conditions. Building on this foundation, emotion-controlled content generation no longer focuses solely on whether the generated image possesses an overall emotional tendency; rather, it further requires generative models to achieve stable, accurate, and controllable emotional expression for specific target objects under different emotional conditions. In particular, how to guide the target object to produce corresponding and reasonable state changes based on different emotional conditions, while keeping the core features of the same target object unchanged, has become an important research question worthy of attention in the field of emotion-controlled content generation.

[0003] Existing methods for generating emotion-controlled images primarily guide the generation process through visual features such as emotion category, color, style, and scene atmosphere, resulting in images that exhibit specific emotional characteristics. For example, the method proposed by EmoEdit et al. introduces emotion tags such as "happy" and "sad" as generation or editing conditions, establishing a correspondence between emotion categories and image visual features, thereby achieving content generation with specific emotional tendencies. Another example is the method proposed by Zhu et al., which enhances the model's understanding and expression of different levels of emotional features by modeling multi-scale emotional visual representations, thus improving the quality of generated emotional images.

[0004] However, emotions in real-world visual content are usually generated by concrete, observable changes in state that align with the subject's own attributes, rather than being directly determined by abstract emotional semantics. Generative models primarily focus on the overall emotional features of generated images. When different emotional conditions are applied to the same target object, the appearance, structure, details, and even identity features of the object in the generated result may change unexpectedly with the emotional conditions, making it difficult to maintain consistency between the generated result and the target object itself under different emotional conditions.

[0005] Therefore, the existing technology still needs to be improved and enhanced. Summary of the Invention

[0006] The technical problem to be solved by this application is to provide a method, apparatus, device and medium for generating emotion-controllable images with subject consistency, in order to address the shortcomings of the existing technology.

[0007] To address the aforementioned technical problems, the first aspect of this application provides a method for generating emotion-controlled images based on subject consistency, wherein the method specifically includes: Construct a training dataset comprising several training sentiment data, wherein the training sentiment data includes at least sentiment categories, basic descriptions reflecting the subject of the image, sentiment descriptions reflecting the emotional state of the subject of the image, and image data; Extract the basic semantic features corresponding to the basic description and the sentiment features corresponding to the sentiment description; Based on the aforementioned basic semantic features and the aforementioned sentiment category, predictive sentiment features are generated; An enhanced sentiment feature is generated based on the basic semantic features and the sentiment features, and a state description is generated based on the enhanced sentiment feature and the basic semantic features; The generated image is determined based on the state description using an initial emotion-controlled generation model; Based on the predicted sentiment features and the sentiment features, a sentiment consistency loss is constructed; based on the enhanced sentiment features, a sentiment feature contrast loss is constructed; and based on the generated image and the image data, an image supervision loss is constructed. The initial emotion-controlled generation model is trained based on the emotion consistency loss, the emotion feature contrast loss, and the image supervision loss to obtain the emotion-controlled generation model. The aforementioned emotion-controlled generation model is used to generate emotion-controlled images with subject-specific consistency.

[0008] The aforementioned method for generating emotion-controlled images with subject-consistency features, wherein constructing a training dataset comprising several training emotion data specifically includes: Obtain an image dataset, which contains image data with the same subject but different emotion categories; Generate basic and sentiment descriptions for image data in the image dataset using a visual language model; Training sentiment data is constructed based on image data and its corresponding sentiment categories, basic descriptions, and sentiment descriptions to obtain a training dataset that includes several training sentiment data.

[0009] The aforementioned subject-consistent emotion-controlled image generation method, wherein the step of generating predicted emotion features based on the basic semantic features and the emotion category specifically includes: The emotion categories are encoded to obtain low-dimensional emotion representations; The correspondence between emotional text and visual state changes is learned by using the low-dimensional sentiment representation and the basic semantic features to obtain predicted sentiment features.

[0010] The aforementioned subject-consistent emotion-controlled image generation method, wherein the step of generating enhanced emotion features based on the basic semantic features and the emotion features specifically includes: Based on the image subject, predict the emotional state residual between the basic semantic features and the emotional features; Based on the basic semantic features and the sentiment features, a gating coefficient is predicted, which is used to constrain the matching relationship between the image subject and the image sentiment. The emotional state residual is fused with the gating coefficient to obtain the enhanced state residual; The enhanced state residual is fused with the emotional feature to obtain the enhanced emotional feature.

[0011] The aforementioned subject-consistency-oriented emotion-controlled image generation method, wherein determining the generated image based on the state description using an initial emotion-controlled generation model specifically includes: Obtain a reference image, which is used to provide image subject consistency constraints; The reference image and the state description are input into the initial emotion controllable generation model, and the generated image is output through the initial emotion controllable generation model.

[0012] The aforementioned method for generating emotion-controlled images with subject-consistency orientation, wherein the specific steps of generating emotion-controlled images with subject-consistency orientation using the emotion-controlled generation model include: Receive the emotion category and generate emotional visual cue text based on the emotion category; Based on the emotional visual cue text, the emotionally controllable generation model is used to generate an emotionally controllable image with subject-consistent characteristics to obtain the target generated image.

[0013] The aforementioned method for generating emotion-controlled images with subject-consistency orientation, wherein the specific steps of generating emotion-controlled images with subject-consistency orientation using the emotion-controlled generation model include: Receive an emotion category and a subject description, and generate emotion state-enhanced text based on the emotion category and the subject description; Based on the emotional state-enhanced text, the emotionally controllable generation model is used to generate subject-consistent emotionally controllable images to obtain the target generated image.

[0014] A second aspect of this application provides an emotion-controlled image generation apparatus oriented towards subject consistency, wherein the emotion-controlled image generation apparatus oriented towards subject consistency specifically includes: A construction module is used to construct a training dataset that includes several training sentiment data, wherein the training sentiment data includes at least sentiment categories, basic descriptions reflecting the subject of the image, sentiment descriptions reflecting the emotional state of the subject of the image, and image data. The training module is used to extract basic semantic features corresponding to the basic description and emotional features corresponding to the emotional description; generate predicted emotional features based on the basic semantic features and the emotional category; generate enhanced emotional features based on the basic semantic features and the emotional features, and generate a state description based on the enhanced emotional features and the basic semantic features; determine the generated image based on the state description through an initial emotionally controllable generation model; construct an emotional consistency loss based on the predicted emotional features and the emotional features, construct an emotional feature contrast loss based on the enhanced emotional features, and construct an image supervision loss based on the generated image and the image data; train the initial emotionally controllable generation model based on the emotional consistency loss, the emotional feature contrast loss, and the image supervision loss to obtain an emotionally controllable generation model. The generation module is used to generate emotion-controlled images with subject-oriented consistency using the emotion-controlled generation model.

[0015] A third aspect of this application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the subject-oriented, consistency-based, emotion-controlled image generation method as described above.

[0016] A fourth aspect of this application provides a terminal device, which includes: a processor and a memory; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps in the subject-oriented, consistency-based, emotion-controlled image generation method as described above.

[0017] Beneficial Effects: This application separates stable subject content from variable emotional states through basic description and sentiment description. It uses basic description and sentiment category to determine predicted sentiment features, and constructs a sentiment consistency loss based on the feature differences between the predicted sentiment features and the sentiment features corresponding to the sentiment description. Sentiment features are enhanced using the basic semantic features of the basic description, and a sentiment feature contrast loss is constructed based on the obtained enhanced sentiment features. The generated image is determined based on the enhanced sentiment features and basic semantic features, and an image supervision loss is constructed based on the generated image. Finally, the sentiment consistency loss, sentiment feature contrast loss, and image supervision loss are integrated to train a controllable sentiment generation model. This allows the model to learn the relationship between the image subject, the real state, and the sentiment semantics, thereby enabling the controllable sentiment generation model to maintain the consistency of the target subject under different sentiment conditions while generating emotional states with clear differences. This improves the subject consistency of the generated content, the accuracy of sentiment control, and the authenticity and rationality of the sentiment content. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart of an emotion-controlled image generation method oriented towards subject consistency provided in an embodiment of this application.

[0020] Figure 2 This is a flowchart illustrating the principles of the training process.

[0021] Figure 3 A flowchart illustrating the process of building the training dataset.

[0022] Figure 4 This is the original flowchart for internal emotional monitoring.

[0023] Figure 5 A schematic diagram of the emotion-controlled image generation device for subject consistency provided in this application embodiment.

[0024] Figure 6 A schematic block diagram of the terminal device provided in the embodiments of this application. Detailed Implementation

[0025] This application provides a method, apparatus, device, and medium for generating emotion-controlled images with subject-specific consistency. To make the objectives, technical solutions, and effects of this application clearer and more explicit, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0026] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0027] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0028] It should be understood that the sequence number and size of each step in this embodiment do not imply the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.

[0029] Research has revealed that with the rapid development of generative models, significant progress has been made in techniques for generating images with specific semantic content based on text, tags, and other conditions. Building upon this foundation, emotion-controlled content generation no longer focuses solely on whether the generated image possesses an overall emotional tendency; rather, it further demands that the generative model achieve stable, accurate, and controllable emotional expression for a specific target object under different emotional conditions. In particular, how to guide the target object to produce corresponding and reasonable state changes based on different emotional conditions, while keeping the core features of the same target object unchanged, has become an important research question worthy of attention in the field of emotion-controlled content generation.

[0030] Existing methods for generating emotion-controlled images primarily guide the generation process through visual features such as emotion category, color, style, and scene atmosphere, enabling the final image to exhibit specific emotional characteristics. For example, the method proposed by EmoEdit et al. introduces emotion tags such as "happy" and "sad" as generation or editing conditions, establishing a correspondence between emotion categories and image visual features, thereby achieving the generation of content with specific emotional tendencies. Another example is the method proposed by Zhu et al., which enhances the model's ability to understand and express emotional features at different levels by modeling multi-scale emotional visual representations, thus improving the quality of generated emotional images.

[0031] However, for emotion-controlled image generation, changes in the target object under different emotional conditions must be based on maintaining the consistency of its core features. In other words, when the emotion category changes, the generative model should not achieve emotional differences by arbitrarily altering the object's category, structure, identity, or main appearance features. Instead, it should maintain the basic consistency of the same target object, controlling only changes in emotion-related state attributes, so that different generated results can be identified as the same target object while also presenting differentiated states corresponding to the specified emotion. Therefore, existing methods focusing on controlling the overall emotional features of the generated image may cause unexpected changes in the appearance, structure, details, and even identity features of the object in the generated result when different emotional conditions are applied to the same target object. Ultimately, this makes it difficult to maintain the consistency of the target object itself in the generated results under different emotional conditions.

[0032] It is understandable that controllable emotion generation requires not only accurate expression of emotion categories but also consideration of the consistency of the target object and the rationality of state changes under different emotional conditions. On the one hand, when generating under different emotional conditions, the core characteristics of the same target object, such as its category, structure, identity, and main appearance, should remain relatively stable to avoid irrelevant changes to the target object itself due to changes in emotional conditions, thus ensuring the consistency of the generated content. On the other hand, it is necessary to control the state changes of the object to present clear emotional differences according to different emotion categories, so that the emotion expressed by the object is consistent with the specified emotion, improving the controllability of emotions under different emotional conditions. At the same time, such state changes should not be generated directly based solely on abstract emotional semantics but should be based on the actual existence of the target object and its state in line with real-world cognition, so that the emotional expression is based on real and reasonable changes in the object.

[0033] Therefore, this application focuses on controlling the state changes of the target object according to different emotional conditions while maintaining the consistency of the core features of the same target object, and achieving controllable emotional expression by using realistic, reasonable, and emotionally meaningful object states. By establishing the connection between the target object, the real state, and the emotional semantics, the model can maintain the consistency of the target subject under different emotional conditions, while forming emotional states with clear differences, thereby improving the subject consistency of the generated content, the accuracy of emotional control, and the authenticity and rationality of the emotional content.

[0034] Specifically, this application provides a method for generating emotion-controlled images with subject consistency. The method includes constructing a training dataset containing several training emotion data; extracting basic semantic features corresponding to the basic description and emotion features corresponding to the emotion description; generating predicted emotion features based on the basic semantic features and the emotion category; generating enhanced emotion features based on the basic semantic features and the emotion features, and generating a state description based on the enhanced emotion features and the basic semantic features; determining the generated image based on the state description using an initial emotion-controlled generation model; constructing an emotion consistency loss based on the predicted emotion features and the emotion features, constructing an emotion feature contrast loss based on the enhanced emotion features, and constructing an image supervision loss based on the generated image and the image data; training the initial emotion-controlled generation model based on the emotion consistency loss, the emotion feature contrast loss, and the image supervision loss to obtain an emotion-controlled generation model; and using the emotion-controlled generation model to generate emotion-controlled images with subject consistency.

[0035] This application separates stable subject content from variable emotional states through basic description and sentiment description. It uses basic description and sentiment category to determine predicted sentiment features, and constructs a sentiment consistency loss based on the feature differences between the predicted sentiment features and the sentiment features corresponding to the sentiment description. The sentiment features are then enhanced using the basic semantic features of the basic description, and a sentiment feature contrast loss is constructed based on the enhanced sentiment features. A generated image is determined based on the enhanced sentiment features and basic semantic features, and an image supervision loss is constructed based on the generated image. Finally, the sentiment consistency loss, sentiment feature contrast loss, and image supervision loss are integrated to train a controllable sentiment generation model. This allows the model to learn the state change patterns of different objects under different emotional conditions, establishing a connection between the image subject, the real state, and the sentiment semantics. This avoids fixing a single appearance feature to a specific emotion, enabling the controllable sentiment generation model to maintain the consistency of the target subject while generating clearly differentiated emotional states under different emotional conditions. This improves the subject consistency of the generated content, the accuracy of sentiment control, and the authenticity and rationality of the sentiment content.

[0036] It should be noted that the execution subject of each embodiment of this application can be a computing service system with data processing, network communication, and program execution functions, such as an electronic system capable of realizing the above functions, an emotion-controlled image generation system oriented towards subject consistency, etc. The following uses an emotion-controlled image generation system oriented towards subject consistency (hereinafter referred to as "the system") as an example to describe the following embodiments.

[0037] The application content will be further explained below with reference to the accompanying drawings and the description of the embodiments.

[0038] This embodiment provides a subject-consistent, emotion-controlled image generation method. This method, through emotion rendering, adaptive emotion enhancement, subject-consistent state-aware image generation, and image supervision, gradually transforms abstract emotional conditions into specific visual states matching the current subject, generating emotion-category images while maintaining subject-semantic consistency. In other words, the core idea of ​​this embodiment is to decouple the stable content information of the subject from the variable emotional state information. The subject category, quantity, identity, structure, core actions, and basic scene in the image data are treated as stable semantic information (reflected through basic descriptions), while subject posture, action mode, local morphology, subject relationships, material representation, lighting, color, and composition are treated as emotion-related states (reflected through emotion descriptions). These are kept as unchanged as possible during emotion transformation, and the emotion-related states are adaptively adjusted according to the emotion category. That is, when changing the emotion category, the emotion-controlled generation model does not reconstruct the complete image semantics, but rather, while maintaining the consistency of the core content of the subject, it selectively enhances the emotional state features related to the changed emotion category.

[0039] like Figure 1 and Figure 2 As shown, the emotion-controlled image generation method based on subject consistency specifically includes: S10. Construct a training dataset that includes several training sentiment data.

[0040] Specifically, the training dataset includes several training sentiment datasets. Each training sentiment dataset includes at least a sentiment category, a basic description, a sentiment description, and image data. The sentiment category is the sentiment label for the corresponding image data, such as happy or sad. The basic description reflects the image subject (i.e., the target object in the image data). Through the basic description, we can obtain fixed and unchanging basic content such as the image subject, scene, action, and local features. The sentiment description corresponds to the basic description and reflects the emotional state of the image subject. Through the sentiment description, we can obtain content that reflects emotions, atmosphere, and subjective feelings, such as the subject's posture, expression, action, shape, spatial relationships, and environmental conditions. This training sentiment dataset contains image content, image sentiment state, and sentiment category, establishing a correspondence between image content, image sentiment state, and sentiment category.

[0041] For example, the training sentiment data is: { Image data; Emotion category: Excitement; Basic description: Three women are laughing heartily at a party, creating a lively atmosphere; Emotional description: joyful, relaxed, warm, full of friendship and happiness.

[0042] }

[0043] In one embodiment, the construction of the training dataset comprising several training sentiment data specifically includes: Obtain the image dataset. Generate basic and sentiment descriptions for image data in the image dataset using a visual language model; Training sentiment data is constructed based on image data and its corresponding sentiment categories, basic descriptions, and sentiment descriptions to obtain a training dataset that includes several training sentiment data.

[0044] Specifically, the image dataset can be obtained by selecting from publicly available datasets or constructed through manual collection. The image dataset contains images with the same subject but different emotional categories. In other words, the image dataset needs to cover different subjects and, for the same subject, provide multiple images with different emotional categories. This ensures that each subject is matched with image data under multiple emotions, providing sufficient training material for the model to learn the emotional state features and expressions of different emotions on the same subject. During the acquisition of the image dataset, each image is labeled with an emotional category, such as happy, sad, or excited.

[0045] Furthermore, after acquiring the image dataset, visual language models (such as the Qwen3-VL multimodal large model) can be used to perform visual understanding and semantic analysis on each image, identifying the subject, scene, and emotion-related state features in the image to generate basic and emotional descriptions for the corresponding image data. Before generating the basic and emotional descriptions using the multimodal large model, structured prompts for the visual language model are pre-constructed. These structured prompts are used to generate basic and emotional descriptions for the same subject from a single image dataset, with the basic description corresponding to the emotional description.

[0046] It is understandable that the structured prompt includes two focal points. The first focal point mainly focuses on basic content such as the subject, object, action, and scene, while excluding emotions, expressions, atmosphere, and subjective information. The second focal point, while maintaining the same basic content as the first focal point, focuses on visual cues related to emotional states that further describe basic content such as facial changes, body movements, character interactions, and environment.

[0047] For example, the structured prompt could be: {We are creating training data for a mood-controlled generative model. The mood categories of the image data are: {Mood Category}. Please generate two descriptions.}

[0048] 1. Basic Description: Only mention the main characters, objects, basic actions, and basic scenes; do not describe emotion categories, facial expressions, moods, atmosphere, or personality; do not use subjective words; keep it very brief, within the first word limit.

[0049] 2. Emotion Category Description: Maintain the same subject as in the basic description; explain how the emotion category is manifested in the scene; the emotion category can be manifested through facial expressions, body movements, character interactions, environment, and visual atmosphere; include relevant visual details that support the judgment of the emotion category; do not invent new objects or events; avoid using poetic or exaggerated language; do not simply add emotion category adjectives; the length should be within the second word limit.

[0050] Please return in JSON format: {"Basic Description": "XXX", "Emotion Category Description": "XXX" } }

[0051] Furthermore, such as Figure 3 As shown, after inputting structured prompts and images (i.e., image data) into the visual language model, the model returns basic and sentiment descriptions in JSON format, along with automatically generated JSON annotation data based on the sentiment category, basic description, and sentiment description. Then, the generated JSON-annotated image dataset can be processed. Specifically, the JSON annotation data generated in different batches is organized, merged, and deduplicated. Field names, field types, and description formats are standardized according to a preset data format to form standardized candidate annotation data. Then, the candidate data is checked for one or more of the following: consistency of subject recognition, completeness of annotation fields, matching of basic description with image content, clarity of sentiment description, and whether the JSON annotation data meets the data construction requirements. For image data that passes the check, it is directly retained to form training sentiment data including both image data and JSON annotation data. For image data that fails the check, it can enter a manual verification and modification stage, where the JSON annotation data is manually checked, supplemented, or corrected. The corrected candidate data is then re-entered into the data processing and detection process for verification. Of course, in practical applications, image data that fails the check can also be directly discarded.

[0052] S20. Extract the basic semantic features corresponding to the basic description and the emotional features corresponding to the emotional description.

[0053] Specifically, the basic semantic features are determined by feature encoding the basic description, and the sentiment features are determined by feature encoding the sentiment description. Specifically, the text encoder in the initial sentiment-controlled generation model can be used to encode the basic and sentiment descriptions, converting the basic description into a low-dimensional basic semantic feature vector containing the core information of the image subject. This basic semantic feature retains only stable semantic information related to fixed attributes such as subject category, structure, and core appearance, while stripping away sentiment-related variable state information to maintain stable semantic information as much as possible during sentiment conversion. The sentiment description is then converted into a low-dimensional sentiment feature vector containing sentiment state information to obtain the sentiment features. These sentiment features include variable state information such as subject posture, expression, and environmental atmosphere corresponding to the sentiment category, providing a basis for subsequent sentiment state adjustment.

[0054] Furthermore, in practical applications, when acquiring basic semantic features, a subject reference provided by a reference image can be fused. For example, features can be extracted from the reference image to obtain subject reference semantic features, and then the basic semantic features and subject reference semantic features can be fused to enhance the basic semantic features. In other words, the fused features obtained by fusing the basic semantic features and subject reference semantic features are used as the basic semantic features for subsequent applications.

[0055] It should be noted that in practical applications, a single text encoder can be used to encode the basic description and the sentiment description separately, or a single text encoder can be used to encode the basic description, while another parallel text encoder encodes the sentiment description. The text encoder can be a pre-trained multimodal text encoder, capable of directly extracting features from natural language descriptions without requiring additional pre-training for this task.

[0056] S30. Generate predicted sentiment features based on the basic semantic features and the sentiment category.

[0057] Specifically, the predicted sentiment features are predicted based on the basic semantic features and the sentiment category, and are used to characterize the sentiment state corresponding to the sentiment category. In other words, the predicted sentiment features are sentiment features that include sentiment state, generated based on the stable semantic information of the main image subject in the basic description and the sentiment category as the sentiment condition.

[0058] For example, generating predicted sentiment features based on the basic semantic features and the sentiment category specifically includes: The emotion categories are encoded to obtain low-dimensional emotion representations; The correspondence between emotional text and visual state changes is learned by using the low-dimensional sentiment representation and the basic semantic features to obtain predicted sentiment features.

[0059] Specifically, low-dimensional sentiment representation is a low-dimensional dense sentiment category feature vector obtained by transforming discrete sentiment categories. For example, a sentiment embedding encoder can be used to embed and encode sentiment categories to transform discrete sentiment categories into low-dimensional dense sentiment category feature vectors, thus obtaining this low-dimensional sentiment representation.

[0060] Then, the low-dimensional sentiment representation is concatenated and fused with the basic semantic features. The sentiment rendering unit learns the mapping relationship between the basic semantic features and sentiment categories to the corresponding sentiment features, and finally outputs the predicted sentiment features that match the current subject and sentiment category. The sentiment rendering unit can be trained using LoRA based on the Qwen2.5-7B model. Through LoRA fine-tuning, Qwen2.5-7B can learn the correspondence between sentiment text and changes in visual state.

[0061] For example, this predicted sentiment feature can be represented as: e =ζ( f b ,z), f b = E text ( S b ), z = E ( C ), in, e Indicates the predicted sentiment features, f b Let z represent the basic semantic features, z represent the low-dimensional sentiment representation, and ζ represent the sentiment rendering unit. S b This represents the basic description. C Indicates the emotion category, E text Indicates a text encoder. E This represents an emotion embedding encoder.

[0062] S40. Generate enhanced emotional features based on the basic semantic features and the emotional features, and generate a state description based on the enhanced emotional features and the basic semantic features.

[0063] Specifically, since sentiment features primarily provide the overall direction of emotion change, but different subjects exhibit different states of the same emotion, joint modeling using basic semantic features and sentiment features is employed to improve the matching degree between sentiment states and the stable semantic information of the image subject. This involves semantically enhancing sentiment features through basic semantic features, ensuring that the enhanced sentiment features retain the state features corresponding to the sentiment category while maintaining consistency between the sentiment state and the target subject's attributes. In other words, the enhanced sentiment features, after adaptively adjusting the sentiment features, result in sentiment state features that integrate the stable semantic constraints of the subject. These features simultaneously preserve the sentiment state corresponding to the sentiment category and the constraint information corresponding to the subject's core attributes, avoiding the generation of unreasonable sentiment states that contradict the subject's physical characteristics and realistic cognition.

[0064] In one embodiment, generating enhanced emotional features based on the basic semantic features and the emotional features specifically includes: Based on the image subject, predict the emotional state residual between the basic semantic features and the emotional features; Predict the gating coefficient based on the basic semantic features and the sentiment features; The emotional state residual is fused with the gating coefficient to obtain the enhanced state residual; The enhanced state residual is fused with the emotional feature to obtain the enhanced emotional feature.

[0065] Specifically, such as Figure 4 As shown, the sentiment state residual is the feature component that needs to be adjusted for sentiment features under the constraints of basic semantic features. It represents the feature difference between the sentiment category and the current sentiment feature. Through this sentiment state residual, sentiment features can be adaptively converted into sentiment category features that match the subject, while maintaining the main content of the image without irrelevant changes. The sentiment state residual can be obtained using a sentiment residual prediction network. That is, the basic semantic features and sentiment features are input into the sentiment residual prediction network, which then predicts the sentiment state residual based on the image subject. This sentiment state residual can be expressed as: Δf = F ( f b , f 0 e ), f 0 e = E text ( S e ), in, Δf Indicates residual emotional state. F This represents an emotional residual prediction network.f 0 e Indicates emotional characteristics, f b Represents basic semantic features. S e Indicates emotional description, E text This indicates a text encoder.

[0066] Furthermore, the gating coefficient is used to assess the suitability of the current emotional state based on the subject's stable characteristics. It constrains the matching relationship between the image subject and the image emotion, assigning different weights to different dimensions of emotional feature components, filtering out invalid adjustment components that do not match the subject's attributes, and retaining reasonable adjustment components that conform to the subject's physical characteristics. The gating coefficient can be expressed as: a = σ ( G a ( f b , f 0 e )), in, Ga (·) denotes an adaptive gating network. a Indicates the gating coefficient. σ This represents the Sigmoid function, used to constrain the gating coefficients to the interval [0,1].

[0067] Furthermore, after obtaining the gating coefficients and sentiment state residuals, the gating coefficients are multiplied point-by-point onto the sentiment state residuals to obtain weighted enhanced state residuals. Then, these enhanced state residuals are superimposed on the sentiment features to obtain enhanced sentiment features that match the basic semantic features. This achieves adaptive constraints on sentiment features by the subject, effectively reducing the probability of generating irrational sentiment states. The enhanced sentiment features are represented as follows: f + e = f 0 e + a ⊙ Δf, in, f + e To enhance emotional characteristics, ⊙ indicates element-wise multiplication. a Indicates the gating coefficient. Δf Indicates residual emotional state. f 0 e It indicates emotional characteristics.

[0068] Furthermore, in step S40 above, after obtaining the enhanced sentiment features, a state description can be generated using the enhanced sentiment features and basic semantic features. This state description is generated by combining the enhanced sentiment features and basic semantic features, and it includes both stable content and sentiment state increments. The stable content is used to clarify the subject's identity, quantity, core actions, and basic scene. The sentiment state increments are used to further describe executable visual states such as posture, form, action mode, subject relationship, material, lighting, color, and composition, thereby transforming high-level sentiment categories into fine-grained state conditions that the initial sentiment controllable generation model can directly respond to.

[0069] The state description generation process can be as follows: Enhanced sentiment features and basic semantic features are projected onto the same common semantic space for feature alignment. The aligned enhanced sentiment features and basic semantic features are then concatenated to obtain concatenated features. These concatenated features are then input into a text decoding network, which decodes each word to generate a state description in natural language. This completes the conversion from features to natural language description, enabling the generation module in the initial controllable sentiment generation model to generate the corresponding image based on the natural language state description. This state description can be represented as: S s = ξ ( f b , f + e ), in, S s Indicates a state description. ξ Represents a text decoder, f + e Indicates enhanced emotional characteristics, f b Represents basic semantic features.

[0070] S50. The generated image is determined based on the state description using the initial emotion-controllable generation model.

[0071] Specifically, the generated image is based on a state description that simultaneously contains stable semantic information and emotional state increments. In the process of generating the image, the initial emotion-controllable generation model adjusts only the visual state corresponding to the target emotion to determine the generated image, while keeping its category, quantity, main structure and core content as unchanged as possible.

[0072] In one embodiment, determining the generated image based on the state description using an initial emotion-controlled generation model specifically includes: Obtain reference image The reference image and the state description are input into the initial emotion controllable generation model, and the generated image is output through the initial emotion controllable generation model.

[0073] Specifically, the reference image is used to provide subject consistency constraints, that is, to provide subject consistency constraints such as subject identity, structure, and basic spatial layout for the initial emotion-controlled generation model, and to specify the specific state changes that the target subject needs to produce under the current emotional conditions through state description. This initial emotion-controlled generation model can be a FLUX.2-based generation model, with incremental adaptation to emotional state conditions using LoRA. Of course, other emotion-controlled generation models can also be used in practical applications.

[0074] For example, the image generation process can be represented as follows: Î = G FLUX.2+LoRA ( I r , S s ), in, I r Indicates a reference image. S s Indicates a state description. Î Indicates the generation of an image. G FLUX.2+LoRA This indicates a model for the controllable generation of initial emotions.

[0075] Compared to existing methods that directly input emotion labels such as "happy," "sad," or "awe" into the generative model, this application utilizes basic and emotion descriptions to convert emotion categories into subject-related state descriptions. Then, an emotion-controlled generative model performs visual rendering to obtain the generated image. Therefore, the emotion-controlled generative model does not need to rely solely on emotion categories to infer all visual changes; instead, it can directly generate images based on the explicitly defined posture, action, relationship, material, lighting, and composition conditions in the state description. Furthermore, during the generation process, subject identity and basic semantics, as stable semantic information, are continuously constrained, while emotion changes primarily affect state features. For the same subject, given different target emotions, the emotion-controlled generative model strives to maintain its category, quantity, subject structure, and core content without irrelevant changes, adjusting only the visual state corresponding to the target emotion. This allows emotion control to be more reflected in the subject itself, rather than simply modifying background color, global hue, or adding environmental elements unrelated to the subject.

[0076] S60. Construct an emotion consistency loss based on the predicted emotion features and the emotion features, construct an emotion feature contrast loss based on the enhanced emotion features, and construct an image supervision loss based on the generated image and the image data.

[0077] Specifically, since the base description and sentiment description correspond to the same image subject, but the sentiment description contains sentiment-related emotional state information, in order to constrain the effectiveness of sentiment features, a sentiment consistency loss is constructed using predicted sentiment features and sentiment features. This sentiment consistency loss is used to supervise the predicted sentiment features, ensuring that the initial sentiment state rendered using the predicted sentiment features closely approximates the sentiment state of the corresponding target sentiment in the sentiment space. The sentiment consistency loss can be expressed as: L e =|| f 0 e - e || 2 , in, L e This indicates a loss of emotional consistency.

[0078] Furthermore, the sentiment feature contrast loss is used to ensure that the enhanced sentiment features maintain the target sentiment direction while possessing good class discrimination ability. This sentiment feature contrast loss uses the enhanced sentiment features as a reference, ensuring high consistency among positive samples with the same sentiment category in the feature space, while simultaneously widening the distance between negative samples with different sentiment categories in the feature space. The sentiment feature contrast loss can be expressed as: , , in, This indicates a loss of contrast in emotional characteristics. Indicates the first i The training sentiment data and the first j Cosine similarity between augmented sentiment features of training sentiment data P ( i )={ p ≠ i | y p = y i} indicates the relationship with the first i A set of positive samples with the same sentiment category in the training sentiment data. This indicates the amount of training sentiment data. Indicates the first i The training sentiment data and the first p The similarity of training sentiment data Indicates the first i The training sentiment data and the first p Cosine similarity between augmented sentiment features of training sentiment data This represents hyperparameters.

[0079] Furthermore, image supervision loss is used to constrain the overall consistency of the generated image in terms of subject structure, content semantics, and target emotional state, enabling the state description to be further translated into realistic visual changes. The image supervision loss can be expressed as: L c = L img ( Î , I t ), in, L c Indicates image supervision loss, I t Represents image data, Î Indicates the generation of an image. L img (·) indicates the distance between images.

[0080] S70. The initial emotion-controllable generation model is trained based on the emotion consistency loss, the emotion feature contrast loss, and the image supervision loss to obtain the emotion-controllable generation model.

[0081] Specifically, the emotional consistency loss and emotional feature contrast loss form an internal emotional supervision mechanism. The emotional consistency loss mainly constrains the consistency between the initial emotional state and the real target state, while the emotional feature contrast loss further constrains the category structure and cross-subject transferability of the enhanced emotional features. This internal emotional supervision mechanism ensures that abstract emotions can form stable and subject-adaptive state features. Image supervision loss acts on the final image generation result, ensuring that these states can be accurately implemented in the visual space. By jointly training with emotional consistency loss, emotional feature contrast loss, and image supervision loss, a two-way combination of internal and external supervision is achieved. This allows the model to learn the extraction rules of abstract emotional features and the correspondence with visual rendering during training, avoiding the emotional control bias caused by relying solely on global image loss in traditional training methods. During training, multiple sets of labeled image data, basic descriptions, and emotional descriptions are input into the model in batches. After each round of training, the weighted sum of the three losses is calculated as the total training loss, which can be expressed as: , in, Indicates total training loss. These are the weight coefficients for sentiment consistency loss, sentiment feature contrast loss, and image supervision loss, respectively. These values ​​can be adjusted based on actual training results and task requirements to balance the impact of different supervision modules on model training. Then, the backpropagation algorithm is used to update all trainable parameters of the model, and training is repeated iteratively until the total loss converges, resulting in a final, usable sentiment-controlled generative model.

[0082] It should be noted that during training, the sentiment rendering module and the initial sentiment-controlled generation model can be jointly trained. After training, the trained initial sentiment-controlled generation model is used as the sentiment-controlled generation model, and this model is used to generate subject-consistent sentiment-controlled images. The sentiment rendering module is used to perform the processes of predicting sentiment features, enhancing sentiment feature determination, and generating state descriptions. This module may include a text encoder, a sentiment rendering unit, a sentiment embedding encoder, an adaptive enhancement unit, and a text decoder. The adaptive enhancement unit may include a sentiment residual prediction network, an adaptive gating network, and a fusion block. Specifically, a text encoder encodes the basic description into basic semantic features and the sentiment description into sentiment features; a sentiment embedding encoder encodes the sentiment type into a low-dimensional sentiment representation; a sentiment rendering unit generates predicted sentiment features based on the basic semantic features and the sentiment category; a sentiment residual prediction network predicts the sentiment state residual between the basic semantic features and the sentiment features based on the image subject; an adaptive gating network uses the basic semantic features and the sentiment features to predict gating coefficients; a fusion block determines the enhanced state residual based on the sentiment state residual and the gating coefficients, and fuses the enhanced state residual with the sentiment features to obtain enhanced sentiment features; and a text decoder combines the enhanced sentiment features and the basic semantic features to generate a state description.

[0083] S80. Utilize the aforementioned emotion-controlled generation model to generate emotion-controlled images with subject-oriented consistency.

[0084] Specifically, the emotion-controlled generation model uses emotion description text as a condition to generate emotion-controlled images with subject consistency. The subject can be provided to the emotion-controlled generation model as a reference image or as a text description. In other words, given the subject, the emotion-controlled generation model is provided with emotion description text, and it can then generate a target image that possesses the target emotion corresponding to the emotion description text while maintaining subject consistency.

[0085] In one embodiment, the generation of emotion-controlled images with subject-consistent characteristics using the emotion-controlled generation model specifically includes: Receive the emotion category and generate emotional visual cue text based on the emotion category; Based on the emotional visual cue text, the emotionally controllable generation model is used to generate an emotionally controllable image with subject-consistent characteristics to obtain the target generated image.

[0086] Specifically, the emotional visual cue text is generated based on the emotional category and describes the specific visual state that the subject corresponding to the target emotional state should present. The emotional visual cue text can contain information such as the subject's posture, movement, lighting, and texture under the target emotional condition, providing a clear direction for the generation model and avoiding the semantic ambiguity caused by directly inputting the emotional category. During the generation of the target image, the emotion-controlled generation model maintains the stability of the subject's category, identity, basic structure, and spatial layout, adjusting the visual features of the subject itself corresponding to the emotional target only according to the requirements of the emotional visual cue text. This ensures that emotional control is accurately applied to the subject and does not cause subject deviation or irrelevant content to interfere with the generation result.

[0087] In one embodiment, the generation of emotion-controlled images with subject-consistent characteristics using the emotion-controlled generation model specifically includes: Receive an emotion category and a subject description, and generate emotion state-enhanced text based on the emotion category and the subject description; Based on the emotional state-enhanced text, the emotionally controllable generation model is used to generate subject-consistent emotionally controllable images to obtain the target generated image.

[0088] Specifically, sentiment-enhanced text is generated while maintaining the core semantic consistency of the subject. It comprises two core parts: a subject description and a sentiment state description. The subject description fully preserves the original subject's category, structure, and semantic information, while the sentiment state description supplements the visual details that the subject should present according to the target sentiment. Compared to directly inputting sentiment categories, sentiment-enhanced text can directly transform the sentiment target into generation conditions that the generative model can directly read, eliminating the need for the model to perform additional semantic parsing and visual reasoning on the sentiment category, thus further improving generation efficiency and the accuracy of sentiment control.

[0089] Based on the aforementioned subject-consistency-oriented emotion-controllable image generation method, this embodiment provides a subject-consistency-oriented emotion-controllable image generation apparatus, such as... Figure 5 As shown, the emotion-controlled image generation device oriented towards subject consistency specifically includes: The construction module 100 is used to construct a training dataset including several training sentiment data, wherein the training sentiment data includes at least sentiment categories, basic descriptions reflecting the subject of the image, sentiment descriptions reflecting the emotional state of the subject of the image, and image data. Training module 200 is used to extract basic semantic features corresponding to the basic description and emotional features corresponding to the emotional description; generate predicted emotional features based on the basic semantic features and the emotional category; generate enhanced emotional features based on the basic semantic features and the emotional features, and generate a state description based on the enhanced emotional features and the basic semantic features; determine the generated image based on the state description through an initial emotionally controllable generation model; construct an emotional consistency loss based on the predicted emotional features and the emotional features, construct an emotional feature contrast loss based on the enhanced emotional features, and construct an image supervision loss based on the generated image and the image data; train the initial emotionally controllable generation model based on the emotional consistency loss, the emotional feature contrast loss, and the image supervision loss to obtain an emotionally controllable generation model. The generation module 300 is used to generate emotion-controlled images with subject-oriented consistency using the emotion-controlled generation model.

[0090] Based on the above-described subject-consistency-oriented emotion-controlled image generation method, this embodiment provides a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement the steps in the subject-consistency-oriented emotion-controlled image generation method as described in the above embodiment.

[0091] Based on the aforementioned subject-consistency-oriented emotion-controlled image generation method, this application also provides a terminal device, such as... Figure 6 As shown, it includes at least one processor 20; a display screen 21; and a memory 22, and may also include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can invoke logical instructions in the memory 22 to execute the methods described in the above embodiments.

[0092] Furthermore, the logical instructions in the aforementioned memory 22 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0093] The memory 22, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of this disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions, or modules stored in the memory 22, thereby implementing the methods in the above embodiments.

[0094] The memory 22 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 22 may include high-speed random access memory (RAM) and non-volatile memory. Examples include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, as well as transient storage media.

[0095] Furthermore, the specific process of loading and executing multiple instruction processors in the aforementioned storage medium and terminal device has been described in detail in the above method, and will not be repeated here.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A subject consistency oriented affect controllable image generation method, characterized in that, The aforementioned subject-consistent emotion-controlled image generation method specifically includes: Construct a training dataset comprising several training sentiment data, wherein the training sentiment data includes at least sentiment categories, basic descriptions reflecting the subject of the image, sentiment descriptions reflecting the emotional state of the subject of the image, and image data; Extract the basic semantic features corresponding to the basic description and the sentiment features corresponding to the sentiment description; Based on the aforementioned basic semantic features and the aforementioned sentiment category, predictive sentiment features are generated; An enhanced sentiment feature is generated based on the basic semantic features and the sentiment features, and a state description is generated based on the enhanced sentiment feature and the basic semantic features; The generated image is determined based on the state description using an initial emotion-controlled generation model; Based on the predicted sentiment features and the sentiment features, a sentiment consistency loss is constructed; based on the enhanced sentiment features, a sentiment feature contrast loss is constructed; and based on the generated image and the image data, an image supervision loss is constructed. The initial emotion-controlled generation model is trained based on the emotion consistency loss, the emotion feature contrast loss, and the image supervision loss to obtain the emotion-controlled generation model. The aforementioned emotion-controlled generation model is used to generate emotion-controlled images with subject-specific consistency.

2. The emotion-controlled image generation method based on subject consistency according to claim 1, characterized in that, The training dataset, which includes several training sentiment data, specifically includes: Obtain an image dataset, which contains image data with the same subject but different emotion categories; Generate basic and sentiment descriptions for image data in the image dataset using a visual language model; Training sentiment data is constructed based on image data and its corresponding sentiment categories, basic descriptions, and sentiment descriptions to obtain a training dataset that includes several training sentiment data.

3. The emotion-controlled image generation method based on subject consistency according to claim 1, characterized in that, The specific steps of generating predicted sentiment features based on the basic semantic features and the sentiment category include: The emotion categories are encoded to obtain low-dimensional emotion representations; The correspondence between emotional text and visual state changes is learned by using the low-dimensional sentiment representation and the basic semantic features to obtain predicted sentiment features.

4. The emotion-controlled image generation method based on subject consistency according to claim 1, characterized in that, The process of generating enhanced emotional features based on the basic semantic features and the emotional features specifically includes: Based on the image subject, predict the emotional state residual between the basic semantic features and the emotional features; Based on the basic semantic features and the sentiment features, a gating coefficient is predicted, which is used to constrain the matching relationship between the image subject and the image sentiment. The emotional state residual is fused with the gating coefficient to obtain the enhanced state residual; The enhanced state residual is fused with the emotional feature to obtain the enhanced emotional feature.

5. The emotion-controlled image generation method based on subject consistency according to claim 1, characterized in that, The step of determining the generated image based on the state description using the initial emotion-controllable generation model specifically includes: Obtain a reference image, which is used to provide image subject consistency constraints; The reference image and the state description are input into the initial emotion controllable generation model, and the generated image is output through the initial emotion controllable generation model.

6. The emotion-controlled image generation method based on subject consistency according to claim 1, characterized in that, The specific steps of generating emotion-controlled images with subject-consistency orientation using the emotion-controlled generation model include: Receive the emotion category and generate emotional visual cue text based on the emotion category; Based on the emotional visual cue text, the emotionally controllable generation model is used to generate an emotionally controllable image with subject-consistent characteristics to obtain the target generated image.

7. The emotion-controllable image generation method based on subject consistency according to claim 1, characterized in that, The specific steps of generating emotion-controlled images with subject-consistency orientation using the emotion-controlled generation model include: Receive an emotion category and a subject description, and generate emotion state-enhanced text based on the emotion category and the subject description; Based on the emotional state-enhanced text, the emotionally controllable generation model is used to generate subject-consistent emotionally controllable images to obtain the target generated image.

8. A subject-consistent emotion-controlled image generation device, characterized in that, The aforementioned subject-consistent emotion-controlled image generation device specifically includes: A construction module is used to construct a training dataset that includes several training sentiment data, wherein the training sentiment data includes at least sentiment categories, basic descriptions reflecting the subject of the image, sentiment descriptions reflecting the emotional state of the subject of the image, and image data. The training module is used to extract basic semantic features corresponding to the basic description and emotional features corresponding to the emotional description; generate predicted emotional features based on the basic semantic features and the emotional category; generate enhanced emotional features based on the basic semantic features and the emotional features, and generate a state description based on the enhanced emotional features and the basic semantic features; determine the generated image based on the state description through an initial emotionally controllable generation model; construct an emotional consistency loss based on the predicted emotional features and the emotional features, construct an emotional feature contrast loss based on the enhanced emotional features, and construct an image supervision loss based on the generated image and the image data; train the initial emotionally controllable generation model based on the emotional consistency loss, the emotional feature contrast loss, and the image supervision loss to obtain an emotionally controllable generation model. The generation module is used to generate emotion-controlled images with subject-oriented consistency using the emotion-controlled generation model.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps in the subject-consistent emotion-controlled image generation method as described in any one of claims 1-7.

10. A terminal device, characterized in that, include: Processor and memory; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps in the subject-consistent emotion-controlled image generation method as described in any one of claims 1-7.

Images