An emotion image generation method, system and application based on visual autoregression

Abstract

The application discloses a kind of based on visual self-regression emotional image generation method, system and application, belong to image generation technical field.The application implementation method is:1, constructs continuous emotional prompt word pairing training data;2, constructs text condition feature cache;3, constructs learnable soft prefix emotional adapter;4, training is carried out to soft prefix emotional adapter using joint loss, and freeze text encoder, visual self-regression text to image backbone and image decoder in training process;5, construct text condition input for visual self-regression backbone;6, generate the image corresponding to input prompt word semantics and conforming to target emotion;Compared with prior art, the present application solves the technical problems of the existing emotional image generation method, such as insufficient continuous control ability, difficulty in adapting to visual self-regression text to image backbone, high training cost and easy to destroy the original prompt word semantics.

Description

Technical Field

[0001] This invention relates to the field of image generation technology, and in particular to a method, system and application for generating emotion-based images based on visual autoregression, which can be applied to emotion-controlled image generation scenarios in emotional content creation, digital art design, interactive visual generation, advertising visual generation and human-computer interaction. Background Technology

[0002] With the development of text-to-image generation technology, generative models are now capable of generating high-resolution images based on natural language prompts. Existing text-to-image generation methods mainly fall into two categories: diffusion-based generation models and visual autoregressive generation models. Diffusion-based generation models are widely used in emotion image generation, image editing, and structural control, while visual autoregressive generation models gradually generate visual content through scale prediction, bit-level visual representation, or other autoregressive methods. Meanwhile, emotion image generation is gradually evolving from discrete emotion category control to continuous emotion control. Commonly used continuous emotion representations include valence-arousal coordinates, where valence represents the positive or negative tendency of emotion, and arousal represents the intensity of emotional activation.

[0003] Existing emotion image generation techniques still suffer from the following shortcomings: First, some methods rely on discrete emotion labels or directly concatenate emotion words into prompt words, making it difficult to express continuously changing emotion intensities. The generated results are prone to discontinuities or unpredictable changes between adjacent emotion points. Second, some methods rely on emotion adapters, attention control, or image denoising modules modified on the diffusion model side, making it difficult to directly transfer to visual autoregressive text-to-image backbone. Third, if emotion control is introduced by using a complete fine-tuning of the generation model, the training cost is high, the memory usage is large, and it may damage the general text generation capabilities of the original generation model. Finally, directly rewriting the original prompt words or rewriting text features word by word can easily change the semantics of the main subjects such as people, objects, and scenes, leading to content drift while enhancing emotions.

[0004] Therefore, how to accurately inject continuous valence-arousal emotion signals into the text conditional space of visual autoregressive text to image skeleton while maintaining the semantic integrity of the original cue words without interference has become an urgent problem to be solved. Summary of the Invention

[0005] The purpose of this invention is to address the technical problems of existing emotion image generation methods, such as insufficient continuous control capability, difficulty in adapting visual autoregressive text to image skeletons, high training costs, and easy destruction of the semantics of original prompt words. This invention proposes a visual autoregressive-based emotion image generation method, system, and application.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] In a first aspect, the present invention discloses a method for generating emotion images based on visual autoregression, comprising the following steps:

[0008] Step 1: Construct continuous emotion cue word pairing training data; the training data includes neutral cue words, emotional cue words, and training records that have undergone scale scaling and normalization processing for valence and arousal labels; wherein, neutral cue words are used to describe the semantics of image subjects, actions, objects, and scenes; emotional cue words are used to describe emotional visual cues such as color, lighting, facial expressions, action intensity, and background atmosphere while preserving the semantics of the subject; valence and arousal labels are used to quantify emotional style and atmosphere;

[0009] Step 1.1: Obtain training data containing training records with neutral cue words, sentiment cue words, valence labels, and arousal labels; where the values ​​of valence labels and arousal labels are both in the range of [-1, 1].

[0010] Step 1.2: Clean the training records using field completeness and numerical parsing checks, remove invalid records, and form valid training records;

[0011] Step 1.3: Generate globally unique sample identifiers by mapping the valid training records; and scale and normalize the valence and arousal labels into continuous emotion conditional vectors, as shown in Equation (1):

[0012]

[0013] Where y represents the normalized continuous emotion conditional vector, v represents the valence label, and a represents the arousal label. Valence normalization scale, R_a represents the arousal normalization scale;

[0014] Step 1.4: Divide the effective training records into training set, validation set and test set according to the preset ratio to form continuous emotion cue word matching training data;

[0015] Step 2: Construct a text conditional feature cache; Encode the cue words in the training data using a frozen text encoder, generate and cache the neutral text hidden state, the sentiment text hidden state and the corresponding attention mask in a cache with sample identifier as the index key and using a sharded storage. The cache structure includes a unique sample identifier, the neutral cue word T5 hidden state, the sentiment cue word T5 hidden state, the valid / filler marker of the neutral word, the mask used to calculate the loss, the original valence arousal value and the normalized valence arousal value;

[0016] Step 2.1: Construct a text conditional feature cache, which is used to avoid repeatedly loading the text encoder;

[0017] Step 2.1.1: Each cached record contains seven fields: sample identifier, neutral text hidden state tensor, sentiment text hidden state tensor, input attention mask, loss attention mask, normalized valence-arousal vector, and original valence-arousal value; multiple records constitute a shard file, and each shard is managed uniformly by an index list; the neutral text hidden state tensor and sentiment text hidden state tensor both adopt a sequence length of 128 × hidden dimension of 2048;

[0018] Step 2.1.2: The text conditional feature cache adopts a fragmented storage method. Each fragment file is serialized and saved in PyTorch tensor format. An index list is stored in the same directory, recording the path of each fragment file and the number of samples in the training set, validation set, and test set.

[0019] Step 2.1.3: During training, the list of shard files is located by reading the index list, each shard is loaded as needed, and records that are targeted to be divided into training set, validation set, and test set are filtered; parallel reading by multiple worker processes and random shuffling by round are supported; after parsing each record, the sample identifier, neutral text features, sentiment text features, attention mask, and valence-wake vector are output in batches to avoid repeatedly calling the frozen text encoder in the training loop;

[0020] Step 2.2: Use the frozen T5-XL text encoder to encode the neutral cue words and sentimental cue words in each valid training record to obtain the neutral text hidden state and the sentimental text hidden state;

[0021] Step 2.3: Use an attention masking mechanism to mark the effective and filler words in the neutral text hidden state and the sentiment text hidden state to obtain the input attention mask and the loss attention mask; wherein, the effective words correspond to a mask value of 1 and the filler words correspond to a mask value of 0;

[0022] Step 2.4: Write the sample identifier, neutral text hidden state, sentiment text hidden state, attention mask, loss attention mask, and continuous sentiment conditional vector into the text conditional feature cache;

[0023] Step 3: Construct a learnable soft prefix sentiment adapter; generate soft sentiment prefixes based on the semantic summary of neutral cue words after masking and pooling and continuous sentiment conditional vectors; the soft sentiment prefixes are independent of the original neutral text lexical units and consist of learnable basic prefixes and prefix increments generated by context vectors; the learnable soft prefix sentiment adapter is used to train lexical prefixes representing sentiment features;

[0024] Step 3.1: Use mask pooling technology to extract the global semantic summary of the neutral text hidden state to obtain the semantic summary of the neutral prompt words. The mask pooling method is shown in Equation (2):

[0025]

[0026] in, This represents a semantic summary of neutral cue words. Let represent the hidden vector of the t-th word in the hidden state of neutral text. Indicates the first The attention mask corresponding to each word, where ϵ represents a constant to prevent the denominator from being zero;

[0027] Step 3.2: Use the first-to-last concatenation method to concatenate the semantic summary of neutral cue words and the continuous sentiment condition vector along the feature dimension to form a context vector;

[0028] Step 3.3: A multilayer perceptron is used to generate a prefix increment based on the context vector, and the prefix increment is added to the learnable basic prefix to obtain a soft emotion prefix. The soft emotion prefix is ​​used to adjust the emotional atmosphere of the generated image; the generation method is shown in Equation (3):

[0029]

[0030] in, Prefix indicating soft emotions Indicates a learnable basic prefix. This indicates that a trainable multilayer perceptron is possible. This means reconstructing the multilayer perceptron output into a tensor that is the prefix length multiplied by the text condition dimension;

[0031] Step 3.4: Set the soft emotion prefix to a continuous vector independent of the original neutral text lexical units, while keeping the neutral text lexical units unchanged;

[0032] Step 4: The soft prefix sentiment adapter is trained using a joint loss consisting of pooling conditional mean square error loss, cosine direction loss, prefix direction loss, and norm constraint loss. The text encoder, visual autoregressive text-to-image backbone, and image decoder are frozen during the training process.

[0033] Step 4.1: Use prefix concatenation to concatenate the soft emotion prefix with the neutral text hidden state dimension to form the overall text condition;

[0034] Step 4.2: Using pooling conditional alignment loss to constrain the pooling representation of the overall text condition, which approximates the pooling representation of the hidden sentiment text state, the concatenated overall conditional pooling representation is defined as follows:

[0035]

[0036] Where P=16, and L is the number of effective lexical units of neutral cue words; thus enabling Tend to ;

[0037] Step 4.3: Use cosine directional loss to constrain the overall text conditions and the emotional text hidden state to be consistent in direction in the text condition space;

[0038] Step 4.4: Use prefix direction loss to constrain the soft emotion prefix to face the emotion text hidden state, and use norm constraint loss to make the vector scale of the soft emotion prefix close to the scale of the natural hidden state output by the frozen text encoder.

[0039] Step 4.5: Train the soft prefix sentiment adapter using the joint loss function consisting of pooling conditional mean square error loss, cosine direction loss, prefix direction loss, and norm constraint loss as shown in Equation (5), and freeze the text encoder, visual autoregressive text-to-image backbone, and image decoder during training: The pooling conditional mean square error loss is used to make the overall text condition after concatenating the prefix closer to the sentiment cue word in the T5 space; the cosine direction loss is used to make the overall semantic direction after concatenating the prefix oriented towards the sentiment cue word; the prefix direction loss is used to make the soft prefix itself carry a strong sentiment direction; the norm constraint loss is used to control the prefix vector scale;

[0040]

[0041] in, Indicates joint loss, This represents the mean squared error loss due to pooling conditions. Indicates loss in the cosine direction. This represents the prefix direction loss. Represents norm-constrained loss. , and These represent the weights of the corresponding losses;

[0042] Step 5: Construct text conditional input for visual autoregressive backbone; During the inference stage, the input prompt words are encoded by the frozen T5-XL text encoder and then fed into the trained soft prefix emotion adapter along with the normalized target valence-arousal coordinates to generate 16 soft emotion lexical units; The above soft emotion lexical units are concatenated to the neutral text hidden state, and then the text condition with emotion prefix is ​​converted into a text condition quadruple that can be directly received by the visual autoregressive text to the image backbone through the compact key-value sequence packaging technique;

[0043] Step 5.1: Obtain the input cue words, target valence, and arousal of the image to be generated, and use the same normalization method as in the training phase to obtain the target continuous emotion conditional vector;

[0044] Step 5.2: Encode the input prompts using a frozen text encoder to obtain the hidden state of the input text and the input attention mask;

[0045] Step 5.3: Input the hidden state of the input text, the input attention mask, and the target continuous emotion conditional vector into the trained soft prefix emotion adapter to obtain the target soft emotion prefix;

[0046] Step 5.4: Use prefix concatenation technology to add the target soft emotion prefix to the hidden state of the input text, forming a text condition with an emotion prefix, as shown in equation (6):

[0047]

[0048] in, This indicates text conditions with an emotion prefix. This indicates concatenation along the dimensions of the text sequence;

[0049] Step 5.5: Compact text conditional packing technology is used to pack the text conditions with sentiment prefixes to obtain a text condition quadruple that can be accepted from visual autoregressive text to image backbone; the text condition quadruple includes a compact key value sequence, the effective length of each sample, the cumulative sequence offset, and the maximum effective length within the batch;

[0050] Step 5.5.1: For each sample, append the sentiment prefix to the effective word to form a compact key-value sequence;

[0051] Step 5.5.2: Use the length of the compact key-value sequence as the effective length of each sample;

[0052] Step 5.5.3: Mark the starting position of each compact key value sequence and construct a cumulative sequence offset with a length of the sequence number plus 1 to record the starting position of each sample;

[0053] Step 5.5.4: Record the maximum valid length within a batch in the compact key-value sequence;

[0054] Step 5.5.5: Pack the text conditions with the emotion prefix into a text condition quadruple;

[0055] Step 6: Input the text conditions with the emotion prefix into the frozen visual autoregressive text, then into the image backbone and image decoder to generate an image that corresponds semantically to the input prompt and matches the target emotion;

[0056] Step 6.1: Input the text conditional quadruple into the frozen Infinity-8B visual autoregressive text-to-image backbone, so that the visual autoregressive text-to-image backbone jointly predicts the visual representation based on the sentiment prefix and the original semantic units;

[0057] Step 6.2: The visual representation is decoded using a frozen image decoder to obtain an emotion image that corresponds semantically to the input cue word and conforms to the target valence-arousal coordinates; the image decoder is the 32-channel variational autoencoder VAE-32 decoder that is compatible with the Infinity-8B.

[0058] Secondly, the present invention discloses a visual autoregressive-based emotion image generation system for implementing the above method, including a data construction module, a text feature precoding module, a soft prefix emotion adaptation module, an adapter training module, a text conditional packaging module, and a visual autoregressive image generation.

[0059] The data construction module is used to acquire training records containing neutral cue words, emotional cue words, valence labels and arousal labels, and to perform field integrity detection, numerical parsing detection, sample identifier mapping and continuous emotion coordinate normalization on the training records to form continuous emotion cue word pairing training data.

[0060] The text feature precoding module is used to call the frozen text encoder to encode neutral cue words and emotional cue words, obtain neutral text hidden state, emotional text hidden state, input attention mask and loss attention mask, and write them into the text conditional feature cache.

[0061] The soft prefix emotion adaptation module is used to perform mask pooling on the hidden state of neutral text, generate a semantic summary of neutral cue words, concatenate the semantic summary of neutral cue words with the continuous emotion condition vector, input it into the multilayer perceptron, and output the soft emotion prefix.

[0062] The adapter training module is used to train the soft prefix sentiment adaptation module using pooling conditional alignment loss, cosine direction loss, prefix direction loss and norm constraint loss under the conditions of frozen text encoder, visual autoregressive text to image backbone and image decoder.

[0063] The text condition packaging module is used to concatenate the soft sentiment prefix with the hidden state of the input text, and convert the concatenated text conditions into a compact key-value sequence, effective length, cumulative sequence offset, and maximum effective length within a batch.

[0064] The visual autoregressive image generation module is used to input the text condition quadruple output by the text condition packaging module into the frozen visual autoregressive text to the image skeleton, and call the frozen image decoder to output the emotion image.

[0065] The evaluation module is used to evaluate the generated images for emotion alignment, semantic consistency, continuous visual response, and ranking consistency.

[0066] Thirdly, an application of the above method and system in the generation of emotional advertising materials in digital media; the advertising creator inputs neutral scene description prompts and target valence-arousal coordinates that express the brand's emotional tone, and the above method is used to automatically generate advertising visual materials that are precisely matched with the brand's emotional positioning, so as to realize the rapid switching and batch customization of the same scene description under different emotional atmospheres.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] 1. This invention solves the problem that discrete emotion labels or simple emotion word combinations cannot express abstract and continuous emotion changes by using a combination of continuous valence-arousal coordinates and learnable soft emotion prefixes. It achieves the effect of fine-grained emotion regulation along the valence and arousal directions under the same cue word.

[0069] 2. This invention solves the problems of high training cost, large memory usage, and easy destruction of the original generation ability of the complete fine-tuned generative model by freezing the text encoder, visual autoregressive text-to-image backbone and image decoder, and training only the soft prefix emotion adapter. It achieves the effect of low cost, lightweight and reusable existing generative backbone.

[0070] 3. This invention solves the problem that word-by-word rewriting or prompt word rewriting can easily lead to semantic drift by retaining the original neutral text words and adding independent soft emotion prefixes in front of them. It achieves the effect of changing the emotional atmosphere of the image while maintaining the semantics of people, objects, actions and scenes.

[0071] 4. This invention uses a compact text conditional packaging technology for visual autoregressive text to image backbone to convert soft emotion prefixes into text conditional quadruples that the backbone model can directly receive. This solves the problem that existing diffusion model emotion adaptation methods are difficult to directly access the visual autoregressive generation process, and achieves the effect of emotion control generation without modifying the visual autoregressive backbone structure.

[0072] 5. This invention, by directly constructing continuous emotion vectors in the frozen T5 text conditional space, reduces the overall distance deviation in the emotion space by approximately 7.1% and the mean absolute error of emotion coordinates by approximately 7.0% compared to directly concatenating valence-arousal values ​​into the cue text to generate an image. Compared to directly inputting emotional cue words into the model to generate an image, this invention further reduces the deviation by approximately 11.3% and 11.1% respectively, achieving a more accurate continuous emotion alignment effect. Attached Figure Description

[0073] Figure 1 This is a schematic diagram of the process for generating emotion-controllable images according to the present invention.

[0074] Figure 2 This is a diagram demonstrating the effect of generating emotion images according to an embodiment of the present invention. Detailed Implementation

[0075] To better illustrate the purpose and advantages of this invention, the invention will be further described below with reference to the accompanying drawings and examples. It should be noted that the implementation of this invention is not limited to the following embodiments, and any modifications or alterations made to this invention will fall within the scope of protection of this invention.

[0076] Example

[0077] like Figure 1 As shown in the figure, the specific implementation steps of the emotion image generation method based on visual autoregression in this embodiment are as follows:

[0078] Step 1: Construct continuous emotion cue word pairing training data; the training data includes neutral cue words, emotional cue words, and training records that have undergone scale scaling and normalization processing for valence and arousal labels; wherein, neutral cue words are used to describe the semantics of image subjects, actions, objects, and scenes; emotional cue words are used to describe emotional visual cues such as color, lighting, facial expressions, action intensity, and background atmosphere while preserving the semantics of the subject; valence and arousal labels are used to quantify emotional style and atmosphere;

[0079] Step 1.1: Obtain training data containing training records with neutral cue words, sentiment cue words, valence labels, and arousal labels; where the values ​​of valence labels and arousal labels are both in the range of [-1, 1].

[0080] In the example, the neutral cue word is "a man with his head in his hands," and the emotional cue word is "A young man in a pensive state, head down, expressing deep sorrow and introspection. The atmosphere is heavy and somber, with muted, dark tone senhancing the feeling of melancholy and contemplation. He appears lost inthought, consumed by the weight of his emotions." While maintaining the semantics of the character and clothing, emotional cues such as brightness, color, facial expressions, posture, or background atmosphere are added. The effectiveness value is -0.7451, the arousal value is -0.712871287, and after normalization, they are -0.248366013 and -0.23762, respectively.

[0081] Step 1.2: Clean the training records using field completeness and numerical parsing checks, remove invalid records, and form valid training records;

[0082] Step 1.3: Generate globally unique sample identifiers by mapping the valid training records; and scale and normalize the valence and arousal labels into continuous emotion conditional vectors, as shown in Equation (1):

[0083]

[0084] Where y represents the normalized continuous emotion conditional vector, v represents the valence label, and a represents the arousal label. Valence normalization scale, R_a represents the arousal normalization scale;

[0085] In this embodiment, the final number of valid prompt word pairings entering the training pipeline was 39,206; in other embodiments, the number of valid samples can be adjusted according to the scale of the training data.

[0086] Step 1.4: Divide the effective training records into training set, validation set and test set according to the preset ratio to form continuous emotion cue word matching training data;

[0087] Step 2: Construct a text conditional feature cache; Encode the cue words in the training data using a frozen text encoder, generate and cache the neutral text hidden state, the sentiment text hidden state and the corresponding attention mask in a cache with sample identifier as the index key and using a sharded storage. The cache structure includes a unique sample identifier, the neutral cue word T5 hidden state, the sentiment cue word T5 hidden state, the valid / filler marker of the neutral word, the mask used to calculate the loss, the original valence arousal value and the normalized valence arousal value;

[0088] Step 2.1: Construct a text conditional feature cache, which is used to avoid repeatedly loading the text encoder;

[0089] Step 2.1.1: Each cached record contains seven fields: sample identifier, neutral text hidden state tensor, sentiment text hidden state tensor, input attention mask, loss attention mask, normalized valence-arousal vector, and original valence-arousal value; multiple records constitute a shard file, and each shard is managed uniformly by an index list; the neutral text hidden state tensor and sentiment text hidden state tensor both adopt a sequence length of 128 × hidden dimension of 2048;

[0090] Step 2.1.2: The text conditional feature cache adopts a fragmented storage method. Each fragment file is serialized and saved in PyTorch tensor format. An index list is stored in the same directory, recording the path of each fragment file and the number of samples in the training set, validation set, and test set.

[0091] Step 2.1.3: During training, the list of shard files is located by reading the index list, each shard is loaded as needed, and records that are targeted to be divided into training set, validation set, and test set are filtered; parallel reading by multiple worker processes and random shuffling by round are supported; after parsing each record, the sample identifier, neutral text features, sentiment text features, attention mask, and valence-wake vector are output in batches to avoid repeatedly calling the frozen text encoder in the training loop;

[0092] In this embodiment, the T5-XL text encoder uses flan-t5-xl, with a maximum text length of 128, a hidden dimension of 2048, and a cached data type of 16-bit floating-point number.

[0093] Step 2.2: Use the frozen T5-XL text encoder to encode the neutral cue words and sentimental cue words in each valid training record to obtain the neutral text hidden state and the sentimental text hidden state;

[0094] Step 2.3: Use an attention masking mechanism to mark the effective and filler words in the neutral text hidden state and the sentiment text hidden state to obtain the input attention mask and the loss attention mask; wherein, the effective words correspond to a mask value of 1 and the filler words correspond to a mask value of 0;

[0095] In this embodiment, the text conditional feature caching eliminates the need to repeatedly load the T5-XL text encoder during subsequent adapter training, as well as the need to load the visual autoregressive backbone or image files.

[0096] Step 2.4: Write the sample identifier, neutral text hidden state, sentiment text hidden state, attention mask, loss attention mask, and continuous sentiment conditional vector into the text conditional feature cache;

[0097] Step 3: Construct a learnable soft prefix sentiment adapter; generate soft sentiment prefixes based on the semantic summary of neutral cue words after masking and pooling and continuous sentiment conditional vectors; the soft sentiment prefixes are independent of the original neutral text lexical units and consist of learnable basic prefixes and prefix increments generated by context vectors; the learnable soft prefix sentiment adapter is used to train lexical prefixes representing sentiment features;

[0098] Step 3.1: Use mask pooling technology to extract the global semantic summary of the neutral text hidden state to obtain the semantic summary of the neutral prompt words. The mask pooling method is shown in Equation (2):

[0099]

[0100] in, This represents a semantic summary of neutral cue words. Let represent the hidden vector of the t-th word in the hidden state of neutral text. Indicates the first The attention mask corresponding to each word, where ϵ represents a constant to prevent the denominator from being zero;

[0101] Step 3.2: Use the first-to-last concatenation method to concatenate the semantic summary of neutral cue words and the continuous sentiment condition vector along the feature dimension to form a context vector;

[0102] Step 3.3: A multilayer perceptron is used to generate a prefix increment based on the context vector, and the prefix increment is added to the learnable basic prefix to obtain a soft emotion prefix. The soft emotion prefix is ​​used to adjust the emotional atmosphere of the generated image; the generation method is shown in Equation (3):

[0103]

[0104] in, Prefix indicating soft emotions Indicates a learnable basic prefix. This indicates that a trainable multilayer perceptron is possible. This means reconstructing the multilayer perceptron output into a tensor that is the prefix length multiplied by the text condition dimension;

[0105] In this embodiment, the prefix length is set to 16, the T5 dimension is set to 2048, the hidden layer dimension is set to 1024, and the output dimension of the output linear layer is 16 times 2048.

[0106] Step 3.4: Set the soft emotion prefix to a continuous vector independent of the original neutral text lexical units, while keeping the neutral text lexical units unchanged;

[0107] In this embodiment, the incremental output of the prefix generation network is reconstructed into a tensor of batch size multiplied by 16 and 2048, and then added to the basic prefix to obtain soft sentiment lexical units; the number of trainable parameters of the soft prefix adapter is approximately 36.78 million, which is a lightweight additional module relative to the visual autoregressive text to image backbone module.

[0108] Step 4: The soft prefix sentiment adapter is trained using a joint loss consisting of pooling conditional mean square error loss, cosine direction loss, prefix direction loss, and norm constraint loss. The text encoder, visual autoregressive text-to-image backbone, and image decoder are frozen during the training process.

[0109] Step 4.1: Use prefix concatenation to concatenate the soft emotion prefix with the neutral text hidden state dimension to form the overall text condition;

[0110] Step 4.2: Using pooling conditional alignment loss to constrain the pooling representation of the overall text condition, which approximates the pooling representation of the hidden sentiment text state, the concatenated overall conditional pooling representation is defined as follows:

[0111]

[0112] Where P=16, and L is the number of effective lexical units of neutral cue words; thus enabling Tend to ;

[0113] Step 4.3: Use cosine directional loss to constrain the overall text conditions and the emotional text hidden state to be consistent in direction in the text condition space;

[0114] Step 4.4: Use prefix direction loss to constrain the soft emotion prefix to face the emotion text hidden state, and use norm constraint loss to make the vector scale of the soft emotion prefix close to the scale of the natural hidden state output by the frozen text encoder.

[0115] Step 4.5: Train the soft prefix sentiment adapter using the joint loss function consisting of pooling conditional mean square error loss, cosine direction loss, prefix direction loss, and norm constraint loss as shown in Equation (5), and freeze the text encoder, visual autoregressive text-to-image backbone, and image decoder during training: The pooling conditional mean square error loss is used to make the overall text condition after concatenating the prefix closer to the sentiment cue word in the T5 space; the cosine direction loss is used to make the overall semantic direction after concatenating the prefix oriented towards the sentiment cue word; the prefix direction loss is used to make the soft prefix itself carry a strong sentiment direction; the norm constraint loss is used to control the prefix vector scale;

[0116]

[0117] in, Indicates joint loss, This represents the mean squared error loss due to pooling conditions. Indicates loss in the cosine direction. This represents the prefix direction loss. Represents norm-constrained loss. , and These represent the weights of the corresponding losses;

[0118] In this embodiment, the cosine direction loss weight is set to 1.0, the prefix direction loss weight is set to 0.25, and the norm constraint loss weight is set to 0.05; the optimizer is AdamW, with a learning rate of 1e-4, a weight decay of 1e-5, a batch size of 128, a training epoch of 50, and gradient pruning is used to control training stability.

[0119] Step 5: Construct text conditional input for visual autoregressive backbone; During the inference stage, the input prompt words are encoded by the frozen T5-XL text encoder and then fed into the trained soft prefix emotion adapter along with the normalized target valence-arousal coordinates to generate 16 soft emotion lexical units; The above soft emotion lexical units are concatenated to the neutral text hidden state, and then the text condition with emotion prefix is ​​converted into a text condition quadruple that can be directly received by the visual autoregressive text to the image backbone through the compact key-value sequence packaging technique;

[0120] Step 5.1: Obtain the input cue words, target valence, and arousal of the image to be generated, and use the same normalization method as in the training phase to obtain the target continuous emotion conditional vector;

[0121] Step 5.2: Encode the input prompts using a frozen text encoder to obtain the hidden state of the input text and the input attention mask;

[0122] Step 5.3: Input the hidden state of the input text, the input attention mask, and the target continuous emotion conditional vector into the trained soft prefix emotion adapter to obtain the target soft emotion prefix;

[0123] Step 5.4: Use prefix concatenation technology to add the target soft emotion prefix to the hidden state of the input text, forming a text condition with an emotion prefix, as shown in equation (6):

[0124]

[0125] in, This indicates text conditions with an emotion prefix. This indicates concatenation along the dimensions of the text sequence;

[0126] Step 5.5: Compact text conditional packing technology is used to pack the text conditions with sentiment prefixes to obtain a text condition quadruple that can be accepted from visual autoregressive text to image backbone; the text condition quadruple includes a compact key value sequence, the effective length of each sample, the cumulative sequence offset, and the maximum effective length within the batch;

[0127] Step 5.5.1: For each sample, append the sentiment prefix to the effective word to form a compact key-value sequence;

[0128] Step 5.5.2: Use the length of the compact key-value sequence as the effective length of each sample;

[0129] Step 5.5.3: Mark the starting position of each compact key value sequence and construct a cumulative sequence offset with a length of the sequence number plus 1 to record the starting position of each sample;

[0130] Step 5.5.4: Record the maximum valid length within a batch in the compact key-value sequence;

[0131] Step 5.5.5: Pack the text conditions with the emotion prefix into a text condition quadruple;

[0132] In this embodiment, the visual autoregressive text-to-image backbone uses Infinity-8B, and the image decoder uses the VAE-32 (32-channel variational autoencoder) decoder that is compatible with Infinity-8B, generating an image with a resolution of 1024*1024; in other embodiments, the visual autoregressive text-to-image backbone can also use other visual autoregressive generation models with text conditional interfaces.

[0133] Step 6: Input the text conditions with the emotion prefix into the frozen visual autoregressive text, then into the image backbone and image decoder to generate an image that corresponds semantically to the input prompt and matches the target emotion;

[0134] Step 6.1: Input the text conditional quadruple into the frozen Infinity-8B visual autoregressive text-to-image backbone, so that the visual autoregressive text-to-image backbone jointly predicts the visual representation based on the sentiment prefix and the original semantic units;

[0135] Step 6.2: The visual representation is decoded using a frozen image decoder to obtain an emotion image that corresponds semantically to the input cue word and conforms to the target valence-arousal coordinates; the image decoder is the 32-channel variational autoencoder VAE-32 decoder that is compatible with the Infinity-8B.

[0136] like Figure 2 As shown, under the same neutral cue word "a man with his head in his hands", after constructing the target grid along the valence and arousal dimensions, the method of the present invention can present a continuous change in emotional atmosphere from low valence and low arousal to high valence and high arousal while maintaining the semantics of the character and action, thereby demonstrating the effect of continuous emotional control and semantic preservation.

[0137] Secondly, the present invention discloses a visual autoregressive-based emotion image generation system for implementing the above method, including a data construction module, a text feature precoding module, a soft prefix emotion adaptation module, an adapter training module, a text conditional packaging module, and a visual autoregressive image generation.

[0138] The data construction module is used to acquire training records containing neutral cue words, emotional cue words, valence labels and arousal labels, and to perform field integrity detection, numerical parsing detection, sample identifier mapping and continuous emotion coordinate normalization on the training records to form continuous emotion cue word pairing training data.

[0139] The text feature precoding module is used to call the frozen text encoder to encode neutral cue words and emotional cue words, obtain neutral text hidden state, emotional text hidden state, input attention mask and loss attention mask, and write them into the text conditional feature cache.

[0140] The soft prefix emotion adaptation module is used to perform mask pooling on the hidden state of neutral text, generate a semantic summary of neutral cue words, concatenate the semantic summary of neutral cue words with the continuous emotion condition vector, input it into the multilayer perceptron, and output the soft emotion prefix.

[0141] The adapter training module is used to train the soft prefix sentiment adaptation module using pooling conditional alignment loss, cosine direction loss, prefix direction loss and norm constraint loss under the conditions of frozen text encoder, visual autoregressive text to image backbone and image decoder.

[0142] The text condition packaging module is used to concatenate the soft sentiment prefix with the hidden state of the input text, and convert the concatenated text conditions into a compact key-value sequence, effective length, cumulative sequence offset, and maximum effective length within a batch.

[0143] The visual autoregressive image generation module is used to input the text condition quadruple output by the text condition packaging module into the frozen visual autoregressive text to the image skeleton, and call the frozen image decoder to output the emotion image.

[0144] The evaluation module is used to evaluate the generated images for emotion alignment, semantic consistency, continuous visual response, and ranking consistency.

[0145] Thirdly, an application of the above method and system in the generation of emotional advertising materials in digital media; the advertising creator inputs neutral scene description prompts and target valence-arousal coordinates that express the brand's emotional tone, and the above method is used to automatically generate advertising visual materials that are precisely matched with the brand's emotional positioning, so as to realize the rapid switching and batch customization of the same scene description under different emotional atmospheres.

[0146] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating emotion images based on visual autoregression, characterized in that: Includes the following steps, Step 1: Construct continuous emotion cue word pairing training data; the training data includes neutral cue words, emotional cue words, and training records that have undergone scale scaling and normalization processing for valence and arousal labels; wherein, neutral cue words are used to describe the semantics of image subjects, actions, objects, and scenes; emotional cue words are used to describe emotional visual cues such as color, lighting, facial expressions, action intensity, and background atmosphere while preserving the semantics of the subject; valence and arousal labels are used to quantify emotional style and atmosphere; Step 2: Construct a text conditional feature cache; Encode the cue words in the training data using a frozen text encoder, generate and cache the neutral text hidden state, the sentiment text hidden state and the corresponding attention mask in a cache with sample identifier as the index key and using a sharded storage. The cache structure includes a unique sample identifier, the neutral cue word T5 hidden state, the sentiment cue word T5 hidden state, the valid / filler marker of the neutral word, the mask used to calculate the loss, the original valence arousal value and the normalized valence arousal value; Step 2.1: Construct a text conditional feature cache, which is used to avoid repeatedly loading the text encoder; Step 2.2: Use the frozen T5-XL text encoder to encode the neutral cue words and sentimental cue words in each valid training record to obtain the neutral text hidden state and the sentimental text hidden state; Step 2.3: Use an attention masking mechanism to mark the effective and filler words in the neutral text hidden state and the sentiment text hidden state to obtain the input attention mask and the loss attention mask; wherein, the effective words correspond to a mask value of 1 and the filler words correspond to a mask value of 0; Step 2.4: Write the sample identifier, neutral text hidden state, sentiment text hidden state, attention mask, loss attention mask, and continuous sentiment conditional vector into the text conditional feature cache; Step 3: Construct a learnable soft prefix sentiment adapter; generate soft sentiment prefixes based on the semantic summary of neutral cue words after masking and pooling and continuous sentiment conditional vectors; the soft sentiment prefixes are independent of the original neutral text lexical units and consist of learnable basic prefixes and prefix increments generated by context vectors; the learnable soft prefix sentiment adapter is used to train lexical prefixes representing sentiment features; Step 3.1: Use mask pooling technology to extract the global semantic summary of the neutral text hidden state to obtain the semantic summary of the neutral prompt words. The mask pooling method is shown in Equation (2): in, This represents a semantic summary of neutral cue words. Let represent the hidden vector of the t-th word in the hidden state of neutral text. Indicates the first The attention mask corresponding to each word, where ϵ represents a constant to prevent the denominator from being zero; Step 3.2: Use the first-to-last concatenation method to concatenate the semantic summary of neutral cue words and the continuous sentiment condition vector along the feature dimension to form a context vector; Step 3.3: A multilayer perceptron is used to generate a prefix increment based on the context vector, and the prefix increment is added to the learnable basic prefix to obtain a soft emotion prefix. The soft emotion prefix is ​​used to adjust the emotional atmosphere of the generated image; the generation method is shown in Equation (3): in, Prefix indicating soft emotions Indicates a learnable basic prefix. This indicates that a trainable multilayer perceptron is possible. This means reconstructing the multilayer perceptron output into a tensor that is the prefix length multiplied by the text condition dimension; Step 3.4: Set the soft emotion prefix to a continuous vector independent of the original neutral text lexical units, while keeping the neutral text lexical units unchanged; Step 4: The soft prefix sentiment adapter is trained using a joint loss consisting of pooling conditional mean square error loss, cosine direction loss, prefix direction loss, and norm constraint loss. The text encoder, visual autoregressive text-to-image backbone, and image decoder are frozen during the training process. Step 4.1: Use prefix concatenation to concatenate the soft emotion prefix with the neutral text hidden state dimension to form the overall text condition; Step 4.2: Using pooling conditional alignment loss to constrain the pooling representation of the overall text condition, which approximates the pooling representation of the hidden sentiment text state, the concatenated overall conditional pooling representation is defined as follows: Where P=16, and L is the number of effective lexical units of neutral cue words; thus enabling Tend to ; Step 4.3: Use cosine directional loss to constrain the overall text conditions and the emotional text hidden state to be consistent in direction in the text condition space; Step 4.4: Use prefix direction loss to constrain the soft emotion prefix to face the emotion text hidden state, and use norm constraint loss to make the vector scale of the soft emotion prefix close to the scale of the natural hidden state output by the frozen text encoder. Step 4.5: Train the soft prefix sentiment adapter using the joint loss function consisting of pooling conditional mean square error loss, cosine direction loss, prefix direction loss, and norm constraint loss as shown in Equation (5), and freeze the text encoder, visual autoregressive text-to-image backbone, and image decoder during training: The pooling conditional mean square error loss is used to make the overall text condition after concatenating the prefix closer to the sentiment cue word in the T5 space; the cosine direction loss is used to make the overall semantic direction after concatenating the prefix oriented towards the sentiment cue word; the prefix direction loss is used to make the soft prefix itself carry a strong sentiment direction; the norm constraint loss is used to control the prefix vector scale; in, Indicates joint loss, This represents the mean squared error loss due to pooling conditions. Indicates loss in the cosine direction. This represents the prefix direction loss. Represents norm-constrained loss. , and These represent the weights of the corresponding losses; Step 5: Construct text conditional input for visual autoregressive backbone; During the inference stage, the input prompt words are encoded by the frozen T5-XL text encoder and then fed into the trained soft prefix emotion adapter along with the normalized target valence-arousal coordinates to generate 16 soft emotion lexical units; The above soft emotion lexical units are concatenated to the neutral text hidden state, and then the text condition with emotion prefix is ​​converted into a text condition quadruple that can be directly received by the visual autoregressive text to the image backbone through the compact key-value sequence packaging technique; Step 5.1: Obtain the input cue words, target valence, and arousal of the image to be generated, and use the same normalization method as in the training phase to obtain the target continuous emotion conditional vector; Step 5.2: Encode the input prompts using a frozen text encoder to obtain the hidden state of the input text and the input attention mask; Step 5.3: Input the hidden state of the input text, the input attention mask, and the target continuous emotion conditional vector into the trained soft prefix emotion adapter to obtain the target soft emotion prefix; Step 5.4: Use prefix concatenation technology to add the target soft emotion prefix to the hidden state of the input text, forming a text condition with an emotion prefix, as shown in equation (6): in, This indicates text conditions with an emotion prefix. This indicates concatenation along the dimensions of the text sequence; Step 5.5: Compact text conditional packing technology is used to pack the text conditions with sentiment prefixes to obtain a text condition quadruple that can be accepted from visual autoregressive text to image backbone; the text condition quadruple includes a compact key value sequence, the effective length of each sample, the cumulative sequence offset, and the maximum effective length within the batch; Step 6: Input the text conditions with the emotion prefix into the frozen visual autoregressive text, then into the image backbone and image decoder to generate an image that corresponds semantically to the input prompt and matches the target emotion.

2. The method for generating emotion images based on visual autoregression as described in claim 1, characterized in that: Step 1 is implemented as follows: Step 1.1: Obtain training data containing training records with neutral cue words, sentiment cue words, valence labels, and arousal labels; where the values ​​of valence labels and arousal labels are both in the range of [-1, 1]. Step 1.2: Clean the training records using field completeness and numerical parsing checks, remove invalid records, and form valid training records; Step 1.3: Generate globally unique sample identifiers by mapping the valid training records; and scale and normalize the valence and arousal labels into continuous emotion conditional vectors, as shown in Equation (1): Where y represents the normalized continuous emotion conditional vector, v represents the valence label, and a represents the arousal label. Valence normalization scale, R_a represents the arousal normalization scale; Step 1.4: Divide the effective training records into training set, validation set and test set according to the preset ratio to form continuous emotion cue word matching training data.

3. The method for generating emotion images based on visual autoregression as described in claim 1, characterized in that: Step 2.1 is implemented as follows: Step 2.1.1: Each cached record contains seven fields: sample identifier, neutral text hidden state tensor, sentiment text hidden state tensor, input attention mask, loss attention mask, normalized valence-arousal vector, and original valence-arousal value; multiple records constitute a shard file, and each shard is managed uniformly by an index list; the neutral text hidden state tensor and sentiment text hidden state tensor both adopt a sequence length of 128 × hidden dimension of 2048; Step 2.1.2: The text conditional feature cache adopts a fragmented storage method. Each fragment file is serialized and saved in PyTorch tensor format. An index list is stored in the same directory, recording the path of each fragment file and the number of samples in the training set, validation set, and test set. Step 2.1.3: During training, locate the list of shard files by reading the index list, load each shard as needed, and filter the records that are divided into training set, validation set and test set; It supports parallel reading of multiple working processes and random shuffling by round; after parsing each record, it outputs sample identifiers, neutral text features, sentiment text features, attention masks and valence-wake vectors in batches, avoiding repeated calls to the frozen text encoder in the training loop.

4. The method for generating emotion images based on visual autoregression as described in claim 1, characterized in that: Step 5.5 is implemented as follows: Step 5.5.1: For each sample, append the sentiment prefix to the effective word to form a compact key-value sequence; Step 5.5.2: Use the length of the compact key-value sequence as the effective length of each sample; Step 5.5.3: Mark the starting position of each compact key value sequence and construct a cumulative sequence offset with a length of the sequence number plus 1 to record the starting position of each sample; Step 5.5.4: Record the maximum valid length within a batch in the compact key-value sequence; Step 5.5.5: Pack the text conditions with the emotion prefix into a text condition quadruple.

5. The method for generating emotion images based on visual autoregression as described in claim 1, characterized in that: Step 6 is implemented as follows: Step 6.1: Input the text conditional quadruple into the frozen Infinity-8B visual autoregressive text-to-image backbone, so that the visual autoregressive text-to-image backbone jointly predicts the visual representation based on the sentiment prefix and the original semantic units; Step 6.2: The visual representation is decoded using a frozen image decoder to obtain an emotion image that corresponds semantically to the input cue word and conforms to the target valence-arousal coordinates; the image decoder is the 32-channel variational autoencoder VAE-32 decoder that is compatible with the Infinity-8B.

6. A visual autoregressive-based emotion image generation system that implements the method described in claim 1, characterized in that: It includes a data construction module, a text feature precoding module, a soft prefix sentiment adaptation module, an adapter training module, a text conditional packaging module, and a visual autoregressive image generation module; The data construction module is used to acquire training records containing neutral cue words, emotional cue words, valence labels and arousal labels, and to perform field integrity detection, numerical parsing detection, sample identifier mapping and continuous emotion coordinate normalization on the training records to form continuous emotion cue word pairing training data. The text feature precoding module is used to call the frozen text encoder to encode neutral cue words and emotional cue words, obtain neutral text hidden state, emotional text hidden state, input attention mask and loss attention mask, and write them into the text conditional feature cache. The soft prefix emotion adaptation module is used to perform mask pooling on the hidden state of neutral text, generate a semantic summary of neutral cue words, concatenate the semantic summary of neutral cue words with the continuous emotion condition vector, input it into the multilayer perceptron, and output the soft emotion prefix. The adapter training module is used to train the soft prefix sentiment adaptation module using pooling conditional alignment loss, cosine direction loss, prefix direction loss and norm constraint loss under the conditions of frozen text encoder, visual autoregressive text to image backbone and image decoder. The text condition packaging module is used to concatenate the soft sentiment prefix with the hidden state of the input text, and convert the concatenated text conditions into a compact key-value sequence, effective length, cumulative sequence offset, and maximum effective length within a batch. The visual autoregressive image generation module is used to input the text condition quadruple output by the text condition packaging module into the frozen visual autoregressive text to the image skeleton, and call the frozen image decoder to output the emotion image. The evaluation module is used to evaluate the generated images for emotion alignment, semantic consistency, continuous visual response, and ranking consistency.

7. An application of the method as described in any one of claims 1 to 5 and the system as described in claim 6 in the generation of emotional advertising materials for digital media, characterized in that: Ad creators input neutral scene description prompts and target valence-awakening coordinates that express the brand's emotional tone. Using the above method, ad visual materials that are precisely matched with the brand's emotional positioning are automatically generated, enabling rapid switching and batch customization of the same scene description under different emotional atmospheres.

Images