An image emotion understanding and emotion generation method and system based on multi-agent cooperation, a terminal, and a storage medium

Abstract

This invention discloses a method, system, terminal, and storage medium for image sentiment understanding and generation based on multi-agent collaboration. The method includes: performing structured sentiment analysis on image samples and original sentiment tags using a multimodal large model to obtain the structured sentiment analysis results, and training a sentiment understanding agent to obtain a target sentiment understanding agent; using the target sentiment understanding agent to perform global semantic sentiment extraction on the image samples to obtain a global sentiment semantic representation, and training a sentiment generation agent based on the image samples and the global sentiment semantic representation to obtain a target sentiment generation agent; acquiring the image to be processed and inputting it into the target sentiment understanding agent to obtain the structured analysis results, and inputting these results into the target sentiment generation agent to obtain multiple candidate images; performing multi-criteria verification processing on the multiple candidate images and outputting the target generated image. This invention effectively improves the accuracy of sentiment image generation.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, system, terminal, and computer-readable storage medium for image emotion understanding and emotion generation based on multi-agent collaboration. Background Technology

[0002] Affective computing aims to enable intelligent systems to perceive, understand, and express emotional information, serving as a crucial technological foundation for fields such as human-computer interaction, digital media creation, intelligent design, and art design. With the development of multimodal learning, visual language models, and image generation models, intelligent systems are now able to extract emotion-related semantic and visual cues from multimodal data such as images and text, further supporting emotional state analysis and the generation of content with specific emotional tendencies.

[0003] Existing image sentiment understanding methods have gradually evolved from traditional closed-set sentiment classification to open-ended multimodal sentiment reasoning, such as sentiment description generation, multi-turn sentiment understanding, and sentiment causal analysis. At the same time, sentiment-based image generation methods are also gaining attention, aiming to generate image content that reflects specific emotional states based on given sentiment categories or semantic conditions.

[0004] However, existing research typically models image sentiment understanding and sentiment image generation as two independent tasks: sentiment understanding models mainly focus on sentiment recognition and interpretation, while sentiment generation models mainly focus on generating images based on text or sentiment labels. There is a lack of effective information interaction and collaborative optimization mechanisms between the two, resulting in inaccurate sentiment images.

[0005] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0006] The main objective of this invention is to provide a method, system, terminal, and computer-readable storage medium for image emotion understanding and emotion generation based on multi-agent collaboration. This invention aims to solve the problem in the prior art where image emotion understanding and emotion image generation are modeled as two independent tasks, lacking an effective information interaction and collaborative optimization mechanism between the two, resulting in inaccurate generated emotion images.

[0007] To achieve the above objectives, this invention provides a method for image emotion understanding and generation based on multi-agent collaboration, which includes the following steps: Image samples and original sentiment labels are acquired. Structured sentiment analysis is performed on the image samples and original sentiment labels using a multimodal large model to obtain structured sentiment analysis results. The sentiment understanding agent is then trained based on the structured sentiment analysis results to obtain the target sentiment understanding agent. The target emotion understanding agent performs global semantic emotion extraction on the image samples to obtain a global emotion semantic representation. The emotion generation agent is then trained based on the image samples and the global emotion semantic representation to obtain the target emotion generation agent. The image to be processed is acquired and input into the target emotion understanding agent to obtain the structured analysis result. The structured analysis result is then input into the target emotion generation agent to obtain multiple candidate images. Multiple candidate images are subjected to multi-criteria verification processing to output the target generated image.

[0008] Optionally, the image sentiment understanding and sentiment generation method based on multi-agent collaboration, wherein acquiring image samples and original sentiment tags, performing structured sentiment analysis on the image samples and original sentiment tags using a multimodal large model to obtain structured sentiment analysis results, and training the sentiment understanding agent based on the structured sentiment analysis results to obtain the target sentiment understanding agent, specifically includes: Image samples and original sentiment labels are acquired, and the image samples and original sentiment labels are input into a multimodal large model. The multimodal large model is used to perform image sample element recognition on the image samples to obtain the sample element recognition result. Multi-layered sentiment evidence is generated based on the sample element identification results, and the support of the original sentiment labels is judged and data preprocessed based on the multi-layered sentiment evidence to obtain structured sentiment analysis results. Obtain the sentiment analysis instruction, and serialize the sentiment analysis instruction, the image sample, and the structured sentiment analysis result to obtain the target token sequence; A sentiment understanding agent is determined, and an initial sentiment understanding agent is obtained by training the sentiment understanding agent with autoregressive loss based on the target token sequence. The initial sentiment understanding agent is fine-tuned using a low-rank adaptation method to obtain the target sentiment understanding agent.

[0009] Optionally, the image sentiment understanding and sentiment generation method based on multi-agent collaboration, wherein the step of extracting global semantic sentiment from the image samples using the target sentiment understanding agent to obtain a global sentiment semantic representation, and training the sentiment generation agent based on the image samples and the global sentiment semantic representation to obtain a target sentiment generation agent, specifically includes: The image samples are input into the visual encoder in the target emotion understanding agent, and the image samples are encoded by the visual encoder to obtain a visual token sequence; The global tokens in the visual token sequence are mapped using a projection semantic network to obtain a global sentiment semantic representation; The image samples are encoded into the latent space by a pre-trained variational autoencoder to obtain the image latent representation. The global sentiment semantic representation and the image latent representation are then fused to obtain a joint latent representation. A target emotion-generating agent is obtained by determining the emotion-generating agent and training it based on the joint latent representation.

[0010] Optionally, the image sentiment understanding and sentiment generation method based on multi-agent collaboration, wherein the target sentiment understanding agent performs global semantic sentiment extraction on the image sample to obtain a global sentiment semantic representation, and trains the sentiment generation agent based on the image sample and the global sentiment semantic representation to obtain a target sentiment generation agent, and then further includes; The target emotion understanding agent outputs multiple types of first information, and a text encoder is used to encode the multiple types of first information to obtain multiple types of emotion information. The first information includes background information, facial expression information, and logical information; The various types of emotional information are mapped to the condition space of the generative model through multiple preset independent projection networks to obtain background condition tokens, expression condition tokens, and logical condition tokens. The background condition token, the expression condition token, and the logical condition token are concatenated to obtain structured emotion conditions. The structured emotional conditions are injected into the target emotion-generating agent to obtain the final target emotion-generating agent.

[0011] Optionally, the image sentiment understanding and sentiment generation method based on multi-agent collaboration, wherein the step of acquiring the image to be processed, inputting the image to be processed into the target sentiment understanding agent to obtain a structured analysis result, and inputting the structured analysis result into the target sentiment generation agent to obtain multiple candidate images, specifically includes: The image to be processed is acquired and input into the target emotion understanding agent to obtain the structured analysis result; The structured analysis results include sentiment tags, background descriptions, facial or action cues, inference text, and metadata. The structured analysis results are converted into generated prompts, and the generated prompts are decomposed to obtain the target sentiment and structured sentiment conditions. The target emotion and the structured emotion conditions are input into the final target emotion generation agent to obtain multiple candidate images.

[0012] Optionally, the image emotion understanding and emotion generation method based on multi-agent collaboration, wherein the step of performing multi-criteria verification processing on multiple candidate images and outputting the target generated image specifically includes: Determine the current image library, and calculate a comprehensive score for each candidate image based on the current image library, the target emotion, and the generated prompts; The comprehensive score is the sum of the emotional semantic clarity score, the cue-image consistency score, the visual quality proxy score, the perceptual diversity score, and the semantic diversity score. A preset threshold is determined, and it is determined whether the comprehensive score of each candidate image is greater than the preset threshold; If so, candidate images that are greater than the preset threshold will be output to obtain the target generated image; If not, candidate images that are less than or equal to the preset threshold will be discarded or regenerated.

[0013] Optionally, the image emotion understanding and emotion generation method based on multi-agent collaboration, wherein the step of performing multi-criteria verification processing on multiple candidate images and outputting the target generated image further includes: Obtain the target sentiment label, target generation prompt, and target structured analysis result corresponding to the target generated image, and add the target generated image, the target sentiment label, the target generation prompt, and the target structured analysis result to the training set to obtain an updated training set; The target emotion understanding agent and the target emotion generation agent are iteratively enhanced based on the updated training set.

[0014] Furthermore, to achieve the above objectives, the present invention also provides an image emotion understanding and emotion generation system based on multi-agent collaboration, wherein the image emotion understanding and emotion generation system based on multi-agent collaboration includes: The emotion understanding agent training module is used to acquire image samples and original emotion labels, perform structured emotion analysis on the image samples and original emotion labels through a multimodal large model to obtain structured emotion analysis results, and train the emotion understanding agent based on the structured emotion analysis results to obtain the target emotion understanding agent; The emotion generation agent training module is used to perform global semantic emotion extraction on the image samples through the target emotion understanding agent to obtain a global emotion semantic representation, and to train the emotion generation agent based on the image samples and the global emotion semantic representation to obtain the target emotion generation agent; A candidate image generation module is used to acquire an image to be processed, input the image to be processed into the target emotion understanding agent to obtain a structured analysis result, and input the structured analysis result into the target emotion generation agent to obtain multiple candidate images; The target generated image output module is used to perform multi-criteria verification processing on multiple candidate images and output the target generated image.

[0015] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a multi-agent collaborative image emotion understanding and emotion generation program stored in the memory and executable on the processor, wherein when the multi-agent collaborative image emotion understanding and emotion generation program is executed by the processor, it implements the steps of the multi-agent collaborative image emotion understanding and emotion generation method described above.

[0016] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an image emotion understanding and emotion generation program based on multi-agent collaboration, and when the image emotion understanding and emotion generation program based on multi-agent collaboration is executed by a processor, it implements the steps of the image emotion understanding and emotion generation method based on multi-agent collaboration as described above.

[0017] In this invention, image samples and original sentiment tags are acquired. A multimodal large model is used to perform structured sentiment analysis on the image samples and original sentiment tags to obtain structured sentiment analysis results. A sentiment understanding agent is trained based on these results to obtain a target sentiment understanding agent. The target sentiment understanding agent then performs global semantic sentiment extraction on the image samples to obtain a global sentiment semantic representation. A sentiment generation agent is trained based on the image samples and the global sentiment semantic representation to obtain a target sentiment generation agent. An image to be processed is acquired and input into the target sentiment understanding agent to obtain structured analysis results. These results are then input into the target sentiment generation agent to obtain multiple candidate images. Multi-criteria verification is performed on these candidate images to output the target generated image. This invention achieves collaborative modeling of image sentiment understanding and generation through structured sentiment data construction, sentiment understanding agent training, sentiment generation agent training, visual evidence-guided prompt planning, and multi-criteria generation result verification, thereby effectively improving the accuracy of sentiment image generation. Attached Figure Description

[0018] Figure 1 This is a flowchart of a preferred embodiment of the image emotion understanding and emotion generation method based on multi-agent collaboration of the present invention; Figure 2 This is a schematic diagram comparing the present invention with existing emotional intelligence paradigms. Figure 2 (a) in the text represents the existing paradigm of emotional intelligence agents. Figure 2 (b) in the text represents the collaborative optimization method for emotion understanding and emotion generation in this invention. Figure 3 This is a schematic diagram of the overall framework of a preferred embodiment of the image emotion understanding and emotion generation method based on multi-agent collaboration of the present invention; Figure 4 This is a schematic diagram of the three-stage collaborative training strategy of a preferred embodiment of the image emotion understanding and emotion generation method based on multi-agent collaboration of the present invention; Figure 5 This is a structural diagram of a preferred embodiment of the image emotion understanding and emotion generation system based on multi-agent collaborative processing of the present invention; Figure 6 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0020] Affective computing aims to enable intelligent systems to perceive, understand, and express emotional information, serving as a crucial technological foundation for fields such as human-computer interaction, digital media creation, intelligent design, and art design. With the development of multimodal learning, visual language models, and image generation models, intelligent systems are now able to extract emotion-related semantic and visual cues from multimodal data such as images and text, further supporting emotional state analysis and the generation of content with specific emotional tendencies.

[0021] Existing image sentiment understanding methods have evolved from traditional closed-set sentiment classification to open-ended multimodal sentiment reasoning, such as sentiment description generation, multi-turn sentiment understanding, and sentiment causation analysis. Simultaneously, sentiment image generation methods have also gained attention, aiming to generate image content that reflects a specific emotional state based on given sentiment categories or semantic conditions. However, current research typically models image sentiment understanding and sentiment image generation as two independent tasks: sentiment understanding models primarily focus on sentiment recognition and interpretation, while sentiment generation models focus on generating images based on text or sentiment tags. An effective information exchange and collaborative optimization mechanism between the two is lacking.

[0022] This task-separation paradigm is prone to the following problems: On the one hand, sentiment understanding results are difficult to provide stable, fine-grained, and interpretable sentiment guidance for the image generation process; on the other hand, generation models usually rely only on sentiment categories or coarse-grained text cues, making it difficult to fully utilize visual evidence such as background scenes, subject expressions, gestures, and emotional logic, resulting in generated images that may have unclear emotional expression, inconsistent semantic conditions, or unstable visual quality. Furthermore, existing generation processes typically lack reliable result verification mechanisms, making it difficult to simultaneously guarantee the emotional clarity, cue consistency, visual quality, and diversity of generated images.

[0023] Therefore, although existing image sentiment understanding and sentiment image generation technologies have made some progress, they still suffer from problems such as insufficient sentiment supervision granularity, independent understanding and generation tasks, lack of structured sentiment constraints on generation conditions, and lack of reliable screening of generation results. There is an urgent need for a technical solution that can achieve collaborative modeling of image sentiment understanding and sentiment image generation within a unified framework, in order to improve the interpretability of sentiment understanding and the semantic consistency and visual quality of sentiment image generation.

[0024] In other words, image sentiment is typically influenced by a combination of factors, including background scene, lighting atmosphere, subject's facial expression, posture and actions, and visual semantic relationships. However, existing methods struggle to systematically distinguish and utilize this visual evidence, resulting in a lack of interpretability in the sentiment judgment process. When the original sentiment labels contain noise, ambiguity, or do not perfectly match the visual content, the model also struggles to effectively validate and correct the labels based on visual evidence.

[0025] Existing image sentiment understanding and sentiment image generation methods are mostly designed and optimized as two independent tasks. Sentiment understanding models primarily analyze the emotional semantics in images, while sentiment generation models mainly generate images based on sentiment categories or textual cues. There is a lack of effective information transfer and collaborative training mechanisms between the two. This task-separate modeling approach makes it difficult to fully utilize sentiment understanding results to guide the generation process, nor can it use generation results to inversely enhance sentiment understanding capabilities, thus limiting the system's collaborative ability between sentiment analysis and sentiment expression.

[0026] Secondly, existing sentiment image generation methods typically rely on simple sentiment labels or coarse-grained textual cues as generation conditions, making it difficult to fully express the fine-grained sentiment semantics in complex images. Due to the lack of structured constraints such as background information, subject expression information, and sentiment logic information, the generation model may produce visually plausible images but with unclear sentiment expression or inconsistent with the target sentiment. Furthermore, directly inputting lengthy sentiment analysis text into the generation model may introduce conditional noise, including inference statements, redundant descriptions, or ambiguous expressions unrelated to visual generation, affecting the semantic consistency and visual quality of the generated images.

[0027] Furthermore, existing emotional image generation processes typically lack reliable result filtering mechanisms. A generative model may generate multiple candidate images under the same emotional condition, but these candidate images vary significantly in terms of emotional clarity, consistency with textual cues, visual quality, and content diversity. Without an effective multi-criteria verification mechanism, low-quality, emotionally ambiguous, or semantically inconsistent images may be directly output or used for subsequent training, thus affecting the overall system performance and stability.

[0028] Existing technologies for image sentiment understanding and sentiment image generation still suffer from several technical challenges, including insufficient structured sentiment supervision, lack of coordination between understanding and generation, lack of effective planning of generation conditions, and lack of reliable screening of generated results. Therefore, a novel collaborative method for image sentiment understanding and generation is urgently needed to achieve, within a unified framework, sentiment understanding based on visual evidence, image generation targeting specific sentiments, and quality control of the generated results. This would improve the interpretability of sentiment understanding, the emotional consistency of generated images, and the overall reliability of the system.

[0029] The preferred embodiment of the image emotion understanding and emotion generation method based on multi-agent collaboration described in this invention, such as... Figure 1 As shown, the image emotion understanding and emotion generation method based on multi-agent collaboration includes the following steps: Step S10: Obtain image samples and original sentiment labels, perform structured sentiment analysis on the image samples and original sentiment labels using a multimodal large model to obtain structured sentiment analysis results, and train the sentiment understanding agent based on the structured sentiment analysis results to obtain the target sentiment understanding agent.

[0030] like Figure 2 As shown, this invention proposes a multi-agent collaborative image sentiment understanding and generation method, mainly used for image sentiment analysis, structured sentiment reasoning, and image generation tasks targeting specific sentiments. This invention unifies the sentiment understanding agent, sentiment generation agent, prompting and planning agent, and multi-criteria verification module into a single framework, enabling the system to perform sentiment understanding based on visual evidence and further generate images that conform to the target sentiment.

[0031] Figure 2 In the example (a), it is indicated that in the existing paradigm, the emotion-generating agent and the emotion-understanding agent are usually independent of each other, and the two lack effective interaction. Figure 2 (b) indicates that the present invention enables structured emotion understanding results to guide the generation of emotion images through the collaboration of an understanding agent, a prompting planner, a generating agent, and a verifier, and to filter candidate images through a verifier, thereby achieving collaborative optimization of emotion understanding and emotion generation.

[0032] like Figure 3 As shown ( Figure 3 This is a schematic diagram of the overall framework of the present invention, which includes modules such as data construction, training, generative reasoning, and analytical reasoning. In the data construction stage, visual reasoning is performed on images and their original labels using emotion understanding skills to obtain structured emotional information such as background, expression, and logic. In the training and reasoning stages, the understanding agent provides emotion analysis results and a global emotional semantic representation, and the generative agent generates emotion-consistent images based on these results. A prompt planner and a validator further improve the reliability of the generated results. The framework of the present invention mainly includes the following components: 1. Structured Sentiment Data Construction Module: This module is used to transform raw sentiment tags into structured sentiment supervision data that includes background, facial expression, and logical information.

[0033] 2. Emotional Understanding Agent: Used to output emotion labels, background descriptions, subject's facial expressions or actions, and emotional reasoning logic based on the input image.

[0034] 3. Emotion-generating intelligent agent: used to generate images based on target emotions and structured emotion conditions.

[0035] 4. Hint Planning Agent: Used to convert structured sentiment analysis results into concise hints suitable for use by generative models.

[0036] 5. Multi-criteria verification module: used to filter generated images based on factors such as emotional clarity, cue consistency, visual quality, and diversity.

[0037] First, structured sentiment analysis data needs to be constructed. The specific steps are as follows: For each image sample and its original sentiment label, both are input into a multimodal large model. Through preset sentiment analysis skills, the model is guided to analyze from three aspects: background, expression, and logic.

[0038] Specifically, image samples and original sentiment labels are acquired, and the image samples and original sentiment labels are input into a multimodal large model. The multimodal large model is used to identify image sample elements of the image samples to obtain sample element identification results. Multi-level sentiment evidence is generated based on the sample element identification results, and support judgment and data preprocessing are performed on the original sentiment labels based on the multi-level sentiment evidence to obtain structured sentiment analysis results.

[0039] like Figure 2 As shown, this invention proposes a method for image sentiment understanding and sentiment generation based on multi-agent collaboration. This method includes structured sentiment data construction, sentiment understanding agent training, sentiment generation agent training, joint optimization of understanding and generation, visual evidence-guided prompt planning, and multi-criteria generation result verification. The overall process consists of a sentiment understanding agent, a sentiment generation agent, a prompt planning agent, and a multi-criteria verification module.

[0040] This invention sets the input image as follows: Its original emotional label is The objective of this invention is to obtain structured sentiment analysis results under the constraint of visual evidence, and further generate images that conform to the target sentiment. .

[0041] The process of constructing structured sentiment analysis data is as follows: For each image sample and its original emotional tags The two are input into a multimodal large model (wherein, a multimodal large model refers to a visual language model with image understanding, text understanding, and graph-text reasoning capabilities, such as GPT-4V / GPT-4o, Qwen-VL / Qwen2-VL / Qwen2.5-VL, InternVL, LLaVA series, MiniCPM-V, etc. This invention does not limit the specific model, as long as it can input images and sentiment tags and output sentiment analysis results based on visual evidence), and guides the model to analyze from three aspects: background, expression, and logic through preset sentiment analysis skills (the analysis process is as follows: first, input the image and its original sentiment tags; then, the multimodal large model sequentially identifies the background scene, lighting atmosphere, subject object, facial expression, posture and movement, and significant visual elements in the image; then, it generates sentiment evidence at the background level, sentiment evidence at the subject expression or movement level, and logical reasoning from visual evidence to sentiment category; finally, it combines the above three types of information to determine whether the original tag is supported by visual evidence and outputs structured sentiment analysis results), and the model outputs. It can be represented as:

[0042] in, Indicates the final emotional label, This indicates a description of the background scene. Indicates clues about the subject's facial expressions, posture, or actions. Representing emotional reasoning logic, This indicates the result of the judgment on the original label.

[0043] The label judgment result is defined as:

[0044] in, To support, Since it cannot be explicitly denied, This is inconsistent.

[0045] When the original label is supported by visual evidence or cannot be explicitly refuted, the original label is retained. When the original label does not match the visual evidence, a revised label is used (if all three types of visual evidence—background, expression, and logic—support the original sentiment label or cannot explicitly refute it, the original label is retained; if the visual evidence is significantly inconsistent with the original label, the multimodal large model re-determines the target sentiment category based on the visual content and uses that category as the revised label. The content of the revised label is a sentiment category from a pre-defined set of sentiment categories, such as amusement, awe, contentment, excitement, anger, disgust, fear, sadness, etc., and can also be expanded to other sentiment categories depending on the specific dataset). The final sentiment label is... The expression is: ; in, This represents the corrected sentiment labels provided by the multimodal large model based on visual evidence. After format normalization, outlier filtering, and manual review, a structured sentiment training set is obtained. (i.e., the structured sentiment analysis results in this invention):

[0046] Where N represents the total number of image samples in the structured emotion training set, which is the number of training samples retained after format normalization, outlier filtering, and manual review.

[0047] Obtain sentiment analysis instructions, and serialize the sentiment analysis instructions, the image samples, and the structured sentiment analysis results to obtain a target token sequence; determine the sentiment understanding agent, and train the sentiment understanding agent using autoregressive loss based on the target token sequence to obtain an initial sentiment understanding agent; fine-tune the initial sentiment understanding agent using low-rank adaptation to obtain the target sentiment understanding agent.

[0048] like Figure 4 As shown (the first stage is to train the sentiment understanding agent to generate structured sentiment analysis text including background, facial expressions, and logic), the training process for the sentiment understanding agent is roughly as follows: The obtained structured sentiment training set (i.e., structured sentiment analysis results) is organized into visual instruction data for training the sentiment understanding agent. Each training sample includes an image sample, a sentiment analysis instruction, and a structured answer.

[0049] During training, the sentiment understanding agent generates structured sentiment analysis text based on the input image and sentiment analysis instructions, including sentiment category, background evidence, subject facial expression or action evidence, and logical reasoning explanations. The model is trained using an autoregressive language modeling approach, enabling it to learn sentiment analysis capabilities based on visual evidence.

[0050] Furthermore, a low-rank adaptation approach can be used to efficiently fine-tune the parameters of the basic multimodal large model, that is, freeze most of the parameters of the basic model and only update a small number of trainable adaptation parameters to reduce training costs.

[0051] The detailed steps for training an emotion-understanding agent are as follows: 1. Organize the structured sentiment analysis results into visual instruction data: For the first... Each sample, the image Sentiment analysis instructions (The sentiment analysis instructions are pre-designed instruction templates by the system, such as "Please determine the sentiment category of the image and explain the visual basis from three aspects: background, expression, and logic." During the reasoning phase, these instructions can be generated by the system by default or entered or adjusted by the user according to specific needs.) and structured answers. (Structured answer) Results from structured sentiment analysis The two are related in meaning but different in form: ={ , x i r i v i} represents a structured collection of fields; It is The emotional tags, background descriptions, facial or behavioral cues, reasoning logic, and tag judgment results are serialized into a natural language answer (used to train the emotional understanding agent) and then serialized into the target token sequence X. i The expression is as follows: X i =Tokenize([I i q i a i ])={x{i,1},x{i,2},…,x{i,T}}; Among them, I i Indicates image input, q i Indicates a sentiment analysis instruction. i Representing a structured answer, X i This represents the target token sequence obtained by serializing the image, instructions, and answers together, where T is the sequence length.

[0052] 2. Input the target token sequence into the sentiment understanding agent to obtain the hidden state sequence. (The specific implementation process is as follows: First, image I is extracted through a visual encoder.) i The visual features are mapped to an embedding space that the language model can process through a projection layer; simultaneously, the sentiment analysis instruction q is applied. i and structured answer a i The visual embedding is then converted into a text token embedding. The visual embedding and text embedding are then concatenated and input into the sentiment understanding agent. After multi-layer self-attention and feedforward network computation, the hidden state sequence H corresponding to each token position is obtained, expressed as:

[0053] Where H represents the hidden state sequence obtained by the emotion understanding agent after performing forward computation on the input token sequence (i.e., the target token sequence), which can be represented as H={ , , ..., }, where T represents the length of the input token sequence. This represents an emotion understanding agent with parameter θ, which is a multimodal autoregressive model used to process images, emotion analysis instructions, and structured answers. This represents the hidden state vector corresponding to the T-th token position in the hidden state sequence H, which is the contextual semantic representation of the last position and can be used for subsequent token prediction or global semantic representation.

[0054] For the At each position, the model predicts the probability of the next token. for:

[0055] in, This represents the conditional probability distribution modeled by the sentiment understanding agent with parameter θ, used to indicate the probability of the model predicting the next token given the context. This represents the token at position t in the sequence; This represents all tokens up to the t-th position, i.e., the historical context that the model has seen. Indicates the t-th The hidden state vector corresponding to each position is used to aggregate the preceding context information and predict the probability distribution of the t-th token by outputting the language modeling head.

[0056] in, Output the language modeling head. The sentiment understanding agent is trained using autoregressive loss. The expression for the calculation is as follows:

[0057] in, This is the loss mask, used to compute the loss only on the response token. This loss enables the model to learn the output sentiment category, contextual evidence, facial expression evidence, and logical reasoning text.

[0058] like Figure 4As shown (the second stage freezes the understanding agent and guides the training of the sentiment generation agent using its textual conditions and CLS tokens), to reduce training costs, a low-rank adaptation method can be used to efficiently fine-tune the parameters of the base model (specifically, the pre-trained multimodal base model used by the sentiment understanding agent, such as Qwen-VL, Qwen2-VL, Qwen2.5-VL, InternVL, LLaVA, etc.). Low-rank adaptation involves adding trainable low-rank matrices to some linear layers of the base model to reduce training costs. For the frozen weight matrix in the base model... Its update format is as follows:

[0059]

[0060] in, and For trainable low-rank matrices, For rank, Let W′ be the scaling factor, representing the equivalent weight matrix after low-rank adaptation update, i.e., the weights after adding the low-rank increment to the original frozen weight matrix W. ΔW represents the weight increment introduced by low-rank adaptation, used to supplement and adjust the model's capabilities through trainable low-rank matrices A and B without directly updating the original weights W. Therefore, W′ = W + ΔW indicates that the model actually uses the equivalent weights obtained by adding the original weights to the low-rank update amount during computation.

[0061] Step S20: The target emotion understanding agent performs global semantic emotion extraction on the image sample to obtain a global emotion semantic representation, and trains the emotion generation agent based on the image sample and the global emotion semantic representation to obtain the target emotion generation agent.

[0062] The general process for global sentiment semantic representation extraction is as follows: After training the sentiment understanding agent, the image is input into the agent's visual encoder to extract the image's visual feature sequence. Since the visual encoder has been trained on the sentiment understanding task, its global features contain the overall sentiment semantic information of the image.

[0063] Subsequently, the global visual features are mapped to the conditional space of the generative model through a semantic projection network to obtain a global sentiment semantic representation. This representation is used to provide overall sentiment constraints to the sentiment-generating agent.

[0064] The training process for the emotion generation agent is as follows: The emotion generation agent uses a diffusion model as the generation backbone. First, the input image is encoded into the latent space to obtain the image latent representation; then, this latent representation is fused with the global emotion semantic representation and used as the input to the diffusion model (i.e., the joint latent representation).

[0065] During training, the model progressively adds noise to the fused latent representation and learns to perform denoising and recovery under structured emotional constraints. Through this process, the emotion-generating agent can learn to generate images consistent with the target emotion.

[0066] Specifically, the image samples are input into the visual encoder in the target emotion understanding agent, and the image samples are encoded by the visual encoder to obtain a visual token sequence; the global tokens in the visual token sequence are mapped by the projection semantic network to obtain a global emotion semantic representation.

[0067] Furthermore, global sentiment semantic representation extraction is performed, and the specific implementation process is as follows: After completing the training of the emotion understanding agent, the image Input its visual encoder to obtain a visual token sequence :

[0068] in, This represents the hidden state vector corresponding to the Nth local visual token output by the visual encoder. Here, N represents the number of local visual tokens obtained after the image is segmented by the visual encoder. , … These represent the visual feature representations of different local regions of the image. (and) Different types of global visual emotional information are used to aggregate them. This represents the feature representation of the last local visual token.

[0069] in, This represents a global token that aggregates global visual sentiment information. Subsequently, it is mapped to the conditional space of the generative model through a semantic projection network.

[0070] in, This represents the global sentiment semantic representation, used to provide overall sentiment constraints to the sentiment-generating agent. This represents a semantic projection network.

[0071] The image samples are encoded into the latent space by a pre-trained variational autoencoder to obtain the image latent representation. The global sentiment semantic representation and the image latent representation are then fused to obtain a joint latent representation. An emotion-generating agent is determined and trained based on the joint latent representation to obtain the target emotion-generating agent.

[0072] The training process for the emotion-generating agent is as follows: The emotion-generating agent uses a diffusion model as its generative backbone. It first processes the input image through a pre-trained variational autoencoder. Encoding into the latent space yields the latent representation of the image. The expression is:

[0073] Subsequently, the latent representation of the image With global sentiment semantic representation By fusion, a joint potential representation is obtained. :

[0074] in, This indicates a splicing operation.

[0075] In the forward noise addition process of the diffusion model, for time step Joint potential representation Added noise The expression is:

[0076] in, , For the diffuse noise scheduler at time step The cumulative retention factor.

[0077] The diffusion model learns to predict noise under structured sentiment conditions, and its training objective can be written as:

[0078] in, This represents a denoising network. This represents a structured emotional conditional sequence.

[0079] Further, multiple types of first information output by the target emotion understanding agent are acquired, and the multiple types of first information are encoded using a text encoder to obtain multiple types of emotion information; wherein, the first information includes background information, facial expression information, and logical information; the multiple types of emotion information are mapped to the generative model condition space through multiple preset independent projection networks to obtain background condition tokens, facial expression condition tokens, and logical condition tokens; the background condition tokens, facial expression condition tokens, and logical condition tokens are concatenated to obtain structured emotion conditions; the structured emotion conditions are injected into the target emotion generation agent to obtain the final target emotion generation agent.

[0080] Furthermore, structured emotional conditioning is injected, roughly as follows: The background information, facial expression information, and logical information output by the emotion understanding agent are input into the text encoder to obtain the corresponding semantic representations. Subsequently, the three types of information are mapped to the condition space of the generative model through independent projection networks, forming a unified structured emotion condition.

[0081] In the diffusion model denoising process, the structured emotional condition is injected into the generative network through a cross-attention mechanism, so that the model pays attention to scene atmosphere, subject expression and emotional logic at the same time when generating images, thereby improving the emotional consistency of the generated images.

[0082] Furthermore, the specific process of structured emotional conditioning injection is as follows: Background information output by the emotion-understanding agent Emoticons and logical information The text encoder is used to encode the expressions as follows:

[0083] Subsequently, different types of emotional information are mapped to the generative model condition space through three independent projection networks:

[0084] in, , and These represent independent projection networks corresponding to background information, facial / action information, and logical information, respectively. The implementation can employ linear layers, such as the Linear-GELU-LayerNorm-Linear structure, to map the semantic vectors output by the text encoder to the conditional token space of the diffusion generative model. The parameters of the three types of projection networks are independent to preserve the differences between different types of emotional evidence.

[0085] in, , , These represent the background condition token, the expression condition token, and the logical condition token, respectively. Concatenating these three tokens yields a unified structured condition sequence (i.e., structured sentiment condition). The expression is:

[0086] In order to structure emotional conditions Injection diffusion model, in the first In each denoising layer, let the latent token of the image be... Conditional reinforcement representation is obtained through cross attention. :

[0087] in, , , These are the query, key, and value projection matrices, respectively. For the attention dimension. This formula represents the structured sentiment condition with the image's latent token as the query. As Key and Value, background, expression, and logical information are injected into the image generation process.

[0088] like Figure 4 As shown in the diagram (the third stage involves joint optimization of the understanding agent and the generative agent to enhance their collaborative capabilities and improve the quality of generated emotional images), the joint optimization of the understanding agent and the generative agent is performed. The general process is as follows: After training the emotion understanding agent and the emotion generation agent separately, they are jointly optimized. During the joint optimization process, the emotion understanding agent continues to learn and output structured emotion analysis results, while the emotion generation agent continues to learn and generate images that conform to the target emotion.

[0089] Meanwhile, an emotional semantic consistency constraint is introduced to ensure that the global emotional semantics extracted by the emotional understanding agent are consistent with the generated prompts.

[0090] The specific implementation steps for the joint optimization of understanding agents and generative agents are as follows: After training the emotion understanding agent and the emotion generation agent separately, further joint optimization is performed, with the overall training objective defined as follows: :

[0091] in, Used to constrain the emotional understanding agent to generate structured sentiment analysis text. This is used to constrain the emotion-generating agent to generate images that conform to the target emotion. Used to constrain the consistency between the global sentiment semantic token and the generation conditions. , , These are the weighting coefficients.

[0092] Among them, token-level sentiment semantic alignment loss It can be defined as:

[0093] in, Indicates the generation of prompts The text semantic representation is used to enhance the consistency between the global sentiment semantics extracted by the understanding agent and the generated prompts.

[0094] Step S30: Obtain the image to be processed, input the image to be processed into the target emotion understanding agent to obtain the structured analysis result, and input the structured analysis result into the target emotion generation agent to obtain multiple candidate images.

[0095] The planning process for visual evidence-guided prompts is as follows: In the reasoning phase, the image to be processed is input into the emotion understanding agent to obtain structured emotion analysis results, which include emotion labels, background descriptions, facial or action cues, logical reasoning text, and metadata.

[0096] Specifically, the process involves acquiring an image to be processed and inputting it into the target emotion understanding agent to obtain a structured analysis result. This structured analysis result includes emotion tags, background descriptions, facial or behavioral cues, inference text, and metadata. The structured analysis result is then converted into generation prompts, which are decomposed to obtain the target emotion and structured emotion conditions. Finally, the target emotion and structured emotion conditions are input into the final target emotion generation agent to obtain multiple candidate images.

[0097] Furthermore, the specific implementation steps for visual evidence-guided prompt planning are as follows: During the inference phase, the emotion understanding agent outputs structured analysis results. :

[0098] in, As an emotional tag, For background description, For facial expressions or action cues, For reasoning text, This is metadata.

[0099] Furthermore, the planning agent is prompted to convert this structured result into a generating prompt. :

[0100] The conversion process is as follows: First, the structured result A={e, b, x, r, m} is read; then, metadata, redundant inference text, and uncertain expressions irrelevant to image generation are filtered out; next, information directly related to visual generation is selected, including the target emotion category, subject, background scene, facial expressions and actions, lighting atmosphere, salient objects, and visual style; finally, this information is compressed into concise generation prompts. For example, "emotion is sadness, background is a rainy street, subject walks with head down, logic is low saturation and lonely posture reflecting sadness" can be converted into "alonely person walking with head down on a rainy street, low saturation, gloomy atmosphere, sadness".

[0101] Furthermore, the planning process is broken down as follows:

[0102]

[0103]

[0104] in, Used to filter metadata, redundant inference descriptions, ambiguous expressions, and non-visual content; Used to select information directly related to visual generation; This is used to compress the filtered information into a compact generated prompt, where Z represents the filtered structured sentiment information; K represents the key visual generation information selected from Z; and p represents the final generated prompt. The specific decomposition process is as follows: 1. Z = Filter(A): Remove metadata, redundant inference, ambiguous expressions, and non-visual content; 2. K = Select(Z): Select visual generation elements such as sentiment category, subject, scene, expression, action, lighting, atmosphere, and salient objects; 3. p = Compress(K): Compress and organize the selected elements into a compact text prompt suitable for the diffusion model input.

[0105] Step S40: Perform multi-criteria verification processing on the multiple candidate images and output the target generated image.

[0106] This invention includes a multi-criteria verification module: After the emotion-generating agent generates multiple candidate images based on the generated prompts and the target emotion, the multi-criteria verification module filters the candidate images, and the evaluation content includes emotional semantic clarity, prompt-image consistency, visual quality, perceptual diversity, and semantic diversity.

[0107] If the overall score of a candidate image reaches a preset threshold, it is used as the final output or added to the image library; if it does not reach the threshold, the candidate image is discarded or a new image is generated. This method can reduce generated results with unclear emotions, semantic inconsistencies, or poor visual quality.

[0108] Specifically, the current image library is determined, and a comprehensive score for each candidate image is calculated based on the current image library, the target emotion, and the generated prompt; wherein, the comprehensive score is the sum of the emotion semantic clarity score, the prompt-image consistency score, the visual quality proxy score, the perceptual diversity score, and the semantic diversity score; a preset threshold is determined, and it is determined whether the comprehensive score of each candidate image is greater than the preset threshold; if so, the candidate images greater than the preset threshold are output to obtain the target generated image; if not, the candidate images less than or equal to the preset threshold are discarded or regenerated.

[0109] Furthermore, the specific process for updating and iteratively enhancing the image library is as follows: For generated images that pass multi-criteria verification, they are combined with the corresponding target emotion, generation prompts, background descriptions, facial or action cues, and logical reasoning information to form new training samples, which are then added to the image library or training dataset.

[0110] Subsequently, the updated data can be used to continue training the emotion understanding agent or the emotion generation agent.

[0111] Furthermore, the specific processing procedure of the multi-criteria verification module is as follows: Candidate images generated by the emotion-generating agent The multi-criteria verification module is based on the target sentiment. Generate prompts and the current image library Calculate the overall score:

[0112] in, Let m represent the scoring item corresponding to the m-th evaluation criterion, where m = 1, 2, 3, 4, 5, corresponding to emotional semantic clarity, cue-image consistency, visual quality proxy score, perceptual diversity, and semantic diversity, respectively. This represents the weight coefficient corresponding to the m-th scoring item, used to control the importance of different evaluation criteria in the overall score. During calculation, firstly, the five scoring items S1 to S5 are calculated separately and normalized to a uniform numerical range; then, according to the preset weights... The candidate image is obtained by weighted summation of the scores for each evaluation item. .in, ≥0, and the sum of the five weights is 1.

[0113] in,

[0114] The five scoring items are defined as follows.

[0115] 1. Clarity of Emotional Semantics :

[0116] in, This represents the semantic clarity score of the candidate image Î with respect to the target emotion e, used to measure whether the candidate image can clearly express the target emotion. This score can be output by the frozen emotion recognizer, i.e., the probability that the candidate image Î is judged to be the target emotion e. The higher the score, the more clearly the emotion expression of the candidate image is made. This indicates that the frozen emotion recognizer determines whether a candidate image belongs to the target emotion. The probability of.

[0117] 2. Hint—Image Consistency :

[0118] in, The cue-image consistency score represents the relationship between the candidate image Î and the generated cue p, and is used to measure whether the content of the generated image is semantically consistent with the input cue. Specifically, it is obtained by extracting features from the candidate image using an image encoder, extracting cue text features using a text encoder, and then calculating the cosine similarity between the two. The higher the score, the better the image content matches the generated cue. and These represent the image encoder and the text encoder, respectively.

[0119] 3. Visual quality proxy score :

[0120] in, For visual feature extractors, and Let represent the mean and covariance of the features of the real image, respectively. This represents the Mahalanobis distance.

[0121] 4. Perceiving diversity :

[0122] in, (Î) represents the visual quality proxy score of candidate image Î, which measures how closely the candidate image's visual feature distribution resembles the distribution of the real image. This score is generated by obtaining candidate image features through a visual feature extractor, calculating the Mahalanobis distance between the candidate image and the mean μ_r and covariance Σ_r of the real image feature distribution, and then converting this distance to a quality score using an exponential function. A smaller distance indicates a closer resemblance to the real image distribution, resulting in a higher visual quality proxy score. The perceptual diversity score represents the candidate image Î relative to the current image library B, and is used to measure the degree of difference between the candidate image and existing images at the visual perception level. This represents the j-th nearest neighbor image sample in the current image library B that is closest to the candidate image Î. This score is calculated by comparing the candidate image Î with its nearest neighbor samples. The LPIPS perceptual distance between the two images is averaged. A higher score indicates lower repetition between the candidate image and existing images, and better perceptual diversity.

[0123] 5. Semantic diversity :

[0124] in, Indicates the current image library In and candidate images closest One sample, This represents a semantic embedding function.

[0125] Finally, based on the threshold The expression for determining whether a candidate image should be accepted is:

[0126] If a candidate image meets the threshold requirement, it is either used as the final output or added to the image library; otherwise, the candidate image is discarded or regenerated.

[0127] Furthermore, the target sentiment label, target generation prompt, and target structured analysis result corresponding to the target generated image are obtained, and the target generated image, the target sentiment label, the target generation prompt, and the target structured analysis result are added to the training set to obtain an updated training set; the target sentiment understanding agent and the target sentiment generation agent are iteratively enhanced based on the updated training set.

[0128] The specific implementation steps for image library updates and iterative enhancements are as follows: For images that pass the multi-criteria verification, they are combined with their corresponding sentiment tags, generated prompts, and structured analysis results to form a new sample. The expression is:

[0129] Then, it was added to the training data to obtain :

[0130] in, This refers to the original structured sentiment training dataset, which is the training data obtained through the aforementioned construction of sentiment understanding skills, format normalization, outlier filtering, and manual review. Each sample in this dataset typically includes structured sentiment supervision information such as images, sentiment labels, background descriptions, facial or behavioral cues, and sentiment inference logic. = ∪ This indicates that the updated dataset is formed by adding newly generated samples that have passed multiple criteria verification to the original structured sentiment training set.

[0131] Furthermore, the updated data is used to continue training the emotion understanding agent or the emotion generation agent, so that the system can gradually improve its emotion understanding ability and emotion generation quality during the iteration process.

[0132] Experimental Comparison: Under the same semantic theme, this invention compares the emotional images generated by existing technologies such as Stable Diffusion, DreamBooth, Textual Inversion, EmoGen, and EmoSym with those generated by the method of this invention, covering multiple emotional categories such as "joy," "anger," "surprise," "sadness," and "excitement." It can be seen that the images generated by the method of this invention are more stable in terms of target emotional expression, scene semantic consistency, and visual realism, and can more accurately reflect the visual atmosphere and subject state corresponding to different emotional categories, thus verifying the effectiveness of this invention in emotional image generation.

[0133] Compared with existing image sentiment understanding and sentiment image generation methods, the present invention has at least the following advantages: 1. This invention achieves structured modeling of image sentiment semantics by decomposing the image sentiment understanding process into background information, subject expression information, and sentiment logic information. Compared to methods that rely solely on sentiment category labels for training, this invention can extract visual evidence from multiple levels, including scene environment, subject behavior, and sentiment reasoning relationships. This allows the sentiment understanding result to not only include the final sentiment category but also provide the corresponding visual evidence and reasoning process. Therefore, this invention improves the interpretability and reliability of image sentiment understanding.

[0134] 2. This invention introduces a label verification and correction mechanism based on visual evidence during the data construction phase. For samples where the original sentiment labels are inconsistent with the image content or are ambiguous, this invention can judge and correct the labels based on background, expression, and logical analysis results, thereby reducing the impact of noisy labels in the original data on model training. Through this method, more granular and reliable structured sentiment supervision information can be obtained, improving the training quality of subsequent sentiment understanding agents.

[0135] 3. This invention places the emotion understanding agent and the emotion generation agent under a unified framework for collaborative modeling, overcoming the problem of independent emotion understanding and emotion generation in existing methods. The structured analysis text and global emotional semantic representation output by the emotion understanding agent can serve as generation conditions to guide the emotion generation agent, enabling the generation model to generate images not only based on simple emotion tags, but also by combining fine-grained emotional information such as background, expression, and logic. Therefore, this invention can improve the emotional consistency and semantic accuracy of generated images.

[0136] 4. This invention employs a training strategy combining phased training and joint optimization. First, the emotion understanding agent is trained to acquire emotion analysis capabilities based on visual evidence. Then, the emotion understanding results guide the emotion generation agent in learning to generate target emotion images. Finally, the understanding agent and the generation agent are jointly optimized, enabling them to update collaboratively under a unified goal. This training method enhances the information interaction between emotion understanding and emotion generation, improving the overall system's collaborative and generalization capabilities.

[0137] 5. This invention establishes a visual evidence-guided prompting planning agent that converts structured sentiment analysis results into compact prompts suitable for the input of an image generation model. This prompting planning process filters redundant text, metadata, abstract reasoning statements, and ambiguous expressions, while retaining information directly related to visual generation, such as sentiment category, subject, expression, posture, scene, lighting, and salient objects. This approach reduces interference from irrelevant text in the image generation process, improving the effectiveness and stability of the generation conditions.

[0138] 6. This invention incorporates a multi-criteria verification module to evaluate generated candidate images in terms of emotional clarity, cue consistency, visual quality, and diversity. Compared to methods that directly output the generated results, this invention can filter out candidate images with unclear emotional expression, inconsistency with cue, poor visual quality, or repetitive content, thereby improving the reliability and overall quality of the final output image.

[0139] 7. This invention can also add high-quality generated images that have passed multi-criteria verification to the image library or training data for subsequent model iteration and optimization. In this way, the system can accumulate high-quality emotional image samples during continuous generation and filtering, further improving the performance of the emotion understanding agent and the emotion generation agent, achieving self-reinforcing optimization.

[0140] In summary, this invention can effectively solve the problems of insufficient sentiment supervision granularity, separation of understanding and generation tasks, large generation condition noise, and lack of reliable screening of generation results in existing image sentiment understanding and sentiment image generation technologies. It has the advantages of improving the interpretability of sentiment understanding, enhancing the consistency of sentiment generation, improving the quality of generated images, and enhancing the system's collaborative capabilities.

[0141] This invention achieves collaborative modeling of image sentiment understanding and sentiment image generation through structured sentiment data construction, sentiment understanding agent training, sentiment generation agent training, visual evidence-guided prompting planning, and multi-criteria generation result verification. Compared with existing methods that process sentiment understanding and sentiment generation separately, this invention can make fuller use of visual evidence, improve the interpretability of sentiment analysis, the sentiment consistency of generated images, and the overall reliability of the system.

[0142] Furthermore, such as Figure 5 As shown, based on the above-mentioned image emotion understanding and emotion generation method based on multi-agent collaboration, the present invention also provides an image emotion understanding and emotion generation system based on multi-agent collaboration, wherein the image emotion understanding and emotion generation system based on multi-agent collaboration includes: The emotion understanding agent training module 51 is used to acquire image samples and original emotion labels, perform structured emotion analysis on the image samples and original emotion labels through a multimodal large model to obtain structured emotion analysis results, and train the emotion understanding agent based on the structured emotion analysis results to obtain the target emotion understanding agent; The emotion generation agent training module 52 is used to perform global semantic emotion extraction on the image samples through the target emotion understanding agent to obtain a global emotion semantic representation, and to train the emotion generation agent based on the image samples and the global emotion semantic representation to obtain the target emotion generation agent; The candidate image generation module 53 is used to acquire the image to be processed, input the image to be processed into the target emotion understanding agent to obtain the structured analysis result, and input the structured analysis result into the target emotion generation agent to obtain multiple candidate images; The target generated image output module 54 is used to perform multi-criteria verification processing on multiple candidate images and output the target generated image.

[0143] Furthermore, such as Figure 6 As shown, based on the above-mentioned image emotion understanding and emotion generation method and system based on multi-agent collaboration, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 6 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0144] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, SmartMediaCard (SMC), SecureDigital (SD) card, or FlashCard. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a multi-agent collaborative image emotion understanding and generation program 40, which can be executed by the processor 10 to implement the multi-agent collaborative image emotion understanding and generation method of this application.

[0145] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the image emotion understanding and emotion generation method based on multi-agent collaboration.

[0146] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display 30 is used to display information on the terminal and to display a visual user interface.

[0147] In one embodiment, when the processor 10 executes the image emotion understanding and emotion generation program 40 based on multi-agent collaboration in the memory 20, it implements the steps of the image emotion understanding and emotion generation method based on multi-agent collaboration as described above.

[0148] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an image emotion understanding and emotion generation program based on multi-agent collaboration, and when the image emotion understanding and emotion generation program based on multi-agent collaboration is executed by a processor, it implements the steps of the image emotion understanding and emotion generation method based on multi-agent collaboration as described above.

[0149] In summary, this invention provides a method, system, terminal, and storage medium for image sentiment understanding and generation based on multi-agent collaboration. The method includes: acquiring image samples and original sentiment tags; performing structured sentiment analysis on the image samples and original sentiment tags using a multimodal large model to obtain structured sentiment analysis results; training a sentiment understanding agent based on the structured sentiment analysis results to obtain a target sentiment understanding agent; performing global semantic sentiment extraction on the image samples using the target sentiment understanding agent to obtain a global sentiment semantic representation; training a sentiment generation agent based on the image samples and the global sentiment semantic representation to obtain a target sentiment generation agent; acquiring an image to be processed and inputting the image to be processed into the target sentiment understanding agent to obtain structured analysis results; inputting the structured analysis results into the target sentiment generation agent to obtain multiple candidate images; performing multi-criteria verification processing on the multiple candidate images and outputting a target generated image. This invention achieves collaborative modeling of image emotion understanding and generation through structured emotion data construction, emotion understanding agent training, emotion generation agent training, visual evidence guidance and prompting planning, and multi-criteria generation result verification, thereby effectively improving the accuracy of emotion image generation.

[0150] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0151] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0152] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for image emotion understanding and emotion generation based on multi-agent collaboration, characterized in that, The image sentiment understanding and sentiment generation method based on multi-agent collaboration includes: Image samples and original sentiment labels are acquired. Structured sentiment analysis is performed on the image samples and original sentiment labels using a multimodal large model to obtain structured sentiment analysis results. The sentiment understanding agent is then trained based on the structured sentiment analysis results to obtain the target sentiment understanding agent. The target emotion understanding agent performs global semantic emotion extraction on the image samples to obtain a global emotion semantic representation. The emotion generation agent is then trained based on the image samples and the global emotion semantic representation to obtain the target emotion generation agent. The image to be processed is acquired and input into the target emotion understanding agent to obtain the structured analysis result. The structured analysis result is then input into the target emotion generation agent to obtain multiple candidate images. Multiple candidate images are subjected to multi-criteria verification processing to output the target generated image.

2. The image emotion understanding and emotion generation method based on multi-agent collaboration according to claim 1, characterized in that, The process of acquiring image samples and original sentiment tags, performing structured sentiment analysis on the image samples and original sentiment tags using a multimodal large model to obtain structured sentiment analysis results, and training the sentiment understanding agent based on the structured sentiment analysis results to obtain the target sentiment understanding agent specifically includes: Image samples and original sentiment labels are acquired, and the image samples and original sentiment labels are input into a multimodal large model. The multimodal large model is used to perform image sample element recognition on the image samples to obtain the sample element recognition result. Multi-layered sentiment evidence is generated based on the sample element identification results, and the support of the original sentiment labels is judged and data preprocessed based on the multi-layered sentiment evidence to obtain structured sentiment analysis results. Obtain the sentiment analysis instruction, and serialize the sentiment analysis instruction, the image sample, and the structured sentiment analysis result to obtain the target token sequence; A sentiment understanding agent is determined, and an initial sentiment understanding agent is obtained by training the sentiment understanding agent with autoregressive loss based on the target token sequence. The initial sentiment understanding agent is fine-tuned using a low-rank adaptation method to obtain the target sentiment understanding agent.

3. The image emotion understanding and emotion generation method based on multi-agent collaboration according to claim 1, characterized in that, The process of extracting global semantic sentiment from the image samples using the target sentiment understanding agent to obtain a global sentiment semantic representation, and then training the sentiment generation agent based on the image samples and the global sentiment semantic representation to obtain the target sentiment generation agent, specifically includes: The image samples are input into the visual encoder in the target emotion understanding agent, and the image samples are encoded by the visual encoder to obtain a visual token sequence; The global tokens in the visual token sequence are mapped using a projection semantic network to obtain a global sentiment semantic representation; The image samples are encoded into the latent space by a pre-trained variational autoencoder to obtain the image latent representation. The global sentiment semantic representation and the image latent representation are then fused to obtain a joint latent representation. A target emotion-generating agent is obtained by determining the emotion-generating agent and training it based on the joint latent representation.

4. The image emotion understanding and emotion generation method based on multi-agent collaboration according to claim 1, characterized in that, The process involves extracting global semantic sentiment from the image samples using the target sentiment understanding agent to obtain a global sentiment semantic representation, and then training the sentiment generation agent based on the image samples and the global sentiment semantic representation to obtain a target sentiment generation agent. The process further includes: The target emotion understanding agent outputs multiple types of first information, and a text encoder is used to encode the multiple types of first information to obtain multiple types of emotion information. The first information includes background information, facial expression information, and logical information; The various types of emotional information are mapped to the condition space of the generative model through multiple preset independent projection networks to obtain background condition tokens, expression condition tokens, and logical condition tokens. The background condition token, the expression condition token, and the logical condition token are concatenated to obtain structured emotion conditions. The structured emotional conditions are injected into the target emotion-generating agent to obtain the final target emotion-generating agent.

5. The image emotion understanding and emotion generation method based on multi-agent collaboration according to claim 4, characterized in that, The process involves acquiring an image to be processed, inputting the image to be processed into the target emotion understanding agent to obtain a structured analysis result, and then inputting the structured analysis result into the target emotion generation agent to obtain multiple candidate images, specifically including: The image to be processed is acquired and input into the target emotion understanding agent to obtain the structured analysis result; The structured analysis results include sentiment tags, background descriptions, facial or action cues, inference text, and metadata. The structured analysis results are converted into generated prompts, and the generated prompts are decomposed to obtain the target sentiment and structured sentiment conditions. The target emotion and the structured emotion conditions are input into the final target emotion generation agent to obtain multiple candidate images.

6. The image emotion understanding and emotion generation method based on multi-agent collaboration according to claim 5, characterized in that, The step of performing multi-criteria verification processing on multiple candidate images and outputting the target generated image specifically includes: Determine the current image library, and calculate a comprehensive score for each candidate image based on the current image library, the target emotion, and the generated prompts; The comprehensive score is the sum of the emotional semantic clarity score, the cue-image consistency score, the visual quality proxy score, the perceptual diversity score, and the semantic diversity score. A preset threshold is determined, and it is determined whether the comprehensive score of each candidate image is greater than the preset threshold; If so, candidate images that are greater than the preset threshold will be output to obtain the target generated image; If not, candidate images that are less than or equal to the preset threshold will be discarded or regenerated.

7. The image emotion understanding and emotion generation method based on multi-agent collaboration according to claim 1, characterized in that, The process of performing multi-criteria verification on multiple candidate images and outputting the target generated image further includes: Obtain the target sentiment label, target generation prompt, and target structured analysis result corresponding to the target generated image, and add the target generated image, the target sentiment label, the target generation prompt, and the target structured analysis result to the training set to obtain an updated training set; The target emotion understanding agent and the target emotion generation agent are iteratively enhanced based on the updated training set.

8. A system for image emotion understanding and emotion generation based on multi-agent collaboration, characterized in that, The image emotion understanding and emotion generation system based on multi-agent collaboration includes: The emotion understanding agent training module is used to acquire image samples and original emotion labels, perform structured emotion analysis on the image samples and original emotion labels through a multimodal large model to obtain structured emotion analysis results, and train the emotion understanding agent based on the structured emotion analysis results to obtain the target emotion understanding agent; The emotion generation agent training module is used to perform global semantic emotion extraction on the image samples through the target emotion understanding agent to obtain a global emotion semantic representation, and to train the emotion generation agent based on the image samples and the global emotion semantic representation to obtain the target emotion generation agent; A candidate image generation module is used to acquire an image to be processed, input the image to be processed into the target emotion understanding agent to obtain a structured analysis result, and input the structured analysis result into the target emotion generation agent to obtain multiple candidate images; The target generated image output module is used to perform multi-criteria verification processing on multiple candidate images and output the target generated image.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and a multi-agent collaborative image emotion understanding and emotion generation program stored in the memory and executable on the processor. When the multi-agent collaborative image emotion understanding and emotion generation program is executed by the processor, it implements the steps of the multi-agent collaborative image emotion understanding and emotion generation method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an image emotion understanding and emotion generation program based on multi-agent collaboration, which, when executed by a processor, implements the steps of the image emotion understanding and emotion generation method based on multi-agent collaboration as described in any one of claims 1-7.

Images