Multi-agent framework for foreground condition image generation

Abstract

The invention provides a multi-agent framework for foreground condition image generation, and particularly relates to the technical field of artificial intelligence, and the multi-agent framework comprises a prompt generation module which is used for carrying out semantic analysis on a foreground image to obtain a foreground description text, and generating a background prompt text according to the foreground description text; the image generation module is used for generating an initial template image according to the background prompt text and the edge image of the foreground image, and performing image optimization on the initial template image to obtain a result image; and the quality evaluation module is used for carrying out image quality evaluation on the result image to obtain an evaluation text, and outputting the result image under the condition that the evaluation text is matched with an image output condition. The multi-agent framework provided by the invention can meet the requirements of low use difficulty and high adaptation degree between an output image result and an image condition.

Description

Technical Field

[0001] The present disclosure relates to the technical field of artificial intelligence, and in particular to a multi-agent framework for foreground conditional image generation. Background Art

[0002] In recent years, significant progress has been made in image-conditioning-based image generation techniques, especially in diffusion models.

[0003] However, it was found in the application that the image generation solution based on image conditions provided by the relevant technology can only obtain image results that are highly adapted to the image conditions when the user provides detailed input control information or accurate segmentation mask, which greatly increases the difficulty of use for users.

[0004] In other words, how to facilitate user use while ensuring a high degree of adaptation between image results and image conditions is an urgent problem to be solved in related technologies. Summary of the invention

[0005] The purpose of the present disclosure is to provide a multi-agent framework for foreground conditional image generation, which is used to solve the technical problem that the image generation solution based on image conditions provided by the related technology cannot take into account both low usage difficulty and high adaptability between image results and image conditions.

[0006] In a first aspect, an embodiment of the present disclosure provides a multi-agent framework for foreground condition image generation, the multi-agent framework comprising:

[0007] A prompt generation module, used for performing semantic analysis on the foreground image to obtain a foreground description text, and generating a background prompt text according to the foreground description text;

[0008] An image generation module, used to generate an initial template image according to the background prompt text and the edge map of the foreground image, and perform image optimization on the initial template image to obtain a result image;

[0009] The quality assessment module is used to perform image quality assessment on the result image to obtain an assessment text, and output the result image when the assessment text matches the image output condition.

[0010] In one embodiment, generating background prompt text according to the foreground description text includes:

[0011] Using preset agent role information and preset output template information as prompt words, the question-answering agent performs question-answering processing on the foreground description text to obtain a plurality of candidate prompt texts, wherein the agent role information is used to indicate a creative role, and the output template information matches the input template of the image generation module;

[0012] The background prompt text is determined from the plurality of candidate prompt texts.

[0013] In one embodiment, determining the background prompt text from the plurality of candidate prompt texts includes:

[0014] Using a large language model to evaluate each of the candidate prompt texts, a plurality of text evaluation scores corresponding to the plurality of candidate prompt texts are obtained, wherein the text evaluation score is determined according to the semantic relevance between the corresponding candidate prompt text and the foreground description text, and the compatibility between the corresponding candidate prompt text and the image generation module, wherein the semantic relevance is positively correlated with the text evaluation score, and the compatibility is positively correlated with the text evaluation score;

[0015] Among the multiple candidate prompt texts, the candidate prompt text corresponding to the highest text evaluation score is determined as the background prompt text.

[0016] In one embodiment, the step of performing image optimization on the initial template image to obtain a result image includes:

[0017] Extracting a first foreground mask from the initial template image, and acquiring a second foreground mask corresponding to the foreground image;

[0018] determining a redundant mask region in the first foreground mask based on the second foreground mask, wherein the second foreground mask does not include the redundant mask region;

[0019] When the area of ​​the redundant mask region is greater than or equal to the set area threshold, the redundant mask region in the initial template image is repaired to obtain a result image.

[0020] In one embodiment, repairing the redundant mask area in the initial template image to obtain a result image includes:

[0021] Deleting the redundant mask area in the initial template image to obtain a deleted template image;

[0022] The deletion template image is optimized to obtain the result image, wherein the image resolution of the result image is greater than or equal to the image resolution of the deletion template image.

[0023] In one embodiment, the image optimization includes at least one of image detail enhancement, image color balance, and image transition optimization.

[0024] In one embodiment, the evaluation text is used to indicate the integration degree between the foreground part and the background part in the result image, and the integration degree is used to indicate at least one of: image correlation between the foreground part and the background part in the result image, semantic correlation between the foreground part and the background part in the result image, and spatial correlation between the foreground part and the background part in the result image;

[0025] In a case where the integration degree indicated by the evaluation text is greater than or equal to a set integration threshold, it is determined that the evaluation text matches the image output condition.

[0026] In one embodiment, the quality assessment module is further configured to:

[0027] When N is less than or equal to the set loop count threshold, and the Nth evaluation text does not match the image output condition, output the evaluation text to the prompt generation module, so that the prompt generation module generates the N+1th background prompt text according to the foreground description text and the evaluation text, and the image generation module generates the N+1th initial template image according to the N+1th background prompt text and the edge map of the foreground image, and performs image optimization on the N+1th initial template image to obtain the N+1th result image;

[0028] Perform image quality assessment on the N+1th result image to obtain the N+1th evaluation text, and output the N+1th result image when the N+1th evaluation text matches the image output condition, wherein N is an integer less than or equal to a set loop count threshold, and the loop count threshold is greater than 1.

[0029] In one embodiment, the quality assessment module is further configured to:

[0030] When N is greater than the loop number threshold, the Nth result image is output.

[0031] In one embodiment, generating background prompt text according to the foreground description text includes:

[0032] Get the user's request text;

[0033] The background prompt text is generated according to the foreground description text and the user requirement text.

[0034] In the present disclosure, a foreground description text is obtained by performing semantic analysis on a foreground image, and a background prompt text is generated based on the foreground description text. Then, an initial template image is generated in combination with an edge map of the foreground image to optimize a result image. When the image quality of the result image is evaluated and the evaluation result indicates that the result image matches the image output condition, the result image is output to ensure adaptive matching of the final output result image with the foreground image by combining the text prompt and the image features of the foreground image. In this process, the user only needs to provide the foreground image as input, which greatly reduces the difficulty of operation for the user. In this process, different modules such as a prompt generation module, an image generation module, and a quality assessment module are used to handle different tasks respectively, which can improve the overall modularity and scalability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a structural schematic diagram of a multi-agent framework for foreground condition image generation provided by an embodiment of the present disclosure;

[0036] Figure 2 is a schematic diagram of a foreground image provided by an embodiment of the present disclosure;

[0037] Figure 3 is a schematic diagram of an edge map of a foreground image provided by an embodiment of the present disclosure;

[0038] Figure 4 is a schematic diagram of an initial background template provided by an embodiment of the present disclosure;

[0039] Figure 5 is a schematic diagram of a foreground mask of an initial background template provided by an embodiment of the present disclosure;

[0040] Figure 6 is a schematic diagram of an actual foreground mask of a foreground image provided by an embodiment of the present disclosure;

[0041] Figure 7 is a schematic diagram of a repaired initial background template provided by an embodiment of the present disclosure;

[0042] Figure 8 This is a multi-agent framework provided by the embodiment of the present disclosure for Figure 2 Schematic diagram of the output results. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0044] The present disclosure provides a multi-agent framework for foreground condition image generation. Figure 1 As shown, the multi-agent framework 100 includes:

[0045] The prompt generating module 101 is used for performing semantic analysis on the foreground image to obtain a foreground description text, and generating a background prompt text according to the foreground description text;

[0046] An image generation module 102 is used to generate an initial template image according to the background prompt text and the edge map of the foreground image, and perform image optimization on the initial template image to obtain a result image;

[0047] The quality assessment module 103 is used to perform image quality assessment on the result image to obtain an assessment text, and output the result image when the assessment text matches an image output condition.

[0048] Among them, the foreground image can be understood as an image of the object subject that the user expects to be presented in the output result. For example, when the user expects to obtain a tourist photo of himself in a specific scenic spot, the foreground image can be understood as a user image including the half-body / full-body of the user; and when the user expects to obtain a placement effect diagram of a certain product in an indoor environment, the foreground image can be understood as a product image of the product.

[0049] The foreground description text is used to represent the image attributes of the foreground image, such as shape, color, posture and other attributes. Furthermore, the foreground description text can also represent the semantic information of the foreground image, such as the type, name, material, application scenario, applicable population and other information of the object indicated by the foreground image.

[0050] Exemplarily, a visual language model (such as a Gemini-Pro-Vision model) may be used to perform semantic analysis on the foreground image to obtain the foreground description text.

[0051] The background prompt text may be understood as text used to guide the generation of the background portion in the initial template image.

[0052] In one example, the foreground description text may be used as question text, and a large model may be used for question answering to obtain the background prompt text.

[0053] The edge map of the foreground image may be understood as an image obtained by edge detection of the foreground image, and the edge map of the foreground image includes detail lines indicating the contour, spatial structure, orientation layout and other contents of the object in the foreground image.

[0054] In one example, the edge map of the foreground image and the background prompt text may be used as image requirement information, and a large model may be used to perform image rendering to obtain the initial template image.

[0055] In some implementations, the initial template image may be used as an image input, and a large model may be used to perform image optimization to obtain the result image.

[0056] In other implementations, the initial template image may also be subjected to denoising, pixel filling, image smoothing and other processing to obtain the result image.

[0057] Among them, the process of evaluating the image quality of the result image can be understood as: the process of evaluating the visual aesthetics of the result image, for example, detecting whether there is unreasonable color matching in the result image, whether there is an abnormal area that is significantly different from other image parts, etc.

[0058] In the present disclosure, a foreground description text is obtained by performing semantic analysis on a foreground image, and a background prompt text is generated based on the foreground description text. Then, an initial template image is generated in combination with an edge map of the foreground image to optimize a result image. When the image quality of the result image is evaluated and the evaluation result indicates that the result image matches the image output condition, the result image is output to ensure adaptive matching of the final output result image with the foreground image by combining the text prompt and the image features of the foreground image. In this process, the user only needs to provide the foreground image as input, which greatly reduces the difficulty of operation for the user. In this process, different modules such as a prompt generation module, an image generation module, and a quality assessment module are used to handle different tasks respectively, which can improve the overall modularity and scalability of the system.

[0059] It should be noted that the multi-agent framework for foreground conditional image generation disclosed in the present invention can be applied in multiple fields, such as e-commerce, content creation, game development, advertising and marketing, and augmented reality, for example: in a virtual try-on scenario, it can accurately display the real-life renderings of the product after wearing it to the user, so as to enhance the user experience; in a content creation scenario, it can provide creators with creative images based on the design core they input, and show the creators diverse creative ideas, thereby accelerating the content production of the creators; in a game development scenario, it can assist developers in quickly generating specific game scenes and character backgrounds to improve game development efficiency; in an advertising and marketing scenario, it can provide creative and personalized advertising images to enhance the output rate and influence of advertising images; in an augmented reality scenario, it can provide AR applications with background images that are more suitable for natural scenes and contexts.

[0060] In one embodiment, generating background prompt text according to the foreground description text includes:

[0061] Using preset agent role information and preset output template information as prompt words, the question-answering agent performs question-answering processing on the foreground description text to obtain a plurality of candidate prompt texts, wherein the agent role information is used to indicate a creative role, and the output template information matches the input template of the image generation module;

[0062] The background prompt text is determined from the plurality of candidate prompt texts.

[0063] In this embodiment, the agent role information indicating the creative role is used as the prompt word. By asking and answering questions with the question-answering agent, multiple richer and more imaginative candidate prompt texts can be quickly obtained based on the foreground description text, that is, the data richness of the obtained multiple candidate prompt texts can be improved. In this process, the output template information is also set as the prompt word to ensure that the data format of the question-answering result output by the question-answering agent matches the input template of the image generation module, so that the background prompt text determined among the multiple candidate prompt texts can be directly sent to the image generation module as input for processing.

[0064] The matching of the output template information with the input template of the image generation module can be understood as that the data structure indicated by the output template information is the same as the data structure indicated by the input template of the image generation module.

[0065] Exemplarily, the creative role may be: an artist, a creator, a painter, a designer, etc.

[0066] In the application, a candidate prompt text may be randomly selected from the multiple candidate prompt texts as the background prompt text, or each candidate prompt text may be evaluated based on a preset evaluation rule and the candidate prompt text with the highest evaluation may be determined as the background prompt text.

[0067] In one embodiment, determining the background prompt text from the plurality of candidate prompt texts includes:

[0068] Using a large language model to evaluate each of the candidate prompt texts, a plurality of text evaluation scores corresponding to the plurality of candidate prompt texts are obtained, wherein the text evaluation score is determined according to the semantic relevance between the corresponding candidate prompt text and the foreground description text, and the compatibility between the corresponding candidate prompt text and the image generation module, wherein the semantic relevance is positively correlated with the text evaluation score, and the compatibility is positively correlated with the text evaluation score;

[0069] Among the multiple candidate prompt texts, the candidate prompt text corresponding to the highest text evaluation score is determined as the background prompt text.

[0070] In this embodiment, the text evaluation score of each candidate prompt text is comprehensively determined by the semantic relevance between the candidate prompt text and the foreground description text, as well as the compatibility between the candidate prompt text and the image generation module, and the candidate prompt text corresponding to the highest text evaluation score is determined as the background prompt text. This not only ensures that a high semantic relevance is maintained between the output background prompt text and the foreground description text, thereby avoiding significant differences between the background part and the foreground image in the initial template image generated based on the background prompt text, but also ensures that the image generation module can accurately identify the text content of the background prompt text, thereby generating an initial template image that is strongly associated with the background prompt text.

[0071] Among them, the compatibility between the candidate prompt text and the image generation module can be understood as: the degree of correlation between the image generated by the image generation module after taking the candidate prompt text as the module input and the candidate prompt text. The higher the correlation, the higher the compatibility between the candidate prompt text and the image generation module, which means that the image generation module can more accurately process the text content of the corresponding candidate prompt text, and vice versa.

[0072] In one example, after obtaining multiple text evaluation scores corresponding to the multiple candidate prompt texts one by one, the multiple candidate prompt texts can also be sorted according to the multiple text evaluation scores, and the candidate prompt text with a higher corresponding text evaluation score has a higher order, and then a candidate prompt text is selected from the sorted multiple candidate prompt texts according to the set rules and used as the background prompt text, wherein the set rules can be: the higher the order of the candidate prompt text, the higher the probability of the corresponding candidate prompt text being selected.

[0073] In one embodiment, the step of performing image optimization on the initial template image to obtain a result image includes:

[0074] Extracting a first foreground mask from the initial template image, and acquiring a second foreground mask corresponding to the foreground image;

[0075] determining a redundant mask region in the first foreground mask based on the second foreground mask, wherein the second foreground mask does not include the redundant mask region;

[0076] When the area of ​​the redundant mask region is greater than or equal to the set area threshold, the redundant mask region in the initial template image is repaired to obtain a result image.

[0077] In this embodiment, the second foreground mask corresponding to the foreground image is obtained, that is, the actual foreground mask is obtained, and based on this, the first foreground mask in the generated initial template image is subjected to redundancy detection, thereby determining the redundant mask area included in the first foreground mask but not included in the second foreground mask, and when the redundant mask area is greater than or equal to a set area threshold, the redundant mask area is repaired, so that the foreground part in the resulting image is as consistent as possible with the foreground image.

[0078] In an application, the first foreground mask in the initial template image may be extracted by image segmentation, by edge detection, or by other mask extraction methods.

[0079] Exemplarily, the area of ​​the redundant mask region may be understood as the pixel area corresponding to the redundant mask region in the initial template image, or may be understood as the pixel area ratio of the redundant mask region in the initial template image.

[0080] In one embodiment, repairing the redundant mask area in the initial template image to obtain a result image includes:

[0081] Deleting the redundant mask area in the initial template image to obtain a deleted template image;

[0082] The deletion template image is optimized to obtain the result image, wherein the image resolution of the result image is greater than or equal to the image resolution of the deletion template image.

[0083] In this embodiment, when the area of ​​the redundant mask region is greater than or equal to the set area threshold, the redundant mask region is deleted so that the foreground part and the foreground image in the obtained result image can be made as consistent as possible. In this process, the template image is optimized after deleting the redundant mask region in order to ensure that the filling content of the blank area left after deleting the redundant mask region is consistent with the undeleted image content, so that the image contents in the result image maintain a high degree of consistency.

[0084] It should be noted that, when the area of ​​the redundant mask region is smaller than the set area threshold, the initial template image is determined as the result image.

[0085] It should be understood that the repair process of a redundant mask area that is too small may lead to the risk of over-repair, which will consume a certain amount of processing time and increase the risk of abnormalities in the resulting image, thereby increasing the difficulty of subsequent image optimization. The setting of the above-mentioned area threshold can avoid the risk of over-repair caused by a redundant mask area that is too small.

[0086] Furthermore, the image optimization includes at least one of image detail enhancement, image color balance, and image transition optimization.

[0087] Among them, image transition optimization can be understood as a smoothing filter process on the deleted template image.

[0088] In the application, image optimization may also include filling pixels of the deleted template image to obtain a result image with higher pixels.

[0089] In one embodiment, the evaluation text is used to indicate the integration degree between the foreground part and the background part in the result image, and the integration degree is used to indicate at least one of: image correlation between the foreground part and the background part in the result image, semantic correlation between the foreground part and the background part in the result image, and spatial correlation between the foreground part and the background part in the result image;

[0090] In a case where the integration degree indicated by the evaluation text is greater than or equal to a set integration threshold, it is determined that the evaluation text matches the image output condition.

[0091] In this embodiment, the evaluation text is formed by analyzing one or more of the image correlation between the foreground part and the background part in the result image, the semantic correlation between the foreground part and the background part in the result image, and the spatial correlation between the foreground part and the background part in the result image. This can flexibly meet different preferences for image evaluation in different scenarios, making the application of the multi-agent framework described in the present application more flexible.

[0092] Among them, when the integration degree is used to indicate: the image correlation between the foreground part and the background part in the result image, the semantic correlation between the foreground part and the background part in the result image, and the spatial correlation between the foreground part and the background part in the result image, a more accurate and reliable evaluation text can be obtained.

[0093] The image correlation may be used to indicate whether the lighting between the foreground part and the background part in the result image is consistent, whether the color matching is harmonious, etc.

[0094] The semantic relevance may be used to indicate whether the semantics between the foreground and background parts in the result image match. For example, it is unreasonable for indoor office equipment to appear against an outdoor background, that is, there is a conflict between the semantics of the foreground and background.

[0095] The spatial correlation may be used to indicate whether the structures between the foreground part and the background part in the result image are coherent, whether the spatial relationship is harmonious, etc.

[0096] In one example, different network model analyses can be used to obtain a first analysis text corresponding to the image relevance, a second analysis text corresponding to the semantic relevance, and a third analysis text corresponding to the spatial relevance, and the first analysis text, the second analysis text, and the third analysis text are fused to form the evaluation text.

[0097] In another example, the quality of the result image may also be evaluated by a quality evaluation agent having an image evaluation function to obtain the evaluation text.

[0098] In the application, the evaluation text may be set to include an integration index indicating the degree of integration. In this case, when the value of the integration index is greater than or equal to a set integration threshold, it is determined that the evaluation text matches the image output condition.

[0099] It may also be arranged that semantic analysis is performed on the evaluation text to obtain an integration index indicating the degree of integration, and when the value of the integration index is greater than or equal to a set integration threshold, it is determined that the evaluation text matches the image output condition.

[0100] In one embodiment, the quality assessment module is further configured to:

[0101] When N is less than or equal to the set loop count threshold, and the Nth evaluation text does not match the image output condition, output the evaluation text to the prompt generation module, so that the prompt generation module generates the N+1th background prompt text according to the foreground description text and the evaluation text, and the image generation module generates the N+1th initial template image according to the N+1th background prompt text and the edge map of the foreground image, and performs image optimization on the N+1th initial template image to obtain the N+1th result image;

[0102] Perform image quality assessment on the N+1th result image to obtain the N+1th evaluation text, and output the N+1th result image when the N+1th evaluation text matches the image output condition, wherein N is an integer less than or equal to a set loop count threshold, and the loop count threshold is greater than 1.

[0103] In this embodiment, in the case where the degree of integration indicated by the evaluation text does not match the image output conditions, by rolling back the process to the process of generating the background prompt text and inputting the evaluation text as supplementary text, the image defects existing in the result image generated previously can be avoided to a certain extent, so that the degree of integration of the result image generated next time is higher, thereby increasing the probability that the result image finally output to the user meets the user's needs.

[0104] In one embodiment, the quality assessment module is further configured to:

[0105] When N is greater than the loop number threshold, the Nth result image is output.

[0106] In this embodiment, a loop count threshold is set to avoid a situation where the multi-agent framework repeatedly loops and occupies excessive computing resources and crashes due to erroneous input or abnormal image processing logic. In the application, the loop count threshold can be set to 2 or 3.

[0107] In one embodiment, generating background prompt text according to the foreground description text includes:

[0108] Get the user's request text;

[0109] The background prompt text is generated according to the foreground description text and the user requirement text.

[0110] The user requirement text may be understood as a requirement text proposed by the user for the target template text, for example: outputting a result image corresponding to a sunset, outputting a result image corresponding to an office scene, and the like.

[0111] In this embodiment, the user requirement text is introduced to increase the user's autonomy in the result image generation process, making the generated background prompt text more flexible and making it more likely that the final output result image meets the user's expectations.

[0112] For ease of understanding, the following examples are provided:

[0113] The multi-agent framework set up for foreground conditional image generation includes: a prompt generation module, an image generation module and a quality assessor.

[0114] The prompt generation module mainly includes a prompt foreground analyzer, a prompt creator, and a prompt selector.

[0115] The foreground analyzer uses the Gemini-Pro-Vision visual language model to extract foreground images (such as Figure 2 The detailed text information of the foreground image (as shown) (which can be understood as the foreground description text mentioned above) is processed as follows: a foreground image input by a user is received, and image analysis is performed on the foreground image to extract the object type, shape, color, posture, material and other attributes in the foreground image, and the text information is output in a structured JSON format.

[0116] The above-mentioned prompt creator uses a large language model (such as Gemini-Pro) to generate diverse background text descriptions (which can be understood as the aforementioned candidate prompt texts) based on the output text of the foreground analyzer. Its processing flow is: receiving the output text of the foreground analyzer, and generating k candidate text prompts (which can be understood as the aforementioned multiple candidate prompt texts) according to the user-specified / pre-set role identity and output template requirements.

[0117] Among them, the role identity can enable the prompt creator to act as a creative engine to explore imaginative composition text content, while the output template requirement can make the text structure of the output data of the prompt creator conform to the input structure requirements of the image generation module.

[0118] In the application, the prompt creator can also receive text input carrying user requirements (which can be understood as the aforementioned user requirement text), and generate k candidate text prompts based on the text input carrying user requirements and the output text of the foreground analyzer, where k is an integer greater than 1.

[0119] The above-mentioned prompt selector uses a large language model (such as Gemini-Pro) to evaluate k candidate text prompts and select the best prompt from them. Its processing flow is: evaluate each candidate text prompt, consider its relevance to foreground details and compatibility with the image generation model, obtain an evaluation score for each candidate text prompt, and sort the k candidate text prompts according to the evaluation score of each candidate text prompt, wherein the higher the evaluation score, the higher the ranking of the corresponding candidate text prompt, and finally output the candidate text prompt with the highest ranking as the best prompt (which can be understood as the aforementioned background prompt text).

[0120] The image generation module mainly includes a prompt template generator (Template Generator), a template repainter (Template Repainter) and an image enhancer (Image Enhancer).

[0121] The above prompt template generator uses the ControlNet SDXL Canny model to create an initial background template (which can be understood as the aforementioned initial template image). Its processing flow is as follows: receiving the best prompt output by the prompt selector and the edge map of the foreground image (such as Figure 3 ), and generates and optimal hint and edge Figure 1 The same initial background template (such as Figure 4 shown).

[0122] The template redrawer uses image segmentation tools (RMBG-1.4 or Grounded-SAM), automatic detection tools (based on Grounding DINO) and image repair tools (such as LaMa) to detect and repair the integrity of the object indicated by the foreground image in the initial background template. The processing flow is as follows: the foreground mask of the initial background template is estimated by the image segmentation tool (such as Figure 5 ), the foreground mask of the initial background template is compared with the actual foreground mask of the foreground image by an automatic detection tool (as shown in Figure 6 The difference between the two images is determined by the image repair tool, and the non-overlapping area between the two images is identified (which can be understood as the redundant mask area mentioned above). When the non-overlapping area exceeds the threshold, the non-overlapping area is selectively repaired through the image repair tool, and the modified template is output (as shown in FIG. Figure 7 As shown), when the non-overlapping area does not exceed the threshold, the initial background template is directly output.

[0123] The above-mentioned image enhancer uses a refinement model (such as SDXL Refiner) for improving image resolution to improve the overall image quality of the output result of the template redrawer. Its processing flow is as follows: receiving the output result of the template redrawer, performing detail enhancement, color balance and transition optimization on the result, and outputting the processed image (which can be understood as the result image) (such as Figure 8 shown).

[0124] The above-mentioned quality assessor uses a visual language model (such as Gemini-Pro-Vision) to assess the quality of the image output by the image enhancer, and provides feedback text and triggers a regeneration mechanism when the feedback text meets the loop condition. The processing flow is as follows: comprehensively assess the visual relationship, content rationality and overall image quality of the image output by the image enhancer, specifically analyzing the foreground-background integration, focusing on lighting consistency, color harmony, structural coherence, spatial relationship and semantic relevance, and generating detailed text feedback (which can be understood as the aforementioned evaluation text). If the text feedback indicates that the image quality of the image output by the image enhancer is unqualified (which can be understood as the evaluation text not matching the image output condition), the text feedback is returned to the prompt creator as a supplementary input to cooperate with the output text of the foreground analyzer to trigger the regeneration mechanism, thereby generating a new image output by the image enhancer. In addition, in order to avoid an infinite loop, the number of loops is set to a maximum of three times.

[0125] If the text feedback indicates that the image quality of the image output by the image enhancer is qualified, the image output by the image enhancer is returned to the user as the final output result.

[0126] The above multi-agent framework effectively solves problems such as object integrity, foreground-background consistency, generation diversity and text consistency through the collaboration of multiple specialized agents, while simplifying the user interaction process and improving the ease of use and flexibility of the system.

[0127] The above-mentioned multi-agent framework setting can be applied to multimodal efficient and controllable visual generation applications to form a new generation of AI-driven image processing tools.

[0128] The terms "comprises," "comprising," or any other variations thereof of the disclosed embodiments are intended to encompass non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article, or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0129] The term "or" in the present disclosure means an inclusive "or" rather than an exclusive "or". A reference to a "first" component does not necessarily require the provision of a "second" component. Furthermore, unless explicitly indicated, a "first" or "second" component does not limit the referenced components to a particular order. The term "based on" means "based at least in part on".

[0130] The embodiments of the present disclosure are described above in conjunction with the accompanying drawings, but the present disclosure is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present disclosure, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present disclosure and the claims, all of which are within the protection of the present disclosure.

Claims

1. A multi-agent framework for foreground conditional image generation, characterized in that: The multi-agent framework includes: A prompt generation module, used for performing semantic analysis on the foreground image to obtain a foreground description text, and generating a background prompt text according to the foreground description text; An image generation module, used to generate an initial template image according to the background prompt text and the edge map of the foreground image, and perform image optimization on the initial template image to obtain a result image; The quality assessment module is used to perform image quality assessment on the result image to obtain an assessment text, and output the result image when the assessment text matches the image output condition.

2. The multi-agent framework according to claim 1, characterized in that: The step of generating background prompt text according to the foreground description text comprises: Using preset agent role information and preset output template information as prompt words, the question-answering agent performs question-answering processing on the foreground description text to obtain a plurality of candidate prompt texts, wherein the agent role information is used to indicate a creative role, and the output template information matches the input template of the image generation module; The background prompt text is determined from the plurality of candidate prompt texts.

3. The multi-agent framework according to claim 2, characterized in that: The determining the background prompt text from the plurality of candidate prompt texts comprises: Using a large language model to evaluate each of the candidate prompt texts, a plurality of text evaluation scores corresponding to the plurality of candidate prompt texts are obtained, wherein the text evaluation score is determined according to the semantic relevance between the corresponding candidate prompt text and the foreground description text, and the compatibility between the corresponding candidate prompt text and the image generation module, wherein the semantic relevance is positively correlated with the text evaluation score, and the compatibility is positively correlated with the text evaluation score; Among the multiple candidate prompt texts, the candidate prompt text corresponding to the highest text evaluation score is determined as the background prompt text.

4. The multi-agent framework according to claim 1, characterized in that: The step of performing image optimization on the initial template image to obtain a result image includes: Extracting a first foreground mask from the initial template image, and acquiring a second foreground mask corresponding to the foreground image; determining a redundant mask region in the first foreground mask based on the second foreground mask, wherein the second foreground mask does not include the redundant mask region; When the area of ​​the redundant mask region is greater than or equal to the set area threshold, the redundant mask region in the initial template image is repaired to obtain a result image.

5. The multi-agent framework according to claim 4, characterized in that: Repairing the redundant mask area in the initial template image to obtain a result image includes: Deleting the redundant mask area in the initial template image to obtain a deleted template image; The deletion template image is optimized to obtain the result image, wherein the image resolution of the result image is greater than or equal to the image resolution of the deletion template image.

6. The multi-agent framework according to claim 5, characterized in that: The image optimization includes at least one of image detail enhancement, image color balance, and image transition optimization.

7. The multi-agent framework according to claim 1, characterized in that: The evaluation text is used to indicate the integration degree between the foreground part and the background part in the result image, and the integration degree is used to indicate at least one of: the image correlation between the foreground part and the background part in the result image, the semantic correlation between the foreground part and the background part in the result image, and the spatial correlation between the foreground part and the background part in the result image; In a case where the integration degree indicated by the evaluation text is greater than or equal to a set integration threshold, it is determined that the evaluation text matches the image output condition.

8. The multi-agent framework according to claim 1, characterized in that: The quality assessment module is also used to: When N is less than or equal to the set loop count threshold, and the Nth evaluation text does not match the image output condition, output the evaluation text to the prompt generation module, so that the prompt generation module generates the N+1th background prompt text according to the foreground description text and the evaluation text, and the image generation module generates the N+1th initial template image according to the N+1th background prompt text and the edge map of the foreground image, and performs image optimization on the N+1th initial template image to obtain the N+1th result image; Perform image quality assessment on the N+1th result image to obtain the N+1th evaluation text, and output the N+1th result image when the N+1th evaluation text matches the image output condition, wherein N is an integer less than or equal to a set loop count threshold, and the loop count threshold is greater than 1.

9. The multi-agent framework according to claim 8, characterized in that: The quality assessment module is also used to: When N is greater than the loop number threshold, the Nth result image is output.

10. The multi-agent framework according to claim 1, characterized in that: The step of generating background prompt text according to the foreground description text comprises: Get the user's request text; The background prompt text is generated according to the foreground description text and the user requirement text.

Images