Emotion image editing method and system based on multi-agent cooperation framework

Abstract

The application provides an emotion image editing method and system based on a multi-agent cooperation framework, wherein the method comprises: in a pre-creation stage, an initial editing scheme is generated based on a pre-acquired source image and a target emotion by a planning agent, the initial editing scheme is adjusted according to an evaluation result by an evaluation agent, and then a preliminary editing operation is performed according to the adjusted initial editing scheme by an editing agent to obtain a preliminary edited image; in an optimization stage, whether the preliminary edited image effectively conveys the target emotion is evaluated by the evaluation agent, and an optimization editing instruction is generated according to the evaluation result of not effectively conveying the target emotion; and the preliminary edited image is iteratively updated according to the optimization editing instruction by the editing agent until the target emotion is effectively conveyed. The method can realize efficient and accurate emotion-oriented image editing, provide a more flexible image editing and emotion expression means, and is suitable for various image processing and creative generation tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image editing in the field of artificial intelligence technology, more specifically, to a sentiment image editing method and system based on a multi-agent collaboration framework. BACKGROUND

[0002] Image editing has always been an important means of image processing and aesthetic expression. Compared with the current semantic image editing method, sentiment image editing can understand and realize abstract emotional expression, aiming to adjust the visual elements in the image from the emotional dimension to evoke special emotional feelings. With the continuous development of deep learning technology, multi-modal technology and visual language model, sentiment image editing has become a more valuable research field because it can provide more accurate emotional expression, more consistent with human cognition and more artistic editing results.

[0003] However, the existing sentiment image editing method mainly focuses on low-level simple editing methods such as color and style, which cannot achieve full and accurate emotional expression. In the real world, images usually express specific emotions with multiple expression methods. In addition, some works convert emotions into simple codes or other features based on semantic image editing to guide image editing. However, due to the complexity of human emotional expression, this often cannot fully and accurately express the target emotion, and is easy to cause image structure distortion. In addition, methods such as EmoEdit, which convert sentiment image editing into a multi-step semantic image editing task, can enhance the target emotional expression by changing the visual elements in the image. However, EmoEdit cannot reasonably and accurately handle the complex interactions and connections between various semantic and emotional elements and various editing methods.

[0004] Therefore, how to ensure the rationality and effectiveness of the image editing result in the target emotional expression has become one of the research directions of the current sentiment image editing task. SUMMARY

[0005] In view of the problem that the current sentiment image editing method cannot achieve reasonable and effective target emotional expression, the purpose of the present application is to provide a sentiment image editing method and system based on a multi-agent collaboration framework, which will use multiple agents to simulate the cognitive behavior of human painters to be responsible for decision-making, editing, and evaluation three functions to complete the sentiment image editing task cooperatively, in order to enhance the explainability and reliability of the operation result.

[0006] The sentiment image editing method based on the multi-agent collaboration framework provided by the present application comprises a pre-creation stage and an optimization stage; wherein,

[0007] In the pre-creation stage, a planning agent generates a preliminary editing scheme based on the obtained source image and target emotion; according to the preliminary editing scheme, an evaluation agent evaluates the preliminary editing scheme, and an editing agent performs a preliminary editing operation in the preliminary editing scheme to obtain a preliminary edited image;

[0008] In the optimization stage, an evaluation agent evaluates the preliminary edited image, and if the preliminary edited image does not meet the target emotion, an optimization editing instruction is generated, and an editing agent adjusts the preliminary edited image according to the optimization editing instruction.

[0009] Optionally, the pre-creation stage includes:

[0010] S111: An emotion and semantic analysis agent analyzes the emotion and semantics of the source image to obtain emotion information and semantic information of the source image;

[0011] S112: Based on the target emotion, a pre-set emotion factor knowledge retriever retrieves emotion editing elements related to the target emotion from an emotion editing element database;

[0012] S113: According to the emotion editing elements, a preliminary editing scheme of the emotion image is generated, and the preliminary editing includes specific editing operations and adjustment sequences;

[0013] S114: An evaluation agent evaluates the preliminary editing scheme, and optimizes the preliminary editing scheme according to the evaluation result to form an optimization editing scheme;

[0014] S115: An editing agent performs an image editing task according to the optimization editing scheme to generate a preliminary edited image.

[0015] Optionally, the optimization stage includes:

[0016] S121: An evaluation agent evaluates the preliminary editing result to determine whether the preliminary editing result effectively conveys the target emotion; if the preliminary editing result does not achieve the expected effect, the evaluation agent provides an optimization editing instruction for the preliminary editing result;

[0017] S122: An editing agent iteratively updates the preliminary editing result according to the optimization editing instruction to obtain an optimization edited image.

[0018] Optionally, the planning agent determines the specific steps of each editing element and editing operation in the preliminary editing scheme through a three-layer decision tree structure; the three-layer decision tree takes the target emotion as the root node, identifies suitable editing elements as branch nodes, and assigns corresponding editing methods as leaf nodes.

[0019] Optionally, the emotion factor knowledge retriever retrieves image elements related to the target emotion from a preset emotion dataset using a retrieval enhancement-based generation method to generate the preliminary editing scheme.

[0020] Optionally, in the pre-creation stage, the editing agent gradually implements the content of all editing instructions in the optimization editing scheme on the source image through iteration according to the optimization editing scheme; the image editing algorithm tool library is integrated in the editing agent, and different editing methods correspond to different image editing algorithms in the image editing algorithm tool library.

[0021] Optionally, for the editing result in the pre-creation stage, the evaluation, adjustment processing, and re-editing operation in the optimization stage are performed in a loop until the evaluation agent obtains an evaluation result that meets the expression of the target emotion.

[0022] The application also provides a multi-agent collaboration framework for implementing the emotion image editing method as described above, comprising:

[0023] A planning agent for receiving a target emotion and a source image and generating a preliminary editing scheme according to the source image and the target emotion;

[0024] An editing agent for editing the source image according to the preliminary editing scheme so that the edited image meets the requirements of the target emotion in terms of vision and emotion;

[0025] An evaluation agent for evaluating the preliminary editing scheme and the editing result according to the preliminary editing scheme, and providing optimization suggestions based on the evaluation result to form an optimization editing scheme; and for evaluating the edited image to determine whether it effectively conveys the target emotion, and providing optimization suggestions for the editing result that does not meet the requirements.

[0026] The application also provides an emotion image editing system based on a multi-agent collaboration framework, which uses the emotion image editing method based on a multi-agent collaboration framework as described above to edit emotion images, comprising a pre-creation unit and an optimization unit; wherein,

[0027] The pre-creating unit is configured to generate a preliminary editing scheme based on the acquired source image and target emotion by a planning agent; and evaluate the preliminary editing scheme by an evaluation agent, and perform a preliminary editing operation by an editing agent according to the preliminary editing scheme to obtain a preliminary edited image.

[0028] The optimization unit is configured to evaluate the preliminary edited image by the evaluation agent, obtain an optimization editing instruction according to the evaluation result, and adjust the preliminary edited image according to the optimization editing instruction by the editing agent.

[0029] The present application also provides an electronic device, which comprises:

[0030] at least one processor; and

[0031] a memory connected with the at least one processor; wherein

[0032] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the steps in the emotion image editing method based on the multi-agent cooperation framework as described above.

[0033] From the above technical solutions, the emotion image editing method and system based on the multi-agent cooperation framework provided by the present application can achieve accurate image editing under the driving of emotion by inputting the source image and target emotion description by the user, and combining emotion editing element retrieval and multi-agent cooperation. First, in the pre-creation stage, the source image is analyzed for emotion and semantics, and relevant emotion editing elements are retrieved by a pre-set emotion factor knowledge retriever to generate a preliminary emotion image editing scheme. Then, the evaluation agent evaluates the effectiveness of the preliminary editing scheme, proposes optimization suggestions, and optimizes the editing scheme based on this to finally generate an optimized editing scheme. Then, the editing agent performs a preliminary editing operation according to the optimized plan to generate a preliminary editing result. In the optimization stage, the evaluation agent evaluates whether the preliminary editing result effectively conveys the target emotion, and generates optimization editing instructions based on the evaluation feedback. The editing agent iteratively generates updated images according to these instructions until the image successfully conveys the target emotion. The technical solution of the present application effectively realizes emotion-driven image editing, improves the accuracy and flexibility of image emotion expression, and is suitable for various creative image generation and emotion-oriented image editing tasks. BRIEF DESCRIPTION OF DRAWINGS

[0034] Other objects and results of the present application will become more apparent and easily understood with reference to the following description of the present application taken in conjunction with the accompanying drawings, and with a more complete understanding of the present application. In the drawings:

[0035] Figure 1 Flow chart of the emotion image editing method based on the multi-agent collaboration framework according to the embodiment of the present application;

[0036] Figure 2 Overview of the three agents according to the embodiment of the present application;

[0037] Figure 3 Workflow schematic diagram of the emotion image editing method based on the multi-agent collaboration framework according to the embodiment of the present application;

[0038] Figure 4 Three-layer decision tree structure schematic diagram of the planning agent according to the embodiment of the present application;

[0039] Figure 5 Action space schematic diagram in the editing agent according to the embodiment of the present application;

[0040] Figure 6 System framework schematic diagram of the emotion image editing based on the multi-agent collaboration framework according to the embodiment of the present application;

[0041] Figure 7 Schematic diagram of the electronic device according to the embodiment of the present application;

[0042] Figure 8 and Figure 9 Effect comparison schematic diagram of applying the present application and other existing image editing schemes;

[0043] Figure 10 and Figure 11 Two effect example diagrams of image editing respectively applying the emotion image editing method and system based on the multi-agent collaboration framework of the present application. DETAILED DESCRIPTION

[0044] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more embodiments. It can be evident, however, that such embodiment(s) can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing one or more embodiments.

[0045] In view of the problems of inaccurate emotional expression and lack of flexibility in the editing process in the existing emotional image editing method, the present application proposes an emotional image editing method and system based on a multi-agent collaboration framework, which combines emotional needs and source images, uses the collaboration framework of multi-agent to edit and optimize images, and more accurately conveys the target emotion; wherein, through the emotional factor knowledge retriever and the fine editing scheme generation process, the present application can effectively solve the problem of inconsistency between emotional understanding and editing instruction execution in the traditional emotional image editing method, and significantly improve the quality and precision of image editing effect.

[0046] In order to better illustrate the technical solutions of the present application, the following will first briefly describe some technical terms involved in the present application.

[0047] Image editing is a highly detailed and detailed modification and adjustment of images. In the image editing task, a source image and related editing instructions, reference images are usually given to generate a target image that meets the editing requirements and retains the characteristics of the original image. The core challenge of this task is how to effectively balance the editing effect and the fidelity to the original content to ensure that the final image can meet the specific editing requirements of the user while retaining the core visual properties of the original image.

[0048] Agent is an intelligent entity built on a large language model with environment perception ability, autonomous understanding, decision making and action execution ability. Agent can simulate independent thinking process, flexibly call various tools, and gradually achieve the preset target. It aims to complete complex tasks through the close combination of perception, thinking and action.

[0049] CoT (Chain of Thought) is a technology that breaks down complex problems into a series of multi-step thinking processes with logical relationships, and explains the entire thinking process by showing a small number of examples to large language models, thereby activating the thinking ability of large models to answer more complex and difficult problems.

[0050] Prompt Engineering is a technology that designs and improves the prompts of large language models to improve their performance. The goal is to create an effective and controllable large language model system that can accurately and reliably perform specific tasks.

[0051] The specific embodiments of the present application will be described in detail below in conjunction with the accompanying drawings.

[0052] It is to be understood that the descriptions of the following exemplary embodiments are merely illustrative in nature and are in no way intended to limit the present application, its application or uses thereof. Techniques and devices known to those of ordinary skill in the relevant art can not be discussed in detail, but should be considered part of the specification when appropriate.

[0053] To illustrate the emotion image editing method and system based on the multi-agent collaboration framework provided by the present application, Figures 1 to 5 The flow and principle of the emotion image editing method based on the multi-agent collaboration framework according to the embodiments of the present application are exemplarily indicated respectively; Figure 6 The logical structure of the emotion image editing system based on the multi-agent collaboration framework according to the embodiments of the present application is exemplarily indicated.

[0054] As Figures 1 to 5 As shown collectively, the emotion image editing method based on the multi-agent collaboration framework provided by the present application mainly includes the following steps:

[0055] S110: In the pre-creation stage, a preliminary editing scheme is generated through multi-step processing of multiple agents, and a preliminary editing operation is performed to finally obtain a preliminary edited image.

[0056] Specifically, in the pre-creation stage, a preliminary editing scheme is generated based on the source image and the target emotion obtained by the planning agent; according to the preliminary editing scheme, the preliminary editing scheme is evaluated by the evaluation agent, and the preliminary editing operation is performed by the editing agent to obtain a preliminary edited image.

[0057] S120: In the optimization stage, optimization editing instructions are obtained through evaluation and feedback of the preliminary edited image, and the preliminary editing result is adjusted according to the optimization editing instructions to better convey the target emotion.

[0058] Specifically, in the optimization stage, the preliminary edited image is evaluated by the evaluation agent, and if the preliminary edited image does not conform to the target emotion, optimization editing instructions are generated, and the preliminary edited image is adjusted by the editing agent according to the optimization editing instructions; if it conforms to the target emotion, the preliminary edited image can be directly outputted.

[0059] Specifically, as an example, the planning agent analyzes the emotion and semantics of the source image through a multi-modal language model to obtain the emotion information and semantic information of the source image. The multi-modal language model used in the present application can use the GPT series multi-modal model of Openai. For the convenience of description, in the following embodiments, GPT4o is taken as an example of the implementation of the multi-modal language model.

[0060] The above steps of the emotion image editing method based on the multi-agent cooperation framework will be described in more detail below in combination with specific embodiments.

[0061] More specifically, as an example, for step S110, the specific method of the preliminary editing image obtained by the pre-creation stage is as follows:

[0062] For the source image input by the user , the target emotion , first, the planning agent analyzes the source image through a multi-modal language model to obtain its semantic information (semantic text description ) and emotional information (original emotion ); then, based on the obtained semantic information and emotional information of the source image, the planning agent uses a pre-set emotional factor knowledge retriever to retrieve appropriate emotional editing elements, and then determines the specific steps of each editing element and editing operation in the preliminary editing scheme through a three-layer decision tree structure, which takes the target emotion as the root node, selects elements with high text similarity as branch nodes, editing methods as leaf nodes, and finally determines editing instructions .

[0063] After multiple rounds of retrieval, the preliminary editing scheme is integrated. The evaluation agent evaluates the target emotion expression degree of the preliminary editing scheme, and modifies the editing elements in the editing instructions that cannot fully express the target emotion, and transmits the adjusted editing scheme to the editing agent for specific editing operations to obtain the edited image in the pre-creation stage , i.e., the preliminary editing image.

[0064] The pre-creation stage proposed in the present application is equivalent to saying that emotion control on the input source image is a large framework, and the single emotion image editing task is the first image editing within this large framework. Specifically, the pre-creation stage includes the following steps:

[0065] S111: The planning agent analyzes the emotion and semantics of the source image to obtain the emotional information and semantic information of the source image; wherein the emotion analysis mainly focuses on the specific original emotion in the source image, and the emotional features of the source image are obtained through emotion analysis, the emotional features being specific emotional elements contained in the source image; the semantic analysis focuses on the semantic features such as objects and scenes in the image;

[0066] S112: Based on the obtained target emotion, the intelligent agent retrieves emotion editing elements through a preset emotion factor knowledge retriever; specifically, based on the target emotion, the emotion factor knowledge retriever is used to retrieve emotion editing elements related to the target emotion from an emotion editing element database.

[0067] The emotion factor knowledge retriever can use a retrieval enhancement based generation method to retrieve image elements related to the target emotion from a preset emotion dataset to generate the preliminary editing scheme.

[0068] S113: Generate a detailed emotion image editing scheme; specifically, according to the emotion editing elements retrieved from the emotion factor knowledge retriever, a preliminary editing scheme for a detailed emotion image is generated, which lists specific editing operations, adjustment order, etc., to ensure that the image can be changed towards the target emotion.

[0069] In one embodiment of the present application, based on the semantic features of the image and the original emotion and the target emotion, appropriate visual elements and editing methods are selected for the generation of the preliminary editing scheme, including retrieval in the emotion factor knowledge retriever and using a large-scale language model to independently conceive editing elements that best match the image semantics and the target emotion, and selecting appropriate editing methods according to the editing elements and the image semantics. The generation algorithm of the preliminary editing scheme is as follows:

[0070]

[0071] Wherein, represents the i-th editing instruction, represents a function of combining the editing element and the editing method . Wherein, the editing elements include: independent objects, scenes, object attributes, colors, filters, facial expressions; the editing methods include: adding, replacing, deleting, expression editing, filter editing, background editing, attribute editing.

[0072] Wherein, the optional scheme is that the emotion factor knowledge retriever includes a constructed emotion factor knowledge base and a constructed emotion factor retrieval engine. The emotion factor knowledge base is constructed based on the emotion dataset EmoSet, each data contains a descriptive text describing a specific editing element, and is labeled with its corresponding emotion and element type; the emotion factor retrieval engine retrieves the descriptive text of the emotion element with the highest semantic similarity to the semantic text description of the original image.

[0073] S114: Optimize the preliminary editing scheme; specifically, evaluate the preliminary editing scheme by the intelligent agent, judge whether the preliminary editing scheme can effectively convey the target emotion, and optimize the preliminary editing scheme according to the evaluation result to form an optimized editing scheme;

[0074] Wherein, the evaluation of the preliminary editing scheme includes understanding the emotion expressed by the image using a multi-modal language model to determine whether it is consistent with the target emotion. The optimization suggestions for optimizing the preliminary editing scheme according to the evaluation results include new editing instructions obtained by updating the editing elements, and adjusting the editing instructions that are not correctly executed and the editing instructions that finally fail to express the target emotion to new editing instructions and commanding the editing intelligent agent to re-edit.

[0075] S115: Perform preliminary editing operation; specifically, the editing intelligent agent performs image editing tasks according to the above-mentioned optimized editing scheme to generate preliminary edited images. These image editing tasks may include color adjustment, light and shadow change, expression modification, object addition, object removal, object replacement, object attribute editing, background editing, etc. Through these editing operations, preliminary edited images are generated.

[0076] Wherein, the optional scheme is that the image editing tasks mentioned above include different image editing algorithms for each editing instruction to perform specific editing operations according to the given editing instructions. Wherein, the specific editing operation includes pre-processing node operation, editing node operation, post-validation node operation; the pre-processing node operation includes using image detection and segmentation algorithms to generate target masks to assist image editing algorithms in editing operations. Wherein, the editing node operation includes using image editing algorithms to perform specific editing operations to obtain edited images; the post-validation node includes using a visual language model to compare the edited images with the editing instructions, and re-editing the edited images that do not meet the editing instructions.

[0077] It should be noted that in the pre-creation stage, the editing intelligent agent can gradually implement all the contents of the editing instructions in the optimized editing scheme for the source image through an iterative manner according to the optimized editing scheme; wherein, the image editing algorithm tool library is integrated in the editing intelligent agent, different editing methods correspond to different image editing algorithms in the image editing algorithm tool library, and the editing methods include object addition, object replacement, object removal, expression editing, background editing, filter editing, and object attribute editing.

[0078] For step S120, the final specific method obtained through the optimization stage is as follows:

[0079] For the preliminary editing result of step S110 , the evaluation intelligent agent will evaluate whether it expresses the target emotion The effectiveness and rationality are evaluated, and for the editing result capable of reasonably and effectively expressing the target emotion, the round of emotion image editing task ends.

[0080] For the editing result incapable of reasonably and effectively expressing the target emotion, the evaluation intelligent agent will optimize and adjust the editing instruction that is not correctly executed , and re-deliver it to the editing intelligent agent for editing. The optimization and adjustment mainly include re-executing the editing operation of the editing instruction or re-adjusting the text description of the editing instruction.

[0081] The optimization stage proposed in the present application is equivalent to subsequent image editing after a single emotion image editing task in the whole emotion image editing framework. Specifically, the optimization stage can further include the following steps:

[0082] S121: evaluating the preliminary editing result and giving an optimization editing instruction; specifically, the evaluation intelligent agent evaluates the preliminary editing result to determine whether the preliminary editing result effectively conveys the target emotion. If the preliminary editing result fails to achieve the expected effect, the evaluation intelligent agent will provide detailed optimization editing instructions indicating the parts that need to be further adjusted.

[0083] S122: iteratively generating an updated image according to the optimization editing instruction; specifically, the editing intelligent agent adjusts the preliminary editing result according to the optimization editing instruction to generate an updated image. The steps S121 to S122 are looped, i.e., the preliminary editing result is iteratively updated until the updated image can effectively convey the target emotion.

[0084] In one specific embodiment of the present application, the process loop of the optimization stage can be executed i times to obtain the final editing result .

[0085] From the overall framework of emotion image editing, the whole process of implementing emotion image editing of the present application includes two flow branches: a pre-creation stage and an optimization stage. The pre-creation stage uses a source image and a target emotion to generate a preliminary editing result , and the optimization stage uses the preliminary editing result to gradually adjust the expression of the target emotion , and generate a final result that can reasonably and effectively express the target emotion .

[0086] In other words, the pre-creation stage and the optimization stage are two iterative editing stages, and the optimization stage needs the preliminary editing result of the pre-creation stage. The optimization stage is performed in a loop to adjust the editing scheme and edit again so that the final editing result can better express the target emotion. For the two process branches, each agent is set with its own workflow and task, and the setting operation can adopt the technologies of Prompt Engineering and CoT.

[0087] As can be seen from the above embodiments, the multi-agent collaboration framework for implementing the above-mentioned emotional image editing method mainly includes three parts of a planning agent, an editing agent and an evaluation agent.

[0088] The planning agent is configured to receive a target emotion and a source image and generate a preliminary editing scheme according to the source image and the target emotion. The editing agent is configured to edit the source image according to the preliminary editing scheme generated by the planning agent, so that the edited image meets the requirements of the target emotion in terms of vision and emotion. The evaluation agent is configured to evaluate the preliminary editing scheme and the editing result according to the preliminary editing scheme, and propose optimization opinions according to the evaluation result to form an optimized editing scheme. If the editing result according to the preliminary editing scheme does not meet the target emotion, an optimization editing instruction is generated. If it meets the target emotion, the preliminary edited image can be directly output. In addition, the evaluation agent is also configured to evaluate the edited image, judge whether the target emotion is effectively conveyed, and propose optimization opinions for the editing result that does not meet the standard.

[0089] In addition, the present application also provides an emotional image editing system based on a multi-agent collaboration framework, which uses the emotional image editing method based on the multi-agent collaboration framework as described above to edit emotional images. The emotional image editing system based on the multi-agent collaboration framework mainly includes a pre-creation unit and an optimization unit. The pre-creation unit is configured to generate a preliminary editing scheme based on the obtained source image and target emotion by the planning agent, and evaluate the preliminary editing scheme by the evaluation agent according to the preliminary editing scheme, and perform the preliminary editing operation in the preliminary editing scheme by the editing agent to obtain a preliminary edited image. The optimization unit is configured to evaluate the preliminary edited image by the evaluation agent, obtain an optimization editing instruction according to the evaluation result, and adjust the preliminary edited image according to the optimization editing instruction by the editing agent.

[0090] From another point of view, the emotional image editing system 300 based on the multi-agent collaboration framework provided by the present application mainly includes an editing generation unit 310, an editing execution unit 320 and an editing evaluation unit 330, as shown in Figure 6

[0091] ​The editing generation unit 310 is configured to generate content according to an editing scheme based on user input data, wherein the user input data includes a source image and a target emotion.

[0092] The editing execution unit 320 is configured to complete actual editing on the content of the editing scheme to obtain an edited image.

[0093] The editing evaluation unit 330 is configured to evaluate and verify the edited image in terms of emotion and semantics.

[0094] The editing generation unit 310 can further include:

[0095] The emotion analysis unit 311 is configured to analyze semantic text description and emotion information of the source image input by the user.

[0096] The emotion element retrieval unit 312 is configured to retrieve suitable reference emotion elements based on the target emotion and the semantic text description of the source image.

[0097] The editing scheme decision unit 313 is configured to combine the semantic text description of the source image and the target emotion with suitable editing elements and content, and formulate an editing scheme matching the source image in combination with the reference emotion elements.

[0098] The editing execution unit 320 can further include:

[0099] The editing tool library unit 321 is configured to integrate different types of image editing algorithms to complete actual editing operations, wherein the image editing algorithms include semantic editing algorithms.

[0100] The editing preprocessing unit 322 is configured to perform corresponding preprocessing operations on the source image and intermediate editing results to assist in completing actual editing, wherein the preprocessing operations include target detection and segmentation, pixel fine-grained optimization, etc.

[0101] The editing decision unit 323 is configured to receive the editing scheme and automatically call suitable image editing algorithms from the editing tool library unit to complete all editing operations to obtain the final edited image.

[0102] The editing evaluation unit 330 can further include:

[0103] The verification evaluation unit 331 is configured to perform emotion and semantic verification evaluation on the edited image or the editing scheme to obtain target emotion matching degree and semantic suitability.

[0104] The editing optimization unit 332 is configured to optimize the editing scheme for the edited image that does not achieve effective target emotion expression to enhance the target emotion expression of the image.

[0105] The emotion image editing system based on the multi-agent collaboration framework corresponds to the emotion image editing control method based on the multi-agent collaboration framework, and the specific execution steps can refer to the specific embodiments of the emotion image editing method based on the multi-agent collaboration framework, which will not be described again.

[0106] As can be seen from the above embodiments, the emotion image editing method and system based on the multi-agent collaboration framework proposed by the present application can realize accurate image editing under the driving of emotion by inputting the source image and the target emotion description by the user, and combining emotion editing element retrieval and multi-agent collaboration. First, in the pre-creation stage, the source image is analyzed for emotion and semantics, and the relevant emotion editing elements are retrieved by the emotion factor knowledge retriever to generate a preliminary emotion image editing scheme. Then, the evaluation agent evaluates the effectiveness of the preliminary editing scheme, proposes optimization suggestions, and optimizes the editing scheme based on this to finally generate an optimized editing scheme. Then, the editing agent executes the preliminary editing operation according to the optimized plan to generate a preliminary editing result. In the optimization stage, the evaluation agent evaluates whether the preliminary editing result effectively conveys the target emotion, and generates optimization editing instructions based on the evaluation feedback, and the editing agent iteratively generates updated images according to these instructions until the image successfully conveys the target emotion. The emotion-driven image editing is effectively realized, and the accuracy and flexibility of image emotion expression are improved.

[0107] Figures 8 to 11 The image editing effects of the emotion image editing method based on the multi-agent collaboration framework of the present application are shown respectively. Among them, Figure 8 、 Figure 9 is a comparison diagram of the effect of applying the present application and other existing image editing schemes, Figure 10 and Figure 11 are two effect examples of image editing by the emotion image editing method and system based on the multi-agent collaboration framework of the present application.

[0108] As Figure 8 、 Figure 9 indicated, the text on the lower side of each row is the target emotion required by the user for image conversion, including eight types.

[0109] The first column of pictures is the original image, and the last column (the ninth column) of pictures "ours" represents the result of image editing by the emotion image editing method and system based on the multi-agent collaboration framework of the present application, and the second column to the eighth column are the result images of other comparison schemes. The other comparison schemes are as follows:

[0110] IP2P: Paper InstructPix2Pix: Learning to Follow Image Editing Instructions

[0111] MagicBrush: Paper MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing

[0112] PnP: Paper Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation

[0113] Insdiff: Paper Instructdiffusion: A Generalist Modeling Interface for Vision Tasks

[0114] CLVA: Language-Driven Artistic Style Transfer

[0115] AIF: Affective Image Filter: Reflecting Emotions from Text to Images

[0116] EmoEdit*: Built following the paper Emoedit: Evoking Emotions through Image Manipulation

[0117] In Figure 10 , every two pictures are a group, the left is the original picture, and the right is the edited picture edited by the emotion image editing method and system based on the multi-agent collaboration framework of the application. The text below the right image is the target emotion.

[0118] In Figure 11 , every nine pictures are a group, the leftmost is the original picture, and the right eight pictures are edited pictures edited by the emotion image editing method and system based on the multi-agent collaboration framework of the application. The text below each image on the right is the target emotion, showing that the emotion image editing method and system based on the multi-agent collaboration framework can convert the image emotion to any target emotion.

[0119] As Figure 7As shown, the present application also provides an electronic device, which comprises:

[0120] at least one processor; and

[0121] a memory in communication with the at least one processor; wherein

[0122] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the steps in the aforementioned emotion image editing method based on the multi-agent cooperation framework.

[0123] It is understood by those skilled in the art that the structure shown does not constitute a limitation on the electronic device 1, and can include fewer or more components than shown, or combine certain components, or different component arrangements.

[0124] For example, although not shown, the electronic device 1 can also include a power source (such as a battery) to power the various components. Preferably, the power source can be logically connected to the at least one processor 10 through a power management device, so that functions such as charge management, discharge management, and power consumption management can be achieved through the power management device. The power source can also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and any other components. The electronic device 1 can also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which are not described here.

[0125] Further, the electronic device 1 can also include a network interface, which can optionally include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), and is typically used to establish a communication connection between the electronic device 1 and other electronic devices.

[0126] Optionally, the electronic device 1 can also include a user interface, which can be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally a standard wired interface, a wireless interface. Optionally, in some embodiments, the display can be an LED display, a liquid crystal display, a touch liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch, etc. The display can also be appropriately referred to as a display screen or a display unit, and is used to display information processed in the electronic device 1 and to display a visualized user interface.

[0127] It should be understood that the embodiments are only for illustration and are not limited in the scope of the patent application by this structure.

[0128] The emotion image editing program 12 stored in the memory 11 in the electronic device 1 based on the multi-agent cooperation framework is a combination of multiple instructions, which, when running in the processor 10, can implement the following steps:

[0129] S110: In the pre-creation stage, a preliminary editing scheme is generated through multi-agent multi-step processing, and a preliminary editing operation is performed, finally obtaining a preliminary edited image;

[0130] S120: In the optimization stage, an optimized editing instruction is obtained through evaluation and feedback of the preliminary edited image, and the preliminary editing result is adjusted according to the optimized editing instruction to better convey the target emotion.

[0131] Among them, the multi-step processing of S110 pre-creation stage includes:

[0132] S111: The planning agent analyzes the emotion and semantics of the source image to obtain the emotion information and semantic information of the source image; wherein the emotion analysis mainly focuses on the emotional features in the source image, and the semantic analysis focuses on the object, scene and other semantic features in the image;

[0133] S112: Based on the obtained target emotion, the planning agent retrieves emotion editing elements through a pre-set emotion factor knowledge retriever; specifically, based on the target emotion, the emotion factor knowledge retriever is used to retrieve emotion editing elements related to the target emotion from the emotion editing element database;

[0134] Among them, the emotion factor knowledge retriever can use a retrieval enhancement generation method to retrieve image elements related to the target emotion from the pre-set emotion data set to generate the preliminary editing scheme.

[0135] S113: Generate a detailed emotion image editing scheme; specifically, according to the emotion editing elements retrieved from the emotion factor knowledge retriever, a detailed emotion image preliminary editing scheme is generated, which is used to list specific editing operations, adjustment order, etc. to ensure that the image can be changed towards the target emotion.

[0136] S114: Optimize the preliminary editing scheme; specifically, the preliminary editing scheme is evaluated by the evaluation agent to determine whether the preliminary editing scheme can effectively convey the target emotion, and the preliminary editing scheme is optimized according to the evaluation result to finally form an optimized editing scheme;

[0137] The preliminary editing scheme is evaluated, including understanding the emotion expressed by the image using a visual language model to determine whether it is consistent with the target emotion. According to the evaluation result, the optimization opinion of the preliminary editing scheme is optimized, including new editing instructions obtained by updating the editing elements, and adjusting the editing instructions that are not correctly executed and the editing instructions that finally fail to express the target emotion to new editing instructions and commanding the editing agent to re-edit.

[0138] S115: Perform a preliminary editing operation; specifically, generate a preliminary edited image by executing image editing tasks according to the above-mentioned optimization editing scheme through the editing agent. These image editing tasks can include color adjustment, light and shadow change, expression modification, and object addition, object removal, object replacement, object attribute editing, background editing, etc. Through these editing operations, a preliminary edited image is generated.

[0139] S120: Multi-step processing in the optimization phase, including: S121: Evaluate the preliminary editing result and give optimization editing instructions; specifically, evaluate the preliminary editing result through the evaluation agent to determine whether the preliminary editing result effectively conveys the target emotion. If the preliminary editing result fails to achieve the expected effect, the evaluation agent will provide detailed optimization editing instructions indicating the parts that need to be further adjusted.

[0140] S122: Iteratively generate updated images according to the optimization editing instructions; specifically, adjust the preliminary editing result according to the above-mentioned optimization editing instructions through the editing agent to generate updated images. Here, steps S121 to S122 are repeated, i.e., the preliminary editing result is iteratively updated until the updated image can effectively convey the target emotion. Specifically, the specific implementation method of the above-mentioned instructions can refer to the description of the corresponding embodiments in the multi-agent collaboration framework emotion image editing method, which is not repeated here.

[0141] Further, the modules / units integrated in the electronic device 1, if implemented in the form of software function units and sold or used as independent products, can be stored in a computer-readable storage medium. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory).

[0142] The emotion image editing method and system based on the multi-agent cooperation framework according to the present application are described above with reference to the drawings by way of example. However, those skilled in the art should understand that various modifications can be made to the emotion image editing method and system based on the multi-agent cooperation framework according to the present application described above without departing from the content of the present application. Therefore, the protection scope of the present application should be determined by the content of the appended claims.

Claims

1. An emotion image editing method based on a multi-agent cooperation framework, characterized in that, The method comprises a pre-creation stage and an optimization stage. In the pre-creation stage, a planning agent generates a preliminary editing scheme based on the obtained source image and target emotion; an evaluation agent evaluates the preliminary editing scheme, modifies the editing elements in the preliminary editing scheme that cannot fully express the target emotion, and transmits the adjusted editing scheme to an editing agent for specific editing operations; and the editing agent executes the editing operations in the preliminary editing scheme to obtain a preliminary edited image; wherein the preliminary editing scheme is used to list specific editing operations and adjustment sequences. In the optimization stage, an evaluation agent evaluates the preliminary edited image to determine whether the preliminary edited image effectively conveys the target emotion; wherein if the preliminary edited image fails to achieve the expected effect, the evaluation agent provides optimization editing instructions for the preliminary edited image; and the editing agent iteratively updates the preliminary edited image according to the optimization editing instructions to obtain an optimized edited image. The generation algorithm of the preliminary editing scheme is as follows: wherein, represents the ith editing instruction, represents a function of combining the editing element and the editing mode ; wherein the editing element includes: independent object, scene, object attribute, color, filter, facial expression; and the editing mode includes: addition, replacement, deletion, expression editing, filter editing, background editing, attribute editing. The method for obtaining the editing elements and the editing methods comprises: analyzing the emotion and semantics of the source image by a planning agent to obtain the emotion features and semantic features of the source image; based on the emotion features and semantic features and the target emotion, using a pre-set emotion factor knowledge retriever to use a large-scale language model to independently conceive the most suitable editing elements for the emotion features, the semantic features, and the target emotion, and selecting appropriate editing methods according to the editing elements and image semantics; The pre-set emotion factor knowledge retriever comprises an emotion factor knowledge base, which is constructed based on an emotion data set EmoSet, wherein each data contains descriptive text describing specific editing elements, and is labeled with its corresponding emotion and element type; The evaluation of the preliminary editing scheme by the evaluation agent comprises using a multi-modal language model to understand the emotion expressed by the image to determine whether it conforms to the target emotion; optimizing the preliminary editing scheme based on the evaluation result, obtaining new editing instructions by updating the editing elements, and adjusting the editing instructions that are not correctly executed and the editing instructions that finally fail to express the target emotion to new editing instructions and instructing the editing agent to re-edit. 2.The emotion image editing method based on the multi-agent collaboration framework according to claim 1, wherein, The pre-creation stage comprises: S111: analyzing the emotion and semantics of the source image by a planning agent to obtain the emotion information and semantic information of the source image; S112: based on the target emotion, retrieving emotion editing elements related to the target emotion from an emotion editing element database by a pre-set emotion factor knowledge retriever; S113: generating a preliminary editing scheme for the emotion image based on the emotion editing elements, wherein the preliminary editing scheme comprises specific editing operations and adjustment sequences; S114: evaluating the preliminary editing scheme by an evaluation agent and optimizing the preliminary editing scheme based on the evaluation result to form an optimized editing scheme; S115: generating a preliminary edited image by executing an image editing task according to the optimized editing scheme by the editing agent. 3.The emotion image editing method based on the multi-agent collaboration framework according to claim 2, wherein, The planning agent determines the specific steps of each editing element and editing operation in the preliminary editing scheme through a three-layer decision tree structure; wherein the three-layer decision tree takes the target emotion as the root node, identifies suitable editing elements as branch nodes, and assigns corresponding editing methods as leaf nodes. 4.The emotion image editing method based on the multi-agent collaboration framework according to claim 2, wherein, The emotion factor knowledge retriever uses a retrieval enhancement-based generation method to retrieve image elements related to the target emotion from a preset emotion dataset to generate the preliminary editing scheme. 5.The emotion image editing method based on the multi-agent collaboration framework according to claim 2, wherein, In the pre-creation stage, the editing agent gradually implements the content of all editing instructions in the optimized editing scheme on the source image through iteration according to the optimized editing scheme; wherein an image editing algorithm tool library is integrated in the editing agent, and different editing methods correspond to different image editing algorithms in the image editing algorithm tool library.

6. The emotion image editing method based on the multi-agent cooperation framework according to claim 1, characterized in that, For the editing result of the pre-creation stage, the evaluation, adjustment, and re-editing operations of the optimization stage are performed in a loop until the evaluation agent obtains an evaluation result that meets the expression of the target emotion.

7. A multi-agent collaboration framework for implementing the method of affective image editing according to any one of claims 1-6, characterized in that, Comprise: a planning agent for receiving a target emotion and a source image and generating a preliminary editing scheme according to the source image and the target emotion; an editing agent for editing the source image according to the preliminary editing scheme so that the edited image meets the requirements of the target emotion in terms of vision and emotion; an evaluation agent for evaluating the preliminary editing scheme and the editing result according to the preliminary editing scheme and providing optimization suggestions based on the evaluation result to form an optimized editing scheme; and for evaluating the edited image to determine whether it effectively conveys the target emotion and providing optimization suggestions for the editing result that does not meet the requirements.

8. A multi-agent collaboration framework-based emotional image editing system for emotional image editing using the multi-agent collaboration framework-based emotional image editing method of any one of claims 1-6, comprising a pre-creation unit and an optimization unit; wherein the pre-creation unit is configured to generate a preliminary editing scheme based on an obtained source image and a target emotion by a planning agent; evaluate the preliminary editing scheme by an evaluation agent; and execute preliminary editing operations in the preliminary editing scheme by an editing agent to obtain a preliminary edited image; the optimization unit is configured to evaluate the preliminary edited image by an evaluation agent, and if the preliminary edited image does not meet the target emotion, optimize the editing instructions and adjust the preliminary edited image according to the optimized editing instructions by an editing agent.

9. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the steps in the multi-agent collaboration framework-based emotional image editing method of any one of claims 1-6.

Images