Method and electronic device for automatic generation of high quality data for image editing applications

Abstract

A method for generating an image editing dataset is provided. The method may include obtaining a candidate image for inputting an AI model from among at least one candidate image. The method may include determining an editing operation to be performed by the AI model on the candidate image from among at least one editing operation. The method may include determining a base prompt and a control prompt based on inputting of the candidate image and the editing operation into the AI model, wherein the base prompt comprises base instructions for detection of at least one object within the candidate image, and the control prompt comprises control instructions relevant to the editing operation to be performed by the AI model on the candidate image. The method may include generating the image editing dataset based on the base prompt and the control prompt.

Description

METHOD AND ELECTRONIC DEVICE FOR AUTOMATIC GENERATION OF HIGH QUALITY DATA FOR IMAGE EDITING APPLICATIONS

[0001] The proposed invention is related to image enhancement technique. More particularly, the embodiment of the disclosure relates to an automatic high quality data generation for image editing applications.

[0002] Digital images have become one of the most significant mediums for information exchange on the internet, providing immense convenience for individuals and organizations alike. With the advancements in image processing theory, digital image inpainting technology has also seen rapid development. Image inpainting refers to the process of completing missing regions in digital images. The primary goal of image inpainting is to fill these missing regions with content that is both semantically reasonable and visually realistic, ensuring that it is consistent with the remaining parts of the image. This technology has various applications, such as an image editing tool to remove unwanted objects from an image or to restore defective regions in damaged paintings.

[0003] In existing mechanism, the pixels that need to be erased are defined using binary masks. From an application standpoint, users are required to generate these masks for the objects they wish to remove. This process is not only time-consuming but also prone to errors, leading to suboptimal results.

[0004] Further, the existing deep learning technologies and image restoration methods face several drawbacks. One significant issue is the computational complexity involved, which affects the real-time performance of these systems. Additionally, the quality of restoration achieved by current methods often falls short of expectations. Training large-scale models for tasks such as text-to-image and image-to-image generation demands an enormous amount of annotated data. For instance, models for text-to-image generation are typically trained on a vast corpus, such as the LAION (Large-scale Artificial Intelligence Open Network) dataset, which contains around 5 billion image-text pairs. While such extensive datasets are available for text-to-image generation, the data available for image-to-image generation―particularly for image editing tasks like adding, removing, or blending elements―is limited and often lacks the desired quality.

[0005] Generating data for instruction-based image editing applications involves creating tuples of source images, edit instructions, and edited images, which is a challenging and labor-intensive process. Existing systems that rely on external datasets with object annotations often lack the capability for intelligent object detection and removal, resulting in unrealistic outputs. Furthermore, these systems typically employ template-based instruction generation, restricting their pipelines to specific types of edits, such as removal.

[0006] The primary issue with existing solutions lies in the low-quality datasets and the lack of intelligent object detection mechanisms. This deficiency leads to unnatural and unrealistic outputs, undermining the effectiveness and applicability of image inpainting technologies in practical scenarios.

[0007] Therefore, there is a need to address these disadvantages or other shortcomings, or at least provide a viable alternative that can efficiently enhance data set generation for image editing application.

[0008] In an embodiment of the disclosure, the objectives are achieved by providing a method for generating an image editing dataset. The method may include obtaining a candidate image for inputting an AI (artificial intelligence) model from among at least one candidate image, wherein the AI model is generative AI model for generating the image editing dataset. The method may include determining an editing operation to be performed by the AI model on the candidate image from among at least one editing operation. The method may include determining a base prompt and a control prompt based on inputting the candidate image and the editing operation into the AI model, wherein the base prompt includes base instructions for detection of at least one object within the candidate image, and the control prompt includes control instructions relevant to the editing operation to be performed by the AI model on the candidate image. The method may include generating the image editing dataset based on the base prompt and the control prompt.

[0009] In an embodiment of the disclosure, the objectives are achieved by providing an electronic device for generating an image editing dataset. The electronic device includes at least one memory comprising instructions and at least one candidate image and at least one processor comprising processing circuitry. The instructions, when executed by the at least one processor individually or collectively, cause the electronic device to: obtain a candidate image for inputting an AI (artificial intelligence) model from among the at least one candidate image, wherein the AI model is generative AI model for generating the image editing dataset; determine an editing operation to be performed by the AI model on the candidate image from among at least one editing operation; determine a base prompt and a control prompt based on inputting the candidate image and the editing operation into the AI model, wherein the base prompt includes base instructions for detection of at least one object within the candidate image, and the control prompt includes control instructions relevant to the editing operation to be performed by the AI model on the candidate image ; and generate the image editing dataset based on the base prompt and the control prompt.

[0010] In an embodiment of the disclosure, a computer-readable medium containing instructions, wherein the instructions, when executed by at least one processor, cause the electronic device to perform the method may be provided. The method may include obtaining a candidate image for inputting an AI (artificial intelligence) model from among at least one candidate image, wherein the AI model is generative AI model for generating the image editing dataset. The method may include determining an editing operation to be performed by the AI model on the candidate image from among at least one editing operation. The method may include determining a base prompt and a control prompt based on inputting the candidate image and the editing operation into the AI model, wherein the base prompt includes base instructions for detection of at least one object within the candidate image, and the control prompt includes control instructions relevant to the editing operation to be performed by the AI model on the candidate image. The method may include generating the image editing dataset based on the base prompt and the control prompt.

[0011] An embodiment of the disclosure will be better appreciated and understood when considered in conjunction with the following description and the accompanying drawings. It should be understood, however, that the following descriptions, while indicating preferred embodiments and numerous specific details thereof, are given by way of illustration and not of limitation. Many changes and modifications be made within the scope of the embodiments herein.

[0012] These and other features, aspects, and advantages of the embodiments of the disclosure are illustrated in the accompanying drawings, throughout which like reference letters indicate corresponding parts in the various figures. The embodiments herein will be better understood from the following description with reference to the drawings, in which:

[0013] Figure 1 is an example of a pipeline for adding a person to a source image using image editing application according to prior art.

[0014] Figure 2A is a block diagram of an electronic device for automatic high-quality data generation according to the embodiment disclosed herein.

[0015] Figure 2B is a block diagram that illustrates a method for automatic high-quality data generation for image editing applications according to the embodiment disclosed herein.

[0016] Figure 2C is a flowchart that illustrates a method for performing controlled multi-object detection in a candidate image or source image according to the embodiment disclosed herein.

[0017] Figure 2D is a block diagram that illustrates a method for performing detection scale guidance based on a control prompt according to the embodiment disclosed herein.

[0018] Figure 2E is a block diagram that illustrates a method for generating a natural language instruction for an edit image according to the embodiment disclosed herein.

[0019] Figure 2F is a block diagram that illustrates a method for generating the depth map using a depth map encoder according to the embodiment disclosed herein.

[0020] Figure 2G and Figure 2H are block diagrams that illustrate a method of identifying the edit region and an architecture of an Edit Region Identification model according to the embodiment disclosed herein.

[0021] Figure 2I is a flowchart that illustrates the working method of a Depth Fusion and a task-Aware Query Former according to the present embodiment disclosed herein.

[0022] Figure 3 is a flowchart that illustrates the working operation of a proposed invention according to the embodiment disclosed herein.

[0023] Figure 4 is a schematic diagram that illustrates an example of data used to train instruction-based image editing models according to the embodiment of the disclosure.

[0024] Figure 5 is a block diagram that illustrates a working method of add or remove operation in an edit image generation pipeline according to the embodiment disclosed herein.

[0025] Figure 6 is a block diagram that illustrates a working method of harmonization in edit image generation pipelines according to the embodiment disclosed herein.

[0026] Figure 7 is a block diagram that illustrates a working method of change attribute or state change in the edit image generation pipelines according to the embodiment disclosed herein.

[0027] Figure 8 is a block diagram that illustrates a working method of partial data completion in the edit image generation pipelines according to the embodiment disclosed herein.

[0028] Figure 9A-Figure 9C are block diagrams that illustrate the working operation of an attribute or state change model, an image blending, and an edit illumination model according to the embodiment disclosed herein.

[0029] Figure 10A-Figure 10B are block diagrams that illustrate the working operation of masking and inpainting respectively according to the embodiment disclosed herein.

[0030] Figure 11A and Figure 11B are images that illustrate an example of control prompt effect in the comparison between the existing system and the proposed invention.

[0031] Figure 12A and Figure 12B are images that illustrate an example of control prompt effect in the comparison between the existing system and the proposed invention.

[0032] Figure 13 is a block diagram that illustrates a method for generating a caption using a source caption generator according to the embodiment disclosed herein.

[0033] Figure 14 is a block diagram that illustrates a method for generating an edit caption for an edit image using an edit caption generator according to the embodiment disclosed herein.

[0034] Figure 15 is a block diagram that illustrates a method for count-based data validation using an edit data validator according to the embodiment disclosed herein.

[0035] Figure 16 is a block diagram that illustrates a method for the semantic filtering of an edit image according to the embodiment disclosed herein.

[0036] Figure 17A-Figure 17C are schematic representations that illustrate the use case of add or remove pipeline for automatic high-quality data generation for image editing applications according to the embodiment disclosed herein.

[0037] Figure 18A-Figure 18C are schematic representations that illustrate the use case of a state change pipeline for automatic high-quality data generation for image editing applications according to the embodiment disclosed herein.

[0038] Figure 19A-Figure 19C are schematic representations that illustrate the use case of an attribute pipeline for automatic high-quality data generation for image editing applications according to the embodiment disclosed herein.

[0039] The principal object of the embodiments herein is to automatically generate high-quality data for image editing applications.

[0040] An embodiment of the disclosure is to provide a method for detection and controlled detection of multiple objects of interest in a candidate image using a base prompt and a control prompt that provides detection guidance of granularity of detections with respect to an edit pipeline.

[0041] An embodiment of the disclosure is to provide a method for identifying an edit region in an edited image and comparing it with a depth map of the candidate image or input image in order to generate precise instructions for the edited image.

[0042] An embodiment of the disclosure is to generate data for multiple types of edits such as add, remove, harmonize, state change, and object completion.

[0043] An embodiment of the disclosure is to automatically detect relevant objects in the candidate image with granularity control using an edit pipeline to perform editing operations on the detected objects.

[0044] An embodiment of the disclosure is to perform controlled multi-object detection with a novel controlled multi-object detection visual language model (CMOD VLM) where control is achieved using control prompts in the object detector.

[0045] The embodiments herein and the various features and advantageous details thereof are explained more fully with reference to the non-limiting embodiments that are illustrated in the accompanying drawings and detailed in the following description. Descriptions of well-known components and processing techniques are omitted so as to not unnecessarily obscure the embodiments herein. Also, the various embodiments described herein are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments. The term "or" as used herein, refers to a non-exclusive or, unless otherwise indicated. The examples used herein are intended merely to facilitate an understanding of ways in which the embodiments herein can be practiced and further enable those skilled in the art to practice the embodiments herein. Accordingly, the examples are not be construed as limiting the scope of the embodiments herein.

[0046] As is existing in the field, embodiments are described and illustrated in terms of blocks that carry out a described function or functions.　These blocks, which referred to herein as managers, units, modules, hardware components or the like, are physically implemented by analog and / or digital circuits such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits and the like, and optionally be driven by firmware and software. The circuits, for example, be embodied in one or more semiconductor chips, or on substrate supports such as printed circuit boards and the like.　The circuits constituting a block be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuitry), or by a combination of dedicated hardware to perform some functions of the block and a processor to perform other functions of the block. Each block of the embodiments be physically separated into two or more interacting and discrete blocks without departing from the scope of the proposed method.　Likewise, the blocks of the embodiments be physically combined into more complex blocks without departing from the scope of the proposed method.

[0047] The accompanying drawings are used to help easily understand various technical features and it is understood that the embodiments presented herein are not limited by the accompanying drawings. As such, the embodiment of the disclosure is construed to extend to any alterations, equivalents and substitutes in addition to those which are particularly set out in the accompanying drawings. Although the terms first, second, etc. used herein to describe various elements, these elements are not be limited by these terms. These terms are generally used to distinguish one element from another.

[0048] Throughout the disclosure, the expression "at least one of a, b or c" indicates only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0049] Advantages and features of the disclosure and methods of accomplishing the same will be more readily appreciated by referring to the following description of embodiments of the disclosure and the accompanying drawings. However, the disclosure should not be construed as being limited to embodiments of the disclosure set forth below but may be embodied in many different forms. Rather, the present embodiments of the disclosure are provided so that the disclosure will be thorough and complete and will fully convey the concept of the disclosure to those of ordinary skill in the art, and the disclosure will only be defined by the appended claims.

[0050] The terms used in the disclosure are general terms currently widely used in the art based on functions described in the disclosure, but may be changed according to an intention of a technician engaged in the art, precedent cases, advent of new technologies, etc. Furthermore, specific terms may be arbitrarily selected by the applicant, and in this case, the meaning of the selected terms will be described in detail in the detailed description of the disclosure. Thus, the terms used herein should be defined not by simple appellations thereof but based on the meaning of the terms together with the overall description of the disclosure.

[0051] Terms used in the disclosure are only used to describe a particular embodiment of the disclosure and are not intended to limit the disclosure. Singular expressions used herein are intended to include plural expressions as well unless the context clearly indicates otherwise. All the terms used herein, which include technical or scientific terms, may have the same meaning that is generally understood by a person of ordinary skill in the art. Furthermore, although the terms including an ordinal number such as "first", "second", etc. may be used herein to describe various elements or components, these elements or components should not be limited by the terms. The terms are only used to distinguish one element or component from another element or component.

[0052] The use of the terms "the" and similar referents used in the specification, especially in the following claims, are to be construed to cover both the singular and the plural. Furthermore, operations of a method according to the disclosure described herein may be performed in any suitable order unless clearly specified herein. The disclosure is not limited to the described order of the operations.

[0053] Expressions such as "in some embodiments of the disclosure" or "in an embodiment of the disclosure" described in various parts of this specification do not necessarily refer to the same embodiment(s) of the disclosure.

[0054] Some embodiments of the disclosure may be described in terms of functional block components and various processing operations. Some or all of such functional blocks may be implemented by any number of hardware and / or software components that execute specific functions. For example, functional blocks of the disclosure may be implemented by one or more microprocessors or by circuit components for performing predetermined functions. Furthermore, functional blocks according to the disclosure may be implemented with various programming or scripting languages. The functional blocks may be implemented using various algorithms executed on one or more processors. Furthermore, the disclosure may employ techniques of the related art for electronics configuration, signal processing, and / or data processing. The terms such as "mechanism", "element", "means", and "construction" may be used in a broad sense and are not limited to mechanical or physical components.

[0055] Furthermore, connecting lines or connectors shown in various figures are intended to represent exemplary functional connections and / or physical or logical couplings between components in the figures. In an actual device, connections between components may be represented by many alternative or additional functional relationships, physical connections, or logical connections.

[0056] Throughout the specification, when a part "includes" or "comprises" an element, unless there is a particular description contrary thereto, it is understood that the part may further include other elements, not excluding the other elements. As used herein, the term "unit" denotes a software element or a hardware element such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC), and performs certain functions. However, the term "unit" is not limited to software or hardware. The "unit" may be formed so as to be in an addressable storage medium, or may be formed so as to operate one or more processors. Thus, in an example, the term "unit" may include elements (e.g., software elements, object-oriented software elements, class elements, and task elements), processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, micro-codes, circuits, data, a database, data structures, tables, arrays, or variables. Functions provided by the elements and "units" may be combined into the smaller number of elements and "units", or may be divided into additional elements and "units".

[0057] An embodiment of the disclosure will now be described more fully hereinafter with reference to the accompanying drawings so that it may be easily implemented by one of ordinary skill in the art. In addition, in the drawings, parts not related to descriptions of the disclosure are omitted to clearly explain the disclosure.

[0058] Figure 1 is the example of the pipeline for adding a person to the source image using the image editing application. Meanwhile, the source image may be referred to as a candidate image, or input image. At step 1-add or upload the image that represent a scene with an object or person (102) (Ex-person, object animal...etc) in the image editing application. At step 2- manually mark the background and foreground areas (103 and 104) to distinguish the person from the surrounding area. At step 3 - create a mask (105) with manually adjusting the noise and channels. At step 4- superimpose the mask (105) to the source image (106) (the term(s) 'source image' or 'input image' are interchangeably mentioned as 'candidate image') and add the person in to the scene or image. This pipeline highlights the manual effort and precision required for task like object addition in the image editing. Since the task requires significant human effort, ensuring the dataset covers a wide range of variations and scenarios becomes challenging. Although methods exist to make local edits to the image, they are not fully automatic because they require some form of human intervention (such as specifying object masks, tuning edit parameters etc.).

[0059] Text to image refers to a technology where an artificial intelligence system generates the image based on a textual description. Existing text to image models utilizes the textual description as an input instruction to generate the image as an output. The Text to image model highly relies on a large data set like LAION corpus.

[0060] Further existing end2end models, such as instructPix2Pix, generate edit images directly from instructions. However, the method generates the dataset where the source and edited images are synthetically generated using a diffusion model, thus lacking realism. Additionally, methods like PaintByInpaint & InstInpaint propose pipelines for adding and removing respectively. However, these method fails to generate a comprehensive edit dataset with a diverse instruction set.

[0061] Existing methods require a combination of source images, scene graphs, and bounding boxes to generate the desired output. This limits the scope of the generated images and makes the process of image generation complex and time-consuming.

[0062] Figure 2A is the block diagram illustrating the operation of the electronic device according to the embodiment disclosed herein. The electronic device (201) may include at least one of at least one processor (202), at least one I / O interface (203), at least one memory (204), and at least one Generative AI-based editing Controller (205).

[0063] For example, the electronic device (201) can include, but is not limited to, a mobile phone, a smartphone, tablets, laptops, Internet of Things (IoT) devices. Further, the processor (202) of the electronic device (201) communicates with the memory (204), the I / O interface (203), and the generative AI-based editing Controller (205). The processor (202) is configured to execute instructions stored in the memory (204) and to perform various processes. The processor (202) can include one or a plurality of processors, can be a general-purpose processor such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), an artificial intelligence (AI) dedicated processor such as a neural processing unit (NPU), and / or processing circuitry. For convenience of explanation, the processor (202) is expressed singularly, but there may be more than one processor.

[0064] In an embodiment of the disclosure, further, the memory (204) of the electronic device (201) includes storage locations to be addressable through the processor (202). The memory (204) is not limited to a volatile memory and / or a non-volatile memory. Further, the memory (204) can include one or more computer-readable storage media. The memory (204) can include non-volatile storage elements. For example, non-volatile storage elements can include magnetic hard disks, optical disks, floppy disks, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. The memory (204) stores the candidate image and the edited images. For convenience of explanation, the memory (204) is expressed singularly, but there may be more than one memory. And, the memory may include instructions, when executed by the at least one processor individually or collectively, cause the electronic device. And, the memory may include at least one candidate image.

[0065] In an embodiment of the disclosure, the I / O interface (203) transmits the information between the memory (204), electronic device (201), and external peripheral devices. The peripheral devices are the input-output devices associated with the electronic device (201). The generative AI-based editing controller (205) communicates with the I / O interface (203) and memory (204) for enhancing captured images by using an imaging sensor. The generative AI-based editing controller (205) is an innovative hardware that is realized through the physical implementation of both analog and digital circuits, including logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive and active electronic components, as well as optical components. Also, generative AI-based editing controller (205) is realized through the physical implementation of both analog and digital circuits, including logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive and active electronic components, as well as optical components.

[0066] In an embodiment of the disclosure, the generative AI-based editing controller (205) is configured to provide the input into the AI model. The input includes minimum one candidate image (106) and minimum one editing operation to be performed by the AI model on the minimum one candidate image (106). The generative AI-based editing controller (205) is configured to input the base prompt into the AI model. The base prompt includes base instructions for detection of minimum one object within the minimum one candidate image (106). The generative AI-based editing controller (205) is configured to receive the control prompt from a user. The control prompt includes control instructions relevant to the minimum one edition operation to be performed by the AI model on minimum one candidate image (106). The generative AI-based editing controller (205) is configured to generate the image editing dataset based on the base prompt and the control prompt. The editing operation may include at least one of an add operation, a remove operation, a rotate operation, a blend operation, a flip operation, a harmonize operation, a state change operation, and an object attribute change operation to change attributes of the at least one object within the candidate image.

[0067] In an embodiment of the disclosure, the electronic device in the generative AI-based editing controller (205) configured to generate the image editing dataset based on the base prompt and the control prompt. The generative AI-based editing controller (205) configured to append the control prompt to the base prompt. The generative AI-based editing controller (205) generates at least one edited image (214) by editing the candidate image (106) using the control prompt appended to the base prompt using the AI model. The generative AI-based editing controller (205) configured to generate an edit instruction (216) by comparing the candidate image (106), the edited image (214) and depth map of the candidate image (106). The generative AI-based editing controller (205) configured to generate the image editing dataset as an output. The image editing dataset includes the at least one edited image (214) along with the edit instruction (216).

[0068] Meanwhile, the operations performed by the generative AI-based editing controller (205) can be performed by at least one processor.

[0069] Meanwhile, in an embodiment of the disclosure, operations performed in blocks of the drawings of the disclosure, such as an encoder and a model, which are omitted for convenience of explanation, may be performed by an electronic device or at least one processor.

[0070] In an embodiment of the disclosure, the edit instructions (216) in the electronic device (201) are the natural language instruction that describes the editing operation.

[0071] In an embodiment of the disclosure, the control instructions in the electronic device (201) includes the first control instruction relevant to first edit operation to be performed by the AI model on the candidate image (106) in an edit pipeline (213). The second control instruction in the electronic device (201) relevant to a second edit operation to be performed by the AI model on the candidate image (106) in the edit pipeline (213). The second edit operation is different from the first edit operation. The first edit operation and the second edit operation includes an add operation, a remove operation, a rotate operation, a blend operation, a flip operation, a harmonize operation, a state change operation, and the object attribute change operation to change attributes of the object.

[0072] In an embodiment of the disclosure, the generative AI-based editing controller (205) configured to edit the object in the candidate image (106) based on the base prompt appended to the control prompts. The generative AI-based editing controller (205) configured to detect the objects in the candidate image (106) using the base prompt appended to the control prompt. The granularity of detection of the object in the candidate image (106) is controlled based on the control prompt. The generative AI-based editing controller (205) configured to assign the labels and bounding boxes to the identified object based on the base prompt and the control prompt. The generative AI-based editing controller (205) configured to　output the edited image (214) with the labels for processing in an edit pipeline (213).

[0073] In an embodiment of the disclosure, the generative AI-based editing controller (205) in the electronic device (201) detects the object in the candidate image (106) using the base prompt and the control prompt. The generative AI-based editing controller (205) configured to extract features from the candidate image (106) using a pre-trained image encoder (231). The generative AI-based editing controller (205) configured to employ a pre-trained large language model (LLM) (233) to process the extracted features and the base prompts. The generative AI-based editing controller (205) configured to integrate the trainable detection scale guidance adapter to guide the LLM (233) in detecting the objects in the candidate image in accordance with the control prompts. The generative AI-based editing controller (205) configured to arrange the object labels and the bounding boxes in a sequence from left to right as they appear in the candidate image (106).

[0074] In an embodiment of the disclosure, the generative AI-based editing controller (205) in the electronic device (201) generates the edit instruction (216) by comparing the candidate image (106), the edited image (214) and depth map of the candidate image (106). The generative AI-based editing controller (205) is configured to obtain the depth map corresponding to the candidate image (106). The depth map includes spatial relationships information's among the object in the candidate image (106). The generative AI-based editing controller (205) configured to generate the edit instruction (216) by comparing attributes of the edit region with respect to the object within the candidate image (106) by using the depth map of the candidate image (106).

[0075] In an embodiment of the disclosure, the AI model in the electronic device (201) is a controlled multi-object detection visual language model (CMOD VLM). The generative AI-based editing controller (205) in the electronic device (201) The attributes in the generative AI-based editing controller (205) may include at least one object comprises at least one of a colour of the at least one object, a size of the at least one object, a shape of the at least one object, a style of the at least one object, an orientation of the at least one object, and a physical displacement of the at least one object.

[0076] In an embodiment of the disclosure, the generative AI-based editing controller (205) in the electronic device (201) generates the image editing dataset for generative-AI based editing application. The generative AI-based editing controller (205) in the electronic device (201) provides the input into the AI model. The input includes the candidate image (106) and the editing operation to be performed by the AI model on the candidate image (106). The generative AI-based editing controller (205) in the electronic device (201) inputs the base prompt into the AI model. The base prompt includes base instructions for detection of the object within the candidate image (106). The generative AI-based editing controller (205) in the electronic device (201) receives the control prompt from the user. The control prompt includes control instructions relevant to the edition operation to be performed by the AI model on the at least one candidate image (106). The generative AI-based editing controller (205) in the electronic device (201) generates the image editing dataset based on the edited image (214), the edit instruction (216), the base prompt and the control prompt.

[0077] In an embodiment of the disclosure, the generative AI-based editing controller (205) in the electronic device (201) generates the image editing dataset based on the base prompt and the control prompt. The generative AI-based editing controller (205) in the electronic device (201) appends the control prompt to the base prompt. The generative AI-based editing controller (205) generates the edited image (214) by editing the candidate image (106) using the control prompt appended to the base prompt using the AI model. The generative AI-based editing controller (205) generates the edit instruction (216) by comparing the candidate image (106), the edited image (214) and the depth map of the at least one candidate image (106). The generative AI-based editing controller (205) generates the image editing dataset as the output. The image editing dataset includes the edited image (214) along with the edit instruction (216).

[0078] In an embodiment of the disclosure, the edit instructions (216) in the electronic device (201) are natural language instruction that describes the at least one editing operation.

[0079] In an embodiment of the disclosure, the control instructions in the electronic device (201) involves the first control instruction relevant to a first edit operation to be performed by the AI model on the candidate image (106) in the edit pipeline (213). The control instructions in the electronic device (201) further involves the second control instruction relevant to the second edit operation to be performed by the AI model on the candidate image (106) in the edit pipeline (213).The second edit operation is different from the first edit operation. The first edit operation and the second edit operation includes the add operation, the remove operation, the rotate operation, the blend operation, the flip operation, the harmonize operation, the state change operation, and the object attribute change operation to change attributes of the object.

[0080] In an embodiment of the disclosure, the generative AI-based editing controller (205) in the electronic device (201) edits the candidate image (106) based on the base prompt appended to the control prompt. The generative AI-based editing controller (205) in the electronic device (201) detects the object in the candidate image (106) using the control prompt appended to the control prompt. The granularity of detection of the object in the candidate image (106) is controlled based on the control prompt. The generative AI-based editing controller (205) in the electronic device (201) assigns the labels and bounding boxes to the identified object based on the base prompt and the control prompt. The generative AI-based editing controller (205) in the electronic device (201) outputs the edited image (214) with the labels for processing in the edit pipeline (213).

[0081] In an embodiment of the disclosure, the generative AI-based editing controller (205) in the electronic device (201) detects object in the candidate image (106) using the base prompt and the control prompt. The generative AI-based editing controller (205) in the electronic device (201) extracts the features from the candidate image (106) using a pre-trained image encoder (231). The generative AI-based editing controller (205) in the electronic device (201) employs the pre-trained large language model (LLM) (233) to process the extracted features and the base prompt. The generative AI-based editing controller (205) in the electronic device (201) integrates the trainable detection scale guidance adapter to guide the LLM (233) in detecting the object in the candidate image (106) in accordance with the control prompt. The generative AI-based editing controller (205) in the electronic device (201) arranges the object labels and the bounding boxes in the sequence from left to right as they appear in the at least one candidate image (106).

[0082] In an embodiment of the disclosure, the generative AI-based editing controller (205) in the electronic device (201) generates the edit instruction (216) by comparing the candidate image (106), the edited image (214) and one depth map of the candidate image (106). The generative AI-based editing controller (205) in the electronic device (201) identifies the attributes of edit region from the candidate image (106). The generative AI-based editing controller (205) in the electronic device (201) obtains the depth map corresponding to the candidate image (106). The depth map includes spatial relationship information among the one object in the candidate image (106). The generative AI-based editing controller (205) in the electronic device (201). The generative AI-based editing controller (205) in the electronic device (201) generates the edit instruction (216) by comparing attributes of the edit region with respect to the object within the candidate image (106) by using the depth map of the candidate image (106).

[0083] In an embodiment of the disclosure, the AI model in the generative AI-based editing controller (205) is a controlled multi-object detection visual language model (CMOD VLM) (212).

[0084] Figure 2B is the block diagram that illustrates the method for automatic high quality data generation for image editing applications. The generative AI-based editing controller (205) includes the candidate image (106), a CMOD-VLM model (212), an edit image generation pipeline (213), an edit image (214) model, a visual edit aware instruction generator (215), an edit instruction (216), a source caption model (217), an edit caption (218), an edit data validator (219) and an image editing dataset (220).

[0085] In an embodiment of the disclosure, the input is received from the data source and the input includes candidate image (106) and task. The candidate image (106) along with the base and control prompts are feed to the CMOD-VLM model (212). The task identifies the type of data to be generated (e.g., removal, addition, or editing of elements within an image). The task can indicate information for operations to be performed on an image editing dataset, which are determined before running the pipeline.

[0086] The electronic device may obtain a candidate image for inputting an AI (artificial intelligence) model from among the at least one candidate image (106), wherein the AI model is generative AI model for generating the image editing dataset. The electronic device may determine an editing operation to be performed by the AI model on the candidate image from among at least one editing operation.

[0087] In an embodiment of the disclosure, the task may include the editing operation (e.g., removal, addition, or editing of at least one object within an image). The electronic device may obtain obtaining a candidate image for inputting an AI (artificial intelligence) model from among at least one candidate image, wherein the AI model is generative AI model for generating the image editing dataset. The electronic device may determine a candidate image from at least one candidate image stored in memory or may receive input from a user.

[0088] The electronic device may determine an editing operation to be performed by the AI model on the candidate image from among at least one editing operation. The electronic device may obtain tasks, such as at least one editing operation, from the user, or the electronic device may determine them randomly or adaptively.

[0089] In an embodiment of the disclosure, the CMOD-VLM model (212) receives the candidate image (106) as well as plurality of prompts. The CMOD-VLM model (212) analyses the input based the plurality of prompts. The plurality of prompts involves the base prompt and the controlled prompt. The CMOD-VLM model (212) identifies and detects all objects present in the candidate image (106) based on the base and control prompts and generates a corresponding detection result.

[0090] In an embodiment of the disclosure, the plurality of prompts involves the base prompt and the controlled prompt. The base prompt provides a general understanding of the image and helps to detect objects. The controlled prompt specifies conditions or constraints for the given task (e.g., remove a specific object, add a new one, etc.). In addition, by utilizing the control prompts, the CMOD-VLM model (212) specifies additional constraints / grouping criteria for fine / coarse grained object detection. At least one of the base prompt and the control prompt can be determined based on the candidate image and editing operation. It is not required that at least one of the base prompt and the control prompt be obtained directly from the user. The base prompt may include base instructions for detection of at least one object within the candidate image, and the control prompt may include control instructions relevant to the editing operation to be performed by the AI model on the candidate image (106). The electronic device may determine a base prompt and a control prompt based on inputting the candidate image and the editing operation into the AI model.

[0091] In an embodiment of the disclosure, the base prompt may include a general directive, such as "Detect all objects in an image". The base prompt may establish a foundational understanding of the candidate image. And, the base prompt may govern to which objects the control prompt applies to. The control prompt may include the instruction obtained or determined from a predefined set of rules to guide the detections for the task (e.g., the edit operation). The electronic device may determine which rules are applicable to the candidate image. The rules need to be defined only once before the start of the dataset generation process for the task. If the task is the object removal operation, the rules are defined as all related objects could be removed.

[0092] In case that an object is detected for removal, any object resting on it should also be detected. In prior arts, although the command is "remove the dining table", sometimes, the model could not detect plates, cutlery and table cloth resting on the dining table. The control prompt may ensure that the removal is holistic, preventing floating or leftover objects in the inpainting operation.

[0093] In an embodiment of the disclosure, the control instructions relevant to the editing operation to be performed by the AI model on the candidate image include detection guidance for object grouping to control a granularity of a detection. In an embodiment of the disclosure, the electronic device may generate the image editing dataset based on the base prompt and the control prompt.

[0094] In an embodiment of the disclosure, the edit image generation pipeline (213) performs the edit operation based on the task and provides the edited images (214) as the output. The edit image generation pipeline (213) includes several sub edit pipeline such as add or remove pipeline (213a), an harmonization pipeline (213b), an attribute or state change pipeline (213c) and an object completion pipeline (213d). For example for adding or removing the objects, the edit image generation pipeline (213) performs object inpainting or removal pipeline, results in the edited image (214).

[0095] In an embodiment of the disclosure, further, the visual edit aware instruction generator (215) receives the edited image (214) as well as the candidate image (106) and a depth map of the candidate image (106). The visual edit aware instruction generator (215) receives the edited image (214), and the candidate image. Based on the edited image (214), and the candidate image (106), the visual edit aware instruction generator (215) generates the edit instruction (216) or natural language instructions for the edit image (214) generated using the edit image generation pipeline (213). The edit instruction (216) provides textual description of the editing task. Additionally, the visual edit aware instruction generator (215) responsible for generating natural language instructions and descriptions of the image editing process for the edited image (214). The electronic device may generate a depth map for the depth information of the at least one object within the candidate image. The electronic device may generate an intermediate feature map for an edited region obtained by comparing the candidate image (106), and the edited image (214). The electronic device may generate the edit instruction based on the depth map for the depth information of the at least one object within the candidate image, and the intermediate feature map for the edited region.

[0096] In an embodiment of the disclosure, the source caption model (217) includes a source caption generator (217a) and an edit caption generator (217b). The source caption generator (217a) receives the candidate image (106) and generates a natural language caption that describes the content of candidate image (106). The source caption generator (217a) analyzes the visual and contextual features of the candidate image (106) and produces a detailed and accurate textual description of its elements, such as objects, scenes, and attributes.

[0097] In an embodiment of the disclosure, the edit caption generator (217b) compares the edited instruction (216), the candidate image (106) and source caption. Based on the comparison the edit caption generator (217b) produces the edit caption (218), which provides the natural language caption of specific changes made in the image.

[0098] In an embodiment of the disclosure, the electronic device may determine whether to validate the image editing dataset by performing count based filtering or semantic filtering.

[0099] In an embodiment of the disclosure, the semantic filtering is performed by the edit data validator (219). The edit data validator (219) combines and evaluates the edited image (214) and the edit caption (218) using a semantic matching technique and assign a score that measure how accurately the caption describes the visual edit. Based on the score, the sample dataset is accepted and rejected. If the score meets the threshold value, a sample is added to the dataset. The count based filtering is performed by the edit data validator (219). The edit data validator (219) is designed to tally the counts of the edited object in both the candidate image (106) and the edited image (214) to ensure consistency. Regarding the count based filtering, it is explained in more detail in Figure. 15.

[0100] In an embodiment of the disclosure, the generated image editing dataset (220) is a synthetic image dataset. Additionally, the generated sample dataset may be used to train the image editing model that may power image editing applications in gallery. User may achieve edit operations such as add, remove, object paste and state change through precise natural language interactions. For example- instructions may include 'remove the man wearing red shirt' or 'add a potted plant to the right of the laptop' or 'flip the painting on the wall horizontally' or 'change the colour of white plate to black'.

[0101] In an embodiment of the disclosure, a series of candidate images (106), the task, the base prompt, and the control prompt, may be provided to the CMOD-VLM model (212). The CMOD-VLM model (212) then performs the edit operations on the candidate images (106b, 106c,..106n) based on the task, first and second base prompts, and control prompts to generate a series of edited images.

[0102] For example, the first control instruction of control prompt is relevant to the editing operation to be performed by the AI model on the candidate image (106) in the edit pipeline (213). The first edit operation may be add or remove edit operation, the object paste or state change edit operation, the object completion edit operation or harmonization edit operation. The edited operation is performed on a first candidate image to obtain a first edit image.

[0103] In an embodiment of the disclosure, similarly, the second control instruction is relevant to the second edit operation to be performed by the AI model on the candidate image (106) in the edit pipeline (213). The second edit operation may be add or remove edit operation, the object paste or state change edit operation, the object completion edit operation or Harmonization edit operation. The edited operation is performed on a second candidate image to obtain a second edit image.

[0104] The Figure 2C is a flow chart that illustrates the method for performing the controlled multi-object detection in the candidate image (106). The CMOD-VLM model (212) used to detect multiple objects in the candidate image (106) using the base prompt and control prompt (235). Through the control prompts, one can specify additional constraints / grouping criteria for fine / coarse grained object detection.

[0105] In an embodiment of the disclosure, the CMOD-VLM model (212) consists of a pre-trained image encoder (231), a detection scale guidance adapter (232) and a pre-trained Large Language Model or LLM (233).The pre-trained image encoder (231) receives the candidate image (106) as source or input image and generates depth information from the candidate image (106) to enable depth-aware instruction generation. The detection scale guidance adapter (232) extracts relevant features from the candidate image (106) based on the provided base prompt and control prompt. The base prompt specifies what objects to detect and the control prompt specifies object grouping criteria based on some prior knowledge.

[0106] For example, the base prompt is "detect all the object in the candidate image" and the control prompt is "while detecting plate, detect plate and food as one object".

[0107] In an embodiment of the disclosure, further, the detection scale guidance adapter (604) guides the LLM (233) to detect the objects according to the provided control prompt. The LLM (233) is trained using the next-token prediction objective, typical in the LLMs (233), where the target consists of object labels along with their bounding box coordinates, arranged from left to right as they appear in the candidate image (106). The electronic device may detect the at least one object to be edited from among the at least one object within the candidate image (106) by the control prompt.

[0108] In an embodiment of the disclosure, the electronic device may detect the at least one object to be edited from among the at least one object within the candidate image. The electronic device may extract features of the least one object from the at least one object within the candidate image based on the base prompt and the control prompt. The electronic device may determine an object label with a bounding box by inputting the extracted features into a pre-trained large language model (LLM) (233).

[0109] In an embodiment of the disclosure, the LLM (233) provide output detection including identifying all the objects present in the candidate image (106) along with the labels, co-ordinates and bounding boxes, in accordance to specified control. For example, in the candidate image (106), the LLM (233) detects and identifies a candle (234a), a plate (234b) and jug (234c).Further, generated the labels along with the four coordinates (Eg- <x11>, <y11>, <x12> , <y12>) that specify the region of the candle, which forms the bounding box.

[0110] Figure 2D is the block diagram that illustrates the method for performing detection scale guidance based on the control prompt.

[0111] In an embodiment of the disclosure, the detection scale guidance adapter (312) provides the detection scale guidance based on the control prompt.

[0112] In an embodiment of the disclosure, a text encoder (238) in the detection scale guidance adapter (604) receives both base prompt and control prompt as a text input (237).The text encoder (238) extracts the relevant features from image patches (240) that align with the base prompt and control prompt. The beauty of the detection scale guidance adapter (312) lies in its ability to align the image features with both the base prompt and control prompt effectively.

[0113] In an embodiment of the disclosure, the detection scale guidance adapter (312) integrates with the LLM (233) and a Swin Transformer block (239) for object detection. The Swin-Transformer block (239) used to guide the detections based on the control prompt. Specifically, the control prompt influences the representation of detected objects through a window-based cross attention (W-CA) (241b) and shifted window based cross attention layers (SW-CA) (242b). The W-CA (241b) and SW-CA (242b) differs in the way the feature maps are partitioned.

[0114] An embodiment discloses a combination of W-MSA (241a) (Window-Based Multi-Head Self-Attention) and the W-CA (241b) and a combination of shifted Window-Based Multi-Head Self-Attention (SW-MSA) (242a) and SW-CA (242b). This process involves dividing the image(s) into multiple patches of varying scales and processing them through the W-CA (241b) and W-MSA (241a).

[0115] In an embodiment of the disclosure, further, the W-CA (241b) enable the interaction not only among the multiple patches themselves but also with guidance prompts provided by the base and control prompt. The W-MSA (241a) enhances the self-attention mechanism within the multiple patches.

[0116] In an embodiment of the disclosure, after processing, the patches are merged (706) to form a composite image. This process can be repeated 'n' times (Xn) to scale up the model.

[0117] In an embodiment of the disclosure, further, the Swin-Transformer block (239) is utilized to guide the detections based on the control prompt. Specifically, the control prompt influences the representation of detected objects through the W-CA (241b) and the SW-CA (242b). The W-CA (241b) and SW-CA (242b) differs in the way the feature maps are partitioned. In a Window based partitioning, the images are evenly partitioned into patches. However, the Window based partitioning lacks connections across windows. The Shifted window based approach solves this problem by displacing the previous window configuration by a certain amount. Let denote the image features input after partitioning to a W-CA (704b) block, and let denote a text representation then the cross attention between the two is calculated as follows:

[0118] , ,

[0119]

[0120] In an embodiment of the disclosure, , , and are Weight matrices for Query, Key, and Value, respectively. And, , , and are query vector, key vector, and value vector used for cross attention calculation. Let may denote the function of the cross attention obtained , as inputs. And, let may denote the softmax function for normalization obtained as a input. The denotes the number of the embedding dimension. The may be bias matrix or positional bias.

[0121] And, the patches from the SW-CA (242b) are merging by the patch merging layer (243).

[0122] Figure 2E is the block diagram that illustrates the method for generating the natural language instruction for the edit image, according to the embodiment disclosed herein.

[0123] In an embodiment of the disclosure, the visual edit aware instruction generator (215) generates the natural language instruction for the edit image (214) using the edit image generation pipeline (213).

[0124] In an embodiment of the disclosure, the visual edit aware instruction generator (215) is trained to generate the natural language instruction. Further, the visual edit aware instruction generator (215) consists of a pre-trained Depth Map Encoder (250), the pre-trained LLM (605) and a Edit Region Identification model (251), a depth fusion model (252) and a Task Aware Query former (253). The visual edit aware instruction generator (215) model is trained to generate precise instruction for the edit on an annotated dataset for the source or the candidate image (106), the edit image (214) and the edit instructions (216).

[0125] In an embodiment of the disclosure, the visual edit aware instruction generator (215) generates the natural language instructions for applying the edits to the candidate image (106). The natural language instructions are utilized to identify objects within the candidate image (106) and are produced in the user's preferred style. For example, if the user chosen style is spatial, the visual edit aware instruction generator (215) model determines the effective or best way to reference the edit region based on its spatial location.

[0126] In an embodiment of the disclosure, the visual edit aware instruction generator (215) generates a concise and descriptive output instruction that describes the edit operation based on the task and style input provided by the user. The tasks includes add, remove, flip, rotate, blend, and complete, enabling the Visual Edit Aware Instruction Generator (215) model to cover a wide range of image-editing operations with precise natural language instructions.

[0127] In Figure 2E, the candidate image (106) and the edit image (214) are provided to the edit region identification model (251).

[0128] In an embodiment of the disclosure, the electronic device may generate a depth map for the depth information of the at least one object within the candidate image. The electronic device may generate an intermediate feature map for an edited region obtained by comparing the candidate image (106), and the edited image (214). The electronic device may generate the edit instruction based on the depth map for the depth information of the at least one object within the candidate image, and the intermediate feature map for the edited region. The edit instruction (216) are natural language instruction that describes the editing operation.

[0129] In an embodiment of the disclosure, the electronic device may generate the image editing dataset based on the base prompt and the control prompt. The electronic device may generate an edited image (214) by editing the candidate image (106) based on the base prompt and control prompt. The electronic device may generate an edit instruction (216) by comparing the candidate image (106), and the edited image (214). The electronic device may generate the image editing dataset as an output, wherein the image editing dataset comprises the candidate image (106), the edited image (214), and the edit instruction (216). The electronic device may generate a depth map for the depth information of the at least one object within the candidate image. The electronic device may generate an intermediate feature map for an edited region obtained by comparing the candidate image (106), and the edited image (214). The electronic device may generate the edit instruction based on the depth map for the depth information of the at least one object within the candidate image, and the intermediate feature map for the edited region.

[0130] In an embodiment of the disclosure, the edit region identification model (251) identifies the edited region in both the candidate image (106) and the edit image (214). The edit region identification model (251) highlights the modified regions. For example, in Figure 2E, the edit region identification model (251) identifies the edited region (255b) "a man wearing red shirt" and highlight edited region (255b).

[0131] In an embodiment of the disclosure, the Depth map encoder (250) receives the candidate image (106). And, the Depth map encoder (250) generates a depth map (255a) indicating the depth of objects in the scene. Further, the depth fusion (252) model fuses the depth map (255a) from the depth map encoder (250) and the intermediate feature map from the edit region identification model (251). The depth fusion (252) model creates a unified representation (255c) of depth and image features. The task encoder (254) process the task and style specified by the user. The task encoder (254) guides the instruction generation based on the provided task and style. For example, user provides the task "remove" and style as 'spatial'. In an embodiment of the disclosure, the task aware query former (253) consider the type of the task needed to generate instructions and the desired instruction type. The task aware query former (253) analyses the candidate image (106), the edit image (214), and outputs suitable instructions. The language decoder (255) process the NLP instruction and generate the edit instruction (216) as a sentence or in a specific format (256).

[0132] In an embodiment of the disclosure, for example, the edit region identification model (251) identifies and highlights the edited region). Further, user provides a task as "remove" and style as spatial. The task aware query former (253) provides the natural language instruction based on the user inputs, specifying the task as "remove" and the style as "spatial". The language decoder (255) generates the edit instruction (216) output as "remove the person at front".

[0133] Figure 2F is the block diagram that illustrates the method for generating the depth map using the depth map encoder (250). The depth map encoder (250) provides the depth information essential to generate depth aware instructions. The depth aware instruction may be given as "Remove <obj1> in front of <obj2>".

[0134] In an embodiment of the disclosure, the depth map encoder (250) process the source or candidate image (106). And, the depth map encoder (250) generates the depth map (255a) as output. The depth map (255a) indicates the depth of objects in the scene in a z-direction with reference to the viewpoint. In an embodiment, the depth map encoder (250) may utilize any pre-trained depth map generation model such as ZoeDepth to produce high-quality depth maps.

[0135] Figure 2G and Figure 2H are block diagrams that illustrates the method of identifying the edit region and the architecture of edit region identification model (251). In an embodiment of the disclosure, the electronic device may identify the edit region by comparing the candidate image and the edited image. And the electronic device may generate an intermediate feature map for an edited region obtained by comparing the candidate image (106), and the edited image (214) by the edit region identification model (251).

[0136] Figure 2G illustrates the method of identifying the edit region using the edit region identification model (251).

[0137] In an embodiment of the disclosure, the edit region identification model (251) process and compares the candidate image (106) and the edited image (214) to identify the modifications. Based on the comparison, the edit region identification model (251) generates an intermediate feature map that indicates the edited region (255b).

[0138] Figure 2H illustrates the architecture of edit region identification model (251). In an embodiment of the disclosure, the architecture utilizes an encoder-decoder framework with multiple functional blocks, that work together to identify modified regions between the candidate image (106) and the edited image (214). An Encoder (901a-901d) is based on either convolutional layers or transformer architectures. The encoder processes both the candidate image (106) and the edited image (214). The encoder (901a-901d) breaks down the images into features and patterns, such as color changes or object alterations. A down block (903a-903n) is the block including down sampling layers (902). An up block (906a-906n) is the block including up sampling layers (908).

[0139] In an embodiment of the disclosure, further, a down sampling block (902) reduces the size of the images while focusing on important details, which helps the edit region identification model (251) to identify significant differences between the candidate image (106) and the edited image (214). The skip connections (904a-904n) links each encoder (901a-901d) layer to its corresponding Up Block (906a-906n) in the decoder. The skip connections (904a-904n) ensure that fine-grained details are preserved throughout the process. The decoder progressively up samples the feature maps, combining the skip features from the encoder (901a-901d) and reconstructed information to restore the spatial resolution. Finally, the output convolution layer (905) generates a high-resolution mask. The generated mask highlights the areas that have been edited, display the probability that each pixel is part of the modified region.

[0140] Figure 2I is the flowchart that illustrates the working method of the depth fusion (252) and the task-aware query former (253), according to an embodiment. The depth fusion model (252) and the task aware query former (253) works together to generate task-specific instructions (1001) by integrating depth, image, and edit region features. The depth fusion block (252) processes the depth, image, and edit region features, convert them into patches and create a unified representation that combines spatial and contextual information.

[0141] In an embodiment of the disclosure, the task aware query former (253) (QFormer) includes learnable queries (1010) that extract relevant information for the task from the fused features and align it to the LLM's input space. For example, if the task is 'remove' and the task prompt mentions the style as 'spatial', then depth level features will be given more importance to generate a suitable 'spatial' instruction for the edit. In the same way, if the style is attribute, then the task aware query former model (806) may extracts relevant attribute level features to describe the edit. The LLM (233) then generates the final instruction from the relevant features provided by the Task Aware Query Former (253).

[0142] In an embodiment of the disclosure, further, the depth fusion model (252) integrates the depth, image, and edit region features to create the unified representation through a series of processing steps. The Depth Fusion module (252) configure a bidirectional self-attention (1002), a layer normalization (1003, 1005), and a feed-forward (1004a) networks to enable effective interaction, resulting in a fused feature. The fused feature (1006) is then passed to the Task Aware Query Former (253), which further processes the data using components like a self-attention (1008), a cross-attention (1007), and feed-forward layers (1004b, 1004c). The Task Aware Query Former (253) incorporates the learnable queries (1010) that dynamically extract task-relevant information and align it with the LLM's input space. Additionally, the Depth Fusion model (252) and the Task Aware Query Former (253) integrates multimodal causal attention mechanisms (1009), enable the effective communication with self-attention layers (1011) to refine the representation. The depth fusion model (252) and the task aware query former (253) together ensures unified integration of depth, visual, and textual features for task-specific, multimodal processing.

[0143] Figure 3 is the flowchart that illustrates the working operation of the proposed invention, according to the embodiment disclosed herein.

[0144] At step 301, the electronic device (201) receives the input from the data source. The input includes minimum one candidate image (106) and the editing operation.

[0145] At step 302, the electronic device (201) transmits the input from the data source to an AI model or the CMOD-VLM (212) model to detect one or more object in the candidate image (106).

[0146] At step 303, the electronic device (201) receives a plurality of prompts into the AI model or the CMOD-VLM (212).The base and control prompts includes minimum one object to be detected within the candidate image (106) and the minimum one edit operation to be performed within the candidate image (106).

[0147] At step 304, the electronic device (201) generates the edited image (214) along with the edit instruction (216) as the image editing datasets by editing the candidate image (106) based on the base and control prompts.

[0148] Figure 4 is the example of the data used to train instruction-based image editing models, according to the embodiment of the disclosure.

[0149] In an embodiment of the disclosure, an automatic data generation pipeline (403) generates (source image or candidate image, edit instruction, edit image) tuples to train generative models like diffusion for instruction based image editing. The automatic data generation pipeline (403) supports edits that involve adding, removing and altering the state of one or more objects and completing incomplete objects in the candidate image (106) and supporting it with a precise natural language instruction. The automatic data generation pipeline (403) significantly reduces the human effort for training data generation.

[0150] In an embodiment of the disclosure, the candidate image (106) represents 'a man looking at a plate with pizza', and the given instruction is to "remove pizza from the plate". The output image (405) represents the 'man is looking at the empty plates', however, the pizza on the plate is removed. Generating such datasets requires creating tuples consisting of the source image (106), the corresponding edit instruction (216), and the edited image (214). By providing the examples, the automatic data generation pipeline (403) learns how to interpret instructions and make the necessary changes to the candidate image (106). This kind of data is important for developing the AI model that assist with tasks such as image manipulation, object removal, and scene editing. Generating accurate and different datasets helps to improve the performance and capabilities of these AI models in various editing applications.

[0151] Figure 5 is the block diagram that illustrates the working method of adding or removal operation in the edit image generation pipelines.

[0152] In an embodiment of the disclosure, add or removal pipeline (213a) is utilized to generate high-quality training pairs (input and target image) for add and removal tasks in the image editing. In this process, the edit image generation pipelines (213) are utilized to generate the edited image (214) by removing one or more objects from the given source image or the candidate image (106). The resulting pair of images (source or candidate image, edited image) is then added to the 'remove' training set, while the pair of images (edited image, source or candidate image) is added to the 'add' training set. This is because the 'add' task is essentially the reverse of the 'remove' task. In an embodiment, the edited image generated by proposed pipeline serves as the target image.

[0153] In an embodiment of the disclosure, for example, the candidate image (106) is provided to the CMOD -VLM model (212). Further, the base prompt and control prompt is provided to the CMOD -VLM model (212). Further, CMOD -VLM model (212) identifies the one or more objects (1022a-1002d) present in the candidate image (106). When the task is to "remove apple," a language and position guided segmentation (1032) identities the position and location of apples. Consequently, the pipeline automatically generates a mask (1024) that identifies the 'apple' in the candidate image (106). The add or remove pipeline (213a) then removes the 'apple' from the candidate image (106) and inpainting model (1014) region to produce the edited image (214) without the apple. This constitutes one sample in the proposed dataset, with the format (candidate image, edited image, edit instruction). Similarly, the removal operation is incorporated into the dataset using the same format, focusing on removing the specified object as instructed.

[0154] Figure 6 is the block diagram that illustrates the working method of harmonization in the edit image generation pipelines (213).

[0155] In an embodiment of the disclosure, the harmonization Pipeline (213b) is utilized to generate (input_image, target_image) training pair for the harmonization task. The source or candidate image (106) from the dataset(s) is provided to the harmonization Pipeline (213b). The harmonization Pipeline (213b) alters the color or illumination of one or more objects in the candidate image (106) and creates an edited version. This results in the training pair (edited image, source image) which is added to the harmonization training set. The purpose of harmonization Pipeline (213b) is to teach models how to modify the object properties, such as color and lighting, while maintaining visual consistency and harmony within the image. This ensures learning for tasks involving realistic adjustments to object appearance.

[0156] In an embodiment of the disclosure, the object is to generate the candidate image (106), the edited image (504) and the edited instruction (216). For example, the candidate image (106) is provided to an auto object segmentation & masking model (501). The auto object segmentation & masking model (501) are achieved using a combination of the CMOD-VLM (212) and the Language & Position Guided Segmentation (1032). And, a output of the auto object segmentation & masking model (501) is input to the Edit illumination and color model (602) When focusing on batsman in the candidate image (106) and mask (1033) is generated and adjustment are made to tweak the color and illumination of the batsman slightly. However, if the output appear unnatural, then the harmonization data set used to refine the result.

[0157] In an embodiment of the disclosure, in the harmonization dataset, the target output (harmonized version) is treated as the input, while the original candidate input (106) becomes the target output. The associated instruction is to harmonize the particular object. This processed data is then provided to the AI model to improve its ability to generate realistic, harmonized outputs for specific objects.

[0158] Figure 7 is the block diagram that illustrates the working method of change attribute / state change in the edit image generation pipelines (213).

[0159] In an embodiment of the disclosure, the attribute or state change pipeline (213c) is used to generate (input_image, target_image) training pair for the attribute or state change task. The auto object segmentation & masking model (501) are achieved using the CMOD-VLM (212). The masked image (1033a) is provided to the attribute or state change model (1042) and the inpainting model (1014). The source or candidate image (106) is given, the attribute or state change pipeline (213c) generates the edited image (214) where the attributes or state of one or more objects has been modified. Attributes includes colour, size, shape, style of the object in an image while state includes the location, orientation of the object. The (source image, edit image) tuple is added to the state change training set.

[0160] In an embodiment of the disclosure, the attribute or state change model (1042) modifies the candidate image (106) by horizontally flipping (1041d) the 'batsman'. The image blending model (1015) combines the horizontally flipping (1041d) 'batsman' image with the inpainted image (1033b) to obtain the edited image (214), where the attribute or state of 'batsman' has been modified.

[0161] Figure 8 is the block diagram that illustrates the working method of the partial data completion in the edit image generation pipelines (213).

[0162] In an embodiment of the disclosure, the object completion pipeline (213d) is utilized to create data for the partial object completion. To generate (input_image, target_image) training pairs for the partial object completion task. Starting with the source or candidate image (106), the object completion pipeline (213d) processes it to produce the edited image (213) containing a partial object. The resulting image, (edited image, source image), forms the partial data completion partial object completion training set.

[0163] For example, In the case of a partial object (e.g., the legs of a person are missing), the output (1051) image is used as the input to the AI model along with the instruction to complete the object.

[0164] Figure 9A -Figure 9C are block diagrams that illustrates the working operation of attribute or state change model, image blending and edit illumination model.

[0165] Figure 9A illustrated the working operation of the attribute or state change model (1042).

[0166] In an embodiment of the disclosure, he attribute or state change model (1042) receives the candidate image (106) and the object mask (1033) to mask out the object from candidate image (106) and modifies the style, size, shape, orientation, color of that object. The attribute or state change model (1042) provides the attribute / state changed object cut out. Further, the suitable mathematical models for object state changes include diffusion-based or GAN-based models for style / attribute alteration.

[0167] In an embodiment of the disclosure, the 'batsman' in the candidate image (106) is masked out to modify the state or orientation of the 'batsman' by horizontally flip (1033) the segmented object. The electronic device may obtain the attribute or state change object (1062),

[0168] In an embodiment of the disclosure, the attributes of the at least one object comprises at least one of a colour of the at least one object, a size of the at least one object, a shape of the at least one object, a style of the at least one object, an orientation of the at least one object, and a physical displacement of the at least one object. Some of the attributes in consideration for edit image generation includes:

[0169]

[0170] Figure 9B illustrates the working operation of the image blending model (1015).

[0171] In an embodiment of the disclosure, the image blending model (1015) receives the attribute or state change object (1062) and the inpainted image (1041c) as input. The image blending model (1015) blends the attribute or state change object (1062) onto the inpainted image (1041c), resulting in the final image (1064) with the object displaying a different attribute / state. This process may be accomplished using any deep image harmonization network or image-to-image blending model.

[0172] In the Figure 9B, initially the 'batsman' in the candidate image (106) is masked and segmented out. In an embodiment of the disclosure, the inpainted image (1041c) is then applied on the masked region so to remove the 'batsman' in the candidate image (106). Then, the transformation is applied on the segmented image of the 'batsman', which is smoothly blended to the inpainted image using an image blending model (1015).

[0173] Figure 9C illustrates the working operation of the edit module.

[0174] In an embodiment of the disclosure, the edit modules (1063) receive the candidate image (106) and the object mask as input. The edit module (1063) change the illumination of the masked object on the candidate image (106), resulting in the edit image (1034) where the object has different illumination, color distribution (1033) compared to the candidate image (106).

[0175] Figure 10A-Figure 10B are the block diagram that illustrates the working operation of masking and inpainting respectively.

[0176] Figure 10A illustrates the working operation of masking.

[0177] In an embodiment of the disclosure, the masking block (1024) receives the segmented object (1021c) and the candidate image (106).The masking block (1024) takes the segmented region (1021c) and mask out that region from the candidate image (106) to generate a masked image (1021d), where the segmented object is masked or removed. Any publically available masking algorithm will mask the object with '0' and all the other pixels will be same as the candidate image (106).

[0178] In Figure 10A, the candidate image (106) includes the apple on the plate and segmented object 'apple' is masked to generate the masked image, where the segmented object 'apple' is masked.

[0179] Figure 10B illustrates the working operation of inpainting. In an embodiment of the disclosure, the inpainting model (1014) receives the masked image (1021d) and the candidate image (106) and inpaints the masked out region of the candidate image (106) created by object removal. This process results in a final inpainted image (1021e). Any diffusion based inpainting model can be used for this inpainting operation

[0180] Figure 11A and Figure 11B are the images that illustrates an example of control prompt effect in the comparison between the existing system and the proposed invention.

[0181] In Figure 11A, the candidate image (106) represents the two person riding bicycles. In an embodiment of the disclosure, the existing system provides the base prompt (1803) as 'detect all bicycle' and the existing system selects and highlight the bicycles (1800a and 1800b). Further, the inpainted operation is performed on first bicycle (1800a). In the existing system, the bicycle (1800a) is removed whereas the person on the bicycle (1800a) left hanging (1800c) resulting in incomplete output. The output generated by the existing system are confusing and unnatural. The incomplete output affects the overall quality of the inpainted image.

[0182] In Figure 11B, the candidate image (106) represents 'the two person riding bicycles'. The proposed invention provides the base prompt (1803) as 'detect all bicycle' and the control prompt (1804) as 'Detect the bicycle and the rider as a single instance'.

[0183] An embodiment selects both the bicycles and the riders. When the inpainted operation is performed on the first bicycle (1800a), the proposed invention removes the bicycle along with the rider thereby maintains the integrity of the scene with both bicycles and riders intact. The control prompt provides specific constraints on the candidate image (106), this helping to edit the image in more efficient way and naturally.

[0184] Figure 12A and Figure 12B are the images that illustrates an example of the control prompt effect in the comparison between the existing system and the proposed invention.

[0185] In Figure 12A, the candidate image (106) represents 'a person holding umbrella'. In an embodiment of the disclosure, the existing system provides the base prompt (1803) as 'detect the person' and the existing system selects and highlights the person (1900a). Further, the inpainted operation is performed on the person (1900a). In the existing system, the person (1900a) is removed whereas the umbrella (1900b) left hanging (1900c) in mid-air resulting in the incomplete output. The inpainted operation on the person does not blend with the rest of the candidate image (106), this leads to a disjointed appearance. Additionally, the base prompt provided to detect only the person limits the scope of the inpainting process, neglecting other important elements such as the umbrella.

[0186] In Figure 12B, the candidate image (106) represents a person holding umbrella'. An embodiment of the disclosure provide the base prompt (1803) as 'detect the person' and the control prompt (1804) as 'detect the person, include any object held by the person". An embodiment of the disclosure is able to identify and differentiate between the person and the person holding the umbrella as one object (1900b). Further, the proposed invention identify and removes the person as well the umbrella while preserving rest of the scene, thereby, maintains the integrity of the scene.

[0187] Figure 13 is the block diagram that illustrates the method for generating a caption using the source caption generator (217a). The source caption generator (217a) is designed to generate the descriptive caption for the candidate image (106). The main objective of the source caption generator (217a) is to create an accurate and meaningful description that reproduces the content of the candidate image (106). The source caption generator (217a) receives the candidate image (106) from the data source as the input and uses computer vision and natural language processing techniques to analyse the visual elements within the candidate image (106). Based on this analysis, the source Caption Generator (217a) generates the descriptive caption or source caption that captures the key features, context, and overall understanding of the candidate image (106) and provides the relevant description.

[0188] For example, In Figure 13, the source Caption Generator (217a) receives the candidate image (106) that represent 'a man looking at the pizza slices on the plate' and it understands the overall scene within candidate image (106) and generates the source caption that describe the candidate image (106) accurately.

[0189] Figure 14 is the block diagram that illustrates the method for generating an edit caption for the edit image (214) using the edit caption generator (217b).

[0190] The edit caption generator (217b) is designed to generate a descriptive caption for the edited image (214). The main objective of edit caption generator (217b) is to provide an accurate and meaningful caption that reproduces the modifications made to the candidate image (106). The edit caption generator (217b) takes the descriptive or source caption and the edit instructions (216) as input and analyses the changes made to the candidate image (106) using an edit caption adapter (2103) and the pre trained LLM (233) and generates the edit caption (218) that describes the modifications or changes made in the edit image (214) .

[0191] For example, In Figure 14, the source caption given as 'a man looking at the pizza slices on the plate' and the edit instruction (216) given as 'remove the pizza slices on the plate'. The edit caption generator (217b) receives both the source caption and the edit instructions (216) as inputs and analyses the changes made to the edit image (214) using the edit caption adapter (2103) and the pre trained LLM (233) and generates the edit caption (218) as 'a man looking at the empty plates'. The edit caption (218) describes the modifications or changes made in the edit image (214).

[0192] Figure 15 is the block diagram that illustrates the method for count based data validation using an edit data validator (219). The count-based data validation may refer to count based filtering. The Count-Based Data Validation using the edit data validator (219) is designed to tally the counts of the edited object in both the candidate image (106) and the edited image (214) to ensure consistency. The Count-Based Data Validation using the edit data validator (219) receives the edited image (214) from the edit pipeline (213) as the input and compares the count of the specific object in the candidate image (106) with the count of the same object in the edited image (214). If the counts match, the count based data validation using the edit data validator outputs a value of '1', indicating that the tally is correct. If the counts do not match, the count based data validation using an edit data validator (219) outputs a value of '0', represents an inconsistency between the two images. The validation method ensure that the edits made to the image align with the proposed changes. The edit data validator (219) may be referred as a pre-trained VLM.

[0193] Figure 16 is the block diagram that illustrates the method for the semantic filtering of the edit image (214) using the edit data validator (219).

[0194] In an embodiment of the disclosure, the semantic filtering model performs the semantic filtering of edited image (214) using the pre-trained image encoder (231) and the text encoder (238) to extract meaningful features from the edit image (214) and the edit caption (218). The semantic filtering model calculates the match score (2032), ranging from -1 to 1, to determine the semantic alignment between the two inputs. By applying text matching techniques (2031), the semantic filtering model ensures that only edits consistent with the caption are retained. The semantic filtering required for maintaining semantic accuracy in automated image editing and captioning process.

[0195] Figure 17A-Figure 17C are the schematic representation that illustrates the use case of the adding or removal pipeline for the automatic high quality data generation for the image editing applications. The Figure 17A represents the candidate image with the object (2401a). The user provide the base prompt as 'remove the object' in the candidate image (106).

[0196] In the Figure 17B, the user provides the control prompt's text command as 'remove the brown chair from the left'. Based on the base and control prompt command, the electronic device (201) detects the specific object i.e. 'brown chair' (2401b) with labels and then perform the edit operation (Ex-add or remove) using the edit pipeline (213).

[0197] The Figure 17C represent the edited image (214) with the specified object i.e. 'brown chair' (2401c) has been removed using the edit operation while maintaining visual quality of the rest of the candidate image (106).

[0198] Figure 18A-18C are schematic representation that illustrates the state change pipeline for automatic high quality data generation for the image editing applications. The Figure 18A illustrates the scenario where the user desired change the orientation of the 'bird' (2502a) in the candidate image (106).The user provides the voice command to bixby to perform the state change edit operation using the attribute or state change pipeline (213c).

[0199] In the Figure 18B, the user provides the control prompt command as 'flip the white bird horizontally'. The white bird (2502b) is identified in the candidate image.

[0200] In the Figure 18C, the state change pipeline (213c) performs a horizontal flip edit operation on the object by detecting the 'white bird' in the candidate image (106). The detection operation is performed using the CMOD-VLM (212). The Figure 18C, the horizontal flip represented the by altering the orientation of the white bird (2502c) while maintaining the visual quality of the candidate image (106).

[0201] Figure 19A-Figure 19C are schematic representation that illustrates the use case of an attribute pipeline for automatic high quality data generation for the image editing applications. The Figure 19A illustrates the scenario where the user desired change the attribute of the object in the candidate image (106). In this scenario, changing the color of the monument (2601a).

[0202] In Figure 19B, the user provides the control prompt command as 'change the monument color to orange'. The monument (2601b) is identified in the candidate image.

[0203] In the Figure 19C, the attribute pipeline detect monument (2601c) and performs the attribute change operation, altering the monument color to orange.

[0204] An embodiment of the disclosure includes an InstGen Model for instruction generation, which allows for a wider range of editing functions such as add, remove, blend, and orientation-based edits using natural language instructions. This expands the scope of the generated images and provides greater flexibility to the user.

[0205] An embodiment provides a novel data pipeline that automates the task of generating high-quality image editing data from the given datasets. This not only saves time and effort but also ensures consistency and accuracy in the generated images.

[0206] An embodiment of the disclosure provides approximately a 300 times reduction in the human effort and time with savings in annotation which can range from $10000 - $100000 with a rough estimation as per costs associated with semi-automatic and fully manual settings.

[0207] In an embodiment of the disclosure, a method for generating an image editing dataset is provided. The method may include obtaining a candidate image (106) for inputting an AI (artificial intelligence) model from among at least one candidate image, wherein the AI model is generative AI model for generating the image editing dataset. The method may include determining an editing operation to be performed by the AI model on the candidate image from among at least one editing operation. The method may include determining a base prompt and a control prompt based on inputting the candidate image and the editing operation into the AI model, wherein the base prompt includes base instructions for detection of at least one object within the candidate image, and the control prompt includes control instructions relevant to the editing operation to be performed by the AI model on the candidate image (106). The method may include generating the image editing dataset based on the base prompt and the control prompt.

[0208] In an embodiment of the disclosure, the method may include generating an edited image (214) by editing the candidate image (106) based on the base prompt and control prompt. The method may include generating an edit instruction (216) by comparing the candidate image (106), and the edited image (214). The method may include generating the image editing dataset as an output, wherein the image editing dataset comprises the candidate image (106), the edited image (214), and the edit instruction (216).

[0209] In an embodiment of the disclosure, the method may include generating a depth map for the depth information of the at least one object within the candidate image. The method may include generating an intermediate feature map for an edited region obtained by comparing the candidate image (106), and the edited image (214). The method may include generating the edit instruction based on the depth map for the depth information of the at least one object within the candidate image, and the intermediate feature map for the edited region.

[0210] In an embodiment of the disclosure, the edit instruction (216) are natural language instruction that describes the editing operation.

[0211] In an embodiment of the disclosure, the editing operation may include at least one of an add operation, a remove operation, a rotate operation, a blend operation, a flip operation, a harmonize operation, a state change operation, and an object attribute change operation to change attributes of the at least one object within the candidate image.

[0212] In an embodiment of the disclosure, the method may include detecting at least one object to be edited from among the at least one object within the candidate image (106) by the control prompt. The control instructions relevant to the editing operation to be performed by the AI model on the candidate image may include detection guidance for object grouping to control a granularity of a detection.

[0213] In an embodiment of the disclosure, the method may include extracting, features from the at least one object within the candidate image based on the base prompt and the control prompt. In an embodiment of the disclosure, the method may include determining an object label with a bounding box by inputting the extracted features into a pre-trained large language model (LLM) (233).

[0214] In an embodiment of the disclosure, the method may include determining whether to validate the image editing dataset by performing count based filtering or semantic filtering.

[0215] In an embodiment of the disclosure, the AI model may include a controlled multi-object detection visual language model (CMOD-VLM) (212).

[0216] In an embodiment of the disclosure, the attributes of the at least one object comprises at least one of a color of the at least one object, a size of the at least one object, a shape of the at least one object, a style of the at least one object, an orientation of the at least one object, and a physical displacement of the at least one object.

[0217] In an embodiment of the disclosure, an electronic device (201) for generating an image editing dataset, may include at least one memory (204) comprising instructions and at least one candidate image; and at least one processor (202) comprising processing circuitry. The instructions, when executed by the at least one processor individually or collectively, cause the electronic device to obtain a candidate image for inputting an AI (artificial intelligence) model from among the at least one candidate image (106), wherein the AI model is generative AI model for generating the image editing dataset, determine an editing operation to be performed by the AI model on the candidate image from among at least one editing operation, determine a base prompt and a control prompt based on inputting the candidate image and the editing operation into the AI model, wherein the base prompt includes base instructions for detection of at least one object within the candidate image, and the control prompt includes control instructions relevant to the editing operation to be performed by the AI model on the candidate image (106), and generate the image editing dataset based on the base prompt and the control prompt.

[0218] In an embodiment of the disclosure, the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to: generate an edited image (214) by editing the candidate image (106) based on the base prompt and the control prompt, generate an edit instruction (216) by comparing the candidate image (106), the edited image (214), and generate the image editing dataset as an output, wherein the image editing dataset comprises the candidate image (106), the edited image (214), and the edit instruction (216).

[0219] In an embodiment of the disclosure, the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to: generate a depth map for the depth information of the at least one object within the candidate image, generate an intermediate feature map for an edited region obtained by comparing the candidate image (106), and the edited image (214), and generate the edit instruction based on the depth map for the depth information of the at least one object within the candidate image, and the intermediate feature map for the edited region.

[0220] In an embodiment of the disclosure, the edit instructions (216) are natural language instruction that describes the editing operation.

[0221] In an embodiment of the disclosure, a computer-readable medium containing instructions, wherein the instructions, when executed by at least one processor, cause the electronic device (150) to perform the method.

[0222] In an embodiment of the disclosure, a method for generating an image editing dataset for generative-AI based editing application, comprising: inputting, by an electronic device (201), an input into an AI model, wherein the input comprises at least one candidate image (106and at least one editing operation to be performed by the AI model on the at least one candidate image (106); inputting, by the electronic device (201), a base prompt into the AI model, wherein the base prompt comprises base instructions for detection of at least one object within the at least one candidate image (106); receiving, by the electronic device (201), a control prompt from a user, wherein the control prompt comprises control instructions relevant to the at least one edition operation to be performed by the AI model on the at least one candidate image (106); and generating, by the electronic device (201), the image editing dataset based on an candidate image (106) the base prompt and the control prompt.

[0223] In an embodiment of the disclosure, a method comprises: appending, by the electronic device (201), the control prompt to the base prompt; and generating, by the electronic device (201), at least one edited image (214) by editing the at least one candidate image (106) using the control prompt appended to the base prompt using the AI model; generating, by the electronic device (201), an edit instruction (216) by comparing the at least one candidate image (106), the at least one edited image (214) and at least one depth map of the at least one candidate image (106); and generating, by the electronic device (201), the image editing dataset as an output, wherein the image editing dataset comprises the at least one edited image (214)along with the edit instruction (216).

[0224] In an embodiment of the disclosure, the edit instructions (216) are natural language instruction that describes the at least one editing operation.

[0225] In an embodiment of the disclosure, a first control instruction relevant to a first edit operation to be performed by the AI model on the at least one candidate image (106) in an edit pipeline (213); and a second control instruction relevant to a second edit operation to be performed by the AI model on the at least one candidate image (106) in the edit pipeline (213), wherein the second edit operation is different from the first edit operation, wherein the first edit operation and the second edit operation comprises at least one of an add operation, a remove operation, a rotate operation, a blend operation, a flip operation, a harmonize operation, a state change operation, and an object attribute change operation to change attributes of the at least one object.

[0226] In an embodiment of the disclosure, the method comprises: detecting, by the electronic device (201), the at least one object in the at least one candidate image (106) using the control prompt appended to the control prompt, wherein the granularity of detection of the at least one object in the at least one candidate image (106) is controlled based on the control prompt; assigning, by the electronic device (201), labels and bounding boxes to the at least one identified object based on the base prompt and the control prompt; and outputting, by the electronic device (201), the at least one edited image (214)with the labels for processing in an edit pipeline (213).

[0227] In an embodiment of the disclosure, the method comprises: extracting, by the electronic device (201), extract features from the at least one candidate image (106) using a pre-trained image encoder (231); employing, by the electronic device (201), a pre-trained large language model (LLM) (233) to process the extracted features and the base prompt; integrating, by the electronic device (201), a trainable detection scale guidance adapter to guide the LLM (233) in detecting the at least one object in the at least one candidate image (106) in accordance with the control prompt; and arranging, by the electronic device (201), the object labels and the bounding boxes in a sequence from left to right as they appear in the at least one candidate image (106).

[0228] In an embodiment of the disclosure, the method comprises: identifying, by the electronic device (201), attributes of at least one edit region from the at least one candidate image (106); obtaining, by the electronic device (201), the at least one depth map corresponding to the at least one candidate image (106), wherein the at least one depth map comprises spatial relationship information among the at least one object in the at least one candidate image (106); and generating, by the electronic device (201), the edit instruction (216) by comparing attributes of the at least one edit region with respect to the at least one object within the at least one candidate image (106) by using the at least one depth map of the at least one candidate image (106).

[0229] In an embodiment of the disclosure, the AI model is a controlled multi-object detection visual language model (CMOD VLM) (212).

[0230] In an embodiment of the disclosure, the attributes of the at least one object comprises at least one of a colour of the at least one object, a size of the at least one object, a shape of the at least one object, a style of the at least one object, an orientation of the at least one object, and a physical displacement of the at least one object.

[0231] In an embodiment of the disclosure, an electronic device (201) for generating an image editing dataset for generative AI-based editing application, comprising: a memory (204) comprising the candidate image (106); a processor (202); and a generative AI-based editing controller (205), coupled to the memory (204) and the processor (202), wherein the generative AI-based editing controller (205) configured to: input an input into an AI model, wherein the input comprises at least one candidate image (106) and at least one editing operation to be performed by the AI model on the at least one candidate image (106); input a base prompt into the AI model, wherein the base prompt comprises base instructions for detection of at least one object within the at least one candidate image (106); receive a control prompt from a user, wherein the control prompt comprises control instructions relevant to the at least one edition operation to be performed by the AI model on the at least one candidate image (106); and generate the trained image dataset based on the candidate image (101), base prompt and the control prompt.

[0232] In an embodiment of the disclosure, the generative AI-based editing controller (205) is configured to generate the image editing dataset based on the base prompt and the control prompt comprises: append the control prompt to the base prompt,; and generate at least one edited image (214)by editing the at least one candidate image (106) using the control prompt appended to the base prompt using the AI model; generate an edit instruction (216) by comparing the at least one candidate image (106), the at least one edited image (214) and at least one depth map of the at least one candidate image (106); and generate the image editing dataset as an output, wherein the image editing dataset comprises the at least one edited image (214) along with the edit instruction (216).

[0233] In an embodiment of the disclosure, the edit instructions (216) are natural language instruction that describes the at least one editing operation.

[0234] In an embodiment of the disclosure, the control instructions comprises: a first control instruction relevant to first edit operation to be performed by the AI model on the at least one candidate image (106) in an edit pipeline (213); and a second control instruction relevant to a second edit operation to be performed by the AI model on the candidate image (106) in the edit pipeline (213), wherein the second edit operation is different from the first edit operation, wherein the first edit operation and the second edit operation comprises at least one of an add operation, a remove operation, a rotate operation, a blend operation, a flip operation, a harmonize operation, a state change operation, and an object attribute change operation to change attributes of the at least one object.

[0235] In an embodiment of the disclosure, the generative AI-based editing controller (205) is configured to edit the at least one object in the at least one candidate image (106) based on the base prompt appended to the control prompts comprises: detect the at least one objects in the candidate image (106) using the base prompt appended to the control prompt, wherein the granularity of detection of the at least one object in the at least one candidate image (106) is controlled based on the control prompt; assign the labels and bounding boxes to the at least one identified object based on the base prompt and the control prompt; and 　output the at least one edited image (214) with the labels for processing in an edit pipeline (213).

[0236] In an embodiment of the disclosure, the generative AI-based editing controller (205) is configured to detect the at least one object in the at least one candidate image (106) using the base prompt and the control prompt comprises: extract features from the at least one candidate image (106) using a pre-trained image encoder (231); employ a pre-trained large language model (LLM) (233) to process the extracted features and the base prompts; integrate a trainable detection scale guidance adapter to guide the LLM (233) in detecting the at least one objects in the at least one candidate image in accordance with the control prompts; and arrange the object labels and the bounding boxes in a sequence from left to right as they appear in the at least one candidate image (106).

[0237] In an embodiment of the disclosure, the generative AI-based editing controller (205) to generate an edit instruction (216) by comparing the at least one candidate image (106), the at least one edited image (214) and at least one depth map of the at least one candidate image (106) comprises: identify attributes of at least one edit region from the at least one candidate image (106); obtain the at least one depth map corresponding to the at least one candidate image (106) wherein the at least one depth map comprises spatial relationships information's among the at least one object in the at least one candidate image (106); and generate the edit instruction (216) by comparing attributes of the at least one edit region with respect to the at least one object within the at least one candidate image (106) by using the at least one depth map of the at least one candidate image (106).

[0238] In an embodiment of the disclosure, the AI model is a controlled multi-object detection visual language model (CMOD VLM).

[0239] In an embodiment of the disclosure, the attributes of the at least one object comprises at least one of a colour of the at least one object, a size of the at least one object, a shape of the at least one object, a style of the at least one object, an orientation of the at least one object, and a physical displacement of the at least one object.

[0240] The various actions, acts, blocks, steps, or the like in the method is performed in the order presented, in a different order or simultaneously. Further, in some embodiments, some of the actions, acts, blocks, steps, or the like are omitted, added, modified, skipped, or the like without departing from the scope of the proposed method.

[0241] The foregoing description of the specific embodiments will so fully reveal the general nature of the embodiments herein that others can, by applying current knowledge, readily modify and or adapt for various applications such specific embodiments without departing from the generic concept, and, therefore, such adaptations and modifications are intended to be comprehended within the meaning and range of equivalents of the disclosed embodiments. It is to be understood that the phraseology or terminology employed herein is for the purpose of description and not of limitation. Therefore, while the embodiments herein have been described in terms of preferred embodiments, those skilled in the art will recognize that the embodiments herein can be practiced with modification within the scope of the embodiments as described herein.

Claims

1.A method for generating an image editing dataset, comprising:obtaining a candidate image (106) for inputting an AI (artificial intelligence) model from among at least one candidate image, wherein the AI model is generative AI model for generating the image editing dataset;determining an editing operation to be performed by the AI model on the candidate image from among at least one editing operation;determining a base prompt and a control prompt based on inputting the candidate image and the editing operation into the AI model, wherein the base prompt includes base instructions for detection of at least one object within the candidate image, and the control prompt includes control instructions relevant to the editing operation to be performed by the AI model on the candidate image (106); andgenerating the image editing dataset based on the base prompt and the control prompt.2.The method of claim 1, wherein generating the image editing dataset based on the base prompt and the control prompt comprises:generating an edited image (214) by editing the candidate image (106) based on the base prompt and control prompt;generating an edit instruction (216) by comparing the candidate image (106), and the edited image (214); andgenerating the image editing dataset as an output, wherein the image editing dataset comprises the candidate image (106), the edited image (214), and the edit instruction (216).3.The method of claim 2, wherein generating the edit instruction comprises:generating a depth map for the depth information of the at least one object within the candidate image;generating an intermediate feature map for an edited region obtained by comparing the candidate image (106), and the edited image (214); andgenerating the edit instruction based on the depth map for the depth information of the at least one object within the candidate image, and the intermediate feature map for the edited region.4.The method of any one of claims 2 to 3, wherein the edit instruction (216) are natural language instruction that describes the editing operation.5.The method of any one of claims 1 to 4, wherein the editing operation comprises:at least one of an add operation, a remove operation, a rotate operation, a blend operation, a flip operation, a harmonize operation, a state change operation, and an object attribute change operation to change attributes of the at least one object within the candidate image.6.The method of any one of claims 2 to 5, wherein the generating the edited image (214) by editing the candidate image (106) based on the base prompt and the control prompt comprises:detecting at least one object to be edited from among the at least one object within the candidate image (106) by the control promptwherein the control instructions relevant to the editing operation to be performed by the AI model on the candidate image include detection guidance for object grouping to control a granularity of a detection.7.The method of claim 6, wherein detecting the at least one object to be edited from among the at least one object within the candidate image (106):extracting, features from the at least one object within the candidate image based on the base prompt and the control prompt;determining an object label with a bounding box by inputting the extracted features into a pre-trained large language model (LLM) (233).8.The method of any one of claims 1 to 7, the method for generating the image editing dataset further comprising:determining whether to validate the image editing dataset by performing count based filtering or semantic filtering.9.The method of any one of claims 1 to 8, wherein the AI model includes a controlled multi-object detection visual language model (CMOD-VLM) (212).10.The method of any one of claims 5 to 9, wherein the attributes of the at least one object comprises at least one of a color of the at least one object, a size of the at least one object, a shape of the at least one object, a style of the at least one object, an orientation of the at least one object, and a physical displacement of the at least one object.11.An electronic device (201) for generating an image editing dataset, comprising:at least one memory (204) comprising instructions and at least one candidate image; andat least one processor (202) comprising processing circuitry;wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:obtain a candidate image for inputting an AI (artificial intelligence) model from among the at least one candidate image (106), wherein the AI model is generative AI model for generating the image editing dataset;determine an editing operation to be performed by the AI model on the candidate image from among at least one editing operation;determine a base prompt and a control prompt based on inputting the candidate image and the editing operation into the AI model, wherein the base prompt includes base instructions for detection of at least one object within the candidate image, and the control prompt includes control instructions relevant to the editing operation to be performed by the AI model on the candidate image (106); andgenerate the image editing dataset based on the base prompt and the control prompt.12.The electronic device (201) of claim 11, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:generate an edited image (214) by editing the candidate image (106) based on the base prompt and the control prompt;generate an edit instruction (216) by comparing the candidate image (106), the edited image (214); andgenerate the image editing dataset as an output, wherein the image editing dataset comprises the candidate image (106), the edited image (214), and the edit instruction (216).13.The electronic device (201) of claim 12, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:generate a depth map for the depth information of the at least one object within the candidate image;generate an intermediate feature map for an edited region obtained by comparing the candidate image (106), and the edited image (214); andgenerate the edit instruction based on the depth map for the depth information of the at least one object within the candidate image, and the intermediate feature map for the edited region.14.The electronic device (201) of any one of claims 11 to 13, wherein the edit instructions (216) are natural language instruction that describes the editing operation.15.A computer-readable medium containing instructions, wherein the instructions, when executed by at least one processor, cause the electronic device (150) to perform the method of any one of claims 1 to 10.

Images