System and method for inserting an object in an image

Abstract

A method (1500) includes: detecting (1502) a stroke input; identifying (1504) the shape and color of the stroke input; generating (1506) a plurality of first candidate objects based on the identified shape and color of the stroke input using an AI model; determining (1508) depth information of one or more objects to be inserted in the image; identifying (1510) a context of the image; identifying (1512) a location of the stroke input within the image, selecting (1514) one or more second candidate objects based on the identified context and the identified location, and updating (1516) the image by inserting the selected one or more second candidate objects based on the determined depth information of the stroke input and the identified location.

Description

Technical Field

[0001] This disclosure relates to image generation using generative artificial intelligence (AI), and more specifically, to a system and method for inserting objects into an image. Background Technology

[0002] Recent advancements in generative artificial intelligence (AI) have facilitated the generation of highly realistic images from minimal input data using diffusion models. Diffusion models have emerged as effective members of the deep generative model family, proving to be valuable tools in image processing and object augmentation due to their ability to generate highly realistic outputs. A wide variety of guided generative techniques have been proposed, focusing on generating images from scratch or on the global manipulation of existing images.

[0003] However, when generating images from text input, these diffusion models struggle to adequately capture the fundamental properties of the intended object, such as color, depth, and other attributes. While generative models (such as sketch-to-image) can provide spatial information to the model, the output does not always respect the strokes drawn by the user and the context within the image. It is often very challenging to instruct the model about the precise location of objects in an image through textual hints or sketch solutions that can guide the generation process. This guidance allows users to convey their intentions in a natural way; however, providing this spatial guidance is often insufficient. Generating images from text-guided sources cannot reliably provide attributes such as the color of the object to be generated. Furthermore, this results in a lack of visual detail in the user's strokes, a lack of image context, and incorrect placement of the generated object. Even additional hints (such as masks) are insufficient to provide accurate location information, such as depth. Therefore, this increases the difficulty of guiding these models to focus on locally adding or replacing objects within the image. Currently, there is no existing solution capable of generating objects within an image while accurately incorporating the stroke color, size, and contextual information present within the image.

[0004] Figure 1A An exemplary scenario of a text-to-image generative model according to the prior art is illustrated. In text-to-image generative models, it is difficult to guide the generation of objects using only text prompts. Furthermore, it is difficult to establish the definition of positional or depth information using text prompts. Therefore, traditional text-to-image generative models cannot generate realistic-looking objects using only text descriptions.

[0005] Figure 1BAn exemplary scenario of a scribbling diffusion generative model according to the prior art is shown. In scribbling diffusion generative models, providing color information through simple strokes is difficult. Furthermore, scribbling diffusion generative models cannot fully understand spatial properties designed specifically for the generated objects. Therefore, it is difficult to generate multiple objects that have never been created in scribbling diffusion generative models. In particular, scribbling diffusion generative models fail to consider any type of depth-related information. New objects cannot be created on existing images, and scribbling diffusion generative models only work on blank canvases. Moreover, in scribbling diffusion generative models, color and texture information cannot be controlled by using pure black strokes.

[0006] Figure 1C An exemplary scenario of a Paint2Pix generative model according to the prior art is illustrated. The Paint2Pix generative model can generate objects for a finite number of classes, and the Paint2Pix generative model is trained for these finite classes. Furthermore, the Paint2Pix generative model can generate objects on a blank canvas, but cannot be used to generate new objects on existing images. Additionally, the Paint2Pix generative model has a finite mapping from strokes to image portions.

[0007] In addition, the following discusses additional issues associated with conventional generative models: One of the most common problems faced by image generative models is object illusion. This problem arises when shadow artifacts create illusions in an image. These illusions can be caused by a variety of factors, such as artifacts (shadows, reflections, etc.) or unclear cues, resulting in fictitious, unusual, and blurry-shaped objects. Furthermore, due to their fundamental structure, content from image generative models may struggle to match user expectations and be difficult to reproduce consistently.

[0008] Lack of visual detail in user strokes: Traditional generative models cannot provide visual detail, such as the shape and color of the object to be generated, through text prompts or ordinary strokes.

[0009] Lack of image context: Traditional generative models lack any mechanism to understand image context. Contextual information helps determine the appropriate objects to be generated.

[0010] Incorrect placement of generated objects: Conventional generative models may have occlusion issues when strokes are drawn on top of another object.

[0011] Indecent / violation-compliant objects are generated using traditional generative models.

[0012] Existing graffiti solutions deploy a simple method to draw anything on an image (sometimes in conjunction with generation). However, these solutions have the limitation that they can only draw on top. If these solutions must place objects at a certain depth, manual adjustments are required. Conventional image generative models tend to always add / replace objects on top. Generative models struggle to identify and place objects in the background. For example, the force cue "generate in background" alters the image's context by removing existing foreground objects from the image.

[0013] Therefore, in view of the above problems, it is beneficial to provide an improved system and method that can overcome the aforementioned problems and limitations associated with object generation on existing images. Summary of the Invention

[0014] Technical solution This summary is provided to introduce selected concepts in a simplified format, which will be further described in the specific embodiments of this disclosure. This summary is not intended to identify any key or essential inventive concepts of this disclosure, nor is it intended to define the scope of this disclosure.

[0015] This disclosure discloses a system for object insertion in an image. The system includes: a memory configured to store at least one instruction; and at least one processor. The at least one processor is communicatively coupled to the memory. The at least one processor is configured to execute the at least one instruction to detect stroke input indicating a request to insert one or more objects into the image. The at least one processor is configured to recognize the shape and color of the stroke input. The at least one processor is configured to generate a plurality of first candidate objects based on the recognized shape and color of the stroke input using an artificial intelligence (AI) model. The at least one processor is configured to determine depth information of the one or more objects to be inserted into the image based on the detected position of the stroke input and one or more adjacent objects within the image. The at least one processor is configured to recognize the context of a scene in the image. The at least one processor is configured to recognize the position of the stroke input within the image. The at least one processor is configured to select one or more second candidate objects from the plurality of first candidate objects based on the recognized scene context and the recognized position of the stroke input. The at least one processor is configured to update the image by inserting one or more selected second candidate objects based on the determined depth information and the position of the identified stroke input.

[0016] In another embodiment, a method for object insertion in an image is also disclosed herein. The method includes detecting stroke input indicating a request to insert one or more objects into the image. Furthermore, the method includes recognizing the shape and color of the stroke input. Additionally, the method includes generating a plurality of first candidate objects based on the recognized shape and color of the stroke input using an artificial intelligence (AI) model. The method includes determining depth information of the one or more objects to be inserted into the image based on the detected position of the stroke input and one or more neighboring objects within the image. The method includes recognizing the context of a scene in the image. The method includes recognizing the position of the stroke input within the image. The method includes selecting one or more second candidate objects from the plurality of first candidate objects based on the recognized context of the scene and the recognized position of the stroke input. Finally, the method includes updating the image by inserting the selected one or more second candidate objects based on the determined depth information and the recognized position of the stroke input.

[0017] To further illustrate the advantages and features of this disclosure, a more specific description of the disclosure will be presented with reference to specific embodiments of the disclosure shown in the accompanying drawings. It is to be understood that these drawings depict only exemplary embodiments of the disclosure and are therefore not intended to limit its scope. The disclosure will be described and explained with additional features and details using the accompanying drawings.

[0018] According to another embodiment of this disclosure, a computer-readable storage medium is disclosed herein for storing computer-readable instructions for performing the methods described above. Attached Figure Description

[0019] The foregoing and other features of the embodiments will become clearer when read in conjunction with the accompanying drawings, based on the following detailed description of the embodiments. In the drawings, the same reference numerals denote the same elements.

[0020] Figure 1A An exemplary scenario of a text-to-image generative model based on existing technology is shown; Figure 1B An exemplary scenario of a writing diffusion generative model based on the prior art is shown; Figure 1C An exemplary scenario of a Paint2Pix generative model based on existing technology is shown; Figure 2 An environment of a system communicatively coupled to a device according to an embodiment of the present disclosure is shown; Figure 3 A block diagram of a system combined with an apparatus for doodle-assisted context-aware object generation (DACAOG) according to an embodiment of the present disclosure is shown. Figure 4 A sequential process for inserting objects into an image according to embodiments of the present disclosure is illustrated; Figure 5 The extraction of graffiti strokes from an image is illustrated according to an embodiment of the present disclosure; Figure 6 A training example of a graffiti-to-object diffusion model according to an embodiment of the present disclosure is shown; Figure 7 This is a training example of the stroke guidance loss of a system according to an embodiment of the present disclosure; Figure 8 A training example of the color-guided loss of a system according to an embodiment of the present disclosure is shown; Figure 9 A color extractor and a shape extractor of a system according to an embodiment of the present disclosure are shown; Figure 10 The determination of the boundary region and binary mask according to an embodiment of the present disclosure is illustrated; Figure 11A This illustrates the selection of one or more second candidate objects by comparing scene-object-location mapping models according to embodiments of the present disclosure; Figure 11B A training example of a contrastive scene-object-location mapping model according to embodiments of the present disclosure is shown; Figure 11C Several examples of a contrasting scene-object-location mapping model associated with embodiments of the present disclosure are shown; Figure 11D The operation of a comparative scene-object-location mapping model according to embodiments of the present disclosure is illustrated; Figure 12A A depth estimation method for a plurality of objects as candidates for depth estimation is illustrated according to embodiments of the present disclosure; Figure 12B A detailed explanation of the depth estimation method according to embodiments of the present disclosure is provided; Figure 12C A flowchart of a depth-based object selection module in a depth estimation method according to an embodiment of the present disclosure is shown; Figure 12D The diagram illustrates objects of the same depth and size in a depth-based object selection module within a depth estimation method according to embodiments of the present disclosure. Figure 12E Small objects of the same depth in a depth-based object selection module of a depth estimation method according to an embodiment of the present disclosure are shown; Figure 12FThe diagram illustrates objects of the same depth and multiple sizes in a depth-based object selection module within a depth estimation method according to embodiments of the present disclosure. Figure 12G The ScaleRef network in a depth estimation method according to an embodiment of the present disclosure is illustrated; Figure 12H (i-iv) illustrate several examples of ScaleRef networks in depth estimation methods according to embodiments of the present disclosure; Figure 13 The operation of an object image synthesizer of a system according to an embodiment of the present disclosure is illustrated; Figures 14A-14B Use cases of systems according to embodiments of the present disclosure are illustrated; and Figure 15 A flowchart of a method for inserting an object into an image according to an embodiment of the present disclosure is shown. Detailed Implementation

[0021] For the purpose of promoting an understanding of the principles of this disclosure, reference will now be made to various embodiments, and these embodiments will be described using specific language. However, it will be understood that this is not intended to limit the scope of the disclosure, and such changes and further modifications to the illustrated systems, as well as such further applications of the principles of the disclosure as illustrated herein, are considered to be common knowledge to those skilled in the art to which this disclosure pertains.

[0022] Those skilled in the art will understand that the foregoing general description and the following detailed description are for the purpose of interpreting this disclosure and are not intended to limit it.

[0023] Whether a feature or element is limited to being used only once, it may still be referred to as "one or more features," "one or more elements," "at least one feature," or "at least one element." Furthermore, the use of the terms "one or more" or "at least one" features or elements does not preclude the absence of that feature or element, unless otherwise specified by restrictive language, including but not limited to "requires one or more..." or "requires one or more elements."

[0024] This document refers to several “Examples”. It should be understood that the embodiments are examples of possible implementations of any feature and / or element of this disclosure. Several embodiments have been described for the purpose of explaining one or more of the potential modes, in which the specific features and / or elements of the disclosure proposed in the potential modes satisfy the requirements of novelty, industrial applicability and non-obviousness.

[0025] The phrases and / or terms used, including but not limited to "first embodiment," "another embodiment," "alternative embodiment," "one embodiment," "embodiment," "multiple embodiments," "some embodiments," "other embodiments," "another embodiment," "yet another embodiment," "additional embodiment," or other variations thereof, do not necessarily refer to the same embodiment. Unless otherwise indicated, one or more specific features and / or elements described in conjunction with one or more embodiments may be found in one embodiment, or in more than one embodiment, or in all embodiments, or may not be found in any embodiment. Although one or more features and / or elements may be described herein in the context of a single embodiment, or in the context of more than one embodiment, or in the context of all embodiments, features and / or elements may alternatively be provided individually, in any suitable combination, or not at all. Conversely, any feature and / or element described in the context of separate embodiments may alternatively be implemented as being present together in the context of a single embodiment.

[0026] Any particular and all details set forth herein are used in the context of some embodiments and should therefore not necessarily be considered as limiting factors of the disclosure presented.

[0027] The terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process or method that includes a list of steps includes not only those steps but may also include other steps not expressly listed or inherent to such a process or method. Similarly, one or more devices, subsystems, elements, structures, or components preceded by “comprising…” do not exclude the presence of other devices or subsystems or elements or structures or components, or additional devices or subsystems or elements or structures or components, without further constraints.

[0028] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0029] For clarity, the first digit of the reference numeral for each component in this disclosure indicates the drawing number of the corresponding component. For example, reference numerals beginning with the number "1" are shown at least in Figure 1. Similarly, at least in... Figure 2 The reference numerals that begin with the number "2" are shown in the figure.

[0030] This disclosure relates to a Doodle-Assisted Context-Aware Object Generation (DACAOG) method for generating objects in images using generative artificial intelligence (AI). The DACAOG method generates new objects from doodles on existing images and appropriately places the generated objects within the image by determining depth information from the 2D image and associating the image with the current object. The DACAOG method is further defined as a contrastive object-to-image mapping model, using which objects are further filtered. Furthermore, user-drawn strokes are used to define the appearance attributes of the generated objects.

[0031] Figure 2 An environment 200 is shown that includes a system 204 communicatively coupled to device 202, according to an embodiment of the present disclosure. Figure 3 A block diagram 300 shows a system 204 for doodle-assisted context-aware object generation (DACAOG) in conjunction with device 202 according to an embodiment of the present disclosure.

[0032] In embodiments, without departing from the scope of this disclosure, device 202 may be a smartphone, tablet computer, or any other electronic device compatible with capturing / searching images.

[0033] When an image is searched / captured by device 202, and the user wants to insert any object into the searched / captured image, system 204 can be configured to insert the object based on one or more user inputs in the form of user strokes drawn on the image. System 204 can be configured to insert the object based on various factors, including but not limited to detecting stroke input, the shape and color of the stroke input, etc. The structure and operational details of system 204 are explained in subsequent paragraphs.

[0034] In one embodiment, without departing from the scope of this disclosure, system 204 may be deployed in device 202. In another embodiment, without departing from the scope of this disclosure, system 204 may be communicatively coupled to device 202.

[0035] In an embodiment, system 204 may include, but is not limited to, at least one processor (referred to herein as one or more processors, processors) 304, memory 308, and multiple modules 312, as explained in detail in other examples in subsequent paragraphs.

[0036] System 204 may include an input / output (I / O) interface 334 and a transceiver 332. Furthermore, in some embodiments where system 204 is implemented as a standalone entity within a server / cloud architecture, system 204 may communicate with multiple devices to receive data from each device. Additionally, the details provided below regarding system 204 and device 202 also apply to system 204 and the multiple devices.

[0037] In an exemplary embodiment, processor 304 may be operatively coupled to each of I / O interface 334, plurality of modules 312, transceiver 332, and memory 308. In one embodiment, processor 304 may include at least one data processor for performing processing in a virtual memory area network. Processor 304 may include dedicated processing units such as integrated system (bus) controllers, memory management control units, floating-point units, graphics processing units, digital signal processing units, etc. In one embodiment, processor 304 may include a central processing unit (CPU), a graphics processing unit (GPU), or both. Processor 304 may be one or more general-purpose processors, digital signal processors, application-specific integrated circuits, field-programmable gate arrays, servers, networks, digital circuits, analog circuits, combinations thereof, or other means now known or hereafter developed for analyzing and processing data. Processor 304 may run software programs (such as manually generated (i.e., programmed) code) to perform desired operations.

[0038] Processor 304 may be configured to communicate with one or more input / output (I / O) devices via I / O interface 334. In some embodiments, processor 304 may use I / O interface 334 to communicate with device 202. In some embodiments, I / O interface 334 may be implemented within device 202. In embodiments, I / O interface 334 may use suitable devices (e.g., but not limited to, cameras, displays, etc.) to implement input to and output from system 204.

[0039] Using I / O interface 334, system 204 can communicate with one or more I / O devices (specifically, device 202), in which system 204 inserts objects into an image according to user requirements. For example, input devices can be antennas, microphones, touchscreens, touchpads, storage devices, transceivers, recording devices / sources, etc. Output devices can be video displays (e.g., cathode ray tube (CRT), liquid crystal displays (LCD), light-emitting diodes (LEDs), plasma displays, plasma display panels (PDP), organic light-emitting diode displays (OLEDs), etc.), audio speakers, etc.

[0040] Processor 304 can be configured to communicate with a communication network via a network interface. In an embodiment, the network interface may be I / O interface 334. The network interface can be connected to the communication network to enable connection between system 204 and device 202. The network interface may employ connection protocols, including but not limited to direct connection, Ethernet (e.g., 10 / 100 / 1000 BaseT twisted pair), Transmission Control Protocol / Internet Protocol (TCP / IP), Token Ring, IEEE 702.11a / b / g / n / x, etc. The communication network may include, but is not limited to, direct interconnect, Local Area Network (LAN), Wide Area Network (WAN), Wireless Network (e.g., using Wireless Application Protocol), Internet, etc. Using the network interface and the communication network, system 204 can communicate with other devices. The network interface may employ connection protocols, including but not limited to direct connection, Ethernet (e.g., 10 / 100 / 1000 BaseT twisted pair), Transmission Control Protocol / Internet Protocol (TCP / IP), Token Ring, IEEE 702.11a / b / g / n / x, etc.

[0041] Transceiver 332 may be configured to receive signals from device 202 and / or send signals to device 202. In one embodiment, the database may be configured to store information required by the plurality of modules 312 and processor 304 to insert objects into an image.

[0042] In some embodiments, memory 308 may be communicatively coupled to processor 304. Memory 308 may be configured to store data and instructions executable by processor 304. In one embodiment, memory 308 may be located within device 202. In another embodiment, memory 308 may be located within system 204, remote from device 202. In yet another embodiment, memory 308 may communicate with processor 304 via a bus within system 204. In yet another embodiment, memory 308 may be located remotely from processor 304 and may communicate with processor 304 via a network. Memory 308 may include, but is not limited to, non-transitory computer-readable storage media, such as various types of volatile and non-volatile storage media, including but not limited to random access memory, read-only memory, programmable read-only memory, electrically programmable read-only memory, electrically erasable read-only memory, flash memory, magnetic tape or disk, optical media, etc.

[0043] In one example, memory 308 may include a cache or random access memory for processor 304. In alternative examples, memory 308 is separate from processor 304, such as processor cache memory, system memory, or other memory. Memory 308 may be an external storage device or database for storing data. Memory 308 may be operable to store instructions that can be executed by processor 304. The functions, actions, or tasks shown or described in the figures can be performed by a programmed processor 304 for executing instructions stored in memory 308. The functions, actions, or tasks are independent of the specific type of instruction set, storage medium, processor, or processing strategy, and can be performed by software, hardware, integrated circuits, firmware, microcode, etc., operating individually or in combination. Similarly, processing strategies may include multiprocessing, multitasking, parallel processing, etc.

[0044] In some embodiments, multiple modules 312 may be included within memory 308. Memory 308 may also include a database for storing data. Multiple modules 312 may include a set of instructions executable to cause system 204 (particularly the processor 304 of system 204) to perform any or more of the methods / processes disclosed herein. Multiple modules 312 may be configured to perform steps of this disclosure using data stored in the database. For example, multiple modules 312 may be configured to perform… Figures 4 to 13 The steps disclosed in the document.

[0045] In this embodiment, each of the plurality of modules 312 may be a hardware unit that may be external to the memory 308. Furthermore, the memory 308 may include an operating system for performing one or more tasks of the system 204, such as those performed by a general-purpose operating system.

[0046] In one example, module 312 may include a detection module 314, a determination module 316, a generation module 318, an extraction module 320, an identification module 322, a selection module 324, an update module 326, a placement module 328, and a filtering module 330. Each of these modules can communicate with each other. Furthermore, each of these modules can communicate with the processor 304.

[0047] Furthermore, this disclosure also describes a computer program product that includes instructions or receives and executes instructions in response to propagated signals. Additionally, instructions are sent or received via a network through a communication port or interface or using a bus (not shown). The communication port or interface may be part of processor 304 or may be a separate component. The communication port may be created in software or may be a physical connection in hardware form. The communication port may be configured to connect to a network, external media, a display, or any other component in system 204. The connection to the network may be a physical connection, such as a wired Ethernet connection, or the connection to the network may be established wirelessly. Similarly, additional connections to other components of system 204 may be physical or may be established wirelessly. Optionally, the network may be directly connected to a bus. For the sake of brevity, the architecture and standard operation of memory 308 and processor 304 are not discussed in detail.

[0048] In an embodiment, the computer program product has machine-readable instructions stored therein, which, when executed by processor 304, cause processor 304 to perform a method of inserting objects into an image. Details regarding the method executed by processor 304 are incorporated in subsequent paragraphs. Figure 3 At least refer to Figures 4 to 13 A detailed explanation was provided.

[0049] Furthermore, this disclosure also examines a non-transitory computer-readable medium encoded with executable instructions. When executed by processor 304, the executable instructions cause processor 304 to perform a method for inserting objects into an image. Details regarding the method executed by processor 304 are incorporated in subsequent paragraphs. Figure 3 At least refer to Figure 4-13 The details are explained in detail. Examples of computer-readable media include non-volatile hard-coded media (such as read-only memory (ROM) or erasable electrically programmable read-only memory (EEPROM)) and user-recordable media (such as floppy disks, hard disk drives, and compact disc read-only memory (CD-ROM) or digital universal disc (DVD)).

[0050] The processor 304, in conjunction with the detection module 314, determination module 316, generation module 318, extraction module 320, identification module 322, selection module 324, update module 326, placement module 328, and filtering module 330, can be configured to perform the specific operations explained in subsequent paragraphs.

[0051] It should be noted that not all modules in the above structural diagram are mandatory; some modules can be omitted depending on actual needs. The division of modules is only for the convenience of describing the functional division adopted. In actual implementation, a module can be implemented as multiple modules, and the functions of multiple modules can also be implemented by the same module. These modules can be located in the same device or in different devices.

[0052] The hardware modules in various implementations can be implemented mechanically or electronically. For example, a hardware module may include specially designed permanent circuitry or logic devices (e.g., dedicated processors such as FPGAs or ASICs) for performing specific operations. A hardware module may also include programmable logic devices or circuitry (e.g., general-purpose processors or other programmable processors) temporarily configured by software to perform specific operations. The specific implementation of the hardware module—whether by mechanical means, dedicated permanent circuitry, or temporarily configured circuitry (e.g., by software configuration)—can be determined based on cost and time considerations.

[0053] Figure 4 A sequential process for inserting objects into an image according to an embodiment of the present disclosure is shown. Figure 5 The following illustration shows the extraction of graffiti strokes 504 from an image according to an embodiment of the present disclosure. Figure 6 A training example of a graffiti-to-object diffusion model for system 204 according to an embodiment of the present disclosure is shown. Figure 7 A training example of the stroke guidance loss of system 204 according to an embodiment of the present disclosure is shown. Figure 8 A training example of the color-guided loss of system 204 according to an embodiment of the present disclosure is shown. Figure 9 A color extractor and a shape extractor of system 204 according to an embodiment of the present disclosure are shown. Figure 10 The determination of the boundary region and binary mask according to an embodiment of the present disclosure is illustrated. Figure 11A An embodiment of the present disclosure illustrates the selection of one or more second candidate objects by comparing a scene-object-location mapping model 1102. Figure 11B A training example of a contrast scene-object-location mapping model 1102 according to an embodiment of the present disclosure is shown. Figure 11C Several examples of the contrast scene-object-location mapping model 1102 associated with embodiments of the present disclosure are shown. Figure 11D The operation of a comparative scene-object-location mapping model 1102 according to an embodiment of the present disclosure is illustrated. Figure 12A A depth estimation method for a plurality of objects as candidates for depth estimation is illustrated according to embodiments of the present disclosure. Figure 12B A detailed explanation of the depth estimation method according to embodiments of the present disclosure is provided. Figure 12CA flowchart of a depth-based object selection module in a depth estimation method according to an embodiment of the present disclosure is shown. Figure 12D The diagram illustrates objects of the same depth and size in a depth-based object selection module within a depth estimation method according to embodiments of the present disclosure. Figure 12E Small objects of the same depth are shown in a depth-based object selection module in a depth estimation method according to an embodiment of the present disclosure. Figure 12F The illustration shows multiple small objects of the same depth in a depth-based object selection module of a depth estimation method according to an embodiment of the present disclosure. Figure 12G The ScaleRef network in a depth estimation method according to an embodiment of the present disclosure is illustrated. Figure 12H (i-iv) illustrate several examples of ScaleRef networks in depth estimation methods according to embodiments of the present disclosure. Figure 13 The operation of an object image synthesizer of system 204 according to an embodiment of the present disclosure is illustrated.

[0054] To seamlessly generate and place objects in an image, processor 304 may use a doodle-assisted context-aware object generation (DACAOG) process 400. The DACAOG process 400 uses at least three interdependent components, including but not limited to a doodle-stroke-based object generation method, a contrastive scene-object-position mapping model, and a relative depth estimation method / module.

[0055] A doodle-stroke-based object generation method is used to generate objects with the aid of strokes of color and size provided by the user. In one or more embodiments that predict multiple objects via doodle strokes, the multiple objects are filtered and classified based on parameters such as, but not limited to, the context of the scene, depth, and the location where the generated objects must be added.

[0056] In step 402, processor 304 receives an input image (referred to herein as an image) from which auxiliary objects need to be generated. In step 404, detection module 314 can be configured to detect stroke input indicating a request to insert one or more objects into the image. In embodiments, without departing from the scope of this disclosure, the input may be provided by a user. In such embodiments, the user uses doodle strokes on the input image to draw and indicate the objects to be generated.

[0057] In steps 406 and 407, the detection module 314 may be configured to detect doodle strokes 504 from user-completed stroke input as shown in box 502, for object generation. In such an embodiment, refer to... Figure 5Extraction module 320 can be configured to extract doodle strokes 504 from stroke input. In embodiments, without departing from the scope of this disclosure, extraction module 320 can be a doodle stroke extraction module. For example, doodle strokes 504 can be drawn on top of an image using strokes based on custom colors and sizes. Doodle strokes 504 can be drawn on a canvas layer overlaid on top of the image. Each drawn doodle stroke can be saved as an image. When a user draws multiple doodle strokes, the individual doodle stroke images can be merged into an image on a blank canvas of the same size. The canvas including the merged doodle strokes can be saved as a separate "doodle stroke image". System 204 requires the user to draw strokes on the image to provide system 204 with spatial information related to the generation and placement of objects in the image. The presence of extraction module 320 enables the extraction of doodle strokes 504 from the image, allowing system 204 to capture the spatial and appearance features of objects that will be generated in the input image.

[0058] In addition, the determination module 316 can be configured to determine the shape and color of the stroke input (i.e., the graffiti stroke 504).

[0059] In step 408, the object generation process based on doodle strokes can be implemented to predict multiple objects corresponding to stroke input provided by the user. In an embodiment, generation module 318 can be configured to use an artificial intelligence (AI) model to generate multiple first candidate objects (p) based on the determined shape and color of the stroke input. In step 410, multiple first candidate objects (p) are generated following the color and shape of the strokes. In such an embodiment, extraction module 320 can be configured to use a low-pass filter to extract color-based features of the detected stroke input. Generation module 318 can be configured to generate a color feature map corresponding to the stroke input based on the extracted color-based features and one or more predefined color-related parameters. Furthermore, extraction module 320 can be configured to use a high-pass filter to extract shape-based features of the detected stroke input. Generation module 318 can be configured to generate a shape feature map corresponding to the stroke input based on the extracted shape-based features and one or more predefined shape-related parameters. Furthermore, generation module 318 can be configured to use a diffusion model to generate a first plurality of candidate objects based at least on the generated color feature map, the generated shape feature map, and multiple random noises.

[0060] Reference Figure 6Specifically, the doodle-based object generation process can be used to generate multiple first candidate objects (p) based on the color and size of doodle stroke 504, as described in later paragraphs. Process 600 details the doodle-to-object diffusion model. In step 602, process 600 takes a random noise image and doodle stroke 504 as input. In step 604, the input obtained in step 602 is shared with the U-net-based diffusion model. The U-net-based diffusion model learns to generate the desired objects by minimizing the color-guided loss and stroke-guided loss, explained in later paragraphs. The stroke-guided loss and color-guided loss are cosine similarity losses between doodle stroke 504 and a benchmark real object image. The U-Net-based diffusion model is trained via a back-diffusion process to generate multiple first candidate objects (p) from the noise image and doodle stroke 504. This training process is called the back-diffusion process. This process involves minimizing the stroke-guided loss and color-guided loss, which evaluates how effectively the model can convert noise samples into objects very similar to doodle stroke 504.

[0061] In the embodiment, for stroke guidance loss, refer to Figure 7 The stroke guidance loss is generated using a doodle-to-object diffusion model. Method 700 is used to compute the stroke guidance loss. In step 702, a stroke input is provided by the user to generate a doodle stroke 504. In step 704, the doodle stroke 504 is passed through a stroke encoder for further processing, and in step 706, a stroke embedding is obtained. In an embodiment, the stroke encoder may include a series of convolutional layers that encode the doodle stroke 504 into a representation capturing stroke features.

[0062] Furthermore, in step 708, doodle strokes 504 are used to generate at least one of a plurality of first candidate objects (p) using a doodle-to-object diffusion model. In step 710, at least one of the generated plurality of first candidate objects (p) is obtained from the doodle-to-object diffusion model. In step 712, at least one of the generated plurality of first candidate objects (p) is passed through an image encoder to further obtain an image embedding in step 714. The stroke encoder and the image encoder are trained in a contrastive manner such that correct stroke and image pairs have similar representations, have similar representations in the latent space, and unrelated stroke and image pairs have distant latent representations. A cosine similarity-based loss method is used to train the stroke encoder and the image encoder, wherein the cosine similarity is maximized for stroke and image pairs and minimized for dissimilar stroke and image pairs.

[0063] (Formula 1) Among them, A i and B i These are embeddings from the stroke encoder and the image encoder, respectively.

[0064] refer to Figure 8 A color-guided loss is generated by utilizing a graffiti-to-object diffusion model. Method 800 is used to calculate the color-guided loss. At step 802, graffiti strokes 504 are provided to generate a plurality of first candidate objects (p). At step 804, graffiti strokes 504 are linearized to obtain a linear stroke image. At step 806, graffiti strokes 504 are binaryized to obtain a binary stroke image. In one embodiment, binaryization refers to converting a red, green, and blue (RGB) image into a binary image. Each image pixel is divided into a set of two pixels, namely black and white. The main goal of binaryization is to segment the image into foreground and background. At step 808, a graffiti-to-object diffusion model is used on graffiti strokes 504, and at step 810, at least one of the plurality of first candidate objects (p) is generated. Furthermore, at step 812, at least one of the generated plurality of first candidate objects (p) is element-wise multiplied with the binary stroke image to obtain at least one of the masked generated plurality of first candidate objects (p). At least one of the generated first candidate objects (p) is masked to ensure that the guide is positioned within the stroke region and does not affect other regions. At step 814, at least one of the generated first candidate objects (p) is linearized to obtain a linearly masked generated image. In one embodiment, linearization refers to converting the RGB channel image into a one-dimensional vector.

[0065] At step 816, a color similarity loss is determined in order to minimize the similarity between the linear stroke image and the generated image of the linear mask.

[0066] (Formula 2) Here, A and B represent the linear stroke image and the generated image from the linear mask, respectively. This configuration ensures that a similarity-based loss is used to match the color of the graffiti stroke 504 with the color of the predicted image.

[0067] In step 606, the doodle strokes 504 are shared with the color and shape infusion module. The color and shape infusion module uses a multi-scale convolutional neural network design, which extracts color and shape features from the doodle strokes 504 and combines them with a U-net-based diffusion model at each diffusion time step to ultimately generate multiple first candidate objects (p).

[0068] In this embodiment, the color and shape infusion module functions by extracting color and shape features from the graffiti strokes and fusing them with each stage of the backdiffusion process to guide object generation. The color and shape infusion module consists of two convolutional encoder-decoder networks that attempt to obtain low-frequency and high-frequency features from the graffiti strokes 504, respectively. Low-frequency features, such as color features, are obtained from the graffiti strokes 504 using a low-pass filter, while high-frequency features, such as edge maps, are obtained using a high-pass filter. The extracted color and shape features are infused into the diffusion blocks of the object-to-graffiti model at each time step.

[0069] Reference Figure 9 Color and shape features can be extracted using a color extractor and a shape extractor. The color extractor can be trained by performing the following steps: Create a GT image from a stroke image using a low-pass filter.

[0070] Set the kernel dimension to a high value to obtain low-level features.

[0071] The color extractor network was trained in a supervised manner using MSE loss.

[0072] Similarly, the shape extractor can be trained by doing the following: Create a GT image from a stroke image using a high-pass filter.

[0073] Set the kernel dimension to a low value to obtain advanced features.

[0074] The shape extractor network is trained in a supervised manner using MSE loss.

[0075] This process ensures that the color features of the generated multiple first candidate objects are based on the colors of the user-drawn strokes. It also ensures that doodle strokes 504 of different sizes can have different color weights across the generated objects. For example, thicker strokes may have a greater color influence on the generated objects compared to thinner strokes. The process receives user-drawn strokes from doodle strokes 504, internally generates random noise, and passes both data to a doodle-to-object diffusion model to generate multiple first candidate objects (p).

[0076] In one embodiment, the recognition module 322 can be configured to recognize the position of a stroke input within an image. In such an embodiment, the determination module 316 can be configured to determine the size of the detected stroke input. The recognition module 322 can be configured to identify boundary regions within the image based on the determined size of the stroke input. The recognition module 322 can be configured to identify the position of the stroke input within the image based on the identified boundary regions.

[0077] Specifically, in steps 411 and 412, the doodle stroke input image created in step 404 is used by the stroke region calculation module for stroke region calculation. In step 414, after determining the stroke region, binary position information is generated to process the placement area of ​​the generated object. In such an embodiment, stroke region calculation refers to calculating the position and area spanned by the doodle stroke 504 drawn by the user on top of the image. This helps identify a location where at least one of the selected one or more second candidate objects can be placed. Based on the boundary region, the generation module 318 can be configured to generate a binary mask that helps to seamlessly blend the object into the image. Specifically, in order to obtain the binary position information or create a stroke region-based binary mask, the determination module 316 can be configured to determine the boundary region, such as a boundary rectangle, from the doodle stroke 504. This binary mask acts as a positioning condition to help the object image synthesizer correctly place the generated object. Therefore, the placement module 328 can be configured to place at least one of the selected one or more second candidate objects based on the generated binary mask and the determined depth information. In this embodiment, the selection of one or more second candidate objects and the determination of the depth information are explained in later paragraphs.

[0078] To determine the boundary area crossed by graffiti stroke 504, refer to... Figure 10 Perform the following operations: The detection module 314 can be configured to detect doodle strokes 504 from the stroke input, as shown in step 1002.

[0079] The recognition module 322 can be configured to recognize the left point, vertex, right point, and bottom point of the graffiti stroke 504, as shown in step 1004. These points define the left edge, top edge, right edge, and bottom edge of the area of ​​the graffiti stroke 504.

[0080] The determination module 316 can be configured to determine a boundary rectangle based on the identified points, as shown in step 1006. This rectangle represents the area spanned by the graffiti stroke 504.

[0081] A binary mask is obtained based on the rectangle from the previous step, where white pixels represent the stroke drawing area and black pixels represent the background area, as shown in step 1008.

[0082] Furthermore, the recognition module 318 can be configured to recognize the context of the scene in which the image is displayed.

[0083] In one embodiment, after identifying the location and context of the scene, the selection module 324 can be configured to select one or more second candidate objects from a plurality of first candidate objects based on the context of the identified scene and the location of the identified stroke input. In such an embodiment, the determining module 316 can be configured to determine a relevance score associated with each of the plurality of first candidate objects based on the identified scene context and the location of the identified stroke input. The selection module 324 can be configured to select one or more second candidate objects from the plurality of first candidate objects based on the determined corresponding relevance scores.

[0084] refer to Figure 4 and Figure 11A At step 416, after determining the stroke region, the output is shared with the comparison scene-object-position mapping model 1102, which also acquires the corresponding output from step 410, including multiple first candidate objects (p), and the corresponding output from step 402 as an image. At step 418, the information acquired at step 416 is processed, and the top relevant object (N), i.e., one or more second candidate objects (N) from step 410, is selected.

[0085] In this embodiment, the contrast scene-object-location module 1102 includes a pre-trained contrast model that learns a mapping between an image, an object to be generated within the image, and the location where the object is placed. Location-based information is encoded along with the image and fed into the pre-trained contrast scene-object-location mapping model. This method attempts to learn the relationship between an object and the entire image at a predetermined location. The model is trained contrastively such that objects with a higher probability of being present in the image have a higher relevance score, while objects with a lower probability of being present at a specific location in the image will have a lower relevance score. For example, "pizza" will have a high relevance to an image of a plate compared to a landscape scene image, while a mountain will have a high relevance to a landscape scene compared to an image including a cooking plate. Location also plays a crucial role in determining the relevance score. It should be reasonable for an object to be present at a particular location.

[0086] refer to Figure 11BThe contrastive scene-object-location mapping model 1102 is a pre-trained model adapted to learn the mapping between the scene to be generated and the objects and their positions within the scene. Locational information is encoded along with the image and fed into the contrastive pre-trained scene-object-location mapping model. Method 1100 is designed to learn the relationship between selected objects and scenes at predefined locations in the image. Initially, natural images from a dataset are selected, and then objects are selected, where each object is associated with the selected image and location information. In step 1104, a triplet dataset is assumed, comprising a natural image (I), objects (O) associated with the natural image (I), and location information (P), represented by boundary coordinates on the objects (O). In step 1106, an object pool is created using the images presented in step 1110. In step 1108, the objects (O) are passed through an object encoder. In step 1110, an image pool is created using the triplet dataset created in step 1104. In step 1112, the image (I) is passed through an image encoder, and the location information (P) is appended to the image encoding and passed through the network. The model jointly trains an image encoder and an object encoder to predict the correct pairing of images and objects. In an embodiment, the image encoder and object encoder are a series of convolutional layers that extract features at different levels from the input image / object. Each layer applies a set of filters to the image and outputs a feature map that highlights specific patterns and structures in the image / object. These specific pattern features are then linearized, and a mapping score is obtained between any set of images or objects.

[0087] In step 1114, the contrastive scene-object-position mapping model learns to recognize the association between the position (P) of an object (O) relative to an image (I) and the visual concept. The contrastive scene-object-position mapping model is trained in a contrastive manner, where for correct image-object-position pairs, the similarity between the object's position relative to the image and its context is maximized, while for incorrect pairs, the similarity is minimized.

[0088] Furthermore, compared to the image-object-location mapping model 1102, training requires the correct pairing of images, objects, and location information. Therefore, a dataset comprising image pools and associated objects can be formed. Image pooling is created such that each image can have one or more associated objects, generating another meaningful image when these objects are placed at specific locations within the image. For example, reference... Figure 11C (i) The pyramid object can be placed in the scene (input image). Additionally, in (ii), the pizza object can be placed on the plate (input image).

[0089] The scene-object-position mapping model has already been used. Figures 11A-11C The discussion in the middle will now refer to Figure 11DTo explain, a contrastive scene-object-location mapping model can have multiple possible outputs, allowing for the existence of more than one object corresponding to a given location (P) in a given image (I). Furthermore, multiple outputs can be selected by the contrastive scene-object-location mapping model, where two outputs can be forwarded for a depth estimation model, thus allowing the contrastive scene-object-location mapping model to have more than one output. Specifically, refer to... Figure 11D (i) and (ii), the doodle strokes 504 drawn by the user are fed into the doodle stroke-based object generation module, which outputs multiple first candidate objects (p). These objects, along with the image (I) and the stroke region boundary rectangles, are fed into the comparison image-object location mapping module 1102 to find the object with the highest relevance to the image. Furthermore, the stroke region boundary rectangles are obtained from the stroke region calculation module based on the position of the doodle strokes drawn by the user on the image. Additionally, there are multiple possible outputs from this model; that is, there can be more than one object {O(1..n)} with similar relevance to a given position in the image (I). In the case where there are multiple objects selected by the comparison image-object-location mapping module, both outputs can be forwarded to the depth estimation module.

[0090] In an embodiment, the determining module 316 may be further configured to determine depth information of the stroke input based on the location of the detected stroke input within the image and one or more adjacent objects. In such an embodiment, the determining module 316 may be configured to determine depth information / image based on the image, one or more selected second candidate objects, the location of the detected stroke input, and the identified boundary regions within the image.

[0091] A depth estimation method is used to estimate the depth at which the stroke input is placed based on the position of the doodle stroke 504 from step 404. The relative depth estimation method is also configured to appropriately select the best object based on the dimensions of the multiple generated first candidate objects (p) present in the image.

[0092] refer to Figure 12AIf multiple objects are candidates for depth estimation, a depth estimation method is used, and this method determines the best object among the multiple objects based on depth relativity. The method begins with a stroke drawn by the user in the area where the object is to be placed, as shown in (i). Furthermore, if the depth of the generated object must change within the scene, the object's size / position also tends to change. The object must be moved backward in the image, and then its size / position must change accordingly, but this directly contradicts the size of the doodle drawn by the user, as shown in (ii) and (iii). Depth information from the stroke input can be determined based on the area where the doodle strokes are drawn, and an optimal object candidate that is best suited for placement on the image at a given depth can also be determined, as shown in (iv).

[0093] Specifically, in step 420, a relative depth estimation method is used to analyze the image received in step 402, the stroke input, and one or more second candidate objects selected in step 418. At step 422, the output generated by the relative depth estimation method in step 420 is used to determine M depth maps. In step 424, objects shared by the relative depth estimation method are filtered.

[0094] In this embodiment, the following operations will be followed to generate depth information: a. Obtain the depth map of the image.

[0095] b. Determine the depth of the stroke input based on the bounding box of the graffiti stroke.

[0096] c. Identify reference objects in an image based on class similarity and proximity to the bounding box of the graffiti strokes, or generate stroke objects as is if no reference object exists.

[0097] d. If a reference object exists, the generated object is filtered based on a comparison of the depth and size of the reference object and the generated object.

[0098] e. Returns a depth map of one or more filtered objects.

[0099] In such an embodiment, reference Figure 12B After determining the depth information and boundary region as shown in boxes 1202 and 1204, the boundary region can be placed on the determined depth information in box 1206. In box 1208, the determination module 316 can be configured to determine D. Gen D can be determined. Gen The bottom edge of the boundary region may overlap with the pixels of the depth information. Furthermore, the determination module 316 can be configured to determine M depth maps, as shown in box 1210.

[0100] Furthermore, as shown in boxes 1212 and 1214, the determining module 316 can be configured to be based on D Gen A ratio is determined, and the determined ratio is passed to the ratio comparison module, as shown in box 1216. Furthermore, as shown in boxes 1218, 1220, 1222, and 1224, the determination module 316 can be configured to determine a list of one or more selected second candidate ratios. Additionally, as shown in box 1216, the ratio list is passed to the ratio comparison module. Finally, after receiving the above input, the ratio comparison module generates M filtered objects, as shown in box 1226.

[0101] In an embodiment, the relative depth method includes an object depth estimation module based on stroke position and an object selection module based on depth.

[0102] refer to Figure 12B Boxes 1202 to 1210 illustrate a stroke-position-based object depth estimation module in a relative depth estimation method, used to determine the depth of the stroke input based on the position of the stroke drawn by the user. Additionally, a depth map of the image is obtained by filtering the generated object based on the depth dimensions of the generated object and the depth dimensions of a reference object in the image. The image is passed through a relative depth estimation module, such as Midas, to obtain the depth map. The depth of the generated object is determined based on the bounding box / boundary region of the doodle stroke, which lies on the obtained depth map. Furthermore, pixels in the obtained depth map of the image that overlap with the bottom edge of the stroke's bounding box define the depth at which the object needs to be placed as D. Gen .

[0103] In addition, refer to Figure 12C Based on the graffiti stroke O r The class similarity and proximity of the bounding boxes are used to determine reference objects in the image. In the absence of a reference object, stroke objects are generated in their original form. In the presence of a reference object, the generated objects are filtered based on a comparison of the depth dimensions of the reference object and the generated objects. Depth maps are returned for one or more filtered objects, ultimately generating objects according to the user's requirements.

[0104] Furthermore, the depth-based object selection module is based on several factors: i) the stroke input received as a doodle stroke at the location in the image where the original object must be generated; and ii) the depth of the stroke input in the scene must change, and then the size / position of the object tends to change as well. The depth of the object can be estimated using a scene depth map. The scene image is processed by a depth estimation model to obtain a depth map. The bounding box obtained from the user-drawn stroke is placed on the obtained depth map. The pixels of the image depth map that overlap with the bottom edge of the stroke's bounding box define the depth at which the object needs to be placed.

[0105] Furthermore, based on the image depth map, the depth of the stroke input, and considering multiple factors, this module identifies the best candidate objects that can be placed on the image. Additionally, it calculates the ratio of depth difference to size difference between the reference and the stroke input. The ratio of the stroke input depth to the object depth estimation module is obtained from the stroke position-based object depth estimation module, and the depth of the reference object is calculated from the image depth map. Compared with the data obtained from the ScaleRef network Compare to select the correct object.

[0106] This module defines a set of equations that determine the proportional ratio of an object to be placed at a location defined by a rectangle calculated from the stroke region. This ratio is determined based on the depth and size differences between the reference and the stroke input, i.e. ( ): Let D Ref For reference object O r depth Let D Gen To generate object O g depth make To generate object O g and O r depth Let Size(O) g () represents the height of the bounding box of the generated object. Let Size(O) r () represents the height of the bounding box of the reference object.

[0107] Let Scale Pred O g The absolute size relative to the O obtained using ScaleRefNet r The proportion of absolute size.

[0108] Let Scale Comp This represents the ratio of the size of Og to the size of Or obtained using the following equation:

[0109] If and only At that time, the generated object will look realistic. The generated objects will be filtered based on the following criteria: (Formula 3) Furthermore, the generation of depth information / graphs may include, but is not limited to, i) objects with the same depth and size, ii) objects with the same depth but smaller size, and iii) multiple objects with the same depth but smaller size, which will be explained in subsequent paragraphs: i) Objects of the same depth and size: refer to Figure 12D When a doodle stroke 504 has the same size as a reference object, and the depth information / map of the stroke input obtained from the object depth estimation module based on stroke position is the same as the depth information / map of the reference object, the ratio of the depth of the reference to the stroke input and the ratio of its size are both 1. Therefore, an object with a ratio of 1 from ScaleRefNet is selected and further passed to be added to the image. Objects with the same depth and size can be identified as:

[0110] Therefore, a choice was made with The object (Formula 4) ii) Objects of the same depth but smaller size: See reference Figure 12E When the graffiti stroke 504 has a smaller size compared to the reference object, and the depth information / map of the stroke input obtained from the object depth estimation module based on stroke position is the same as the depth of the reference object, the ratio of the reference to the stroke input depth is 1, and the ratio of the reference to the stroke input size is 0.5. Therefore, according to the algorithm, It will be 0.5. Therefore, objects with a scale of 0.5 from ScaleRefNet are selected. Objects with the same depth and smaller size can be identified as:

[0111] therefore,

[0112]

[0113] Therefore, a choice was made with The object (Formula 5) iii) Multiple small objects at the same depth: See reference Figure 12F When the graffiti stroke 504 has a smaller size compared to the reference object, and the depth information / map of the stroke input obtained from the object depth estimation module based on stroke position is the same as the depth of the reference object, the ratio of the reference to the stroke input depth is 1, and the ratio of the reference to the stroke input size is 0.5. Therefore, according to the algorithm, The scale will be 0.5. Therefore, objects with a scale of 0.5 from ScaleRefNet are selected and further passed to be added to the image. In this specific case, there is more than one object with a scale of 0.5 from ScaleRefNet, so more than one object will be passed further, and there can be more than one output of the entire doodle-assisted context-aware object generation method. Multiple small-sized objects of the same depth can be identified as:

[0114] therefore,

[0115]

[0116] Therefore, a choice was made with All objects (Formula 6) In the embodiment, reference Figure 12G The ScaleRef network is used to determine the scale of a reference object relative to the stroke input. The scale value represents the ratio of the size of the reference object to the size of multiple first candidate objects (p) generated from the doodle stroke while maintaining the same depth level. The ScaleRef network follows a regression network architecture based on a convolutional neural network design. Training data is prepared by annotating the object scale of one object relative to another. Furthermore, the range of scale values ​​can be described as follows: If the ratio value is greater than 1, and Size(O) Gen ) <Size(O Ref ) The ratio value = 1, if Size(O) Gen ) <Size(O Ref ) If the ratio value is less than 1, and Size(O) Gen Size(O) Ref ) (Formula 7) refer to Figure 12H (i-iv) illustrate several examples of the ScaleRef network. As shown in (i), the scale value of the reference object relative to the multiple first candidate objects (p) generated from the doodle strokes can be 2.1. As shown in (ii), the scale value of the reference object relative to the multiple first candidate objects (p) generated from the doodle strokes can be 0.7. As shown in (iii), the scale value of the reference object relative to the multiple first candidate objects (p) generated from the doodle strokes can be 0.5. Finally, as shown in (iv), the scale value of the reference object relative to the multiple first candidate objects (p) generated from the doodle strokes can be 200.

[0117] Furthermore, at step 426, the filtered object from step 424, the output generated from the M depth maps at step 422, the shared binary location information at step 414, and the image from step 402 are shared with the object image synthesizer, which uses the aforementioned inputs at step 428 to synthesize the final output image. The object image synthesizer employs a mask-based hybrid diffusion technique by utilizing and combining a control network model to guide editing toward local regions provided by the binary location information and a denoising diffusion probability model to generate a seemingly natural result. To seamlessly blend the final output image with the image background, the object image synthesizer spatially blends a noisy version of the image with a new object diffusion latent image at a noise level.

[0118] In one embodiment, the update module 326 may be configured to insert at least one of one or more selected second candidate objects (e.g., based on determined depth information and the recognition position of the stroke input) by an object image synthesizer. Figure 13 As shown and in Figure 10 and Figure 12B (See the explanation below) to update the image. Specifically, refer to... Figure 13 and 12B The determining module 316 can be configured to determine a depth map of the image based on at least one depth information of at least one object and at least one reference object within the image. Furthermore, the determining module 316 can be configured to determine the depth of each of one or more selected second candidates. The determining module 316 can be configured to determine the similarity and proximity of each of the selected one or more second candidates based on the corresponding depth and depth map of the image. Additionally, the filtering module 330 can be configured to filter at least one of the selected one or more second candidate objects based on the at least determined similarity and proximity.

[0119] Figures 14A-14B Use cases of system 204 according to embodiments of the present disclosure are shown.

[0120] refer to Figure 14A Through applications such as from Figures 4 to 13 The process described above involves system 204 generating the final output image according to the user's needs.

[0121] refer to Figure 14B Through applications such as from Figures 4 to 13 The process described above involves system 204 generating a complete final output image based on the user's needs.

[0122] Figure 15 A method 1500 for doodle-assisted context-aware object generation (DACAOG) performed by system 204 according to an embodiment of the present disclosure is shown.

[0123] Method 1500 can be executed by a programmed computing device, for example, based on instructions retrieved from a non-transitory computer-readable medium. The computer-readable medium may include machine-executable or computer-executable instructions to perform all or part of the described method. The computer-readable medium may be, for example, a digital storage device, a magnetic storage medium (such as disks and magnetic tapes), a hard disk drive, or an optically readable data storage medium.

[0124] Method 1500 includes Figure 15 Steps 1502 to 1516 illustrate a series of operations. Method 1500 can be executed by system 204 in conjunction with module 312, the details of which are as follows: Figures 3 to 13 The explanation has been provided and will not be repeated in this disclosure for the sake of brevity. Method 1500 begins with step 1502.

[0125] In step 1502, method 1500 includes detecting stroke input indicating a request to insert one or more objects into the image.

[0126] In step 1504, method 1500 includes recognizing the shape and color of the stroke input.

[0127] In step 1506, method 1500 includes using an artificial intelligence (AI) model to generate a plurality of first candidate objects based on the shape and color of the identified stroke input. Method 1500 includes using a low-pass filter to extract color-based features of the detected stroke input. Method 1500 includes generating a color feature map corresponding to the stroke input based on the extracted color-based features and one or more predefined color-related parameters. Method 1500 includes using a high-pass filter to extract shape-based features of the detected stroke input. Method 1500 includes generating a shape feature map corresponding to the stroke input based on the extracted shape-based features and one or more predefined shape-related parameters. Method 1500 uses a diffusion model to generate a plurality of first candidate objects based at least on the generated color feature map, the generated shape feature map, and multiple random noises.

[0128] In step 1508, method 1500 includes determining depth information of one or more objects to be inserted into the image based on the location of the detected stroke input and one or more adjacent objects within the image.

[0129] In one embodiment, method 1500 includes determining depth information of the stroke input based on the location of the detected stroke input and one or more neighboring objects within the image. Method 1500 also includes determining a depth map based on the image, one or more selected second candidate objects, the location of the detected stroke input, and identified boundary regions within the image.

[0130] In step 1510, method 1500 includes recognizing the context of the scene of the image.

[0131] In step 1512, method 1500 includes identifying the location of a stroke input within an image based on the detected stroke input. Method 1500 includes determining the size of the stroke input based on the detected stroke input. Method 1500 includes identifying a boundary region within the image based on the determined size of the stroke input. Method 1500 includes identifying the location of the stroke input within the image based on the identified boundary region.

[0132] In step 1514, method 1500 includes selecting one or more second candidate objects from a plurality of first candidate objects based on the context of the identified scene and the location of the identified stroke input. Method 1500 includes determining a relevance score associated with each of the plurality of first candidate objects based on the identified scene context and the identified stroke input location. Method 1500 includes selecting one or more second candidate objects from the plurality of first candidate objects based on the determined corresponding relevance scores.

[0133] In step 1516, method 1500 includes updating the image by inserting one or more selected second candidate objects based on determined depth information and the position of the identified stroke input. To update the image, method 1500 includes generating a binary mask on the image based on identified boundary regions. Method 1500 includes placing at least one of the selected one or more second candidate objects based on the generated binary mask and the determined depth information. Furthermore, to update the image using at least one of the one or more second candidate objects, method 1500 includes determining a depth map of the image based at least on determined depth information of at least one object within the image and at least one reference object. Method 1500 includes determining the depth of each of the selected one or more second candidates. Method 1500 includes determining the similarity and proximity of at least one of the selected one or more second candidates based on the corresponding depth and the depth map of the image. Method 1500 includes filtering at least one of the selected one or more second candidate objects based on the at least determined similarity and proximity.

[0134] This disclosure presents us with various advantages, such as: This method provides a more controlled way of generating objects in images.

[0135] Defining and locating objects is accomplished using a single user input.

[0136] Multiple previous derivations can be inferred based on the strokes drawn by the user and the input image provided by the user.

[0137] This method can generate images of objects that the model cannot see or that do not exist in the real world.

[0138] Specifically, the doodle-stroke-based object generation engine effectively overcomes the problems mentioned in the prior art because object generation is tightly guided by user-provided colored strokes. The engine is designed so that object generation strictly adheres to the appearance attributes defined by the shape, color, and size of the doodle strokes, remaining unaffected by any other content or artifacts in the image.

[0139] The image-object-location mapping module has the ability to estimate whether any object at a given location in an image is appropriate. This ability can be used by any image generation model to identify whether the generated image has appropriate meaning by analyzing objects in the image. Unlike existing technologies, this engine is used to limit generative AI models from generating inappropriate / dirty objects in images by detecting each object in the image and identifying whether any object is obscene or misplaced by understanding the context of the image. The engine also suggests appropriate objects that can replace any obscene objects present in the image.

[0140] Furthermore, the depth estimation module / method provides the ability to estimate and control the depth at which objects can be placed. This module identifies the depth at which objects can be placed and can also suggest whether the object is suitable for placement in the image, which differs from existing technologies.

[0141] In this application, unless otherwise specifically stated, the use of the singular includes the plural, and the use of "or" means "and / or". Furthermore, the use of the terms "comprising" or "having" is not limiting. Any scope described herein will be understood to include all values ​​between endpoints. Features of the disclosed embodiments may be combined, rearranged, omitted, etc., within the scope of this disclosure to produce additional embodiments. Furthermore, certain features may sometimes be advantageously used without the corresponding use of other features.

[0142] Although at least one exemplary embodiment has been presented in the foregoing detailed description, it should be understood that numerous variations exist.

[0143] In this document, where appropriate, one or more computer-readable non-transitory storage media may include one or more semiconductor-based or other integrated circuits (ICs) such as, for example, field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical discs, optical disc drives (ODDs), magneto-optical drives, floppy disks, floppy disk drives (FDDs), magnetic tape, solid-state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these. Where appropriate, computer-readable non-transitory storage media may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0144] In this document, unless otherwise expressly stated or the context otherwise indicates, "or" is inclusive rather than exclusive. Therefore, in this document, unless otherwise expressly stated or the context otherwise indicates, "A or B" means "A, B, or both". Furthermore, unless otherwise expressly stated or the context otherwise indicates, "and" is both union and multiple. Therefore, in this document, unless otherwise expressly stated or the context otherwise indicates, "A and B" means "A and B, either unionly or individually".

[0145] The scope of this disclosure covers all changes, substitutions, variations, alterations, and modifications to the exemplary embodiments described or illustrated herein that will be understood by those skilled in the art. The scope of this disclosure is not limited to the exemplary embodiments described or illustrated herein. Furthermore, although this disclosure describes and illustrates corresponding embodiments herein as including specific components, elements, features, functions, operations, or steps, any of these embodiments may include any combination or arrangement of any components, elements, features, functions, operations, or steps described or illustrated anywhere herein that will be understood by those skilled in the art.

[0146] A method (1500) for object insertion in an image includes: detecting (1502) a stroke input indicating a request to insert one or more objects into the image; identifying (1504) the shape and color of the stroke input; generating (1506) a plurality of first candidate objects based on the identified stroke input using an artificial intelligence (AI) model; determining (1508) depth information of the one or more objects to be inserted into the image based on the position of the detected stroke input and one or more adjacent objects within the image; identifying (1510) the context of the scene of the image; identifying (1512) the position of the stroke input within the image; selecting (1514) one or more second candidate objects from the plurality of first candidate objects based on the identified context of the scene and the identified position of the stroke input; and updating (1516) the image by inserting the selected one or more second candidate objects based on the determined depth information and the position of the identified stroke input.

[0147] Identifying the location of a stroke input within an image includes: identifying the size of the stroke input based on the detected stroke input, identifying a boundary region within the image based on the identified stroke input size, and identifying the location of at least one stroke input within the image based on the identified boundary region.

[0148] Determining the depth information to be inserted into one or more objects in an image includes: determining a depth map based on the image, one or more selected second candidate objects, the location of detected stroke inputs, and identified boundary regions within the image.

[0149] To update the image, the method (1500) includes: generating a binary mask on the image based on the identified boundary region, and placing at least one of the selected one or more second candidate objects based on the generated binary mask and the determined depth information.

[0150] To generate multiple first candidate objects, the method (1500) includes: extracting color-based features of the detected stroke input using a low-pass filter; generating a color feature map corresponding to the stroke input based on the extracted color-based features and one or more predefined color-related parameters; extracting shape-based features of the detected stroke input using a high-pass filter; generating a shape feature map corresponding to the stroke input based on the extracted shape-based features and one or more predefined shape-related parameters; and generating multiple first candidate objects using a diffusion model based at least on the generated color feature map, the generated shape feature map, and multiple random noises.

[0151] Selecting one or more second candidate objects from a plurality of first candidate objects includes: determining a relevance score associated with each of the plurality of first candidate objects based on the identified scene context and the position of the identified stroke input, and selecting one or more second candidate objects from the plurality of first candidate objects based on the determined corresponding relevance scores.

[0152] Updating an image using one or more selected second candidate objects includes: determining a depth map of the image based at least on determined depth information of one or more objects to be inserted into the image and at least one reference object within the image; determining the depth of each of the one or more selected second candidates; determining the similarity and proximity of each of the one or more selected second candidates based on the corresponding depth of the image and the depth map; and filtering at least one of the one or more selected second candidate objects based on the determined similarity and proximity.

[0153] A system (204) for object insertion in an image, the system (204) comprising: a memory (308) configured to store at least one instruction; at least one processor (304) communicatively coupled to the memory (308), the at least one processor (304) being configured to execute at least one instruction to: detect a stroke input indicating a request to insert one or more objects into the image; identify the shape and color of the stroke input; generate a plurality of first candidate objects based on the identified shape and color of the stroke input using an artificial intelligence (AI) model; determine depth information of the one or more objects to be inserted into the image based on the position of the detected stroke input and one or more adjacent objects within the image; identify the context of a scene in the image; identify the position of the stroke input within the image; select one or more second candidate objects from the plurality of first candidate objects based on the identified context of the scene and the identified position of the stroke input; and update the image by inserting the selected one or more second candidate objects based on the determined depth information and the position of the identified stroke input.

[0154] In order to identify the location of a stroke input within an image, at least one processor (304) is configured to identify the size of the stroke input based on the detected stroke input, identify a boundary region within the image based on the identified stroke input size, and identify the location of the stroke input within the image based on the identified boundary region.

[0155] In order to determine the depth information of one or more objects to be inserted into an image, at least one processor (304) is configured to determine a depth map based on the image, one or more selected second candidate objects, the location of detected stroke inputs, and the identified boundary regions within the image.

[0156] In order to update the image, at least one processor (304) is configured to generate a binary mask on the image based on the identified boundary regions, and place at least one of the selected one or more second candidate objects based on the generated binary mask and the determined depth information.

[0157] To generate multiple first candidate objects, at least one processor (304) is configured to use a low-pass filter to extract color-based features of the detected stroke input, generate a color feature map corresponding to the stroke input based on the extracted color-based features and one or more predefined color-related parameters, use a high-pass filter to extract shape-based features of the detected stroke input, generate a shape feature map corresponding to the stroke input based on the extracted shape-based features and one or more predefined shape-related parameters, and generate multiple first candidate objects using a diffusion model based at least on the generated color feature map, the generated shape feature map, and multiple random noises.

[0158] In order to select one or more second candidate objects from a plurality of first candidate objects, at least one processor (304) is configured to determine a relevance score associated with each of the plurality of first candidate objects based on the identified scene context and the identified stroke input position, and to select one or more second candidate objects from the plurality of first candidate objects based on the determined corresponding relevance scores.

[0159] In order to update an image using one or more selected second candidate objects, at least one processor (204) is configured to determine a depth map of the image based at least on determined depth information of one or more objects to be inserted into the image and at least one reference object within the image, determine the depth of each of the selected one or more second candidates, determine the similarity and proximity of each of the selected one or more second candidates based on the corresponding depth of the image and the depth map, and filter at least one of the selected one or more second candidate objects based on the determined similarity and proximity.

[0160] One or more non-transitory computer-readable storage media storing instructions and coupled to one or more processors, the processors being operable to execute the instructions to: detect stroke input indicating a request to insert one or more objects into an image; identify the shape and color of the stroke input; generate a plurality of first candidate objects based on the identified shape and color of the stroke input using an artificial intelligence (AI) model; determine depth information of the one or more objects to be inserted into the image based on the position of the detected stroke input and one or more adjacent objects within the image; identify the context of the scene of the image; identify the position of the stroke input within the image; select one or more second candidate objects from the plurality of first candidate objects based on the identified context of the scene and the identified position of the stroke input; and update the image by inserting the selected one or more second candidate objects based on the determined depth information and the position of the identified stroke input.

Claims

1. A method (1500) for object insertion in an image, comprising: Detection (1502) indicates a stroke input requesting the insertion of one or more objects into the image; Identify the shape and color of the stroke input described in (1504); Using an artificial intelligence (AI) model, multiple first candidate objects (1506) are generated based on the shape and color of the identified stroke input; Based on the detected position of the stroke input and one or more adjacent objects within the image, the depth information of the one or more objects to be inserted into the image is determined (1508); Identify the context of the scene of the image (1510); Identify (1512) the position of the stroke input within the image; Based on the context of the identified scene and the position of the identified stroke input, one or more second candidate objects are selected from the plurality of first candidate objects (1514); as well as The image is updated (1516) by inserting one or more selected second candidate objects based on the determined depth information and the position of the identified stroke input.

2. The method (1500) according to claim 1, wherein, The steps of identifying the position of the stroke input within the image include: The size of the stroke input is identified based on the detected stroke input; Identify boundary regions within the image based on the size of the identified stroke input; and The location of the at least one stroke input within the image is identified based on the identified boundary region.

3. The method (1500) according to claim 2, wherein, The steps of determining the depth information of the one or more objects to be inserted into the image include: A depth map is determined based on the image, the selected one or more second candidate objects, the detected position of the stroke input, and the identified boundary region within the image.

4. The method (1500) according to claim 3, wherein, To update the image, the method (1500) includes: A binary mask is generated on the image based on the identified boundary region; and At least one of the selected one or more second candidate objects is placed based on the generated binary mask and the determined depth information.

5. The method (1500) according to any one of the preceding claims, wherein, To generate the plurality of first candidate objects, the method (1500) includes: The detected stroke input is extracted using a low-pass filter based on color. A color feature map corresponding to the stroke input is generated based on the extracted color-based features and one or more predefined color-related parameters. The detected stroke input is extracted using a high-pass filter based on shape features. Based on the extracted shape-based features and one or more predefined shape-related parameters, a shape feature map corresponding to the stroke input is generated; and The plurality of first candidate objects are generated using a diffusion model, based at least on the generated color feature map, the generated shape feature map, and multiple random noises.

6. The method (1500) according to any one of the preceding claims, wherein, The step of selecting one or more second candidate objects from the plurality of first candidate objects includes: A relevance score associated with each of the plurality of first candidate objects is determined based on the context of the identified scene and the position of the identified stroke input; and Based on the determined corresponding relevance scores, one or more second candidate objects are selected from the plurality of first candidate objects.

7. The method (1500) according to any one of the preceding claims, wherein, The step of updating the image using the selected one or more second candidate objects includes: The depth map of the image is determined based at least on the depth information of the one or more objects to be inserted into the image and at least one reference object within the image; Determine the depth of each of the selected one or more second candidate objects; The similarity and proximity of each of the selected one or more second candidate objects are determined based on the corresponding depth and the depth map of the image; and At least one of the selected one or more second candidate objects is filtered based on the determined similarity and proximity.

8. A system (204) for object insertion in an image, the system (204) comprising: The memory (308) is configured to store at least one instruction; At least one processor (304), communicatively coupled to the memory (308), is configured to execute the at least one instruction to perform the following operations: Detect stroke inputs that indicate a request to insert one or more objects into the image; Identify the shape and color of the input strokes; Using an artificial intelligence (AI) model, multiple first candidate objects are generated based on the shape and color of the identified stroke input; The depth information of the one or more objects to be inserted into the image is determined based on the detected position of the stroke input and one or more adjacent objects within the image; Identify the context of the scene depicted in the image; Identify the position of the stroke input within the image; Based on the context of the identified scene and the position of the identified stroke input, select one or more second candidate objects from the plurality of first candidate objects; as well as The image is updated by inserting one or more selected second candidate objects based on the determined depth information and the identified position of the stroke input.

9. The system (204) according to claim 8, wherein, In order to identify the location of the stroke input within the image, the at least one processor (304) is configured to: The size of the stroke input is identified based on the detected stroke input; The boundary regions within the image are identified based on the size of the identified stroke input; as well as The location of the stroke input within the image is identified based on the identified boundary region.

10. The system (204) according to claim 9, wherein, In order to determine the depth information of the one or more objects to be inserted into the image, the at least one processor (304) is configured to: A depth map is determined based on the image, the selected one or more second candidate objects, the detected position of the stroke input, and the identified boundary region within the image.

11. The system (204) according to claim 10, wherein, In order to update the image, the at least one processor (304) is configured to: A binary mask is generated on the image based on the identified boundary region; as well as At least one of the selected one or more second candidate objects is placed based on the generated binary mask and the determined depth information.

12. The system (204) according to any one of the preceding claims, wherein, In order to generate the plurality of first candidate objects, the at least one processor (304) is configured to: A low-pass filter is used to extract color-based features from the detected stroke input; A color feature map corresponding to the stroke input is generated based on the extracted color-based features and one or more predefined color-related parameters. The detected stroke input is extracted using a high-pass filter based on shape features. Based on the extracted shape-based features and one or more predefined shape-related parameters, a shape feature map corresponding to the stroke input is generated; as well as The plurality of first candidate objects are generated using a diffusion model, based at least on the generated color feature map, the generated shape feature map, and multiple random noises.

13. The system (204) according to any one of the preceding claims, wherein, In order to select one or more second candidate objects from the plurality of first candidate objects, the at least one processor (304) is configured to: A relevance score associated with each of the plurality of first candidate objects is determined based on the context of the identified scene and the position of the identified stroke input; as well as Based on the determined corresponding relevance scores, one or more second candidate objects are selected from the plurality of first candidate objects.

14. The system (204) according to any one of the preceding claims, wherein, In order to update the image using the selected one or more second candidate objects, the at least one processor (204) is configured to: The depth map of the image is determined based at least on the depth information of the one or more objects to be inserted into the image and at least one reference object within the image; Determine the depth of each of the selected one or more second candidates; The similarity and proximity of each of the selected one or more second candidate objects are determined based on the corresponding depth and the depth map of the image. as well as At least one of the selected one or more second candidate objects is filtered based on the determined similarity and proximity.

15. A non-transitory computer-readable storage medium storing instructions for performing the method according to any one of claims 1 to 7.

Images