Electronic devices, and the methods employed by these electronic devices to create images.

Abstract

The invention relates to an electronic device, and the method by which such electronic device produces images.An electronic device comprises: a display unit; at least one processor including a processing circuit; and memory that stores instructions which, when executed individually or collectively by at least one processor, cause the electronic device to: display the first image comprising the sketch, based on the first drawing input; receive the second drawing input corresponding to the circular drawing around at least part of the sketch contained in the first image; identify the second drawing input, corresponding to the circular drawing around at least part of the sketch contained in the first image, as a control instruction to create an image from the sketch; recognize, based on the identification of the second drawing input as a control instruction to create an image from the sketch, one or more first attributes of the sketch; recognize an object comprising one or more second attributes corresponding to one or more first attributes recognized of the sketch; create a second image comprising the recognized object; and provide the second image.

Description

Electronic device, method and storage medium for generating images

[0001] Embodiments of the present document relate to electronic devices, methods and storage media, for example, electronic devices, methods and storage media for generating images.

[0002] Electronic devices provide users with a variety of functions. For example, electronic devices can capture images and display them. They also provide image editing capabilities. Typically, electronic devices can provide image editing functions that crop images or stitch two images together. Furthermore, electronic devices can provide image editing functions that extract objects from images, move and arrange objects within an image, or add new objects.

[0003] The above information may be provided solely as background information to aid in understanding the present disclosure. None of the above-described matters are claimed as prior art related to the present disclosure or can be used in determining prior art.

[0004] A method for generating an image according to various embodiments of the present document may include an operation of receiving a second drawing input around an object included in the image. The method may include an operation of identifying an attribute of the object when the second drawing input is determined to be a control command based on the object and the second drawing input. The method may include an operation of generating a first preliminary object image based on the attribute of the identified object. The method may include an operation of providing the generated first preliminary object image.

[0005] An electronic device according to various embodiments of the present document may include a display for displaying an image, at least one processor including a processing circuit, and a memory for storing instructions that are individually or collectively executed by the at least one processor. The instructions stored in the memory may be configured to cause the electronic device to receive a second drawing input around an object included in the image. The instructions may be configured to cause the electronic device to identify an attribute of the object when the second drawing input is determined to be a control command based on the object and the second drawing input. The instructions may be configured to cause the electronic device to generate a first preliminary object image based on the attributes of the identified object. The instructions may be configured to cause the electronic device to provide the generated first preliminary object image.

[0006] A non-transitory computer-readable storage medium having recorded thereon a program for performing a method for generating an image according to various embodiments of the present document may include an operation for receiving a second drawing input around an object included in the image. The storage medium may include an operation for identifying an attribute of the object when the second drawing input is determined to be a control command based on the object and the second drawing input. The storage medium may include an operation for generating a first preliminary object image based on the attribute of the identified object. The storage medium may include an operation for providing the generated first preliminary object image.

[0007] Various embodiments of this document may provide high-level image processing capabilities that take into account scenes or subjects contained in an image, for example, with intuitive user input.

[0008] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0009] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0010] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to various embodiments.

[0011] FIGS. 3A and 3B are drawings illustrating a process of editing an image according to various embodiments.

[0012] FIGS. 4A, 4B, and 4C are drawings illustrating a process of recognizing an object based on drawing elements according to various embodiments.

[0013] FIG. 5 is a diagram illustrating a process for generating an edited image according to various embodiments.

[0014] FIGS. 6A and 6B are drawings illustrating examples of editing an image according to various embodiments.

[0015] FIG. 7A and FIG. 7B are diagrams illustrating examples of determining an object by analyzing a user's input according to various embodiments.

[0016] FIGS. 8A, 8B, 8C, and 8D are drawings illustrating examples of moving an object within an image while maintaining realism according to various embodiments.

[0017] FIGS. 9A, 9B, and 9C are drawings illustrating examples of moving an object using depth information according to various embodiments.

[0018] FIG. 10a, FIG. 10b, FIG. 10c, FIG. 10d, FIG. 10e, and FIG. 10f are drawings illustrating examples of adding and moving objects according to various embodiments.

[0019] FIGS. 11a, 11b, 11c, 11d, 11e, and 11f are drawings illustrating examples of adding and moving multiple objects according to various embodiments.

[0020] FIGS. 12a, 12b, 12c, 12d, 13a, 13b, 13c and 13d are drawings illustrating examples of changing the shape of a portion of an object according to various embodiments.

[0021] FIGS. 14a, 14b, and 14c are diagrams illustrating examples of adding objects using different information according to various embodiments.

[0022] FIGS. 15a, 15b, 16a, and 16b are drawings illustrating examples of changing the color of an area within an image according to various embodiments.

[0023] FIG. 17A and FIG. 17B are drawings illustrating examples of editing an image by motion of an electronic device according to various embodiments.

[0024] FIGS. 18a, 18b, and 18c are drawings illustrating examples of displaying images of various sizes using a completed object image according to various embodiments.

[0025] FIGS. 19 and 20 are drawings illustrating examples of generating images based on drawings according to various embodiments.

[0026] FIG. 21 is a diagram illustrating a mixed of experts (MoE) layer according to various embodiments.

[0027] FIG. 22 is a diagram illustrating an operation for processing text input according to various embodiments.

[0028] FIGS. 23a, 23b, 23c, 23d, 23e, 24a, 24b, 25a, 25b, 25c, 26a, 26b, 27a, 27b, 28, 29, 30a, 30b, 30c, 30d and 31 are drawings illustrating examples of generating and / or modifying an image based on input drawing properties and / or drawing-based control functions according to various embodiments.

[0029] Figure 32 is a flowchart illustrating an image editing method according to various embodiments.

[0030] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0031] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0032] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0033] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0034] The memory (130) can store various data used by at least one component (e.g., the processor (120) or the sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., the program (140)) and input data or output data for commands related thereto. The memory (130) can include a volatile memory (132) or a non-volatile memory (134). The non-volatile memory (134) can include at least one internal memory (136) and an external memory (138).

[0035] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0036] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0037] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0038] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0039] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0040] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0041] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0042] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0043] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0044] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0045] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0046] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0047] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0048] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0049] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0050] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0051] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0052] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0053] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0054] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to various embodiments.

[0055] Referring to FIG. 2, the electronic device (101) may include a memory (130), a processor (120), and a display (160).

[0056] The memory (130) (e.g., the memory (130) of FIG. 1) may store data, algorithms, programs, commands, etc. that perform functions of the electronic device (101). For example, the memory (130) may store images, image editing algorithms, preliminary object images, and image editing models learned by machine learning or deep learning. For example, the image editing model may include an image analysis model, a user input analysis model, a preliminary object generation model, an atmosphere / lighting model, and / or a depth map generation model. For example, the artificial intelligence-based image editing model may include a plurality of artificial neural network layers. The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above. Each of the artificial intelligence-based image editing models may be included in one artificial intelligence computing module included in the electronic device (101) or may be included in different artificial intelligence computing modules. For example, each of the artificial intelligence-based image editing models may be included in one artificial intelligence computing module included in a separate server (e.g., server (108) of FIG. 1) or may be included in different artificial intelligence computing modules. The artificial intelligence-based image editing model may be a logic, firmware, software, and / or hardware module that can perform an operation using at least one artificial intelligence computing module.

[0057] The processor (120) (e.g., the processor (120) of FIG. 1) can control each component of the electronic device (101) by executing instructions stored in the memory (130). The electronic device (101) may include one or more processors (120). For example, the processor (120) may correspond to a plurality of processors that collectively perform a plurality of functions by dividing them among the processors.

[0058] The processor (120) can display an image (e.g., an original image) on the display (160) (e.g., the display module (160) of FIG. 1). For example, the image can include a still image (e.g., a still image) and a dynamic image (e.g., a video). For example, the dynamic image (e.g., a video) can be provided frame by frame or scene by scene. For example, a selected frame or a key scene from the video can be displayed for image editing, and editing of the frame or scene can be applied to other related frames or scenes. When a user input is received, the processor (120) can identify the user input. For example, the user input can be an input for adding at least one object to an image being displayed on the display, an input for modifying an object, or a control input. In various embodiments of the present document, the object can include an object, a person, and / or a background. For example, the background can be an object such as the sea, a mountain, the sky, or a river behind an object or person. For example, inputs for adding objects may include inputs for adding new objects and / or people. Inputs for modifying objects may include inputs for changing the size or shape of an object, inputs for changing the color, and / or inputs for adding colors. Control inputs may be inputs for controlling an object or adding information about the object in a way other than drawing the shape of the object. For example, if an arrow is input on (or adjacent to) an object, the processor (120) may move the object based on the direction and length of the arrow. For example, if the text “log” is input on (or adjacent to) an object in the shape of a house, the processor (120) may identify the object as a log cabin. Arrows or text input by the user may be examples of control inputs. That is, the processor (120) may identify whether the user’s input is an input for adding an object, an input for modifying an object, or a control input.

[0059] For example, the processor (120) can identify an object that the user wants to add based on a shape (drawing shape) formed by the user input. For example, the user input is a touch gesture, and the processor (120) can identify an object that the user wants to add from a shape formed by the touch gesture. As an example, the user input is a motion gesture by the user's hand (or finger), and the processor (120) can identify an object that the user wants to add from a shape formed by the motion gesture. As an example, the user input may include an input using a remote controller capable of motion recognition in an XR (Extended Reality) environment or a user gesture recognized using a camera. For example, the user input is an input distinguished by a touch / measurement on the display (160) (e.g., a touch screen), and the processor (120) can identify the user input based on the presence or absence of a touch. For example, if a finger (or an electronic pen) comes into contact with the display (160) and draws, the processor (120) can identify this as a user input for adding an object. If a finger (or an electronic pen) hovers and draws, the processor can identify this as a user input for modifying an object (e.g., changing a texture, changing a color, or adding a color). For example, if the user input is a drawing input on the display (160), it can be identified as an input for adding an object, and if the drawing is input through a motion sensor (e.g., an acceleration sensor), it can be identified as an input for modifying an object.

[0060] According to one embodiment, the user input may be input using various input devices and / or input using various input methods, and the processor (120) may identify the user input based on the type of the input device and / or the input method. For example, if the user input is drawing using an electronic pen, the processor (120) may identify it as an input for adding an object, and if the user input is drawing using a finger, the processor (120) may identify it as an input for modifying an object. For example, if the user input is drawn without pressing the button of the electronic pen, the processor (120) may identify it as an input for adding an object, and if the user input is drawn while the button of the electronic pen is pressed, the processor (120) may identify it as an input for modifying an object.

[0061] If the user input is identified as an input for adding an object (e.g., a first object), the processor (120) may request an artificial intelligence-based computing device (e.g., an artificial intelligence-based computing module or an artificial intelligence-based image editing model) to generate a preliminary object image (e.g., a preliminary image for the first object). As an example, the preliminary object image may be generated as image information used to display an image for the object, in addition to being generated as an image. The image information for displaying or generating the object may include, for example, image information expressing a two-dimensional overall view of the object, image information expressing a three-dimensional overall view of the object, image information expressing various views that may appear differently depending on the angle at which the object is viewed, image information expressing various views that may appear differently depending on the focal length, and image information expressing various views that may appear differently depending on the angle of view, etc. The first preliminary object image may be generated as first object image information.

[0062] For example, an artificial intelligence-based computing device may include an image editing model trained by machine learning or deep learning, and the image editing model may be a generative artificial intelligence model. The generative artificial intelligence model may be a deep learning text-to-image model that receives a prompt and generates an image corresponding to the prompt. For example, the prompt may include various types of content such as text, images, videos, web addresses, and files. For example, the generative artificial intelligence model may generate images using various computational models such as a generative adversarial network (GAN), a vector quantization GAN (VQ-GAN), a variational auto-encoder (VAE), a VQ-VAE, a Diffusion, or a Diffusion-GAN model.

[0063] The generative adversarial network (GAN) model is an artificial intelligence neural network that uses an adversarial neural network structure and can utilize fake data similar to real data for training. The GAN model trains the generator by having a discriminator compare and evaluate images generated by the generator with original images. The variable-area neural network (VAE) model is a probabilistic artificial neural network that can perform generative modeling by encoding data as a probability distribution in a latent space. The VAE model can act as a generator. Through an encoder, the VAE model extracts and samples features (e.g., eyes, nose, mouth, etc.) that best represent the input image (X) to generate a latent vector (Z). Through a decoder, the VAE model can generate new data from Z that is most similar to X. Compared to the GAN model, the VAE model has an additional encoder / decoder and can train the latent vector, encoder, and decoder simultaneously. The diffusion model can be trained using a forward process (diffusion process) that gradually adds noise to the data to completely denoise the data, and a reverse process that gradually restores the noise to create new data.

[0064] For example, the processor (120) may load an image editing model from the memory (130) and request generation of a preliminary object image. In one example, the image editing model may be stored in an external device (e.g., the server (108) of FIG. 1). The electronic device (101) includes a communication interface (e.g., the communication module (190) of FIG. 1), and the processor (120) may request generation of a preliminary object image using the image editing model through the communication interface. That is, the image editing model may be a general-purpose model provided entirely by the external device. In one example, a part of the image editing model may be provided by the electronic device (101), or the entire model may be provided by the electronic device (101) (on-device form).

[0065] For example, if the image editing model is provided entirely from an external device, the electronic device (101) can transmit the original image selected by the user and user input information to the external device. The external device can identify objects (including the background) added from the user input information. The external device can generate preliminary object images for the identified objects and objects extracted from the original image, and transmit a first edited image reflecting the object's rearrangement and lighting characteristics to the electronic device (101). The electronic device (101) can display the received first edited image. If an object in the first edited image is moved by the user, the electronic device (101) can transmit information related to the object whose position has changed (e.g., information on the moved object, coordinate information, etc.) to the external device. The external device can transmit a second edited image reflecting the lighting characteristics and depth information to the electronic device (101) based on the received information. The electronic device (101) can display the second edited image received from the external device.

[0066] According to one embodiment, the preliminary object generation model and the depth map generation model may be provided by an external device (e.g., a server), and other models (e.g., an image analysis model, a user input analysis model, an atmosphere / lighting model) may be provided by the electronic device (101). For example, the electronic device (101) may identify an object to be added from user input information. In addition, the electronic device (101) may analyze an original image and identify information related to an object included in the original image. The electronic device (101) may transmit information on an object to be added and information on an existing object included in the original image to the external device. The external device may generate preliminary object images for the object to be added and the existing object based on the information received from the electronic device (101). In addition, the external device may generate depth maps for the object to be added and the existing object. The external device may transmit the preliminary object image and depth map information to the electronic device (101). The electronic device (101) can reposition objects based on depth map information and preliminary object images and generate a first edited image reflecting lighting characteristics.

[0067] According to one embodiment, all image editing models may be provided in the electronic device (101). When the electronic device (101) receives drawing information by a user input, it may generate a preliminary object image through a preliminary object generation model and generate a first edited image including the generated preliminary object image. Lighting characteristics may be applied to the first edited image. The electronic device (101) may obtain depth information of the first edited image through a depth map generation model. When the electronic device (101) receives an image editing command (e.g., object movement) from a user, the electronic device (101) may edit the first edited image according to the user input. When the object is moved, the electronic device (101) may update the depth information and generate a second edited image to which lighting characteristics according to the object movement are applied.

[0068] An artificial intelligence computing device can identify attributes of a first object (e.g., an object formed by a user input) based on a shape formed by a user input. For example, the artificial intelligence computing device can analyze an original image and identify attributes of the first object together with information of the original image. For example, information of the image may include information about the scene represented by the image (e.g., landscape, building, person, food, city, countryside, indoor, outdoor, etc.), temporal / visual information represented by the image (e.g., sunset, sunrise, night, etc.), mental images associated with the image (e.g., comfort, warmth, liveliness, etc.), specific information about an object (e.g., subject) included in the image (e.g., the user, a specific person, the user's house, the user's car, etc.), the relative sizes of objects included in the image, and / or image / object information surrounding the user input. For example, attributes of an object may include the type, shape, texture, color, characteristics, and / or additional information of the object. As an example, if the user's input in an indoor image has a rectangular shape, the artificial intelligence computing device can identify the attribute of the first object as furniture. For example, an artificial intelligence computing device can identify the properties of an object drawn on a person's head as a hat.

[0069] When the artificial intelligence computing device is stored in the memory (130) of the electronic device (101), the artificial intelligence computing device can be loaded into the processor (120) and perform an operation. In this case, the processor (120) can identify the properties of the first object.

[0070] The processor (120) (e.g., an artificial intelligence computing device) may generate a first preliminary object image of the first object based on the identified attributes. For example, the first preliminary object image may refer to an image of the overall shape of the first object. The processor (120) may identify an object from an image that includes an object with attributes identical to or related to the attributes of the identified first object among images stored in the memory (130), and generate the first preliminary object image. According to an example, the processor (120) may search for images of a shopping mall site, a portal site, and / or a cloud of a user account through a network. The processor (120) may identify an object from an image that includes an object with attributes identical to or related to the attributes of the identified first object among the searched images, and generate the first preliminary object image. The processor (120) may generate the first preliminary object image using an object included in the stored images and / or the searched images. In one example, the processor (120) can generate a first preliminary object image from user input by referring to stored images and / or retrieved images.

[0071] The processor (120) may generate one first preliminary object image and may generate a plurality of first preliminary object images. For example, the plurality of first preliminary object images may be a plurality of images including respective shapes that can be captured at different focal distances with respect to the first object. In one example, the plurality of first preliminary object images may include a plurality of object images each including different styles. For example, the processor (120) may generate a preliminary object image of the entire appearance of a person, including the lower body, based on a user's drawing depicting the upper body of the person. The processor (120) may pre-generate a plurality of first preliminary objects that differ in clothing, posture, expression, etc. In one example, the plurality of first preliminary object images may be images of similar objects that match the user input information and fall within the type of the first object. For example, if the user input is a red shape and the first object according to the shape of the user input is identified as a hat, the processor (120) may generate a plurality of red hats as the first preliminary object images. When the processor (120) generates a first preliminary object image of a plurality of similar first objects, the processor (120) may display the first preliminary object images of the generated plurality of similar first objects and determine one first preliminary object according to a user's selection. For example, the first preliminary object image may be referred to as a candidate image since it may be used in an image editing process, and may be referred to as a complete image since it is generated as an image that includes the overall shape of the object.

[0072] According to one embodiment, if the background of the original image and the properties of the first object do not match, the processor (120) may insert a new object (e.g., another object or another part of the object that can be expressed together with the first object to assist the first object) to generate a first preliminary object image. For example, the processor (120) may identify a relationship (or suitability, consistency) between the properties of the first object and the environmental information of the image (or the context of the image). The properties of the first object may include the type and / or characteristics (features) of the first object. The environmental information of the image may include location information, time information, and / or weather information. The processor (120) may generate a first preliminary object image (or information of the first preliminary object image) that includes a new object related to the first object based on the relationship. The generated first preliminary object image (or information of the first preliminary object image) may include an object (or object information) identified from a shape drawn by a user input and a new object (or information of the new object) related to the first object.

[0073] For example, one of the attributes of the background of the image may be indoors or a living room, and one of the attributes of the first object may be a tree or a plant. The processor (120) may determine that the information of the image background is indoors, and may determine that the object identified by the user input is a tree. The processor (120) may determine the relevance between the location information of indoors and the attribute of the object of tree. It may be realistically impossible for a tree to be planted on the floor of an indoor space. For example, based on the determined relevance, the processor (120) may insert a flowerpot that was not input by the user as a new object, and generate a tree planted in the flowerpot as a first preliminary object image.

[0074] According to one embodiment, the processor (120) may identify an existing object (e.g., a second object) included in an original image and request generation of a preliminary image for the existing object (e.g., a preliminary image for the second object). If there are multiple second objects, the processor (120) may select the second objects according to priorities. For example, if the number of second objects is greater than or equal to a preset number and / or at least some of the second objects are smaller than or equal to a preset size, the processor (120) may select some of the second objects based on priorities. Then, the processor (120) may request generation of preliminary images only for the selected second objects. For example, the priorities may be set in the order of people, animals, home appliances, and furniture. As an example, the priorities may be set in the order of object sizes. The second preliminary object image may be referred to as a candidate image because it may be used in an image editing process, and may be referred to as a complete image because it is generated as an image that includes the overall shape of the object.

[0075] In addition to being generated as an image, a preliminary object image according to embodiments of the present disclosure may also be generated as image information used to display an image of an object. Image information for displaying or generating an object may include, for example, image information expressing a two-dimensional overall view of an object, image information expressing a three-dimensional overall view of an object, image information expressing various views that may appear differently depending on the angle at which the object is viewed, image information expressing various views that may appear differently depending on the focal length, and image information expressing various views that may appear differently depending on the angle of view. Each of the first preliminary object image and the second preliminary object image may also be generated as first object image information or second object image information.

[0076] The processor (120) may display a first edited image (e.g., an image with an object added, a color changed, or a color added) associated with the original image based on the first preliminary object image and / or the second preliminary object image. The first edited image may be an image obtained by correcting the original image according to a user's editing input. The first edited image may include a first object and / or a second object. The processor (120) may display a first portion and / or a second portion of the first object and / or the second object on the first edited image.

[0077] The second preliminary object image may include a first part and a second part of the second object. For example, in an original image including an upper body of a person, the person may be the second object, and an image of the entire shape of the person (e.g., an entire shape including the upper and lower bodies of the person) may be the second preliminary object image. The upper body of the person included in the original image may be the first part of the second object, and the lower body not included in the original image may be the second part of the second object. Similarly, the first preliminary object image may also include the first part and the second part of the first object. As an example, the processor (120) may generate a first edited image to which a first object identified by a user's drawing is added. If the processor (120) displays only a part of the first object in the first edited image, the part displayed in the first edited image may be the first part (or the second part) of the first object, and the part not displayed in the first edited image may be the second part (or the first part). In one example, a first part and / or a second part of a first object in a first preliminary object image may be displayed depending on a displayed display area. In one example, a first part and / or a second part of a second object in a second preliminary object image may be displayed depending on a displayed display area. The processor (120) may display the second object in the first edited image based on a relationship with the first object. For example, the size of the second object included in the first edited image may be different from the size of the second object included in the original image.

[0078] The processor (120) can display (or add, provide) a lighting effect according to the location of the first object and / or the second object based on the analysis information of the original image.

[0079] When the processor (120) receives an input related to a first object on the first edited image, it may display a second edited image associated with the first edited image based on the first preliminary object image and / or the second preliminary object image. The second edited image may be an image that is obtained by correcting the first edited image according to the user's editing input. The input related to the first object on the first edited image may be an input for moving the object. For example, the input for moving the object may be an input for moving the object to another location on the screen (e.g., a drag input) while the object is selected, or a control input. As an example, the control input may be an input in the form of an arrow corresponding to a distance and direction in which the object is to be moved. When the processor (120) receives an input for moving the first object on the first edited image, it may display a second edited image in which the location of the first object is changed based on the first preliminary object image. The processor (120) can acquire depth information of the first object based on a change in the position of the first object, and display the first object based on the acquired depth information of the first object. In addition, the processor (120) can automatically change the position of the second object on the second edited image to a position different from the position of the second object on the first edited image based on the relationship with the first object.

[0080] For example, when receiving an input for moving a second object on a first edited image, the processor (120) may display a second edited image in which the position of the second object is changed based on a second preliminary object image. The second edited image may be an image obtained by correcting the first edited image according to the user's editing input. The processor (120) may obtain depth information of the second object based on the change in position of the second object, and display the second object based on the obtained depth information of the second object. In addition, the processor (120) may automatically change the position of the first object on the second edited image to a position different from the position of the first object on the first edited image based on the relationship with the second object.

[0081] For example, if the position of the second object is moved according to the user's input, the processor (120) may display the first part and / or the second part of the second object in the second edited image based on the relationship with the first object. As an example, if the position of the first object is moved according to the user's input, the processor (120) may display the first part and / or the second part of the first object in the second edited image based on the relationship with the second object.

[0082] When the positions of the first object and / or the second object are changed, the processor (120) may display (or, provide, change) a lighting effect based on the change in the positions of the first object and / or the second object. As an example, when the sun is displayed in the upper central area of ​​the first edited image and the first object (or the second object) is displayed in the lower left area, the processor (120) may display a shadow effect of the first object (or the second object) in the left direction on the first edited image. When the position of the first object (or the second object) is changed to the lower right area according to a user input, the processor (120) may display (or, change) a shadow effect of the first object (or the second object) in the right direction on the second edited image.

[0083] The display (170) can output data processed by the processor (120) as an image. The display (170) can display an original image, a first edited image (e.g., an image with an object added), or a second edited image (e.g., an image with an object moved). In addition, the display (170) can display a UI (User Interface) related to an input for adding an object, an input for modifying an object, or a control input. For example, the display (170) can be implemented as a touch screen and receive a user's input on the display.

[0084] FIGS. 3A and 3B are drawings illustrating a process of editing an image according to various embodiments.

[0085] Referring to FIG. 3A, an electronic device (101) can display an image (e.g., an original image) and receive a user's input from the displayed image. The user's input may be an input using a finger, a gesture (or, motion), a mouse, a keyboard, a camera, and / or an electronic pen. The electronic device (101) can analyze an image (or, an image area, image data, an original image) and drawing information based on the user input among the images (305) containing the user input (e.g., a drawing).

[0086] For example, the image analysis model (320) can analyze an image among images (305) including user input (e.g., a drawing). The image analysis model (320) can analyze information and / or lighting characteristics of the image. The image analysis model (320) can be an artificial intelligence model trained based on machine learning or deep learning. The image analysis model (320) can generate a prompt (e.g., cozy, shining sun light, before sunset, etc.) that includes information of the analyzed image and / or information of lighting characteristics (e.g., direction of light). The electronic device (101) can place an object to be added so as to match the information and / or lighting environment of the image based on the prompt generated by the image analysis model (320).

[0087] Additionally, the image analysis model (320) can extract objects (including the background) from the image. For example, the image analysis model (320) can identify objects to be extracted. For example, the processor (120) can display object candidates to the user and identify objects to be extracted based on the user's selection. The image analysis model (320) can identify objects in an area unrelated to the drawing area. Additionally, the image analysis model (320) can determine key objects based on a preset priority based on the number, size, and / or type of objects. For example, if the object to be extracted is a person, the image analysis model (320) can provide a UI that receives input from the user about the person's style to generate a preliminary object image for the person (e.g., a skirt, pants, slippers, shoes, sneakers, etc.). The image analysis model (320) can generate a prompt including the person and the input style information to generate the preliminary object image.

[0088] The user input analysis model (330) can identify an input drawing. For example, the user input analysis model (330) can identify an input for adding an object, an input for adding (or changing) a color to an object (e.g., a background), and / or a control input.

[0089] For example, the user input analysis model (330) can determine whether to add an object or add (or change) a color based on whether the drawing is in the form of a simple geometry (e.g., a point, a line, a plane) or a combination of multiple shapes. The user input analysis model (330) can learn about adding colors using a data set including simple geometric images of various colors. When simple geometric images including colors are grouped according to a preset standard and include a certain shape, the grouped simple geometric images can be recognized as objects. The user input analysis model (330) can learn about objects using a data set including grouped simple geometric images.

[0090] The user input analysis model (330) can analyze the input drawing and convert the object of the image into a drawing form. The user input analysis model (330) can obtain drawing information through a reverse transformation process that converts the object of the image into a drawing form. The user input analysis model (330) can correct the image using the obtained drawing information. The user input analysis model (330) can improve its performance by retraining it with a data set that includes the corrected image.

[0091] The user input analysis model (330) may receive information about the thickness of a line input through the drawing UI before receiving the user's input. For example, if a thick line is selected, the user input analysis model (330) may determine that it is a drawing for the color of an object (e.g., changing the color of the background, adding a color), and if a thin line is selected, it may determine that it is a drawing for, for example, adding an object. In one example, the user input analysis model (330) may determine whether to add an object and / or change its color based on a preset drawing pattern.

[0092] For example, the user input analysis model (330) can group a plurality of drawing elements based on the color, continuity, input location, and relationship between the image area and the drawing input by the user, and determine whether to add an object and / or add (or change) a color based on the grouped drawing elements. For example, the drawing elements can include individual lines that form a part of an object with identity, a shape including a plurality of lines, or an area. The user input analysis model (330) can determine each drawing element based on the color, continuity, and / or input location of the input drawing, and group the plurality of drawing elements based on the relationship between the input location and the image area of ​​each drawing element. In addition, the user input analysis model (330) can determine whether to add an object, add a color, change a color, or issue a control command based on the grouped drawing elements. As an example, if the input drawing (or a plurality of grouped drawing elements) is in the form of a diagonal pattern or a zigzag pattern within the object area or has the same color and repeating shape inside after drawing the boundary of a specific area, the drawing analysis unit (20) can identify the input drawing as input for color addition.

[0093] In one example, the user input analysis model (330) can identify an input drawing based on the input tool or input method. For example, if a finger and / or an electronic pen comes into contact with the display (160) and draws, the user input analysis model (330) can identify it as an input for adding an object, and if a finger and / or an electronic pen hovers over the touchscreen and draws, the user input analysis model (330) can identify it as an input for adding (or changing) a color to an object. In one example, if a drawing is made while the button of the electronic pen is not pressed, the user input analysis model (330) can identify it as an input for adding an object, and if a drawing is made while the button of the electronic pen is pressed, the user input analysis model (330) can identify it as an input for adding a color to an object. According to one example, the user input analysis model (330) can identify a drawing input on the display (190) as an input for adding an object, and a drawing input by a motion of the user's hand identified through a camera or a motion of the electronic device (101) identified through a motion sensor can be identified as an input for adding a color.

[0094] The user input analysis model (330) can generate information about an object related to an identified drawing as a prompt. For example, the user input analysis model (330) can generate a prompt for object creation and / or color addition based on the judgment results regarding the area and coloring area of ​​the image into which the drawing is input. The prompt including the object information generated by the user input analysis model (330) can be input to the preliminary object creation model (340). For example, the prompt can be in the form of text, visual, and / or sound. The textual prompt can be a prompt input in the form of text. The textual prompt can be in the form of natural language. The visual prompt can be a prompt in the form of an image. The visual prompt can be used to generate information in a different form from a general image. For example, the visual prompt can be used when generating depth information from an image (e.g., an original image or a first edited image). As an example, the depth map generation model (380) can generate depth information acquired from an image as a visual prompt (e.g., a depth map in the form of an image). For example, a visual prompt can be used to acquire skeletal information for acquiring shape and / or motion information from a human image and / or an animal image. As an example, the image analysis model (320) can generate skeletal information of an object as a visual prompt. For example, when a visual prompt is used with text information, it can be used to acquire an image converted according to the input text information. As an example, when a user inputs the text “log” along with a drawing of a house shape, the preliminary object generation model (350) can acquire a preliminary object image of a log cabin based on the visual prompt and text in the form of a house generated by the user input analysis model (330). A sound prompt is similar to a text prompt, but can be a prompt input as sound (e.g., voice) instead of text.Prompts input as sound can be converted into text prompts. For example, a model trained on a specific frequency can output the results of performing a pre-trained action when a sound prompt is input.

[0095] In various embodiments of the present document, the prompt input to the preliminary object creation model (330) may be a prompt converted into textual form as a result of drawing analysis and / or a prompt converted into visual form as an input image. For example, if a user draws a tree in the lower right corner of an input image, a textual prompt such as “The tree is currently centered on the bottom-right coordinate center pixels 250 (x-coordinate) by 145 (y-coordinate) in the figure” may be input to the preliminary object creation model (340).

[0096] The user input analysis model (330) can identify whether an object addition input or a color addition / change input is input, and analyze the relationship with the area within the image where the drawing is input to identify the properties of the drawing (e.g., type, shape, etc.). For example, the prompt can include a sentence describing the background or object according to the thickness of the pen specified by the user (e.g., 'an object of…', 'a landscape of…'). The prompt can include words describing the properties and colors of the drawing analyzed by the user input analysis model (330).

[0097] The user input analysis model (330) provides a preset pattern to the user and can identify drawing properties based on the input drawing and the preset pattern. For example, the preset pattern may distinguish between object types (or drawing properties) or control commands.

[0098] For example, the user input analysis model (330) can identify one vertical line as a tree, and three or more vertical lines as a forest. One circle and two vertical lines can be identified as a cat, and two overlapping circles and two vertical lines can be identified as a fox. If the attribute of the input drawing is identified as a fox, the user input analysis model (330) can generate a prompt for generating a fox image. As an example, the user input analysis model (330) can generate a prompt containing natural language information such as “Generate an image of a fox expressing emotions in its own unique way” and input it to the preliminary object generation model (340).

[0099] The user input analysis model (330) can determine whether there is a match (or, fit, realism (e.g., actual feasibility), suitability) based on the relevance of the identified object and the region in the image where the drawing is input. If the identified object and the region where the drawing is input do not match, the user input analysis model (330) can determine whether a new object is created. As an example, the user input analysis model (330) can determine that the attribute of the user input received from the indoor image is a tree. However, the indoor floor and the tree may not match (or, do not match, lack realism). The user input analysis model (330) can determine that a flower pot is to be created as a new object.

[0100] For example, the user input analysis model (330) can determine whether the received user input is a control command. If the user input is a control command, the electronic device (101) can perform an operation according to the control command. As an example, the control command may include a command providing specific information about a drawing or a command for image editing.

[0101] For example, the electronic device (101) can input text. The user input analysis model (330) can additionally use the input text to identify the properties of the drawing. As an example, a drawing of a triangle on a square can be input, and the text “log” can be input. The drawing analysis unit (20) can identify the property of the drawing as a house based on the shape of the drawing, and can identify the property of the drawing as a log cabin, which is a subcategory of the house, based on the input text.

[0102] For example, the control command may include a drawing for image editing. The electronic device (101) may provide an object movement function using a control command through drawing. When a user inputs a circular drawing around a drawing of a human figure and inputs a drawing in the shape of a left arrow in a left area close to the circular drawing, the user input analysis model (330) may recognize this as a control command to move the human object in the circular drawing in the direction of the arrow. The processor (120) may use the preliminary object generation model (340) to generate a preliminary object image of the human, and may move the human object to the left when rearranging the object to generate a first edited image. In one example, a second edited image may be generated in which the object is moved using a control command through drawing in the first edited image. For example, the type of control command may be distinguished by the shape or size of the control drawing. In the above example, the movement distance of the object may be determined according to the length of the arrow. That is, the selected object may be moved by the length of the arrow, or may be moved by a preset distance proportional to the length of the arrow.

[0103] The processor (120) can extract information required for generating a preliminary object image and use the extracted information to generate a preliminary object image using a preliminary object generation model (340). As an example, the extracted information may include images stored in the memory (130) of the electronic device (101) and / or a cloud server, searched images, schedule data, a shopping list, a shopping list image, an interest list, an interest image, and / or a screenshot. The preliminary object generation model (340) can receive a prompt (or output data) generated from the image analysis model (320) and / or the user input analysis model (330). According to an example, the preliminary object generation model (340) can receive information extracted for generating a preliminary object image. The preliminary object generation model (340) can generate an artificial intelligence-based image (e.g., a preliminary object image) (350) based on the received image and the received information. When the user input analysis model (330) determines the creation of a new object, the preliminary object creation model (340) can create a preliminary object image including the new object. The electronic device (101) can secure a variety of preliminary object images that can be created. The preliminary object creation model (340) can create a preliminary object image including the entire shape of the object. An object included in an original image and / or an object identified from a user input can be created in a completed object shape even if a portion thereof is not displayed on the display (160). The preliminary object image can include, for example, image information expressing a two-dimensional overall shape of the object, image information expressing a three-dimensional overall shape of the object, image information expressing various shapes that may appear differently depending on the angle at which the object is viewed, image information expressing various shapes that may appear differently depending on the focal length, and image information expressing various shapes that may appear differently depending on the angle of view.When an image is edited (e.g., an object is moved) or the display area of ​​the display is changed, the processor (120) can process and display the image in real time using the generated preliminary object image.

[0104] The image information / lighting model (360) can receive a prompt containing image information and / or information on lighting characteristics and a generated artificial intelligence-based image (e.g., a preliminary object image) (350). The image information / lighting model (360) can arrange the generated preliminary object image to match the original image and generate a first edited image (370) with updated lighting information reflecting information on lighting characteristics.

[0105] The depth map generation model (380) can obtain depth information of an object from a first edited image (e.g., an image to which image information and / or lighting characteristics are applied) (370). For example, the depth information may be perspective-related data according to the spatial arrangement of objects. The depth information may include information related to the surface distance of an object included in an image (e.g., the first edited image) from a view point. The depth map generation model (380) may be an artificial intelligence model that calculates depth information of an object from an individual image. The depth map generation model (380) can obtain depth information for an object (including the background) in an image. As an example, the depth information may include information obtained using a sensor that obtains depth information, such as LiDAR (Light Detection And Ranging) or ToF (Time of Flight), when capturing an image. The depth information may be stored in metadata of the image.

[0106] The depth map generation model (380) can store the depth information of each object in the acquired first edited image (370) in the memory (130). When a command to move an object is received from the user, the depth map generation model (380) can update the depth information of the moved object based on the change in the position of the object. The depth map generation model (380) can update the depth information and store the updated depth information in the memory (130).

[0107] The processor (120) can generate a second edited image (390) based on the updated depth information according to the movement of the object. The processor (120) can change the object display area based on the state of the object before the movement and the changed position information. For example, the processor (120) can display only the upper body area of ​​a person (e.g., the first part) before the movement, and display the full body area of ​​the person (e.g., the first part and the second part) after the movement. The processor (120) can change the lighting characteristics according to the movement of the object by using the image information / illumination model (360) according to the movement of the object. Therefore, when the electronic device (101) of the present document edits an image that moves an object, it can edit the image in real time by using a preliminary object image. In addition, the electronic device (101) can continuously preserve the sense of perspective and edit the image by updating the depth information of the object included in the image according to the change in the object position.

[0108] The distinction, use, connection relationship, operation order, etc. of the image analysis model (320), user input analysis model (330), preliminary object generation model (340), image information / lighting model (360), and depth map generation model (380) presented as examples in this document can be modified in various ways. In the above-described example, a process is presented in which a first edited image (370) with a preliminary object image applied is generated through the image information / lighting model (360), and a second edited image (380) is generated from the generated first edited image (370) through the depth map generation model (380). However, embodiments of the present invention are not limited thereto. For example, image information and lighting-related information acquired through the image information / lighting model (360), preliminary object images acquired through the preliminary object generation model (340), and depth map information (e.g., data related to perspective according to the spatial arrangement of objects) acquired through the depth map generation model (380) may be processed by at least one artificial intelligence computing model to generate an edited image (e.g., a first edited image (370)). According to one embodiment, the electronic device (101) may separately generate a preliminary object image through the preliminary object generation model (340) in parallel or sequentially while generating a first edited image (370) in which an object drawn by a user is expressed through the image analysis model (320) and the user input analysis model (330), and may then use the preliminary object image in the process of generating a second edited image (390).

[0109] Referring to FIG. 3B, the electronic device (101) can receive a drawing (or user input) (310). For example, the electronic device (101) can identify the input drawing. For example, a user input analysis model (330) can identify the input drawing. For example, a single generation model (315) that performs multiple model functions can identify the input drawing.

[0110] In one example, a drawing may include a sketch (311), text (312), and control input (313). The sketch (311) may include a drawing that forms a certain shape for creating (or adding) an object. The text (312) may include a drawing that adds attribute information (e.g., color, texture, shape, material of the object) when creating a sketch object. For example, the text (312) may include words, phrases, or sentences handwritten by a user.

[0111] The control input (313) may be an input for controlling an object or adding information about the object in a manner other than drawing the object shape. The control input (313) may be a drawing input for controlling an object or adding information about the object in a manner other than drawing to indicate the object shape. For example, text (312) may provide a function for adding attribute information, and the control input (313) may provide a function for identifying an editing command. The control input (313) may include a sketch type (e.g., a shape, a symbol) and / or a text type. The distinction between the sketch (311), text (312), and control input (313) in this document may be a classification according to function.

[0112] As an example, if an arrow is input on (or adjacent to) an object, the processor (120) can move the object based on the direction and length of the arrow. An arrow or text input by a user can be an example of a control input.

[0113] For example, a controlled drawing input may be performed based on a user selection of a preset menu or user interface element provided by the device. The controlled drawing input may be performed through color or thickness settings, etc., provided in a user interface element for setting properties for performing a drawing. The controlled drawing input may be performed based on the type of input means (e.g., hand gesture, pen input, virtual keyboard, input pad type, input area, etc.).

[0114] The control input (313) relates to a command related to editing an image, and may include a drawing that changes (or modifies) an object. As an example, the electronic device (101) may identify an object to be created based on a sketch (311) and identify object attribute information to be added based on text (312). The electronic device (101) may change the object or perform a general control operation based on the control input (313). As an example, the control input (313) may include a line, a shape, a symbol, and / or a gesture.

[0115] For example, the electronic device (101) can generate a drawing-based image (e.g., a preliminary object image) using an image generation model (315) trained based on machine learning or deep learning (e.g., an artificial intelligence-based image editing model, an artificial intelligence-based computing device, or an artificial intelligence-based computing module).

[0116] In one example, the image generation model (315) may generate an image based on a drawing object based on input drawing information (e.g., sketches for object creation / modification / coloring, text drawings, drawings for control, etc.) and / or input additional drawing information and / or input additional object information (e.g., additional content data such as images, texts, etc.) and / or input prompts (e.g., prompts generated based on drawings and / or additional object information passed as input to the generation model).

[0117] According to one example, the image generation model (315) can generate an image that reflects perspective information based on lighting information and / or depth information based on input drawing information and / or additional drawing information and / or input additional object information.

[0118] As an example, the models described in FIG. 3a may each refer to different artificial intelligence models, and may be components (e.g., modules, units) that provide different functions included in one artificial intelligence model.

[0119] As an example, image information and lighting-related information acquired through the image information / lighting model (360), preliminary object images acquired through the preliminary object generation model (340), and depth map information (e.g., data related to perspective according to the spatial arrangement of objects) acquired through the depth map generation model (380) may be processed by at least one artificial intelligence computing model to generate an edited image (e.g., a first edited image (370)).

[0120] As an example, as illustrated in FIG. 3b, the electronic device (101) may include one image generation model (315).

[0121] As an example, as illustrated in FIG. 3b, the electronic device (101) may include a generation model (315) that generates an image that reflects image / lighting / depth information when generating an image based on drawing object properties in at least one artificial intelligence computing model.

[0122] In one example, an image generation model (315) can generate a new image (365) based on input drawing objects, additional images, and / or object properties, and prompt information.

[0123] In one example, an image generation model (315) can analyze an input drawing object, additional image and / or object, obtain information, and generate a new image (365) based on the obtained information.

[0124] In one example, the image generation model (315) can generate an image based on a sketch (311), text (312), and / or control input (313). The image generation model (315) can, for example, identify an object that a user intends to draw from the sketch (311). As an example, a sketch of a triangle (1131a of FIG. 23e) and a square (1131b of FIG. 23e) under the triangle and a text of a log (1133 of FIG. 23e) can be input. The image generation model (315) can identify an object called a house (1131 of FIG. 23e) based on the shape of the sketch of the triangle (1131a of FIG. 23e) and the square (1131b of FIG. 23e) under the triangle. The image generation model (315) can determine the properties of an object based on, for example, text (312) and / or control input (313). In one example, if a circular drawing (1135 of FIG. 23e) is input around a sketch of a triangle (1131a of FIG. 23e) and a square (1131b of FIG. 23e) below the triangle, the image generation model (315) can recognize the sketch within the circular drawing (1135 of FIG. 23e) as a control command to identify (or generate a preliminary object image). The image generation model (315) can identify the property of a log cabin, which is a subcategory of a house (1131 of FIG. 23e), based on the input text. The image generation model (315) can generate an object image (or an image including an object) to which the properties identified from the text (312) and / or the control input (313) are applied. As an example, the image generation model (315) can generate an object of a log cabin (1137 in FIG. 23e) based on the identified properties.

[0125] In one example, the image generation model (315) can analyze a drawing, an image in which the drawing is input, information about the image, and / or lighting characteristics. The image generation model (315) can generate text information (e.g., keywords or prompts such as cozy, shining sun light, before sunset, etc.) that includes information about the analyzed image and / or information about lighting characteristics (e.g., direction of light). If the drawing is received on an original image, the image generation model (315) can identify other objects (including the background) included in the original image. The image generation model (315) can identify the input drawing. The image generation model (315) can identify whether the input drawing (e.g., sketch (311)) is an input that adds an object or an input that modifies an object. The image generation model (315) can generate information about an object related to the identified drawing as text information (e.g., a prompt). The image generation model (315) can identify depth information related to the drawing. The image generation model (315) can generate an object image (or an image including the object) based on information about lighting characteristics and / or depth information as the drawing is received. For example, the image generation model (315) can generate a preliminary object image including the overall shape of the object. The image generation model (315) can generate an image (365) based on the image information, the information about lighting characteristics, the depth information, and / or the generated preliminary object image.

[0126] FIGS. 4A, 4B, and 4C are drawings illustrating a process of recognizing an object based on drawing elements according to various embodiments.

[0127] Referring to FIG. 4A, the electronic device (101) can determine a drawing element based on continuity at the time of input. For example, a drawing element may include an individual line forming a part of an object with identity or consistency, a shape including multiple lines, or an area. The electronic device (101) can determine each drawing element based on the color, position, and continuity of the input drawing. As an example, the electronic device (101) can receive a single line in a first color. The single line can be input substantially without a break. The electronic device (101) can determine a single line that is continuously input as a first drawing element (1).

[0128] Referring to FIG. 4B, the electronic device (101) can determine a drawing element based on the same color. For example, the electronic device (101) can receive a plurality of second lines in a second color. The plurality of second lines are individual lines, but can be input with the same color at adjacent locations where some portions overlap. The electronic device (101) can determine the plurality of second lines input with the same color at adjacent locations as the second drawing element (3).

[0129] Referring to FIG. 4C, the electronic device (101) (or an artificial intelligence-based image editing model) can group a plurality of drawing elements. For example, the electronic device (101) can group a plurality of drawing elements based on at least one of proximity of positions between drawing elements, relevance, continuity based on input time, and relevance between input positions and original image areas. For example, a first drawing element (1) and a second drawing element (3) can be input at adjacent positions with a very short time interval and can include relevance between each other. The electronic device (101) can determine that the first drawing element (1) and the second drawing element (3) are grouped as a group (5) of drawing elements. Based on the group (5) of drawing elements, the electronic device (101) can determine whether an object addition, color addition, color change control command is issued, or an attribute of the object.

[0130] FIG. 5 is a diagram illustrating a process for generating an edited image according to various embodiments.

[0131] Referring to FIG. 5(a), an electronic device (101) can display an image (e.g., an original image) (11) and receive user input from the displayed image. The user input may be input using a finger, a gesture (or motion), and / or an electronic pen. As an example, the electronic device (101) can display a UI (21) for selecting a color.

[0132] Referring to FIG. 5(b), the electronic device (101) receives a user input in a color selected on a displayed image, and referring to FIG. 5(c), the electronic device can identify a drawing input (22) formed by the user input using an image editing model (e.g., a user input analysis model).

[0133] For example, the electronic device (101) can determine a drawing element based on continuity, drawing color, and / or input location based on the input time. For example, the drawing element can include an individual line that forms a part of an object with identity, a shape including multiple lines, or an area. The electronic device (101) can group a plurality of drawing elements based on the relationship between the input location of the drawing element and the image area. Then, the electronic device (101) can determine whether to add an object, add a color, change a color, or provide a control command based on the grouped drawing elements. If the electronic device (101) determines that the user input (or drawing input) is an input for adding an object, the electronic device (101) can identify the object based on the shape drawn by the user input.

[0134] For example, if the screen is small or the area for drawing user input is small, the electronic device (101) can enlarge the image according to the input of a magnification gesture (e.g., a zoom gesture). The electronic device (101) can analyze the relevance of the drawing elements based on the input drawing elements and automatically group the drawing elements. The electronic device (101) can provide a UI that can display the grouped drawing elements or modify the grouped drawing group. In addition, the electronic device (101) can adjust the size and position of the input drawing to place the drawing group at the position intended by the user.

[0135] As an example, the electronic device (101) may determine whether the color of the drawing in the background area is lighter or darker than the color of the background, and may provide a UI for the user to select time / visual options (e.g., am / pm, morning / lunch / evening, sunset time, etc.) and / or weather options that the image represents.

[0136] The electronic device (101) can generate a prompt to change the time / visual information expressed by the image based on the capture time extracted from the image metadata. For example, if the capture time of the image is 12:00 PM and a blue colored drawing is input in the sky area, the electronic device (101) can generate a prompt to create a clear sky during a bright day. As another example, if the capture time of the image is 5:00 PM and a blue colored drawing is input in the sky area, the electronic device (101) can generate a prompt to create a clear sky during an evening.

[0137] In parallel or sequentially, the electronic device (101) may extract objects (including the background) and analyze information and / or lighting characteristics of the image using an image editing model (e.g., an image analysis model). For example, information of the image may include information about the scene of the image, temporal / visual information, an image associated with the image, specific information related to objects included in the image, relative sizes of objects, and / or information about images / objects surrounding a user input. Lighting characteristics may include information about the type, direction, intensity, angle, and / or range of light.

[0138] Referring to FIG. 5(d), the electronic device (101) can generate a preliminary object image (25a, 27a) for an object using an image editing model (e.g., a preliminary object generation model). For example, the preliminary object image (25a, 27a) may include a preliminary object image for an object determined from the shape of a drawing and / or an object included in the image. For example, the object included in the image may be an object (e.g., a person, a main object, etc.) included in the original image rather than by the drawing input (22). The electronic device (101) can generate a preliminary object image (25a, 27a) including the entire shape of the object. For example, the electronic device (101) can generate a preliminary object image (25a, 27a) of a completed shape for an object (or, information on the object) determined from the shape of the drawing input (22). As an example, the electronic device (101) can extract an object (or information about the object) included in the original image (11) rather than the drawing input (22), and generate a preliminary object image (25a, 27a) in a completed form for the extracted object (or information about the object). The preliminary object image (25a, 27a) of an object included in the original image (11) and / or an object identified from a user's input can be generated in a completed object form regardless of whether it is displayed on the display (160).

[0139] For example, the preliminary object image (25a, 27a) may include image information representing a two-dimensional overall appearance of the object, image information representing a three-dimensional overall appearance of the object, image information representing various appearances that may appear differently depending on the angle at which the object is viewed, image information representing various appearances that may appear differently depending on the focal length, and image information representing various appearances that may appear differently depending on the angle of view. When the image is edited (e.g., the object is moved) or the display area of ​​the display (160) is changed, the electronic device (101) may process and display the image in real time using the generated preliminary object image.

[0140] As an example, the electronic device (101) can identify an object that the user wants to add based on the input drawing shape and information of the identified image. The electronic device (101) can generate one or more preliminary object images for the identified object. In addition, the electronic device (101) can provide a UI that allows selection of one of the plurality of preliminary object images. When one preliminary object image is selected, the electronic device (101) can generate a first edited image (12) with an object (25b) added based on the selected preliminary object image (25a).

[0141] As an example, the electronic device (101) can use metadata of an image (e.g., information such as date / time / location) to extract an object from another related image (e.g., obtain information about the type of object, the appearance of the object, etc.). For example, the electronic device (101) can identify the location where the image was taken from the metadata. The electronic device (101) can search for and extract an object of the same type as the identified object from another image taken at the same or adjacent location as the identified shooting location. As another example, the electronic device (101) can search for images that include the same or similar information about the location and object through crawling (or web search) using keywords related to the identified shooting location and information about the identified object, thereby obtaining information about an object that the user wants to add. For example, the electronic device (101) can identify the time when the image was taken from the metadata. The electronic device (101) can search for and extract an object of the same type as the identified object from another image taken at a similar time to the identified time.

[0142] As another example, the electronic device (101) can search for and extract (e.g., obtain information about the type of object, the appearance of the object, etc.) an object of the same type as an object identified based on information about an existing object included in an image (e.g., an original image) from another image. As an example, the electronic device (101) can receive a drawing on a face of a specific person included in the image. The electronic device (101) can identify sunglasses as an object (or a type of object) based on the shape of the input drawing. The electronic device (101) can search for another image of a specific person wearing sunglasses. The electronic device (101) can generate a preliminary object image based on information about the sunglasses image included in the searched image.

[0143] The electronic device (101) can generate one preliminary object image based on image information of one of the extracted objects. In one example, the electronic device (101) can generate a plurality of preliminary object images based on image information of a plurality of extracted objects. The electronic device (101) can display a plurality of preliminary object images and generate a first edited image with an object added based on one preliminary object image selected by the user.

[0144] For example, if a user has previously drawn a specific type of object, the electronic device (101) may give priority to the previously drawn specific type of object. For example, if a user has previously added a cat as a new object on a sofa, the electronic device (101) may, when determining the type of animal drawn on the sofa in another image, give priority to the cat. For example, if a user draws an object to add to an image, the electronic device (101) may suggest adding other objects by considering the image information. For example, if a user draws an umbrella on an image, the electronic device (101) may suggest adding clouds and raindrops or may automatically add them.

[0145] Referring to FIG. 5(e), the electronic device (101) can arrange objects (25b, 27b) generated based on preliminary object images (25a, 27a) generated using an image editing model (e.g., image information / illumination model) to match the original image (11), and generate a first edited image (12) that updates lighting information by reflecting information on lighting characteristics, as illustrated in FIG. 5(f).

[0146] For example, if a first edited image (12) is generated from an original image (11), the electronic device (101) can apply similar (or consistent) editing results to other images captured at a similar time and / or location as the original image (11). For example, if a palm tree is added next to the user in the original image, the electronic device (101) can collectively reflect the editing results to other images captured at the same location as the original image (11). For example, if the location of a person in the original image (11) is moved, the electronic device (101) can collectively reflect the editing results to other images captured at the same location as the original image (11). For example, the electronic device (101) can provide a UI that performs automatic batch editing. The electronic device (101) can reflect the editing results to a plurality of selected images. For example, when an image edited by the user is selected together with another image, the electronic device (101) may provide a UI that inquires whether to apply the editing result of the edited image to the other image. For example, the editing result reflected in the other image (e.g., the appearance of an added object) may be adjusted using a preliminary object image generated based on the distance between the object to be added and another object (included in the other image), the type of the other object, the size of the other object, etc. For example, the electronic device (101) may adjust the size, gaze, and / or posture of another object included in the original image (11) based on its relationship with the newly added object. For example, when a puppy is drawn next to a person included in the original image (11), the electronic device (101) may automatically adjust the size, gaze, and / or posture of the person and / or the puppy so that the person looks at the puppy using the preliminary object image of the person.

[0147] An electronic device (101) can obtain depth information of an object from a first edited image (e.g., an image to which image information and / or lighting characteristics are applied) (12) using an image editing model (e.g., a depth map generation model). For example, the depth information may be perspective-related data according to the spatial arrangement of objects. The electronic device (101) can store the depth information of each object in the obtained first edited image (12) in a memory (130).

[0148] Referring to FIG. 5(g), the electronic device (101) can receive an object movement command from the user (9). When the object movement command is received from the user (9), the electronic device (101) can change the position of the object (27b) according to the user's command, and update the depth information of the moved object (27b) based on the change in the position of the object (27b). As an example, as illustrated in FIG. 5(g), the electronic device (101) can move the human object (27b) to a position further than the position of the human object (27b) included in the image of FIG. 5(f) according to the object movement command of the user (9). Depending on the position movement, the human object (27b), which was only partially (e.g., a first portion) displayed in the image of FIG. 5(f), can be displayed in its entirety in the image of FIG. 5(g). For example, the electronic device (101) can display the overall shape of a human object (27b) using a preliminary object image (27a).

[0149] The electronic device (101) can generate a second edited image (13) based on the updated depth information. In addition, the electronic device (101) can change the lighting characteristics according to the movement of the object using an image editing model (e.g., image information / lighting model).

[0150] FIGS. 6A and 6B are drawings illustrating examples of editing an image according to various embodiments.

[0151] Referring to FIG. 6A, a drawing element (1001) is added to an original image (31a) by a user input. The electronic device (101) can identify whether the user input is an input for adding an object. If the electronic device (101) identifies the input as an input for adding an object, the electronic device (101) can identify the properties of the drawing (1001) and identify information and / or lighting characteristics of the original image (31a). The electronic device (101) can generate a prompt for adding an object based on the identified properties of the drawing (1001) and the information and / or lighting characteristics of the original image (31a). As an example, the generated prompt can include information such as “Generate light blue sofa in the style of avant-garde design.”

[0152] Referring to FIG. 6B, an edited image (31b) (e.g., a first edited image) including an added object (1002) is illustrated. The electronic device (101) may generate an object image (e.g., a preliminary object image) using a preliminary object generation model based on the generated prompt. The generated preliminary object image may be an image including the entire shape of the object. The electronic device (101) may generate an edited image (31b) including the object generated based on the preliminary object image. For example, the edited image (31b) including the added object (1002) may be generated using a portion of the preliminary object image. When an additional editing request, such as movement of the added object (1002) or insertion of another object, is received, the electronic device (101) may display a modified appearance of the added object (1002) using the preliminary object image.

[0153] FIG. 7A and FIG. 7B are diagrams illustrating examples of determining an object by analyzing a user's input according to various embodiments.

[0154] Referring to FIG. 7A, an image (32) including a user's input is illustrated. For example, a user may input a drawing element (1003a, 1003b) by touch or motion. The user may select a specific color and input a desired drawing element (1003a, 1003b). For example, the user may input a drawing element (1003a, 1003b) corresponding to a preset pattern. The electronic device (101) may display the drawing element (1003a, 1003b) on the image (32) according to the user's input.

[0155] The electronic device (101) can identify a user input based on a shape formed by the user input. The processor (120) can group multiple user inputs according to certain criteria and identify the user input. For example, the certain criteria can include color, proximity of input locations, continuity based on the input time, and the relationship between the input location and the original image area. In FIG. 7A, the first drawing element (1003a) can be a single drawing element including multiple lines input in the same color at adjacent locations and having overlapping portions. Additionally, the second drawing element (1003b) can be a drawing element including a single line input continuously. The first drawing element (1003a) and the second drawing element (1003b) can be input at very short time intervals at adjacent locations and can include a relationship between them. The electronic device (101) can group the first drawing element (1003a) and the second drawing element (1003b) and determine the user input. The electronic device (101) can determine that the grouped drawing elements (1003a, 1003b) do not match the preset control drawing elements and include a certain shape, and thus can be used as input to add an object identified based on the grouped drawing elements (1003a, 1003b).

[0156] The electronic device (101) can identify information, lighting characteristics, and / or objects in the image (32). In addition, the electronic device (101) can identify attributes of the drawn object based on the information of the identified image (32), the object included in the image (32), and the shape of the input drawing elements (1003a, 1003b). As an example, the image (32) includes the sea, the information of the image (32) may be beach, summer, vacation, travel, etc., and the drawing elements (1003a, 1003b) may be determined to be in the shape of a tree. For example, the electronic device (101) can identify attributes of the object (e.g., the type, characteristics, and additional information of the object) as a tree on the beach, a tree in a tropical region, a palm tree, etc. The electronic device (101) can generate a preliminary object image for the identified object. The electronic device (101) can extract an object from an image that includes an object related to an identified object among stored images and / or searched images to generate a preliminary object image. As an example, the electronic device (101) can generate various types of preliminary object images based on the properties of the identified object. The object image can be generated in the overall shape of the object by taking into account a subsequent image editing process. Therefore, the generated object image can be referred to as a complete object image. The electronic device (101) can provide a plurality of generated preliminary object images through the UI (51). For example, the preliminary object image can be referred to as preliminary image information or candidate image information.

[0157] Referring to FIG. 7B, an image including at least a portion of a generated preliminary object image (1004) is illustrated. A user may select one preliminary object image from a UI (51) including samples (e.g., thumbnails) of a plurality of preliminary object images. The electronic device (101) may place the selected preliminary object image (1004) in a drawn area of ​​the image (32). Then, the electronic device (101) may generate an edited image by reflecting the identified lighting characteristics to the preliminary object image (1004). In addition, the electronic device (101) may identify depth information for a main object within the image including the added object and store the identified depth information.

[0158] FIGS. 8A, 8B, 8C, and 8D are drawings illustrating examples of moving an object within an image while maintaining realism according to various embodiments.

[0159] Referring to FIG. 8A, an image to which an object has been added (e.g., a first edited image) is illustrated. A user can move an object within the image. When an object within the image is moved, the electronic device (101) can edit the image by changing the size and position of the object using depth information. As an example, an object of a plant (1005) within the image can be selected by the user. According to one embodiment, the electronic device (101) can express a lighting effect that changes according to the position and / or size of the object using lighting characteristic information of the image. For example, the change in the lighting effect can include a change in the hue, saturation, brightness, texture, etc. of the object surface or a change in shadows considering the intensity and direction of natural lighting or indoor lighting, and the influence of other objects.

[0160] Referring to FIG. 8B, when a plant (1005), which is one of the objects in the image, is selected, the electronic device (101) may display a scroll bar (52) for receiving input for moving the position of the plant (1005). In response to a user input through the scroll bar (52), the electronic device (101) may change the position of the selected plant (1005). According to one embodiment, the electronic device (101) may acquire or update depth information in real time (or periodically) to maintain occlusion and perspective between objects. The electronic device (101) may determine whether there is occlusion between objects based on the position and depth information between the objects.

[0161] Referring to FIG. 8c, the electronic device (101) updates depth information in response to a user's input through a scroll bar (52), and based on the updated depth information, the position of the plant (1005), the part of the plant to be displayed, the part of the plant to be hidden, and the size can be changed in real time (or periodically).

[0162] Referring to FIG. 8D, when the user removes the touch from the scroll bar (52), the electronic device (101) can identify the position and size of the plant (1005) at the time the touch is removed as the final position and size. The electronic device (101) can change the lighting effect within the image based on the final position, size, and identified lighting characteristic information of the plant (1005). The electronic device (101) can generate an edited image (e.g., a second edited image in which the position of the plant is edited) by applying a lighting effect corresponding to the change in the position and size of the plant (1005).

[0163] FIGS. 9A, 9B, and 9C are drawings illustrating examples of moving an object using depth information according to various embodiments.

[0164] Referring to FIG. 9A, a beach image (33) is illustrated, which includes an object of a tree (1006) and an object of a person (1007). A user can touch the tree (1006) and the person (1007) and move them in opposite directions.

[0165] Referring to FIG. 9B, the electronic device (101) can change the positions of the tree (1006) and the person (1007) using the depth information (33-1) along with the image (33). The electronic device (101) can update the depth information (33-1) to maintain the relationship and perspective between objects. In addition, the electronic device (101) can identify the lighting characteristic information of the image (33).

[0166] Referring to FIG. 9C, when the user removes the touch from the tree (1006) and the person (1007), the electronic device (101) can identify the final position and size of the tree (1006) and the person (1007) when the touch is removed. The electronic device (101) can change the lighting effect within the image based on the final position, size, and identified lighting characteristic information of the tree (1006) and the person (1007). The electronic device (101) can generate an edited image (e.g., a second edited image) in which the objects are moved by applying a lighting effect corresponding to the change in the position and size of the tree (1006) and the person (1007).

[0167] As an example, the electronic device (101) may provide a short cut reality function that generates data in the form of a GIF that allows the user to feel the movement of an object for a certain period of time based on the original image or the edited image after image editing and generation. The image generation model may be pre-trained on direction vectors so that the recognized object can move naturally.

[0168] FIG. 10a, FIG. 10b, FIG. 10c, FIG. 10d, FIG. 10e, and FIG. 10f are drawings illustrating examples of adding and moving objects according to various embodiments.

[0169] Referring to FIG. 10a, an electronic device (101) may display an image (e.g., an original image) (34a) and receive user input from the displayed image (34a). As an example, the electronic device (101) may display a UI (1081) for selecting a color.

[0170] Referring to FIG. 10b, the electronic device (101) can receive a drawing input on the image (34a). The electronic device (101) can determine a drawing element based on continuity at the input time, the color of the drawing, and / or the input location. If the electronic device (101) determines that the drawing element is an input for adding an object, the electronic device (101) can identify the object that the user wants to add based on the input drawing shape and information of the identified image. As an example, if the electronic device (101) receives a green drawing element (1082) with an overall triangular shape in an indoor image, the electronic device (101) can determine that the input drawing element (1082) is an object of a tree or a Christmas tree.

[0171] Referring to FIG. 10c, the electronic device (101) can generate a preliminary object image (1083-1) for an object. The electronic device (101) can generate a preliminary object image (1083-1) that includes the entire shape of the object. An object identified from a user's input can be generated in a completed object shape regardless of the shape displayed on the display (160).

[0172] Referring to FIG. 10d, the electronic device (101) may arrange an object (1083-2) generated based on the generated preliminary object image (1083-1) to match other objects of the original image (34a) and generate a first edited image (34b) by reflecting information on lighting characteristics. As an example, the object (1083-2) added to the first edited image (34b) may be displayed only in part, not in its entirety, depending on its location within the image (34d). The electronic device (101) may obtain depth information of the added object (1083-2) from the first edited image (34b).

[0173] The electronic device (101) can receive an object selection command from a user. For example, when an object selection command is received from a user, the electronic device (101) can display an indicator (1084) indicating an area of ​​the selected object (1083-2).

[0174] Referring to FIG. 10E, the electronic device (101) can receive a command to move a selected object (1083-2). The electronic device (101) can change the position of the object (1083-2) including the indicator (1084) according to the user's command, and update the depth information of the moved object (1083-2) based on the change in the position of the object (1083-2). As an example, the user can adjust the size of the object (1083-2) through the indicator (1084). The electronic device (101) can update the depth information of the object (1083-2) based on the changed position and / or size.

[0175] Referring to FIG. 10f, the electronic device (101) can generate a second edited image (34c) including an object (1085) whose position has been moved. According to one embodiment, the electronic device (101) can generate the second edited image (34c) using updated depth information. The electronic device (101) can display the entire shape (1085) of the object according to the moved position of the object. The electronic device (101) can display the entire shape (1085) of the moved object using the generated preliminary object image (1083). According to one embodiment, the electronic device (101) can change the lighting characteristics according to the movement of the object (1085).

[0176] FIGS. 11a, 11b, 11c, 11d, 11e, and 11f are drawings illustrating examples of adding and moving multiple objects according to various embodiments.

[0177] Referring to FIG. 11a, the electronic device (101) can display an image (e.g., an original image) (35a) and receive a user's input from the displayed image (35a).

[0178] Referring to FIG. 11B, the electronic device (101) can receive a drawing on an image (35a). If the electronic device (101) determines that the input drawing is an input for adding an object, the electronic device (101) can identify an object that the user wants to add based on the input drawing shape and information of the identified image. The electronic device (101) can receive a plurality of drawing elements (2001, 2002, 2003). The electronic device (101) can determine whether the plurality of drawing elements (2001, 2002, 2003) are included in one object or form individual objects. For example, the electronic device (101) can determine whether the plurality of drawing elements (2001, 2002, 2003) are included in one object or form individual objects based on the color, position, and continuity of the input drawing.

[0179] Referring to FIG. 11C, the electronic device (101) can generate preliminary object images (2004, 2005, 2006) for an object. If the electronic device (101) determines that a plurality of drawing elements (2001, 2002, 2003) each form an individual object (e.g., a wireless charger, a smartphone, a smart watch), the electronic device (101) can generate preliminary object images (2004, 2005, 2006) that include the overall shape of each object.

[0180] Referring to FIG. 11d, the electronic device (101) can arrange a plurality of objects (2101, 2102, 2103) generated based on a plurality of generated preliminary object images (2004, 2005, 2006) to match the original image, and generate a first edited image (35b) by reflecting information on lighting characteristics. The plurality of objects (2101, 2102, 2103) added to the first edited image (35b) may display only a portion, not the entire shape, depending on their positions relative to each other. The electronic device (101) can obtain depth information of the plurality of objects (2101, 2102, 2103) added from the first edited image (35b).

[0181] Referring to FIG. 11e, the electronic device (101) receives a movement command for a selected object (2102) (e.g., a smartphone) among a plurality of objects (2101, 2102, 2103), and as illustrated in FIG. 11f, the electronic device (101) can change the position of the selected object (2102) according to the user's command.

[0182] The electronic device (101) can correct a first object (2101) (e.g., a wireless charger) affected by the movement of the selected object (2102). As an example, an obscured area of ​​the first object (2101) can change as the selected object (2102) moves over the first object (2101). The electronic device (101) can correct the first object (2101) based on the preliminary object image (2004) of the first object (2101) by taking into account the movement position of the selected object (2102). The electronic device (101) can generate a second edited image (35c) in which the selected object (2102) is moved and the first object (2101) is corrected. For example, the electronic device (101) can display the first object (2101) including the covered portion by using a preliminary object image (2004) for the first object (2101) without a separate in-painting process to create a portion of the first object (2101) that was covered by the existing position of the second object (2102).

[0183] FIGS. 12a, 12b, 12c, 12d, 13a, 13b, 13c and 13d are drawings illustrating examples of changing the shape of a portion of an object according to various embodiments.

[0184] Referring to FIG. 12A, as an example, an electronic device (101) may display an image including a face (1010). The electronic device (101) may analyze a major portion of the face (1010) and display information on the analyzed major portion. For example, the electronic device (101) may analyze an eye portion (1011), a mouth portion (1012), an ear portion (1013), and an eyebrow portion (1014), and display the eyes, mouth, ear, and eyebrows.

[0185] Referring to FIG. 12B, a user may input a drawing to a portion of a face (1010). For example, the user may input a drawing to change the shape of an eyebrow portion (1014). The electronic device (101) may display the input drawing on the face (1010). In addition, the electronic device (101) may display display information corresponding to the portion where the drawing was input, such as by highlighting.

[0186] Referring to FIG. 12C, the electronic device (101) may generate multiple images for a portion where a drawing is input based on an input drawing. The electronic device (101) may generate multiple images using a preliminary object generation model (340). The electronic device (101) may generate a shape of the drawing and an input portion as a prompt, and input the generated prompt into the preliminary object generation model (340) to generate an image. As an example, the electronic device (101) may generate a prompt such as “straight lined eyebrows” and generate multiple eyebrow images (1016a, 1016b) using the preliminary object generation model (340). The electronic device (101) may display the generated multiple eyebrow images (1016a, 1016b) on the display (160). The user may select one of the displayed multiple eyebrow images (1016a, 1016b).

[0187] Referring to FIG. 12d, the electronic device (101) can display a completed face with the second eyebrow image (1016b) selected by the user applied on the display (160). The electronic device (101) can display the original image (1017) together with the edited image.

[0188] Referring to FIG. 13A, the electronic device (101) can analyze key parts of a face (1010) and display information about the analyzed key parts. For example, the electronic device (101) can analyze the eye part (1011), the mouth part (1012), the ear part (1013), and the eyebrow part (1014) and display the eyes, mouth, ears, and eyebrows. The user can select the part to be edited. As an example, as illustrated in FIG. 13A, the user can select the mouth part (1012).

[0189] Referring to FIG. 13B, the electronic device (101) may display an indicator on a selected mouth portion (1012) of a face (1010) and display an area (1020) for receiving a drawing input on an area of ​​the display (160). When a user inputs a drawing in the shape of an “O,” the electronic device (101) may generate a prompt based on the shape of the input drawing. For example, the electronic device (101) may generate a prompt such as “mouth shape like letter O” and input it into the preliminary object creation model (340).

[0190] Referring to FIG. 13C, the preliminary object generation model (340) can generate multiple facial images (1021a, 1021b) with edited mouth shapes based on an input prompt. The electronic device (101) can display the generated multiple facial images (1021a, 1021b) on a display (160). The user can select one of the displayed multiple facial images (1021a, 1021b). For example, the user can select the second facial image (1021b).

[0191] Referring to FIG. 13d, the electronic device (101) can display the selected second face image (1021b) as an edited image.

[0192] FIGS. 14a, 14b, and 14c are diagrams illustrating examples of adding objects using different information according to various embodiments.

[0193] Referring to FIG. 14A, an indoor image including a person is illustrated. A user may input a green drawing (1031) into a human head area and a brown rectangular drawing (1032) into one area. The electronic device (101) may determine the properties of the drawing based on information about the image, information about the object, the shape of the input drawing, the color of the drawing and / or the drawn area, and the relationship between the drawing and the object. For example, the electronic device (101) may determine the object of the green drawing (1031) input into the human head area as a green hat, and may determine the object of the brown rectangular drawing (1032) input into one area as brown furniture. The electronic device (101) may generate a preliminary object image based on the determined properties of the drawing.

[0194] Referring to FIG. 14B, the electronic device (101) may generate a preliminary object image using another image. For example, the electronic device (101) may extract objects having the same properties as the determined drawing from images searched on the web, images in a storage space (e.g., memory, cloud server, etc.) associated with a user account, images related to a shopping list, and / or images related to a list of interests. As an example, the electronic device (101) may extract a plurality of green hats (1033) and a plurality of brown furniture (1034), and display them on the display (160). The user may select one of the displayed plurality of green hats (1033) and one of the displayed plurality of brown furniture (1034).

[0195] Referring to FIG. 14c, the electronic device (101) can add a green hat (1035) selected by the user to a portion where a green drawing is input (e.g., a human head portion), and can add brown furniture (1036) selected by the user to a portion where a brown drawing is input (e.g., a work area). The electronic device (101) can generate and display an edited image (e.g., a first edited image) with objects added.

[0196] For example, an original image captured at a wide angle reflects the corresponding viewing angle, and if the existing viewing angle is maintained when the object moves to a different location within the image, an awkward image that is different from the actual situation may be generated depending on the characteristics of the original image and the characteristics of the object shape. When generating a preliminary image of a separated and extracted object (e.g., furniture), the electronic device (101) may analyze the space within the original image to additionally generate a preliminary object image of the object so that it can be matched with a different viewing angle. As an example, when the generated object moves to a different location, if the electronic device (101) determines that the viewing angle of the object at the moved location is not realistic using an image analysis model, the electronic device (101) may additionally generate a preliminary object image so that the moved object can be viewed at a realistic viewing angle.

[0197] In addition, the electronic device (101) may be implemented as an augmented reality (AR), virtual reality (VR), or mixed reality (MR) device. The electronic device (101) implemented as an AR, VR, or MR device may provide a simulation function that changes and displays the color and / or texture of an object at a user's point of view. The electronic device (101) may capture an image at a specific point of view and segment the image (or group objects) so that the user can select an object. For example, the electronic device (101) may receive a user input (e.g., a 2D or 3D drawing) for adding an object to a two-dimensional (2D) original image or a three-dimensional (3D) original image in a virtual space, and may generate a preliminary object image for the added object and / or the existing object as 3D image information. For example, when a user moves in a virtual space or an object moves in a virtual space, the electronic device (101) can provide a virtual space that reflects the movement of the user or the object by using a preliminary object image generated as 3D image information.

[0198] For example, the electronic device (101) may be linked to online shopping to display products identical / similar to an object selected by a user from a shopping list in the form of thumbnails. According to one embodiment, the electronic device (101) may display a UI for selecting a color and / or texture in 3D space. When the user selects a specific color and / or texture displayed on the UI and drags it to move it to the location of the object, the electronic device (101) may generate an image of the object to which the selected color and / or texture has been applied. The electronic device (101) may display a 3D modeling result of the object to which the selected color and / or texture has been applied using an image editing model.

[0199] The electronic device (101) can display the generated results by adjusting transparency if the wearer's viewing point is similar to the time at which the task was requested. For example, if the wearer's viewing point is different from the time at which the task was requested, the electronic device (101) can display a message indicating that the user's requested task has been completed on a portion of the screen.

[0200] FIGS. 15a, 15b, 16a, and 16b are drawings illustrating examples of changing the color of an area within an image according to various embodiments.

[0201] Referring to FIG. 15A, an image including a refrigerator (1040) as an object (e.g., subject) is illustrated. A user can color the surface of the refrigerator. When an editing operation is performed on the image, the electronic device can provide a color palette in the form of a UI. The user can select a color and draw a user input on the image using a finger or a pen. For example, the electronic device (101) can display a UI (1043) including a type of color to be colored, and color the area where the drawing is input with the color selected by the user. For example, the user can input a drawing to color the upper surface (1041) of the refrigerator (1040) with a second color (1047), and input a drawing to color the lower surface (1042) of the refrigerator (1040) with a first color (1046). The electronic device (101) can analyze an image area containing a drawing (e.g., an upper surface (1041), a lower surface (1042)) to determine an area to be colored. Then, the electronic device (101) can color the determined area with a determined color (e.g., a first color (1047), a second color (1046)).

[0202] Referring to FIG. 15B, a refrigerator (1040) is illustrated, the upper surface (1041) being colored with a second color (1047) and the lower surface (1042) being colored with a first color (1046). The electronic device (101) can identify the overall shape of the colored object (e.g., the refrigerator (1040)) to generate a preliminary object image. The electronic device (101) can provide the generated preliminary object image so that it can be edited. If the refrigerator (1040) is moved to another area, the electronic device (101) can in-paint the background area where the refrigerator (1040) was located. In-painting may mean filling an empty space with a color identical to the surrounding space. For example, the electronic device (101) can generate a preliminary object image for a background as well as a preliminary object image for a specific object (e.g., the refrigerator (1040)). The preliminary object image for the background may include image information for the in-painted background. When the refrigerator (1040) is moved within the image, an edited image may be provided using the preliminary object image for the background without performing separate in-painting.

[0203] Referring to Fig. 16a, an indoor image is illustrated. A user can color a wall. For example, the electronic device (101) can display a UI (1053) including a type of color to be colored, and color an area where a drawing is input with a color selected by the user. For example, the user can input a drawing to color a side (1051) with a first color (1056), and a drawing to color a ceiling (1052) with a second color (1057). The electronic device (101) can analyze an image area including a drawing (e.g., the side (1051), the ceiling (1052)) to determine an area to be colored. Then, the electronic device (101) can color the determined area with a determined color (e.g., the first color (1056), the second color (1057)).

[0204] Referring to FIG. 16B, an indoor image is illustrated in which the side (1051) is colored with a first color (1056) and the ceiling (1052) is colored with a second color (1057). If another object moves to reveal the covered area of ​​the side (1051) and / or the ceiling (1052), the electronic device (101) can use the preliminary image object in-painted against the background to in-paint the background area (e.g., the covered area) where the other object was located.

[0205] FIG. 17A and FIG. 17B are drawings illustrating examples of editing an image by motion of an electronic device according to various embodiments.

[0206] Referring to FIG. 17A, an electronic device (101) is illustrated that receives objects (1061, 1062, 1063) of different colors from a user. The electronic device (101) can receive objects (1061, 1062, 1063) of a specific color on the display (160) through a user input. The electronic device (101) can create a new object based on a shape drawn by the user input. Accordingly, the electronic device (101) can create an image based on a shape composed of simple dots and lines. The electronic device (101) can provide a random art effect based on a sensor (e.g., the sensor module (176) of FIG. 1). For example, the electronic device (101) can detect shaking of the electronic device (101) through the sensor and provide various effects based on the detected shaking. For example, the sensor may include a gravity sensor, an acceleration sensor, a gyro sensor, and / or a geomagnetic sensor.

[0207] Referring to FIG. 17b, the electronic device (101) can provide an effect (1066, 1067, 1068) of color spreading around an input color area based on the direction and speed of movement of the electronic device (101) detected by the sensor. The electronic device (101) can identify a color spreading area as a color drawing input and generate a preliminary object image based on the identified color drawing input. For example, the electronic device (101) can determine that a drawing input using a sensor (e.g., the sensor module (176) of FIG. 1) is an input for modifying an object or background included in an original image.

[0208] FIGS. 18a, 18b, and 18c are drawings illustrating examples of displaying images of various sizes using preliminary object images according to various embodiments.

[0209] Referring to FIG. 18A, a first display area (1070) is illustrated that displays a plurality of objects generated based on a preliminary object image. The electronic device (101) can display an image including at least a portion of the preliminary object image on a display (160) of various sizes and / or shapes. For example, the electronic device (101) can include a rollable display or a foldable display. In the case of a rollable display, the physical size of the display can change in real time. As described above, the electronic device (101) can generate a preliminary object image for an object (including a background) included in the image. Accordingly, the electronic device (101) can display an image corresponding to the size of the display in real time even if the size of the display changes.

[0210] Since the generated preliminary object image includes the entire portion as well as the portion visible on the display (160), the image can be displayed in real time to suit the changing screen size of the rollable form factor. For example, when generating the preliminary object image, the electronic device (101) can generate a prompt for generating the preliminary object image at the maximum resolution and size of the displayable image based on the type of form factor (e.g., rollable, foldable), and input the generated prompt into the preliminary object generation model. For example, the prompt can include information such as “Create the image at a resolution of 2272x1984, and perform out-painting on the extension area at the bottom.”

[0211] As illustrated in FIG. 18A, as an example, the electronic device (101) may generate preliminary object images for a person (81) and a tree (91), and display a portion (e.g., the first portion) of the person (81) and a portion (e.g., the first portion) of the tree (91) according to an area (or size) (1070) of the current first display. However, although not displayed in the display of FIG. 18A, the preliminary object image of the person (81) may include another portion (e.g., the second portion), and the preliminary object image of the tree (91) may also include another portion (e.g., the second portion).

[0212] Referring to FIG. 18B, the display area of ​​the electronic device (101) can be expanded in the left and right directions. For example, the display (160) can be a rollable display or a foldable display. The display area can be expanded to include a second display area (1071-1) to the left of the existing first display area (1070) and a third display area (1071-2) to the right. The electronic device (101) can display a previously undisplayed portion of an image in the second display area (1071-1) and the third display area (1072-2) using a preliminary object image generated according to the expansion of the display area. For example, the electronic device (101) can display a second portion of a tree (91) that is not displayed in the first display area (1070) in the second display area (1071-1).

[0213] Referring to FIG. 18C, the display area of ​​the electronic device (101) can be expanded downward. The display area can be expanded to an area including a fourth display area (1072) below the existing first display area (1070). The electronic device (101) can display a previously undisplayed portion of an image in the fourth display area (1072) using a preliminary object image generated according to the expansion of the display area. For example, the electronic device (101) can display a second portion of a person (81) that is not displayed in the first display area (1070) in the fourth display area (1072).

[0214] FIGS. 19 and 20 are drawings illustrating examples of generating images based on drawings according to various embodiments.

[0215] Referring to FIG. 19, an example is illustrated in which an image is generated by specifying the location of an object within an image by a user. For example, the drawing for image generation may include a sketch and a control input (or, a control drawing). The electronic device (101) may provide additional functions related to image generation based on the control input received through the drawing.

[0216] In one example, the sketch may include a drawing of a human figure (1091), and the control input may include a circular drawing (1093) and / or an arrow shape (1095). When a user inputs a circular drawing (1093) around the drawing of the human figure (1091) and inputs a drawing of a rightward arrow shape (1095) in a right area close to the circular drawing (1093), the electronic device (101) may recognize this as a control input to move the drawing of the human figure (1091) within the circular drawing (1093) in the direction of the arrow. The electronic device (101) may generate an image by moving the object of the human figure (1097) to the right.

[0217] For example, a second edited image may be generated in which an object is moved through a control input via drawing in a first edited image. For example, if a user inputs a drawing (1091) of a human shape, the electronic device (101) may determine that this is a user input to add an object. The electronic device (101) may generate a first edited image including an object of a human (1097) at a location where the drawing (1091) of the human shape is input based on the input drawing (1091) of the human shape. If the user inputs a circular drawing (1093) around the object of the human (1097) and inputs a drawing in the shape of a right arrow (1095) in a right area close to the circular drawing (1093), the electronic device (101) may recognize this as a control input to move the object of the human (1097) in the direction of the arrow. The electronic device (101) can generate a second edited image in which the object of the person (1097) is moved to the right.

[0218] An electronic device can provide a user with preset patterns (e.g., circular drawings, arrow drawings, etc.) and identify the properties of the drawing based on the input drawing and the preset patterns. For example, the preset patterns may distinguish the type of object (or properties of the drawing) or distinguish control commands.

[0219] For example, the type of control input can be distinguished by the shape or size of the control drawing. In the example described above, the distance an object moves can be determined based on the length of the arrow. That is, the selected object can be moved by the length of the arrow, or by a preset distance proportional to the length of the arrow.

[0220] Referring to FIG. 20, an example of modifying an object image is illustrated. For example, inputs for modifying an object may include inputs for changing the size of the object, inputs for changing the shape of the object, inputs for changing the color, and / or inputs for adding color. Inputs for modifying an object may be included in control inputs.

[0221] According to one example, the electronic device (101) can receive an input of a drawing (1101) (or sketch) in the shape of a cup from a user. The electronic device (101) can identify whether the user's input is an input for adding an object (e.g., creating an image). If the electronic device (101) identifies the user's input as an input for adding an object, it can identify the properties of the drawing (1101). If the electronic device (101) identifies the properties of the drawing (1101) as a cup in the shape of a drawn sketch, it can generate an image including the object of the first cup (1103).

[0222] For example, the electronic device (101) may receive an input of a line-shaped drawing (1105) on an object of the first cup (1103). The electronic device (101) may determine the line-shaped drawing (1105) as a control input for changing the object. In one example, the electronic device (101) may determine the line-shaped drawing (1105) as a control input for changing the shape of the object of the first cup (1103). The electronic device (101) may edit an image including an object of the first cup (1103) into an image including an object of the second cup (1107) whose shape has been changed, depending on the properties of the input control input.

[0223] FIG. 21 is a diagram illustrating a MoE (mixture of experts) layer according to various embodiments.

[0224] Referring to FIG. 21, an artificial intelligence-based image editing model of an electronic device (101) may include an MoE structure. The electronic device (101) may transmit a user's drawing input (2110) to an MoE (2120). For example, the MoE and / or the gating network (2121) may determine the type of the drawing input (2110) (e.g., sketch, text, control input). The gating network (2121) may select which expert (e.g., depth estimation (2123), re-lighting (2125), drawing-to-image (2127), etc.) to use based on the determination result. Among the experts, only the experts selected by the gating network (2121) may be activated. The drawing input (2110) may be transmitted to the activated expert.

[0225] For example, depth estimation (2123) can analyze the depth of an object included in an image and obtain depth information. Re-lighting (2125) can analyze the lighting characteristics of an image and obtain information on the lighting characteristics. Drawing-to-image (2127) can analyze an input drawing and obtain information on a preliminary object image.

[0226] For example, the MoE (2120) may apply a weight to each expert output from the gating network (2121) to the output results of each expert. The output results of each expert with the weight applied may be transmitted to a Gen AI (generative artificial intelligence) model (2129). The input transmitted to the Gen AI model (2129) may vary depending on the number of activated experts. For example, the Gen AI model (2129) may include a large multimodal model (LMM), a large language model (LLM), or a large vision model (LVM). As an example, the LLM model may be implemented as a separate configuration from the Gen AI model (2129). For example, the LLM model may analyze text drawings. The Gen AI model (2129) can generate a preliminary object image based on the type of input drawing input (2110) and / or information of text drawing analyzed in the LLM model, and generate an edited image (generated image (2130)) including the preliminary object image.

[0227] For example, experts in MoE (2120) can be selectively used depending on the input data type. Experts in MoE (2120) can be modularized and added and removed as needed. When an AI-based image editing model is implemented as MoE, the overall model size may increase, but data processing can be accelerated because the number of experts actually used during the inference process is small.

[0228] For example, an artificial intelligence-based image editing model including MoE (2120) may be included in a single artificial intelligence computing module contained in a separate server (e.g., server (108) of FIG. 1) or may be included in different artificial intelligence computing modules. The distinction, use, connection relationship, operation sequence, etc. of the artificial intelligence-based image editing model including MoE (2120) may be modified in various ways.

[0229] FIG. 22 is a diagram illustrating an operation for processing text input according to various embodiments.

[0230] Referring to FIG. 22, a process of generating an image using the LLM and Gen AI models is illustrated. As an example, the electronic device (101) may receive a drawing (e.g., a sketch, text, control input, etc.) from a user. The received drawing may include a control sign (or, control input) (2205). The received drawing may be transmitted to a sign analyzer (2210). The sign analyzer (2210) may determine whether the transmitted drawing is a control sign (2205) or whether the transmitted drawing includes a control sign (2205). As an example, the drawing may include a control sign (2205), an object specifying word for image generation, and / or various object description words. If the drawing is a control sign (2205) (or includes a control sign (2205)), the sign analyzer (2210) may transmit the drawing and / or the determination result to the LLM (2220). The LLM (2220) can receive at least one control element (2215a, 2215b). As an example, the control element (2215a, 2215b) can be an element representing a control type (e.g., move, change, composite, etc.). As an example, the control element (2215a, 2215b) can include object description words (e.g., object characteristic keywords such as color, shape, texture, etc.).

[0231] For example, the LLM (2220) can identify the type of the control sign (2205) based on the transmitted control sign (2205) and the control elements (2215a, 2215b). As an example, the LLM (2220) can adaptively determine the control sign (2205) based on the type of object description word (e.g., color, shape, texture). The LLM (2220) can output a prompt (2230) based on the object description word and / or the control signal.

[0232] For example, when generating a new image based on a drawing, a prompt (2230) is passed as input to a generative artificial intelligence model (2240, 340, 315), and the generative artificial intelligence model (2240) can output an image (e.g., a first edited image) (2235) generated based on the prompt (2230).

[0233] For example, when changing (or editing) a generated image (e.g., a first edited image) (2235), a prompt (2230) may be passed as input to a generative artificial intelligence model (2240) along with an image (2235) (e.g., a first edited image) generated from object-specific words and object-descriptive words. The generative artificial intelligence model (2240) may change an existing image (2235) into a different type of image (e.g., a second edited image) based on the prompt (2230) and output it.

[0234] FIGS. 23a, 23b, 23c, 23d, 23e, 24a, 24b, 25a, 25b, 25c, 26a, 26b, 27a, 27b, 28, 29, 30a, 30b, 30c, 30d, and 31 are drawings illustrating examples of generating and / or modifying an image based on input drawing (e.g., sketch, handwritten text, predefined pattern drawing, etc.) properties and / or drawing-based control functions according to various embodiments.

[0235] For example, the electronic device (101) can create an object based on a sketch and text (e.g., a word, phrase, or sentence handwritten by a user). The electronic device (101) can receive the sketch and text as a drawing input from the user. The electronic device (101) can additionally use the input text to identify the properties of the sketch. The electronic device (101) can create or modify an object based on the properties of the identified sketch. As described above, the objects of the present document can include objects, people, and / or backgrounds. For example, the background can be an object such as the sea, mountains, sky, or river behind an object or person.

[0236] Referring to FIG. 23a, as an example, a triangle (1111a) and a square (1111b) may be input, and a text of log (1113a) may be input. The electronic device (101) may group the drawing elements of the triangle (1111a) and the square (1111b), and identify an attribute of the sketch called house (1101) based on the shape of the grouped sketch. The electronic device (101) may identify an attribute of the sketch called log cabin, which is a subcategory of house (1101), based on the input text. The electronic device (101) may create an object of log cabin (1115a) based on the identified attribute.

[0237] In one example, the electronic device (101) may select some text related to the sketch based on the meaning represented by the text among the text (e.g., multiple words) surrounding the sketch. In one example, the electronic device (101) may display an object image of a log cabin (1115a) generated through a generation model instead of the sketch at the location where the sketch was input. In one example, the electronic device (101) may continuously display the surrounding text object reflected in the generated object image on the screen until an additional user input occurs, or may maintain the input text for a certain period of time or remove the text when displaying the object.

[0238] Referring to FIG. 23b, as an example, a triangle (1111a) and a square (1111b) may be input, and the text log cabin in the forest (1113b) may be input. The electronic device (101) may group the drawing elements of the triangle (1111a) and the square (1111b), and identify an attribute of the sketch called house (1111) based on the shape of the grouped sketch. The electronic device (101) may identify attributes of the sketch called log cabin and in the forest, which are subcategories of house (1111), based on the input text. In one example, the electronic device (101) may identify “log cabin” among the texts as an attribute of the sketched object, and “in the forest” as an attribute of the background of the image. The electronic device (101) may generate an image representing a log cabin in the forest (1115b) based on the identified attributes. The electronic device (101) can generate a background image not included in the sketch drawing form based on text properties.

[0239] Referring to FIG. 23c, as an example, a triangle (1121a) and a square (1121b) may be input, and the text of log (1123) may be input. The electronic device (101) may group the drawing elements of the triangle (1121a) and the square (1121b), and identify an attribute of the sketch called house (1121) based on the shape of the grouped sketch. The electronic device (101) may identify an attribute of the sketch called log cabin, which is a subcategory of house (1121), based on the input text. The electronic device (101) may create an object of log cabin (1125) based on the identified attribute. In an example, the electronic device (101) may receive a changed text of log cabin in the forest (1127) at the location where the text of log (1123) is input. As an example, the electronic device (101) may receive an input to modify the text of log (1123) to log cabin in the forest (1127) with “cabin in the forest” added. The electronic device (101) may identify the changed attributes of log cabin and in the forest, which are subcategories of house, based on the changed text. The electronic device (101) may change the displayed object of log cabin (1125) to the object of log cabin in the forest (1129) based on the changed attribute. In one example, the electronic device (101) may change the object of log cabin in the forest (1129) based on the displayed object of log cabin (1125). In one example, the electronic device (101) may create an object of log cabin in the forest (1129) and change the displayed object of log cabin (1125) to the created object of log cabin in the forest (1129). In one example, the electronic device (101) can generate an image of a log cabin in a forest (1129) by adding a forest background image to the image of a log cabin (1125). The electronic device (101) can separately store objects of the previously generated log cabin (1125).

[0240] Referring to FIG. 23d, as an example, a triangle (1121a) and a square (1121b) may be input, and the text of log (1123) may be input. The electronic device (101) may group the drawing elements of the triangle (1121a) and the square (1121b), and identify an attribute of the sketch called house (1121) based on the shape of the grouped sketch. The electronic device (101) may identify an attribute of the sketch called log cabin, which is a subcategory of house (1121), based on the input text. The electronic device (101) may create an object of log cabin (1125) based on the identified attribute. As an example, the electronic device (101) may receive an input to modify the text of log (1123) to log cabin in the forest (1127a) by adding the text of cabin in the forest. The electronic device (101) can identify the added attribute "in the forest" based on the changed text. Based on the added attribute, the electronic device (101) can change the displayed log cabin (1125) object into an object with a forest background. In other words, the electronic device (101) can maintain the displayed log cabin (1125) object and create an object of a log cabin in the forest (1129a) by adding a forest background.

[0241] Referring to FIG. 23E, as an example, after texts of a triangle (1131a), a square (1131b), and a log (1133) are input, a control command may be input. The electronic device (101) may identify the properties of the drawing based on the input drawing and a preset pattern. As an example, if a circular drawing (1135) is input around a sketch including a triangle (1131a) and a square (1131b), the electronic device (101) may recognize the control command to identify (or generate a preliminary object image) the sketch within the circular drawing (1135). The electronic device (101) may group the drawing elements of the triangle (1131a) and the square (1131b), and identify the properties of the sketch called a house (1131) based on the shape of the grouped sketch. The electronic device (101) can identify an attribute of the sketch called log cabin, which is a subcategory of house (1131), based on the input text. The electronic device (101) can create an object of log cabin (1137) based on the identified attribute. The electronic device can create an object of log cabin (1137) based on the attributes of the input sketch, text, and circular drawing. The electronic device can create an object of log cabin (1137) based on the input sketch and objects (e.g., text, circular drawing, etc.) input around the sketch area.

[0242] Referring to FIG. 24A, as an example, a sketch in the shape of a cup (1141) may be input, and text of a blue cup (1143) may be input. The electronic device (101) may identify an attribute of the sketch called a cup based on the shape of the sketch, and may identify additional attributes (e.g., a color attribute) of the sketch called a blue cup based on the input text. The electronic device (101) may generate an object of a blue cup (1145) based on the identified attributes. The electronic device (101) may store text information (e.g., location information) reflected in the generation of the object image of the blue cup (1145). As an example, the electronic device (101) may receive an input for modifying the text of the blue cup (1143).

[0243] Referring to FIG. 24B, as an example, the electronic device (101) may receive an input to change the text of the blue cup (1143), which is an attribute reflected in the generation of the blue cup (1145) object image, to the text of the purple cup (1147). The electronic device (101) may identify the changed attribute, purple cup, based on the changed text. The electronic device (101) may generate the purple cup object image based on the changed attribute. The electronic device (101) may change the object of the blue cup (1145) displayed on the screen to the newly created object of the purple cup (1149). The electronic device (101) may display the newly created object image of the purple cup (1149) by replacing it with the object image of the blue cup (1145) displayed on the screen, or may additionally display the newly created object image of the purple cup (1149) on the screen and provide a user interface element for the user to confirm whether to replace the image. The purple cup (1149) image generated based on the user's selection can be deleted or replaced with the blue cup (1145) object image that was previously displayed.

[0244] Referring to FIG. 25A, as an example, texts of green (1151a), purple (1151b), blue (1151c), and cup (1151d) are input, and a preset indicator (e.g., highlighting) may be input for the text of green (1151a). For example, the preset indicator may be a control command for selecting at least one of a plurality of elements. As an example, the preset indicator may be in the shape of a star (1155). A sketch of a cup shape (1157) may be input together with the text. The electronic device (101) may identify an attribute of the sketch called a cup based on the shape of the sketch. The electronic device (101) may identify an additional attribute of the sketch called a green cup based on a text in which an indicator is displayed among the plurality of input texts. The electronic device (101) may create an object of a green cup (1159) based on the identified attribute.

[0245] Referring to FIG. 25b, as an example, the indicator may be an underline (1163). The electronic device (101) may identify the underline (1163) drawn on the text of green (1161) as a preset indicator and determine an additional attribute of the sketch as a green cup.

[0246] Referring to FIG. 25c, as an example, the indicator may be the size of the text. The size of the text in green (1165) may be larger than the sizes of the texts in purple and blue. The electronic device (101) may determine additional properties of the sketch as a green cup based on the size of the text in green (1165). As an example, the indicator may include a symbol, a shape, a check, an arrow, a highlight, a text color, a text thickness, and / or a border.

[0247] Referring to FIG. 26A, an electronic device (101) is illustrated in which multiple applications (61, 63) are executed. In one example, the multiple applications (61, 63) may each include an image-related application and a text input application. For example, the electronic device (101) may display the multiple applications (61, 63) on each of multiple displays. For example, the electronic device (101) may split the screen and display the multiple applications (61, 63) on each of the split screens. For example, the electronic device (101) may display one application (61) and display another application (63) in a pop-up window. For example, the electronic device (101) may sequentially and alternately display the multiple applications (61, 63). For example, the electronic device (101) may display an image on a first application (61) (e.g., an image editing application) and receive a sketch drawing on the image. A text may be received on a second application (63) (e.g., a note application). In one example, the electronic device (101) may receive a sketch in the shape of a cup (1171) through the first application (61) and may receive a text of a purple cup (1173) through the second application (63). The electronic device (101) may receive a control input through the second application (63). In one example, the control input may be a circular drawing (1175), and the circular drawing (1175) may be a control command for identifying the sketch (or generating a preliminary object image). In one example, the control input may be performed based on a specific menu selection or may be distinguished based on a predefined drawing pattern. The electronic device (101) may identify an attribute of the sketch as a cup based on the shape of the sketch, and may identify an additional attribute of the sketch as a purple cup based on the text. The electronic device (101) may generate an object of a purple cup (1177) based on the identified attribute.The electronic device (101) can display an image with an object of a purple cup (1177) added to the first application (61).

[0248] Referring to FIG. 26B, an electronic device (101) including a plurality of screens (66, 67) is illustrated. As an example, the plurality of screens (66, 67) may each be displayed on a separate display (160). In one example, the plurality of screens (66, 67) may be split screens from a single display (160). In one example, the plurality of screens (66, 67) may each be a separate application.

[0249] In one example, the electronic device (101) can receive a sketch of a human figure (1172) through the first screen (66). The electronic device (101) can create an object of a human figure (1174) in real time based on the characteristics of the sketch received through the first screen (66) and display it on the second screen (67). In one example, when a user inputs a circular drawing (1176a) around the sketch of a human figure (1172) displayed on the first screen (66) and inputs a drawing in the shape of a leftward arrow (1176b) in a left area close to the circular drawing (1176a), the electronic device (101) can recognize this as a control input to move the object of the human figure within the circular drawing (1176a) in the direction of the arrow. The electronic device (101) can move the object of the human figure (1174) displayed on the second screen (67) to the left in real time.

[0250] Referring to FIG. 27A, as an example, sketches of a first cup shape (1181a) and a second cup shape (1181b) may be input, and texts of green (1183a), purple (1183b), blue (1183c), and cup may be input. The electronic device (101) may identify attributes of the sketches as first cup and second cup based on the shapes of the sketches. The electronic device (101) may identify a plurality of input texts and generate an item controlled by the user. As an example, the electronic device (101) may identify the texts of green (1183a), purple (1183b), and blue (1183c) as attributes of the cup. The electronic device (101) can generate control items (1185a, 1185b, 1185c) corresponding to green (1183a), purple (1183b), and blue (1183c), respectively. Each control item can be moved by the user. In one example, the first control item (1185a) corresponding to the attribute of green (1183a) can be moved to the sketch of the first cup shape (1181a), and the third control item (1185c) corresponding to the attribute of blue (1183c) can be moved to the sketch of the second cup shape (1181b).

[0251] Referring to FIG. 27b, the electronic device (101) may receive a control input in a sketch area. As an example, the control input may be a circular drawing (1187), and the circular drawing (1187) may be a control command for identifying a sketch (or generating a preliminary object image). The electronic device (101) may identify additional properties of the sketch based on properties corresponding to the control item moved to the sketch location. As an example, the electronic device (101) may identify an additional property of a sketch of a first cup shape (1181a) as a green cup, and may identify an additional property of a sketch of a second cup shape (1181b) as a blue cup. Based on the identified properties, the electronic device (101) may generate an object of a green first cup (1189a) and an object of a blue second cup (1189b).

[0252] Referring to FIG. 28, as an example, a screen is illustrated in which a generated object of a first green cup (1201a) and an object of a second blue cup (1201b) and texts of green (1203a), purple (1203b), and blue (1203c) are displayed. The object of the first green cup (1201a) and the object of the second blue cup (1201b) may be objects generated (or changed) based on the input texts of green (1203a) and blue (1203c). The electronic device (101) may include attribute information corresponding to the generated object. According to an example, the electronic device (101) may receive an input for selecting the object of the first green cup (1201a). The electronic device (101) may determine the attribute of green corresponding to the object of the first green cup (1201a). The electronic device (101) can determine the text attributes and text location information used when creating the object of the green first cup (1201a). The electronic device (101) can display an indicator (1205) in the text area of ​​green (1203a) corresponding to the text representing the attribute of the green first cup (1201a) selected by the user. As an example, the indicator (1205) can be a display indicating text of an attribute corresponding to the selected object.

[0253] Referring to FIG. 29, the electronic device (101) can change (or create) an object with multiple attributes. For example, texts of yellow (1193a) and blue (1193b) can be input. For example, if there are object-specific words input for image generation and various object description words (or object attribute words), the object description words can be selected in a mixed manner. The number of selectable object description words may not be limited.

[0254] A selection control indicator may be input for the text of yellow (1193a) and blue (1193b), and an object description word control indicator (1195) may be input. As an example, the selection control indicator may be a control command for selecting at least one of a plurality of elements, and may include a symbol, a star, a shape, a check, an arrow, a highlight, an underline, a text size, a text color, a text thickness, and / or a border. The object description word control indicator (1195) may be a command for selecting an object. The object description word control indicator (1195) may be a control command for summing the selected elements, and may include a plus sign. For example, the object description word control indicator (1195) may operate adaptively depending on the type of the object description word (e.g., color, shape, texture). An image (or, object, sketch) in the shape of a cup (1191) may be input together with the text. The electronic device (101) can identify additional attributes of yellow (1193a) and blue (1193b) as attributes of the image in the shape of a cup (1191) based on the selection control indicator. The electronic device (101) can identify an additional attribute that adds up yellow (1193a) and blue (1193b) based on the object description word control indicator (1195). The electronic device (101) can identify green, which is the sum of yellow (1193a) and blue (1193b), as an additional attribute of the image in the shape of a cup. The electronic device (101) can change (or generate) the input image in the shape of a cup (1191) into an image in the shape of a green cup (1197) based on the identified attributes.

[0255] Referring to FIG. 30A, an electronic device (101) implemented as an AR, VR, or MR device may receive a user input (e.g., a 2D or 3D sketch drawing) for adding an object to a two-dimensional (2D) original image or a three-dimensional (3D) original image in a virtual space or an area corresponding to a real environment, and text including attribute information. As an example, the electronic device (101) may execute a text input application (e.g., a note application) and receive text through a virtual tool (e.g., a virtual pen) and / or gestures. According to an example, the electronic device (101) may receive a sketch of a chair shape (1211) and a text of a wooden sofa leg (1213). The electronic device (101) may identify an attribute of the sketch as a chair based on the shape of the sketch. The electronic device (101) may identify additional attributes (e.g., material attributes, shape attributes) of the sketch as a wooden sofa leg based on the input text. The electronic device (101) can generate a 2D or 3D object of a chair (1215a) including a wooden sofa leg based on the identified properties. The electronic device (101) can display the generated 3D object.

[0256] Referring to FIG. 30b, as an example, the electronic device (101) may execute a text input application and receive text through a virtual tool and / or gesture. According to an example, the electronic device (101) may receive a text of a sketch of a chair shape (1211) and a wooden sofa leg (1213). The electronic device (101) may receive a control command. As an example, the electronic device (101) may receive a first circular drawing (1217a) input into a sofa area included in an image corresponding to an image or a real environment, and may receive a second circular drawing (1217b) input into a text area. When the first circular drawing (1217a) and the second circular drawing (1217b) are input, the electronic device (101) may identify the properties of the sketch by considering the elements included in the first circular drawing (1217a) and the second circular drawing (1217b). As an example, the electronic device (101) can identify an attribute of the sketch called a chair based on the shape of the sketch. The electronic device (101) can identify an attribute (e.g., a color attribute, a material attribute, a shape attribute) of the sofa for which the first circular drawing (1271a) was received. The electronic device (101) can identify additional attributes of the sketch by considering the attributes of the identified sofa and the text of the wood sofa leg (1213). In one example, the electronic device (101) can identify the wood sofa leg, the fabric material of the sofa, and / or the color pink of the sofa as additional attributes of the sketch. Based on the identified attributes, the electronic device (101) can generate a 2D or 3D object of the chair (1215b) including the wood sofa leg, the fabric material, and the color pink. The electronic device (101) can add or display the generated 2D or 3D object to a displayed image.

[0257] Referring to FIG. 30c, as an example, the electronic device (101) may execute a text input application and receive text through a virtual tool and / or gesture. In one example, the electronic device (101) may receive a text of a sketch of a chair shape (1211) and a wooden sofa leg (1213). The electronic device (101) may receive a control command. As an example, the electronic device (101) may receive a circular drawing (1217c) including a sofa area and text included in the image. When the circular drawing (1217c) is input, the electronic device (101) may identify attributes of the sketch by considering elements included in the circular drawing (1217c). As an example, the electronic device (101) may identify attributes of the sketch called a chair based on the shape of the sketch. The electronic device (101) can identify the properties of the sofa (e.g., color properties, material properties, shape properties) and / or the properties of the floor, etc., for which the circular drawing (1271c) was received. The electronic device (101) can identify additional properties of the sketch by considering the properties of the area of ​​the identified circular drawing (1271c) and the text of the wood sofa leg (1213). In one example, the electronic device (101) can identify the wood sofa leg, the fabric material of the sofa, and / or the color of the sofa (pink), and the color of the floor (white) as additional properties of the sketch. Based on the identified properties, the electronic device (101) can generate a 2D or 3D object of the chair (1215c) including the wood sofa leg, the fabric material, and the bright pink color that is a mixture of pink and white. The electronic device (101) can display the generated 2D or 3D object or add it to a displayed image.

[0258] Referring to FIG. 30d, as an example, the electronic device (101) may execute a text input application and receive text via a virtual tool and / or gesture. In an example, the electronic device (101) may receive a sketch of a chair shape (1211). The electronic device (101) may receive a control command. In an example, the electronic device (101) may receive a circular drawing (1217d) on a plant included in an image. In an example, the circular drawing (1271d) may include an emphasis attribute. In an example, if the circular drawing (1271d) is received as a bold line and / or in a continuous multiple number of times, the electronic device (101) may determine that the circular drawing (1271d) includes an emphasis attribute. When a circular drawing (1217d) including an emphasis attribute is input, the electronic device (101) can identify the attribute of the sketch by assigning weights to elements included in the circular drawing (1217d). As an example, the electronic device (101) can identify the attribute of the sketch as a chair based on the shape of the sketch. The electronic device (101) can identify the attribute (e.g., type attribute, color attribute) of the plant for which the circular drawing (1271d) is received. The electronic device (101) can identify additional attributes of the sketch by assigning weights to the attributes of the identified plant. As an example, the electronic device (101) can identify a wood sofa leg, a type of plant, and a dark green color of the plant as additional attributes of the sketch. Based on the identified attributes, the electronic device (101) can generate a 2D or 3D object of a wooden chair (1215d) made of wood, including a wood sofa leg and a dark green color, and a type of plant. The electronic device (101) can display the generated 2D or 3D object or add it to a displayed image.

[0259] Referring to FIG. 31, as an example, a drawing input application may be executed. The drawing input application may receive sketches and / or text. The electronic device (101) may receive various texts (e.g., words (nouns)) through the drawing input application. In one example, the electronic device (101) may receive texts of green (1221a), sun (1221b), ball (1223c), and rain (1221d) at any location in the drawing input application.

[0260] As an example, the electronic device (101) may receive a selection control indicator on the text. For example, the selection control indicator may be a control command for selecting at least one of a plurality of elements and may include circular drawings (1223a, 1223b, 1223c). In one example, among the input texts, a first circular drawing (1223a) may be input to green (1221a), a second circular drawing (1223b) may be input to sun (1221b), and a third circular drawing (1223c) may be input to ball (1221c). The texts of green (1221a), sun (1221b), and ball (1221c) may be selected based on the input of the selection control indicator. As an example, the electronic device (101) may accurately recognize the text to be selected based on the selection control indicator even when the texts overlap. The text selected by the selection control indicator can be generated as a prompt and passed to an AI-based image editing model (e.g., a generative AI model). In one example, the AI-based image editing model can generate an object of grass (1227a) based on the text of green (1221a), an object of sun (1227b) based on the text of sun (1221b), and an object of ball (1227c) based on the text of ball (1221c).

[0261] For example, an artificial intelligence-based image editing model can generate an object corresponding to a selected text and generate an image in which the generated object is arranged by itself. For example, the electronic device (101) can receive a placement control indicator on the text. For example, the placement control indicator can be a control command for determining the placement position of an object and can include arrow drawings (1225a, 1225b, 1225c). For example, among the input texts, a downward arrow drawing (1225a) can be input to green (1221a), an upward arrow drawing (1225b) can be input to sun (1221b), and a left-right horizontal arrow drawing (1225c) can be input to ball (1223c). An artificial intelligence-based image editing model can generate an image in which an object of grass (1227a) is placed downwards, an object of the sun (1227b) is placed upwards, and an object of a ball (1227c) is placed on a flat surface (e.g., grass (1227a)) based on the input of the placement control indicator.

[0262] As an example, the electronic device (101) can generate an image (or object) by reflecting the properties of surrounding text. The electronic device (101) can generate a first image through an artificial intelligence-based image editing model (e.g., a generative AI model) based on a drawing by a user gesture input and some of the text displayed around the drawing. The electronic device (101) can display the generated first image in place of the drawing. When a user input occurs in the text reflected in the prompt for generating the first image, the electronic device (101) can regenerate the first image by reflecting the modification of the text. As an example, the user input occurring in the text reflected in the prompt can include an additional drawing input (e.g., an underline, a circle, a check mark, etc.) corresponding to a control command for deleting text, adding text, or modifying text. As an example, the electronic device (101) can replace the first image with a regenerated image. As an example, the electronic device (101) can display the regenerated image around the first image. As an example, the electronic device (101) can display a regenerated image by emphasizing (e.g., highlighting) a changed portion of the first image.

[0263] As an example, the electronic device (101) can analyze the properties of the modified text and determine whether to regenerate the entire first image or a portion of the first image.

[0264] As an example, the electronic device (101) can assign a relatively high weight to text that is emphasized (e.g., highlighted, underlined, checked, sketched, etc.) among the surrounding text of the drawing and generate an image. As an example, the electronic device (101) can assign a relatively high weight to text that is larger than the surrounding text or to text displayed on the screen without any obscured portions.

[0265] For example, when the electronic device (101) generates an image based on drawing properties, the image can be generated by reflecting not only surrounding text but also sound information and properties of information provided by a linked device in a multi-device environment. According to one example, when a user watches a lecture video played on a linked TV or tablet PC in a multi-device environment and takes notes through a note application of the electronic device (101), the electronic device (101) can obtain sound (e.g., voice) information and / or image information corresponding to the time at which the user takes notes to generate an object image. According to one example, when multiple applications are executed and / or multiple applications are executed through a multi-screen function in an electronic device and / or a multi-device environment, the electronic device (101) can obtain information (e.g., text, image, sound, etc.) provided by other applications (e.g., media player, browser, SNS, etc.) other than the currently used application (e.g., note, gallery, image editing, etc.), and reflect it in a prompt for generating an image. For example, the electronic device (101) can generate a sound prompt based on acquired sound information and a visual prompt based on acquired image information. The electronic device (101) can convert the sound prompt and / or visual prompt into a text prompt and identify the properties of the image.

[0266] In one example, if there are multiple selected attributes, multiple selectable options related to the attributes, and / or multiple selected attributes are in conflict, the electronic device (101) may display selectable candidates (e.g., text, thumbnail images, etc.) related to the attributes. In one example, if the multiple selected attributes are in conflict, the electronic device (101) may select an area of ​​an image to which the attribute is to be reflected based on a user input, or position text corresponding to the attribute in an area of ​​an image to which the attribute is to be reflected.

[0267] As an example, the electronic device (101) may determine text recommended for image generation from among text data (e.g., notes) displayed around the drawing. The electronic device (101) may provide a UI (e.g., a control item) that can reflect the determined text in the image generation.

[0268] For example, when the electronic device (101) displays an object image corresponding to a drawing input, if it cannot be displayed as a single image (e.g., handwritten texts express multiple subjects that cannot be reflected in a single image, or express properties that change over time), the electronic device (101) can generate and provide each candidate image reflecting different texts.

[0269] According to one example, the electronic device (101) can share a generated object (or an image of an object) and / or an image including a generated object with another device. When the electronic device (101) shares an object generated and / or an image including a generated object with another device through an artificial intelligence-based image editing model (e.g., a generative AI model), the electronic device (101) can share (e.g., store as metadata of the object and / or image) data related to the generation of the object and / or image (e.g., sketches / generated objects / surrounding objects / text, etc.). When the electronic device (101) receives an object generated and / or an image including a generated object through an artificial intelligence-based image editing model (e.g., a generative AI model) from another device, the electronic device (101) can receive data related to the generation of the object and / or image (e.g., sketches / generated objects / surrounding objects / text, etc.) together. When the electronic device (101) displays a shared object and / or image on a screen (or, AR environment, VR environment), it can display one or more related data around the object and / or image.

[0270] For example, the electronic device (101) may receive drawings and texts according to user commands and apply visual effects (e.g., effects indicating main keywords using highlighting, underlining, etc.) to the texts. When an image is generated through an artificial intelligence-based image editing model based on a drawing input (e.g., a sketch, text), and a visual effect is applied to text displayed in the vicinity, the electronic device (101) may extract the text with the visual effect and reflect it in the image generation. As an example, the electronic device (101) may also apply the visual effect to a portion of the image related to the text with the visual effect and display it. For example, when the text of a 'pillar' reflected in the image generation is highlighted or underlined, the electronic device (101) may display the highlight or text together in the pillar area of ​​the generated image.

[0271] For example, the electronic device (101) may display a visual effect (e.g., an emphasis effect using an indicator, etc.) on text related to an area selected by the user on an image among displayed text. According to one example, the electronic device (101) may receive an input for selecting an object of a green first cup (1201a). The electronic device (101) may determine a green attribute corresponding to the object of the green first cup (1201a). The electronic device (101) may display an indicator (1205) on the text of green (1203a) among the texts indicating the attribute. As an example, the indicator (1205) may be a display indicating text of an attribute corresponding to the selected object.

[0272] As an example, when an additional drawing input (e.g., changing the shape / color of an image object included in the image, deleting at least a portion of it, adding an object) is received on a generated first image and a second image is generated in which the properties of the additional drawing input are reflected, the electronic device (101) may display text related to the properties of the additional drawing input around the image or add text related to the properties of the additional drawing input to the displayed text. As an example, the electronic device (101) may process text related to the properties of the additional drawing input by considering the context of the displayed text and add the text to the displayed text.

[0273] As an example, the electronic device (101) may determine recommended text for image generation from among text data (e.g., notes) displayed around the sketch. The electronic device (101) may provide a UI (e.g., a control item) that can reflect the determined text in the image generation. As an example, the UI may be displayed with a graphic element that can add text in a floating form.

[0274] As an example, the electronic device (101) can analyze and extract image and / or text data included in the location and area where the circular drawing is input according to the size of the circular drawing and reflect the data in the prompt for image generation.

[0275] As an example, the electronic device (101) may determine whether to generate only an object image or a background image based on text properties surrounding the sketch. For example, a tree-shaped sketch and text of a beach and palm trees may be input. For example, if only the text of a palm tree is selected, the electronic device (101) may generate only a palm tree object. For example, if the text of a beach and palm trees is selected, the electronic device (101) may generate an object of a palm tree with a beach background.

[0276] For example, when an additional drawing is received on an image, the electronic device (101) can regenerate the image by reflecting the additional drawing properties. The image into which the additional drawing is input may be an original image or a generated image.

[0277] For example, if a drawing is input to change some properties (e.g., a door of a house) within an object (or an image of an object), the electronic device (101) can create an object (e.g., create a house including a door) based on the changed properties. As an example, the electronic device (101) can reflect information in the prompt to create only areas corresponding to some properties (e.g., create only an area of ​​a door) and maintain the remaining areas.

[0278] For example, when the input means or input device changes, the electronic device (101) may perform different actions with respect to the drawing received based on the input means / device. In one example, when a drawing is received based on a gesture input using a hand, the electronic device (101) may generate an image. In one example, when a drawing using a pen is received, the electronic device (101) may determine it as a control input. In one example, a function may be set so that a drawing tool may perform different actions for each of a plurality of selectable input means for drawing.

[0279] As an example, when generating an image, the electronic device (101) may generate and output a voice description related to the image, a sound effect related to a location expressed in the image (e.g., the sound of ocean waves), and a sound related to an object included in the image (e.g., the sound of seagulls). When the electronic device (101) generates a voice description related to the image, the electronic device (101) may utilize information from a prompt used to generate the image. As an example, the electronic device (101) may generate and output a text description instead of a voice.

[0280] As an example, when the electronic device (101) generates an image, it may provide a link (e.g., a web page) containing information related to the image around the image. For example, the link may correspond to a web page that searches for and / or discloses content related to the surrounding text used to generate the image. The link may be provided in the generated image. For example, when a user inputs a specific gesture (e.g., double-tap) to the generated image, the electronic device (101) may provide a UI that displays the content of the link. The electronic device (101) may summarize content searched on the web related to the generated image, and when a specific gesture is input to the generated image, it may provide a UI that displays the summarized content.

[0281] As an example, when a specific area is selected (e.g., by specifying the area with a pen) in a generated image, the electronic device (101) can emphasize (e.g., highlight) text related to the selected area (or involved in generating the area).

[0282] For example, different images can be generated depending on the input method of a circular drawing surrounding a sketch. For example, only the sketch and text contained within the circular drawing may be reflected in the image. For example, drawing a circular drawing once generates an image, and drawing it repeatedly (e.g., overlapping the drawing three times) can generate a video. Drawing a circular drawing once can only reflect the contents (text) of the corresponding page in the image, while drawing it twice can also reflect the contents of at least one other page in the document. The more circular drawings are overlapped, the more elaborate the image can be generated.

[0283] For example, differences between previously input drawings and / or images and the currently generated image can be reflected in the image. For example, differences from other drawings / images can be highlighted. If the text includes temporal change as part of its subject matter, a video showing the changes between the drawings / images (e.g., highlighting and playing back the changes) can be generated and presented.

[0284] For example, a drawing may be input on a specific page of a note, memo, or document. Each page may contain handwritten text. In one embodiment, when a user draws on a specific page, the text on that page may be reflected in an image. In one embodiment, when a user draws on a specific page, the electronic device (101) may identify other pages containing text, drawings, and / or images associated with that page, and may reflect the text, drawings, and / or images contained in the identified pages in the image.

[0285] For example, if texts are written in multiple languages, including a first language and a second language, the electronic device (101) can generate multiple images for each language. For example, the electronic device (1010) can translate the second language into the first language and use it as prompts, metadata, and / or tag information.

[0286] For example, an image reflecting the content / subject expressed by a drawing of an object input by a user using a touchscreen with a hand or a pen can be generated. The drawing of the object input by the user can include the shape of an object, the shape of a person, and / or writing. When a drawing of a specific shape (e.g., a circle, a square, an arrow) is identified around the drawing of the object input by the user, the electronic device (101) can generate an image using the drawing of the objects designated by the drawing of the specific shape. The generated image can be generated in a blank space on the screen / page and can be generated in a size appropriate to the size of the blank space. The area where the generated image is located can be a blank space on the screen / page or an area designated by the user's hand or pen.

[0287] In an AR and / or VR environment (e.g., an electronic device is an XR device), an image can be generated based on an object in the image corresponding to the real environment (e.g., information acquired by a camera) or a drawing input for background modification and text included in an application (e.g., a note app, a browser app, an SNS app, etc.) running on an electronic device (101) (e.g., a visual wearable device).

[0288] As an example, in an AR and / or VR environment, when an image is generated based on a drawing object and surrounding text input from a real environment or application, the electronic device (101) can generate an image suitable for the real environment by extracting elements such as shape, color, design, etc. of surrounding objects (e.g., furniture, space style, etc.) included in an image corresponding to the real environment or depth value of objects in the real environment and reflecting them in a prompt for image generation.

[0289] As an example, when an image is generated in an AR and / or VR environment, the electronic device (101) may extract information about the real environment area where the drawing object is located and reflect this information in the prompt for image generation. For example, the information about the real environment area may include depth information of the real environment.

[0290] As an example, when an image is generated in an AR and / or VR environment, elements representing the shape / style / color / design, etc. of other images or UIs included in one or more running applications can be extracted. The electronic device (101) can reflect the extracted data in a prompt and generate an image that matches the content or UI characteristics provided by the running application.

[0291] As an example, when an image is generated in an AR and / or VR environment, the electronic device (101) may select an object for reference. For example, the object for reference may include an image, text, and / or an object included in a real environment displayed on a screen within a running application. The electronic device (101) may receive a gesture or drawing (e.g., a circle) corresponding to a control command related to the object for reference. The electronic device (101) may set a different reflection intensity depending on the input characteristics of the control command related to the object for reference. For example, when a circular gesture or drawing is received for the object for reference, the electronic device (101) may generate a prompt to generate an image by reflecting relatively more weights of object characteristics when many circles are drawn, and to generate an image by reflecting relatively less weights when few circles are drawn.

[0292] As an example, when an image is generated in an AR and / or VR environment, if a drawing in the shape of a hat is received on an actual person's head, the electronic device (101) can generate a 3D image (e.g., a person wearing a hat) that includes the drawing in the shape of a hat and information about the person related to the drawing. The electronic device (101) can provide a function for rotating the generated 3D image.

[0293] Figure 32 is a flowchart illustrating an image editing method according to various embodiments.

[0294] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0295] According to one embodiment, steps 3210 to 3240 may be understood to be performed in a processor (e.g., processor (120) of FIG. 2) of an electronic device (e.g., electronic device (101) of FIG. 2).

[0296] Referring to FIG. 32, the electronic device (101) may receive a second drawing input around an object included in an image (3210). For example, the object may include at least one drawing element received by the first drawing input. As an example, the first drawing input may include a drawing that creates the object (e.g., a sketch, text), and the second drawing input may include a drawing that controls the object (e.g., a control input).

[0297] For example, the electronic device (101) can identify the properties of an object (3220). As an example, when the electronic device (101) determines the second drawing input as a control command based on the object and the second drawing input, the electronic device (101) can identify the properties of the object. The electronic device (101) can identify the type of the control command based on the second drawing input. For example, the control command can include a command to identify a sketch (or create a preliminary object image), move an object, select at least one of a plurality of elements (or attributes), sum the selected elements, and / or determine the placement position of an object. The electronic device (101) can identify the properties of the object based on at least one of the shape of at least one drawing element and the relevance of the drawing elements. For example, the image can include text. The electronic device (101) can additionally use the text to identify the properties of the object.

[0298] For example, the electronic device (101) can generate a first preliminary object image based on the properties of the identified object (3230). If the type of the identified control command is a command to generate the first preliminary object image, the electronic device (101) can provide at least one drawing element by replacing it with the generated first preliminary object image. As an example, the first preliminary object image can be generated based on image information of an object of the same type as an object included in another image. As an example, the electronic device (101) can generate a plurality of first preliminary object images.

[0299] For example, the electronic device (101) may provide the generated first preliminary object image (3240). As an example, when a plurality of first preliminary object images are generated, the electronic device (101) may provide the generated plurality of first preliminary object images and receive a selection command from the user.

[0300] For example, the electronic device (101) may receive a third drawing input that includes additional text in an area of ​​text. The electronic device (101) may identify a modified attribute of the object based on the text and the additional text. The electronic device (101) may generate a second preliminary object image based on the modified attribute of the object and provide the generated second preliminary object image. For example, the electronic device (101) may receive a fourth drawing input that modifies text. The electronic device (101) may identify a modified attribute of the object based on the modified text. The electronic device (101) may generate a third preliminary object image based on the modified attribute of the object and provide the generated third preliminary object image.

[0301] According to one example, the electronic device (101) may receive a first user input on a displayed original image (e.g., the original image (11) of FIG. 5) (3210). The electronic device (101) may identify a first object based on a shape drawn by the first user input, and may determine whether the first user input is an input for adding the first object (3220). For example, the user input may be an input for adding an object, an input for modifying an object, or a control input. The input for adding an object may include an input for adding a new object and / or person. The input for modifying an object may include an input for changing the size or shape of an object, an input for changing a color, and / or an input for adding a color. The control input corresponds to a command related to editing an image, and may include an input for changing the position of an object and / or an input for adding information about the object. That is, the electronic device (101) may determine whether the user input is an input for adding an object, an input for modifying an object, or a control input.

[0302] If the first user input is an input for adding the first object, the electronic device (101) may request generation of a first preliminary object image (e.g., preliminary object image (25a) of FIG. 5) including properties of the first object (3230). As an example, the electronic device (101) may request generation of a second preliminary object image (e.g., preliminary object image (27a) of FIG. 5) including properties of a second object. For example, the first object may be an object to be added, and the second object may be an object included in the original image. For example, the electronic device (101) may request generation of the preliminary object image from an artificial intelligence-based computing device. The artificial intelligence-based computing device may be included in the electronic device (101), and the processor (120) may request generation of the preliminary object image from the artificial intelligence-based computing device. As an example, the artificial intelligence-based computing device may be included in an external device (e.g., the server (108) of FIG. 1). The electronic device (101) can request the generation of a preliminary object image from an artificial intelligence-based computing device via a communication interface (e.g., the communication module (190) of FIG. 1).

[0303] The electronic device (101) can generate a first edited image (e.g., the first edited image (12) of FIG. 5) based on the first preliminary object image. The first edited image may include a portion (e.g., the first part) of a first object (e.g., the object (25b) of FIG. 5) and may be an image associated with the original image. As an example, the first edited image may include a portion (e.g., the first part) of a second object (e.g., the object (27b) of FIG. 5). The electronic device (101) can display the first edited image (3240).

[0304] When the electronic device (101) receives a second user input related to the first object on the first edited image, the electronic device (101) may display a second edited image (e.g., the second edited image (13) of FIG. 5) in which the size and / or position of the first object is changed according to the second user input based on the first preliminary object image (3250). For example, the input related to the first object on the first edited image may be an input for moving the first object. For example, the input for moving the first object may be an input for moving the first object to another position on the screen (e.g., a drag input) or a control input. As an example, the control input may be an input in the form of an arrow corresponding to the distance and direction in which the first object is to be moved. As an example, when the input for moving the first object on the first edited image is received, the electronic device (101) may display a second edited image in which the position of the first object is changed based on the first preliminary object image. The electronic device (101) can acquire depth information of the first object based on a change in the position of the first object, and display the first object based on the acquired depth information of the first object. In addition, the electronic device (101) can automatically change the position of the second object on the second edited image to a position different from the position of the second object on the first edited image based on the relationship with the first object.

[0305] As an example, a method for generating an image may include receiving a second drawing input around an object included in the image. The method may include identifying an attribute of the object when the second drawing input is determined to be a control command based on the object and the second drawing input. The method may include generating a first preliminary object image based on the attribute of the identified object. The method may include providing the generated first preliminary object image.

[0306] As an example, the object may include at least one drawing element received by the first drawing input.

[0307] As an example, the method may further include an operation of identifying the type of the control command. The operation of providing the generated first preliminary object image may provide the generated first preliminary object image by replacing at least one drawing element with the generated first preliminary object image, if the type of the identified control command is a command for generating the first preliminary object image.

[0308] As an example, the act of identifying a property of the object may identify the property of the object based on at least one of a shape of the at least one drawing element and a relevance of the drawing elements.

[0309] As an example, the image may further include text. The operation of identifying the properties of the object may additionally utilize the text to identify the properties of the object.

[0310] As an example, the method may further include receiving a third drawing input including additional text in an area of ​​the text. The method may further include identifying a modified property of the object based on the text and the additional text. The method may further include generating a second preliminary object image based on the modified property of the object. The method may further include providing the generated second preliminary object image.

[0311] As an example, the method may further include receiving a fourth drawing input that modifies the text. The method may further include identifying a modified property of the object based on the modified text. The method may further include generating a third preliminary object image based on the modified property of the object. The method may further include providing the generated third preliminary object image.

[0312] As an example, the first preliminary object image may be generated based on image information of an object of the same type as the object included in another image.

[0313] As an example, the operation of generating a first preliminary object image may generate a plurality of first preliminary object images. The operation of providing the generated first preliminary object image may provide the plurality of generated first preliminary object images.

[0314] As an example, the method may further include an operation of identifying the type of the control command. The operation of providing the generated preliminary object image may, if the type of the identified control command is a command to move an object, delete the object and display the generated first preliminary object image at a location different from the location where the object is displayed.

[0315] As an example, an electronic device may include a display for displaying an image, at least one processor including a processing circuit, and a memory for storing instructions that are individually or collectively executed by the at least one processor. The instructions stored in the memory may be configured to cause the electronic device to receive a second drawing input around an object included in the image. The instructions may be configured to cause the electronic device to identify an attribute of the object when the second drawing input is determined to be a control command based on the object and the second drawing input. The instructions may be configured to cause the electronic device to generate a first preliminary object image based on the attribute of the identified object. The instructions may be configured to cause the electronic device to provide the generated first preliminary object image.

[0316] As an example, the object may include at least one drawing element received by the first drawing input.

[0317] As an example, the command may be configured to cause the electronic device to identify a type of the control command, and, if the type of the identified control command is a command to generate a first preliminary object image, provide the at least one drawing element by replacing it with the generated first preliminary object image.

[0318] As an example, the command may be configured to cause the electronic device to identify a property of the object based on at least one of a shape of the at least one drawing element and a relevance of the drawing elements.

[0319] As an example, a non-transitory computer-readable storage medium having recorded thereon a program that performs a method for generating an image may include an operation for receiving a second drawing input around an object included in the image. The storage medium may include an operation for identifying an attribute of the object when the second drawing input is determined to be a control command based on the object and the second drawing input. As an example, the storage medium may include an operation for generating a first preliminary object image based on the attribute of the identified object. As an example, the storage medium may include an operation for providing the generated first preliminary object image. As an example, a method for editing an image may include receiving a first user input on a first image (e.g., an original image (11) of FIG. 5). The method may identify whether the first user input is an input for adding a first object (e.g., an object (25b) of FIG. 5). The method may request generation of a first preliminary object image (e.g., preliminary object image (25a) of FIG. 5) for the first object when the first user input is an input for adding the first object. The method may include a first object generated based on the first preliminary object image, and may display a second image (e.g., a first edited image (12) of FIG. 5) associated with the first image. When the method receives a second user input related to the first object on the second image, the method may display a third image (e.g., a second edited image (13) of FIG. 5) in which at least one of a size and a position of the first object is changed according to the second user input based on the first preliminary object image.

[0320] As an example, the action of identifying whether the input is adding the first object may identify the first object based on a shape drawn by the first user input.

[0321] As an example, the first preliminary object image may be generated based on image information of an object of the same type as the first object included in another image.

[0322] As an example, the first preliminary object image may include a first portion and a second portion of the first object. The operation of displaying the second image may display only the first portion of the first object. The operation of displaying the third image may display the first portion and the second portion of the first object.

[0323] As an example, the action of displaying the third image may display a lighting effect based on a change in at least one of the size and position of the first object.

[0324] As an example, the method may identify a second object (e.g., object (27b) of FIG. 5) included in the first image and request generation of a second preliminary object image (e.g., preliminary object image (27a) of FIG. 5) for the second object. The second preliminary object image may include a first portion and a second portion of the second object. The operation of displaying the second image may display the second object generated based on the second preliminary object image.

[0325] As an example, the operation of displaying the third image may obtain depth information of the first object that is greater than the depth information of the second object when the position of the first object changes to be behind the second object. The operation of displaying the third image may obtain depth information of the first object that is less than the depth information of the second object when the position of the first object changes to be in front of the second object. The operation of displaying the third image may display the first object and the second object based on the depth information of the first object and the depth information of the second object.

[0326] As an example, the operation of displaying the third image may be such that when the position of the first object is changed to a position overlapping with the area where the second object is located, if the depth information of the first object is smaller than the depth information of the second object, only the first part of the second object excluding the second part of the second object that overlaps with the first object may be displayed. The operation of displaying the third image may be such that when the depth information of the first object is greater than the depth information of the second object, only the first part of the first object excluding the second part of the first object that overlaps with the second object may be displayed.

[0327] As an example, the first image may include the first portion and the second portion of the second object. The operation of displaying the third image may display the first portion among the first portion and the second portion of the second object based on the relationship with the first object.

[0328] As an example, the first image may include the second object of a first size. The act of displaying the second image may display the second object at a second size different from the first size based on a relationship with the first object.

[0329] As an example, the operation of identifying the first user input may identify the first user input based on the type of input method of the first user input.

[0330] As an example, an electronic device (101) may include a display (160), at least one processor (120), and a memory (130) that stores instructions executed by the at least one processor (120). The instructions may cause the electronic device (101) to perform an operation of requesting generation of first object image information including a first part and a second part of the first object when a first user input for adding a first object on a first image displayed on the display (160) is received. The instructions may cause the electronic device (101) to perform an operation of displaying, on the display (160), a second image in which the first part of the first part and the second part of the first object is added to the first image based on the first object image information. The above command may cause the electronic device (101) to perform an operation of displaying a third image in which the second part of the first object is added to the second image based on the first object image information when a second user input related to the first object is received on the second image.

[0331] As an example, the command may cause the electronic device (101) to perform an operation of identifying an attribute of the first object, including a type or characteristic of the first object, based on a shape formed by the first user input using an artificial intelligence computing device.

[0332] As an example, the command may cause the electronic device (101) to perform an operation of identifying a relationship between an attribute of the first object and information of the first image background. The first object image information may include information of a new object related to the first object based on the relationship.

[0333] As an example, the information of the first image background may include at least one of location information, time information, and weather information.

[0334] As an example, the command may cause the electronic device (101) to perform an operation of displaying a plurality of objects of the same type as the first object included in at least one of the images stored in the memory (130) and the searched images. The command may cause the electronic device (101) to perform an operation of generating the first object image information based on an image of one object selected from among the displayed plurality of objects.

[0335] As an example, the command may cause the electronic device (101) to perform an operation of selecting a second object from among a plurality of objects included in the first image according to a preset priority. The command may cause the electronic device (101) to perform an operation of requesting the generation of second object image information including a first portion and a second portion of the selected second object.

[0336] As an example, the command may cause the electronic device (101) to automatically change the location of the second object on the third image to a different location than the location of the second object on the second image based on the relationship with the first object.

[0337] As an example, a non-transitory computer-readable storage medium having recorded thereon a program for performing a method of editing an image may include an operation for identifying, when a first touch gesture input is received on a first image through a touch screen, whether the first touch gesture input is an input for adding a first object on the first image or an input for modifying a second object included in the first image. The storage medium may include an operation for requesting an artificial intelligence computing device to generate first object image information including a first part and a second part of the first object, if the first touch gesture input is an input for adding the first object. The storage medium may include an operation for displaying at least one of the first part and the second part of the first object on the first image based on the first object image information. The storage medium may include an operation for requesting the artificial intelligence computing device to generate second object image information including a third part and a fourth part of the second object, if the first gesture input is an input for modifying the second object. The storage medium may include an operation of displaying at least one of the third part and the fourth part of the second object on the first image based on the second object image information.

[0338] As an example, the input for modifying the second object included in the first image may include at least one of a change in color and texture of at least a portion of the second object.

[0339] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0340] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0341] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0342] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0343] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0344] The effects of this document are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the above description.

Claims

1. In the method of generating an image, An action of receiving a second drawing input around an object included in the above image; An operation for identifying a property of the object when the second drawing input is determined to be a control command based on the object and the second drawing input; An operation of generating a first preliminary object image based on the properties of the identified object; and A method comprising: providing a first preliminary object image generated as described above.

2. In paragraph 1, The above object is, A method comprising at least one drawing element received by a first drawing input.

3. In paragraph 2, An operation for identifying the type of the above control command; further comprising: The operation of providing the first preliminary object image generated above is: A method for providing a method of replacing at least one drawing element with the generated first preliminary object image, when the type of the identified control command is a command for generating a first preliminary object image.

4. In paragraph 2, The action of identifying the properties of the above object is: A method for identifying a property of an object based on at least one of a shape of at least one drawing element and a relevance of the drawing elements.

5. In paragraph 4, The image above contains more text, The action of identifying the properties of the above object is: A method of identifying properties of an object by additionally using the above text.

6. In paragraph 5, An action of receiving a third drawing input containing additional text in an area of ​​said text; An action to identify modified properties of the object based on the text and the additional text; An operation of generating a second preliminary object image based on the modified properties of the above object; and A method further comprising: providing the second preliminary object image generated above.

7. In paragraph 5, An action to receive a fourth drawing input that modifies the above text; An action to identify modified properties of said object based on said modified text; An operation of generating a third preliminary object image based on the modified properties of the above object; and A method further comprising the action of providing the third preliminary object image generated above.

8. In paragraph 1, The above first preliminary object image is, A method generated based on image information of an object of the same type as the object contained in another image.

9. In paragraph 1, The action of generating the first preliminary object image is: Generate multiple first preliminary object images, The operation of providing the first preliminary object image generated above is: A method for providing a plurality of first preliminary object images generated above.

10. In paragraph 1, An operation for identifying the type of the above control command; further comprising: The action of providing the above generated preliminary object image is: A method of deleting the object and displaying the generated first preliminary object image at a location other than the location where the object is displayed, when the type of the identified control command is a command to move an object.

11. In electronic devices, A display that displays images; At least one processor comprising a processing circuit; and A memory storing instructions to be executed individually or collectively by at least one processor; The instructions stored in the above memory cause the electronic device to: Receive a second drawing input around the object included in the above image, When the second drawing input is determined to be a control command based on the above object and the second drawing input, the property of the object is identified, Generating a first preliminary object image based on the properties of the identified object, An electronic device configured to provide the first preliminary object image generated above.

12. In paragraph 11, The above object is, An electronic device comprising at least one drawing element received by a first drawing input.

13. In paragraph 12, The instructions stored in the above memory cause the electronic device to: An electronic device configured to identify the type of the control command, and if the type of the identified control command is a command to generate a first preliminary object image, provide at least one drawing element by replacing it with the generated first preliminary object image.

14. In paragraph 12, The instructions stored in the above memory cause the electronic device to: An electronic device configured to identify a property of the object based on at least one of a shape of said at least one drawing element and a relevance of said drawing elements.

15. A non-transitory computer-readable storage medium having recorded thereon a program for performing a method of generating an image, An action of receiving a second drawing input around an object included in the above image; An operation for identifying a property of the object when the second drawing input is determined to be a control command based on the object and the second drawing input; An operation of generating a first preliminary object image based on the properties of the identified object; and A non-transitory computer-readable storage medium having recorded thereon a program for performing a method, comprising: providing the first preliminary object image generated above.

Images