Electronic device for generating image based on sketch input and control method thereof

Abstract

An electronic device is disclosed. The electronic device of the present disclosure comprises: a processor; a display; and a memory for storing instructions which, when individually or collectively performed by the processor, cause the electronic device to: display an image on the display; display, on the basis of receiving a sketch input on the image, the image including a drawing image corresponding to the sketch input; and provide an edited image including an object image by generating the object image on the basis of the sketch input, wherein the object image is obtained as output data from a second artificial intelligence model by using, as input data of a first artificial intelligence model, the drawing image corresponding to the sketch input and using, as input data of a second artificial intelligence model, a prompt obtained as output data from the first artificial intelligence model, the first artificial intelligence model is trained to obtain, as output data, a prompt for image generation by using the drawing image as input data, and the second artificial intelligence model may be trained to obtain, as output data, the object image by using the prompt as input data.

Description

Electronic device for generating an image based on sketch input and method for controlling the same

[0001] Embodiments of the present disclosure relate to an electronic device for generating an image based on a sketch input and a method for controlling the same.

[0002] The variety of services and additional features offered through electronic devices, such as smartphones, is steadily increasing. To enhance the utility of these devices and satisfy the diverse needs of users, telecommunications service providers and electronic device manufacturers are competitively developing electronic devices to offer a variety of features and differentiate themselves from competitors. Consequently, the various functions offered through electronic devices are also becoming increasingly sophisticated.

[0003] Additionally, electronic devices provide various GUIs (graphical user interfaces) for user interaction through displays.

[0004] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0005] According to one embodiment, an electronic device may include at least one processor, a display, and a memory that stores instructions that, when individually or collectively executed by the at least one processor, cause the electronic device to perform the following operations:

[0006] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display an image on the display.

[0007] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to, based on receiving a sketch input on the image, display on the image a drawing image corresponding to the sketch input.

[0008] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to provide an edited image including the object image by generating an object image based on the sketch input.

[0009] According to one embodiment, the object image may be obtained as output data from the second artificial intelligence model, with the drawing image corresponding to the sketch input as input data of the first artificial intelligence model, and a prompt obtained as output data from the first artificial intelligence model as input data of the second artificial intelligence model.

[0010] According to one embodiment, the first artificial intelligence model may be trained to obtain a prompt for image generation as output data, using a drawing image as input data.

[0011] According to one embodiment, the second artificial intelligence model may be trained to obtain an object image as output data by using a prompt as input data.

[0012] According to one embodiment, a server may include communication circuitry, at least one processor, and a memory storing instructions that, when individually or collectively executed by the at least one processor, cause the server to perform the following operations:

[0013] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the server to receive, through the communication circuit, from an external electronic device a drawing image corresponding to a sketch input received from the external electronic device.

[0014] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the server to transmit the drawing image to a first external server where the first artificial intelligence model is stored via the communication circuit.

[0015] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the server to receive the object image corresponding to the drawing image from a second external server in which the second artificial intelligence model is stored through the communication circuit.

[0016] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the server to transmit the object image to the external electronic device via the communication circuit.

[0017] According to one embodiment, the first artificial intelligence model may be trained to obtain a prompt for image generation as output data, using a drawing image as input data.

[0018] According to one embodiment, the second artificial intelligence model may be trained to obtain an object image as output data by using a prompt as input data.

[0019] According to one embodiment, a method of controlling an electronic device may include an action of displaying an image on a display of the electronic device.

[0020] According to one embodiment, the control method of the electronic device may include an action of displaying on the image a drawing image corresponding to the sketch input, based on receiving a sketch input on the image.

[0021] According to one embodiment, the control method of the electronic device may include an operation of providing an edited image including the object image by generating an object image based on the sketch input.

[0022] According to one embodiment, the object image may be obtained as output data from the second artificial intelligence model, with the drawing image corresponding to the sketch input as input data of the first artificial intelligence model, and a prompt obtained as output data from the first artificial intelligence model as input data of the second artificial intelligence model.

[0023] According to one embodiment, the first artificial intelligence model may be trained to obtain a prompt for image generation as output data, using a drawing image as input data.

[0024] According to one embodiment, the second artificial intelligence model may be trained to obtain an object image as output data by using a prompt as input data.

[0025] According to one embodiment, a non-transitory computer-readable recording medium storing one or more programs may include instructions that cause an electronic device to display an image on a display of the electronic device.

[0026] In one embodiment, the one or more programs may include instructions that cause the electronic device to, based on receiving a sketch input on the image, display on the image a drawing image corresponding to the sketch input.

[0027] In one embodiment, the one or more programs may include instructions that cause the electronic device to generate an object image based on the sketch input, thereby providing an edited image including the object image.

[0028] According to one embodiment, the object image may be obtained as output data from the second artificial intelligence model, with the drawing image corresponding to the sketch input as input data of the first artificial intelligence model, and a prompt obtained as output data from the first artificial intelligence model as input data of the second artificial intelligence model.

[0029] According to one embodiment, the first artificial intelligence model may be trained to obtain a prompt for image generation as output data, using a drawing image as input data.

[0030] According to one embodiment, the second artificial intelligence model may be trained to obtain an object image as output data by using a prompt as input data.

[0031] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0032] FIG. 2 is a flowchart illustrating an operation of providing an image generated based on a sketch input of an electronic device according to one embodiment.

[0033] FIG. 3 is a drawing for explaining an operation of providing an image generated based on a sketch input of an electronic device according to one embodiment.

[0034] FIG. 4 is a drawing for explaining an operation of providing an image generated based on a sketch input of an electronic device according to one embodiment.

[0035] FIG. 5 is a drawing for explaining an operation of providing an image generated based on a sketch input of an electronic device according to one embodiment.

[0036] FIG. 6 is a drawing for explaining a crop operation based on a sketch input of an electronic device according to one embodiment.

[0037] FIG. 7 is a flowchart illustrating a cropping operation based on a sketch input of an electronic device according to one embodiment.

[0038] FIG. 8 is a flowchart illustrating an operation of determining a margin included in a generation area of ​​an object image corresponding to a sketch input of an electronic device according to one embodiment.

[0039] FIG. 9 is a drawing for explaining an operation of determining a margin included in a generation area of ​​an object image corresponding to a sketch input of an electronic device according to one embodiment.

[0040] FIG. 10 is a flowchart illustrating an operation of determining the shape of a generation area of ​​an object image corresponding to a sketch input of an electronic device according to one embodiment.

[0041] FIG. 11 is a flowchart illustrating the shape of a generation area of ​​an object image corresponding to a sketch input of an electronic device according to one embodiment.

[0042] FIG. 12A is a drawing for explaining an operation of displaying a drawing image corresponding to a sketch input on an image including a face of a person of an electronic device according to one embodiment.

[0043] FIG. 12b is a drawing for explaining whether a face area of ​​a person is deformed depending on whether the face area of ​​the person is included in an object image generation area according to one embodiment.

[0044] Figure 13 is a diagram for explaining the operation of an artificial intelligence model according to one embodiment.

[0045] FIG. 14 is a diagram for explaining a filtering operation performed on output data of an artificial intelligence model according to one embodiment.

[0046] FIG. 15 is a flowchart illustrating an operation of personalizing a sketch input of an electronic device according to one embodiment.

[0047] FIG. 16 is a diagram illustrating an operation for reducing latency when displaying an edited image each including a plurality of generated object images of an electronic device according to an embodiment.

[0048] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100), according to one embodiment. Referring to FIG. 1 , in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0049] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0050] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0051] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0052] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0053] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0054] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0055] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0056] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0057] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0058] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0059] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0060] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0061] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0062] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0063] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0064] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0065] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0066] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0067] In one embodiment, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0068] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0069] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0070] FIG. 2 is a flowchart illustrating an operation of providing an image generated based on a sketch input of an electronic device according to one embodiment.

[0071] Referring to FIG. 2, an electronic device (e.g., an electronic device (101) of FIG. 1 or a processor (120) of FIG. 1) may display an image on a display (e.g., a display module (160) of FIG. 1) in operation 210.

[0072] According to one embodiment, the electronic device may display an image selected by a user input on a display. According to one embodiment, the selected image may be a background image from which an object image is generated. According to one embodiment, the object image may include an image generated through an artificial intelligence (AI) model based on a sketch input.

[0073] In one embodiment, when an object image is generated without a background image, the background image may be displayed as a white screen or operation 210 may be omitted.

[0074] In one embodiment, in operation 220, the electronic device may display on the image a drawing image corresponding to the sketch input based on receiving a sketch input on the image. In one embodiment, the electronic device may receive the sketch input through a touch and / or drag using a user's finger or an electronic pen on a display that is a touch screen. In one embodiment, the electronic device may receive information corresponding to the sketch input through an external device, such as a tablet.

[0075] According to one embodiment, the electronic device may display a drawing image corresponding to the touched trajectory or received information.

[0076] In one embodiment, in operation 230, the electronic device may provide an edited image including the object image by generating an object image based on a sketch input.

[0077] According to one embodiment, the object image may be obtained as output data from the second artificial intelligence model, with a drawing image corresponding to a sketch input as input data of the first artificial intelligence model, and a prompt obtained as output data from the first artificial intelligence model as input data of the second artificial intelligence model.

[0078] In one embodiment, when generating a prompt using the first artificial intelligence model, an optimal prompt can be generated by using additional data including information corresponding to a background image (e.g., a result of analyzing the background image, at least a portion of the background image) or a result of analyzing the user's previously created images, in addition to the drawing image corresponding to the sketch input, as input data for the first artificial intelligence model together with the drawing image.

[0079] In one embodiment, when inputting a prompt to the second artificial intelligence model, additional data including information corresponding to the background image (e.g., a result of analyzing the background image, at least a portion of the background image) or a result of analyzing the user's previously created images may be used as input data for the second artificial intelligence model along with the prompt.

[0080] In one embodiment, the first AI model may be a sketch-to-text model that takes a sketch as input data and obtains text as output data. In one embodiment, the first AI model may be trained to take a drawing image as input data and obtain a prompt for image creation as output data.

[0081] In one embodiment, the second AI model may be a text-to-image model that obtains an image as output data from text as input data. In one embodiment, the second AI model may be trained to obtain an object image as output data from a prompt as input data. In one embodiment, the prompt, which is input data of the second AI model, may be output data of the first AI model.

[0082] According to one embodiment, at least one of the first artificial intelligence model and the second artificial intelligence model may be stored in the memory of the electronic device (e.g., memory (130) of FIG. 1 ). Accordingly, even if the electronic device is not connected to an external electronic device, the first artificial intelligence model and the second artificial intelligence model can be used to generate an object image based on a sketch input.

[0083] According to one embodiment, at least one of the first artificial intelligence model or the second artificial intelligence model may be stored in an external server. According to one embodiment, the electronic device may transmit a drawing image to a first external server in which the first artificial intelligence model is stored via a communication module (or communication circuit) (e.g., the communication module (190) of FIG. 1). According to one embodiment, the electronic device may receive an object image corresponding to the drawing image from a second external server in which the second artificial intelligence model is stored via the communication module. According to one embodiment, the first external server or the second external server may be the same server or different servers. According to one embodiment, when the first external server and the second external server are different servers, the first external server may transmit a prompt, which is output data of the first artificial intelligence model, to the second external server, and the second external server may use the prompt received from the first external server as input data of the second artificial intelligence model.

[0084] In one embodiment, either the first artificial intelligence model or the second artificial intelligence model may be stored in the memory of the electronic device. In one embodiment, when the first artificial intelligence model is stored in the memory of the electronic device and the second artificial intelligence model is stored in an external server, the electronic device may transmit a prompt, which is output data of the first artificial intelligence model, to the external server to be used as input data of the second artificial intelligence model, and receive an object image, which is output data of the second artificial intelligence model, from the external server.

[0085] In one embodiment, when a first artificial intelligence model is stored in an external server and a second artificial intelligence model is stored in an electronic device, the electronic device may receive a prompt, which is output data of the first artificial intelligence model, from the external server, and use the received prompt as input data of the second artificial intelligence model to obtain an object image as output data of the second artificial intelligence model.

[0086] In one embodiment, at least one of the first AI model or the second AI model may be variably applied, depending on the user's settings (e.g., selection based on privacy), by selecting an AI model stored on an electronic device for privacy protection purposes (e.g., an on-device AI model) or an AI model stored on an external server for performance purposes.

[0087] According to one embodiment, the electronic device may crop an area related to a sketch input including a drawing image displayed on a background image. According to one embodiment, the area related to the sketch input is used as input data of a first artificial intelligence model and may include an area where an object image is generated. According to one embodiment, the area where the object image is generated may be referred to as a 'mask'. According to one embodiment, when the mask is in the shape of a rectangle, a polygon, or a closed curve, the area related to the sketch input to be cropped may be identical to or include the mask.

[0088] According to one embodiment, the electronic device may provide an edited image by editing an image based on a cropped area where an object image is generated. For example, the edited image may be provided by generating an object image in a mask included in the cropped area and merging the cropped area including the generated object image with a background image.

[0089] In one embodiment, the electronic device may crop a region of the image associated with the sketch input to include an identified margin (or margins) based on at least one of an aspect ratio of the drawing image or a length ratio of one side of the drawing image to the image.

[0090] According to one embodiment, the electronic device can check a margin having a length corresponding to a set number of pixels from each side of the drawing image based on a length ratio of one side of the drawing image to the image being less than a set first value. For example, if the length ratio of one side of the drawing image to the image is less than 10%, the electronic device can check a margin having a length corresponding to a set number of pixels from each side of the drawing image. Accordingly, when the size of the drawing image is small, the margins on the top, bottom, left, and right sides can have the same length regardless of the size of the drawing image. Since the image generation rate tends to decrease when the size of the area related to the sketch input is small, the image generation rate can be improved by checking the area related to the sketch input to be greater than a certain size in this way.

[0091] According to one embodiment, the electronic device can determine a margin having a length corresponding to a set ratio of the product of the horizontal length and the vertical length of the drawing image based on the fact that the length ratio of one side of the drawing image to the image is greater than or equal to a set first value and the aspect ratio of the drawing image is greater than or equal to a set second value. For example, if the length ratio of one side of the drawing image to the image is greater than or equal to 10% and the aspect ratio of the drawing image is greater than or equal to 1:2, the electronic device can determine 30% of the square root of the area of ​​the drawing image as the margin. According to one embodiment, the aspect ratio of the drawing image being greater than or equal to 1:2 means that the difference between the horizontal length and the vertical length of the drawing image is large, and the horizontal length:vertical length may be greater than or equal to 1:2, or the vertical length:horizontal length may be greater than or equal to 1:2. Accordingly, when the difference between the horizontal length and the vertical length of the drawing image is large, the margins on the top, bottom, left, and right sides of the drawing image may have the same length, and the length of the margins may vary depending on the horizontal length and the vertical length of the drawing image.

[0092] According to one embodiment, the electronic device can determine a horizontal margin corresponding to the set ratio of the horizontal length of the drawing image and a vertical margin corresponding to the set ratio of the vertical length of the drawing image based on a length ratio of one side of the drawing image to the image being greater than or equal to a set first value and an aspect ratio of the drawing image being less than or equal to a set second value. For example, if the length ratio of one side of the drawing image to the image is greater than or equal to 10% and the aspect ratio of the drawing image is less than or equal to 1:2, the electronic device can determine the left and right margins as 40% of the horizontal length of the drawing image and the top and bottom margins as 40% of the vertical length of the drawing image. Accordingly, if the difference between the horizontal length and the vertical length of the drawing image is not large, different left and right margins and top and bottom margins can be determined based on the length of the drawing image.

[0093] According to one embodiment, the operation of identifying an area related to a sketch input including a margin around a drawing image based on the size of the drawing image is described in more detail with reference to FIGS. 9 and 10 below.

[0094] According to one embodiment, the above description assumes that the area related to the sketch input to be cropped is a rectangle, but the area in which the object image is generated included in the area related to the sketch input may be a polygon or a closed curve rather than a rectangle.

[0095] According to one embodiment, the electronic device can determine the shape of the generation area of ​​the object image as a rectangle based on whether the area ratio of the drawing image to the image is less than a third value, or whether the area ratio of the drawing image to the image is greater than or equal to the third value and the ratio of empty space around the outermost trajectory included in the drawing image is less than a fourth value. For example, if the area of ​​the drawing image is less than 50% of the background image, the electronic device can determine the shape of the generation area of ​​the object image as a rectangle. According to one embodiment, if the area of ​​the drawing image is greater than or equal to 50% of the background image and the ratio of the margin included in the drawing image is less than a set value, the electronic device can determine the shape of the generation area of ​​the object image as a rectangle. If the margin outside the outermost trajectory included in the drawing image is greater than or equal to the set value, there is a high possibility that the background image of the margin within the drawing image will be deformed due to the generation of the object image. Therefore, if the area of ​​the drawing image is greater than or equal to 50% of the background image, the electronic device can determine the shape of the generation area of ​​the object image as a rectangle only when the ratio of the margin included in the drawing image is less than the set value.

[0096] According to one embodiment, the electronic device may determine the shape of the generation area of ​​the object image by further considering the complexity of the background image of the margin within the drawing image. For example, if the complexity of the background image of the margin within the drawing image (e.g., the shape of the border, the number of borders, the number of colors) is low, the influence is not great even if it is deformed. Therefore, the electronic device may determine the shape of the generation area of ​​the object image as a rectangle if the complexity of the background image of the margin is low even if the area of ​​the drawing image is 50% or more of the background image and the ratio of the margin included in the drawing image is a set value or more.

[0097] In one embodiment, when the generation area (or mask) of the object image is rectangular, the generation area of ​​the object image may be the same as the area associated with the sketch input being cropped.

[0098] According to one embodiment, the electronic device can determine the shape of the generation area of ​​the object image as a polygon or a closed curve based on the outermost trajectory included in the drawing image, based on a ratio of the area of ​​the drawing image to the image being a third value or greater and a ratio of empty space around the outermost trajectory included in the drawing image being a fourth value or greater. According to one embodiment, the electronic device can determine the shape of the generation area of ​​the object image as a polygon or a closed curve based on the outermost trajectory included in the drawing image, if the area of ​​the drawing image is 50% or greater of the background image and the ratio of empty space included in the drawing image is a set value or greater.

[0099] According to one embodiment, the operation of determining the shape of the generation area of ​​an object image will be described in more detail with reference to FIGS. 10 and 11 below.

[0100] According to one embodiment, the electronic device can determine the shape of the generated area of ​​the object image as a polygon excluding the face area based on whether the image includes a face area. For example, if the background image includes a face area and a sketch input is received around the face area, the electronic device can generate a mask excluding the face area since the face area is likely to be deformed when generating a background image corresponding to the sketch input.

[0101] According to one embodiment, the operation of generating a mask excluding the face area included in the background image will be described in more detail with reference to FIGS. 11A and 11B below.

[0102] According to one embodiment, the electronic device may use a template for at least one of the number of words to be included in a prompt, which is output data of the first artificial intelligence model, or an attribute of the words, as input data of the first artificial intelligence model. According to one embodiment, if too much text is used as input data of the second artificial intelligence model, the image generation efficiency may rather decrease, so the electronic device may further use a template that allows a set number of words and words with a set attribute to be obtained as output data of the first artificial intelligence model, which is used as input data of the second artificial intelligence model, as a template of the first artificial intelligence model. According to one embodiment, the template used as input data of the first artificial intelligence model will be described in more detail below with reference to FIG. 13.

[0103] According to one embodiment, the electronic device may filter the prompt, which is the output data of the first artificial intelligence model, to exclude preset words including a brand name or logo. According to one embodiment, in order to prevent legal issues such as copyright issues of object images acquired from the second artificial intelligence model and / or to prevent unintended style images from being output from the second artificial intelligence model, the electronic device may perform a filtering operation to exclude preset words including a brand name or logo from the prompt, which is the output data of the first artificial intelligence model. According to one embodiment, the filtering operation of the electronic device will be described in more detail below with reference to FIG. 14.

[0104] According to one embodiment, when the ratio of the area of ​​the sketch input area to the background image exceeds a set value (e.g., 25% or 33%), the electronic device may display an indicator (e.g., a message) that guides to indicate an area (or mask) where an object image is to be generated and to limit the size of the area.

[0105] According to one embodiment, the electronic device displays an area in which a drawing image and an object image corresponding to a sketch input are to be generated, and can move, resize, and / or scale the area in which the drawing image and the object image are to be generated through a user input.

[0106] In this way, if the ratio of the area of ​​the sketch input area to the background image is too large, damage to the background image that may occur due to the generation of an object image corresponding to the sketch input can be prevented.

[0107] In one embodiment, the electronic device may provide a user with a prompt, which is output data of a first artificial intelligence model, and modify the prompt through user input. In one embodiment, the electronic device may use (or transmit) the modified prompt as input data of a second artificial intelligence model.

[0108] In one embodiment, the electronic device may use a template reflecting the type of pen and / or information on the type of pen as input data of the first artificial intelligence model to generate different object images depending on the type of pen used when inputting a sketch (e.g., pencil, colored pencil, brush, pen, highlighter).

[0109] According to one embodiment, the electronic device may provide an edited image by overlaying the generated object image with a background image in a different layer, thereby enabling editing of the object image, such as moving, rotating, and resizing, through user input.

[0110] According to one embodiment, the electronic device can perform personalization of object image generation based on sketch input using the received sketch history, the prompt history used for object image generation, the prompt feedback history, and / or the object image feedback history for object image generation.

[0111] In one embodiment, all operations for image generation based on sketch input are described as being performed on an electronic device in FIG. 2, but this is not limited thereto, and some operations may be performed on at least one server. An embodiment in which some operations for image generation based on sketch input are performed on at least one server will be described in more detail below with reference to FIGS. 3 and 4.

[0112] FIG. 3 is a drawing for explaining an operation of providing an image generated based on a sketch input of an electronic device according to one embodiment.

[0113] Referring to FIG. 3, LVM management (large vision model management) (1000) for providing an image generated based on a sketch input can be performed through an electronic device (101) (e.g., the electronic device (101) of FIG. 1 or the processor (120) of FIG. 1), an external server (108-1) (e.g., the server (108) of FIG. 1), and an AI server (108-2).

[0114] In one embodiment, LVM management may be implemented on a per-application basis (e.g., single application) or as a service of an electronic device and applied to multiple applications (e.g., multi-application).

[0115] According to one embodiment, the electronic device (101) may display a background image (310) and receive a sketch input on the background image (310). According to one embodiment, the electronic device (101) may display a drawing image (311) corresponding to the received sketch input on the background image (310).

[0116] According to one embodiment, the electronic device (101) may obtain input data (320, 321, 322, 323) of the first artificial intelligence model (340) based on the background image (310) and / or the drawing image (311). According to one embodiment, the electronic device may obtain at least one of a drawing image without a background (320), a drawing image on a background image (321), a mask (322) or a background image (323) as input data. According to one embodiment, the drawing image without a background (320), the drawing image on a background image (321), the mask (322) or the background image (323) as an area where an object image is to be generated, and the background image (323) may be cropped with respect to the sketch input. According to one embodiment, when the input data (320, 321, 322, 323) are cropped images, the generation efficiency and quality of the artificial intelligence model can be improved, and network resources can be reduced.

[0117] According to one embodiment, the electronic device (101) may transmit the acquired input data (320, 321, 322, 323) to an external server (108-1). According to one embodiment, the external server (108-1) may be an intelligent proxy server (IPS). According to one embodiment, the external server (108-1) may transmit a sketch-based text output request (request sketch to text) (330) including the input data (320, 321, 322, 323) received from the electronic device (101) to the AI ​​server (108-2) via a communication module to be used as input data of the first artificial intelligence model (340). According to one embodiment, the sketch-based text output request (330) may further include a template for at least one of the number of words to be included in the prompt, which is the output data of the first artificial intelligence model, or an attribute of the words.

[0118] According to one embodiment, an external server (108-1) may transmit information about a plurality of recommended templates to an electronic device (101) through a communication module, and receive information about one template selected from among the plurality of recommended templates from the electronic device (101) through the communication module. According to one embodiment, the external server (108-1) may transmit a template corresponding to information about the selected template to an AI server (108-2).

[0119] According to one embodiment, the external server (108-1) may perform a sketch-based text output request (330) after processing an error that occurs when generating text for prompt generation or after generating the prompt (e.g., when a drawing image is identified as a wrong area, when a drawing image is cropped incorrectly, or when an error occurs in prompt generation).

[0120] According to one embodiment, the sketch-based text output request (330) may further include a prompt describing at least one of the input data (320, 321, 322, 323).

[0121] According to one embodiment, the template used as input data of the first artificial intelligence model will be described in more detail with reference to FIG. 13 below.

[0122] According to one embodiment, the AI ​​server (108-2) may include a first external server storing a first artificial intelligence model (340) and a second external server storing a second artificial intelligence model (341). According to one embodiment, the first external server and the second external server may be the same server or different servers.

[0123] In one embodiment, the first AI model may be a sketch-to-text model that uses a sketch as input data and obtains text as output data. In one embodiment, the second AI model may be a text-to-image model that uses text as input data and obtains an image as output data.

[0124] According to one embodiment, an external server (108-1) may receive a prompt, which is output data of a first artificial intelligence model (340), from an AI server (108-2).

[0125] According to one embodiment, the external server (108-1) may obtain a filtered prompt configuration (config) (332) by filtering (refine description prompt) (331) to exclude preset words including brand names or logos from the received prompt.

[0126] According to one embodiment, the external server (108-1) may transmit an image output request (333) including a prompt configuration (332) to the AI ​​server (108-2) via a communication module to be used as input data of the second artificial intelligence model (341). According to one embodiment, at least one of a mask (322) or a background image (323), which is an area where an object image is to be generated, may further be used as input data of the second artificial intelligence model (341).

[0127] According to one embodiment, the external server (108-1) may receive an image (334), which is output data of the second artificial intelligence model (341), from the AI ​​server (108-2). According to one embodiment, the image (334) may include an object image generated based on the sketch input (311). According to one embodiment, the image (334) may include an object image generated on a background image.

[0128] In one embodiment, an external server (108-1) may transmit an image (334) to an electronic device (101).

[0129] According to one embodiment, the electronic device (101) may edit the background image (310) by merging the received image (334) into the background image (310) and provide the edited image (350). According to one embodiment, the edited image (350) by merging the received image (334) into the background image (310) may be one layer.

[0130] According to one embodiment, the electronic device (101) may edit the background image (310) by overlapping the received image (334) onto the background image (310) and provide the edited image (350). According to one embodiment, the edited image (350) by overlapping the received image (334) onto the background image (310) may be a plurality of layers in which the background image (310) and the image (334) are configured as separate layers.

[0131] FIG. 4 is a drawing for explaining an operation of providing an image generated based on a sketch input of an electronic device according to one embodiment.

[0132] Referring to FIG. 4, LVM management (large vision model management) (1001) for providing an image generated based on a sketch input can be performed through an electronic device (101) (e.g., the electronic device (101) of FIG. 1 or the processor (120) of FIG. 1) and an AI server (108-2) (e.g., the server (108) of FIG. 1).

[0133] According to an embodiment, the electronic device (101) of FIG. 4 may perform the operations of the external server (108-1) of FIG. 3. For example, the electronic device (101) may perform the operations of transmitting a sketch-based text output request to the AI ​​server (108-2), filtering a prompt received from the AI ​​server (108-2), transmitting an image output request including the filtered prompt to the AI ​​server (108-2), and receiving an image generated from the AI ​​server (108-2). According to an embodiment, the AI ​​server (108-2) may include a first external server storing a first artificial intelligence model and a second external server storing a second artificial intelligence model. According to an embodiment, the first external server and the second external server may be the same server or different servers.

[0134] FIG. 5 is a drawing for explaining an operation of providing an image generated based on a sketch input of an electronic device according to one embodiment.

[0135] Referring to FIG. 5, large vision model management (LVM) management (1002) for providing an image generated based on a sketch input may be performed through an electronic device (101) (e.g., the electronic device (101) of FIG. 1 or the processor (120) of FIG. 1). According to an embodiment, the electronic device (101) of FIG. 5 may perform the operation of the AI ​​server (108-2) of FIG. 4. According to an embodiment, a memory of the electronic device (e.g., the memory (130) of FIG. 1) may store a first artificial intelligence model, which is a sketch-to-text model that obtains text as output data using a sketch as input data, and a second artificial intelligence model, which is a text-to-image model that obtains an image as output data using text as input data.

[0136] According to one embodiment, the electronic device may obtain a prompt corresponding to a sketch input using a stored first artificial intelligence model, input the prompt obtained from the first artificial intelligence model into a stored second artificial intelligence model, and obtain an image generated as output data of the second artificial intelligence model.

[0137] According to one embodiment, although FIG. 5 illustrates that both the first artificial intelligence model and the second artificial intelligence model are stored in the electronic device (101), this is not limited thereto, and the first artificial intelligence model or the second artificial intelligence model may be stored in the electronic device (101), and the other artificial intelligence model may be stored in an external server. For example, the first artificial intelligence model may be stored in the electronic device (101), and the second artificial intelligence model may be stored in an AI server, or the first artificial intelligence model may be stored in an AI server, and the second artificial intelligence model may be stored in the electronic device (101).

[0138] FIG. 6 is a drawing for explaining a crop operation based on a sketch input of an electronic device according to one embodiment.

[0139] Referring to FIG. 6, an electronic device (e.g., an electronic device (101) of FIG. 1 or a processor (120) of FIG. 1) may obtain at least one of a background image (610) including an area (611) related to a sketch input as input data of a first artificial intelligence model (e.g., a sketch to text model), a background image (620) including a drawing image (621) corresponding to the sketch input, and a mask (640) corresponding to the entire background image including a drawing image (630) or a drawing image area (641) and an object image generation area (642) including a margin.

[0140] According to one embodiment, when the electronic device uses input data related to a background image (610), it can obtain a background image (650) including an object image (651) generated based on a sketch input as output data of a second artificial intelligence model (e.g., a text to image model). According to one embodiment, the background image (650) obtained as output data can be at least partially identical to the background image (610) used as input data.

[0141] According to one embodiment, the electronic device may obtain, as input data, an image cropped from an area (611) related to a sketch input included in a background image (610).

[0142] According to one embodiment, the electronic device may obtain, as input data of a first artificial intelligence model (e.g., a sketch to text model), at least one of a cropped background image (612) including an area (611) related to a sketch input, a cropped background image (623) including a drawing image (621) corresponding to the sketch input, and a mask (643) corresponding to a cropped background image including a drawing image (630) or a drawing image area (641) and an object image generation area (642) including a margin.

[0143] In one embodiment, when the electronic device uses input data related to a cropped background image (612), the electronic device may obtain a cropped background image (660) including an object image (661) generated based on a sketch input as output data of a second artificial intelligence model (e.g., a text to image model). In one embodiment, the background image (660) obtained as output data may be at least partially identical to the cropped background image (612) used as input data. In one embodiment, the background image (660) obtained as output data may be identical to the size of the cropped background image (612) used as input data.

[0144] By cropping the area related to the sketch input in this way and using it as input data, the performance of the AI ​​model (e.g., generation rate and quality) is improved as the image area to be processed is reduced, and network resources (e.g., time and memory) are reduced, which can reduce traffic degradation.

[0145] According to one embodiment, the operation of obtaining input data (610, 612, 620, 623, 630, 640, 643) will be described in more detail with reference to FIG. 7 below.

[0146] FIG. 7 is a flowchart illustrating a cropping operation based on a sketch input of an electronic device according to one embodiment.

[0147] Referring to FIG. 7, an electronic device (e.g., the electronic device (101) of FIG. 1 or the processor (120) of FIG. 1) may receive a user sketch input (710). According to one embodiment, the electronic device may receive the sketch input through a touch and / or drag using a user's finger or an electronic pen on a display (e.g., the display module (160) of FIG. 1) that is a touch screen. According to one embodiment, the electronic device may receive information corresponding to the sketch input through an external device, such as a tablet.

[0148] According to one embodiment, the electronic device can recognize (720) a user sketch area. For example, the electronic device can recognize a drawing image area including a drawing image corresponding to a sketch input. According to one embodiment, the electronic device can recognize a drawing image area of ​​a rectangle that touches the outermost trajectory of the drawing image.

[0149] According to one embodiment, the electronic device may specify (730) a crop area (or sketch input related area or object image generation area) including a drawing image area and a margin (or area) surrounding the drawing image area. According to one embodiment, the margin may be determined based on at least one of an aspect ratio of the drawing image or a length ratio of one side of the drawing image to the image.

[0150] According to one embodiment, the electronic device may determine at least one of the size, shape, or position of the drawing image area based on the sketch input method, and may determine the size and shape of the margin area for the drawing image area differently. According to one embodiment, the electronic device may also determine the weight of the margin area differently based on the similarity and confidence level between sketches by using a separate engine for managing sketches (e.g., a sketch management engine (not shown)).

[0151] According to one embodiment, the electronic device may generate a mask (740) based on a crop area. For example, a mask (640) corresponding to the entire background image including the crop area including the drawing image area and the margin, or a mask (643) corresponding to the cropped background image including the drawing image area and the margin, may be obtained.

[0152] According to one embodiment, the electronic device may obtain a drawing image (630) including a drawing image area and a margin (or space) around the drawing image area.

[0153] According to one embodiment, the electronic device may obtain (741) input data including a sketch input and a background image based on a crop area. For example, the electronic device may obtain a background image (620) including a drawing image (621) corresponding to the sketch input or a cropped background image (623) including a drawing image (621) corresponding to the sketch input based on the crop area.

[0154] According to one embodiment, the electronic device may acquire (742) as input data a background image corresponding to the drawing image area and margin based on the crop area. For example, the electronic device may acquire as input data a background image (610) including an area related to the sketch input or a background image (612) in which the area related to the sketch input is cropped.

[0155] In one embodiment, when applying margins, a uniform margin can be applied to prevent problems that may occur when drawing images with extreme aspect ratios are used, thereby improving the generation rate and quality of object images.

[0156] According to one embodiment, a margin ratio applied to a drawing image will be described in detail with reference to FIG. 8 below. FIG. 8 is a flowchart for explaining an operation of determining a margin included in a generation area of ​​an object image corresponding to a sketch input of an electronic device according to one embodiment.

[0157] FIG. 9 is a drawing for explaining an operation of determining a margin included in a generation area of ​​an object image corresponding to a sketch input of an electronic device according to one embodiment.

[0158] Referring to FIG. 8, an electronic device (e.g., the electronic device (101) of FIG. 1 or the processor (120) of FIG. 1) may determine, as part of operation 730 of FIG. 7, which determines a margin to designate a crop area, at operation 810, determine whether a ratio of the length of one side of the drawing image to the background image is greater than or equal to a set value. For example, the electronic device may determine whether a ratio of the length of one side of the drawing image to the background image is greater than or equal to 10%.

[0159] According to one embodiment, if the ratio of the length of one side of the drawing image to the background image is less than 10% (operation 810 - No), in operation 820, the electronic device may add a fixed number of pixels as a margin to each side. For example, the electronic device may check a margin having a length corresponding to a set number of pixels from each side of the drawing image. For example, as illustrated in FIG. 9, the electronic device may check a crop area (920) having a length corresponding to a fixed number of pixels as a margin (911) of each side of the drawing image (910). Accordingly, when the size of the drawing image is small, the margins on the top, bottom, left, and right may have the same length regardless of the size of the drawing image. In this way, the image generation rate can be improved by checking an area related to sketch input to be greater than a certain size.

[0160] In one embodiment, if the ratio of the length of one side of the drawing image to the background image is 10% or more (operation 810 - Yes), in operation 830, the electronic device may determine whether the aspect ratio of the drawing image is greater than or equal to a set value. For example, the electronic device may determine whether the ratio of the length to the width of the drawing image or the ratio of the length to the width of the drawing image is 1:2 or more.

[0161] According to one embodiment, if the aspect ratio of the drawing image is equal to or greater than a set value (operation 830 - Yes), in operation 840, the electronic device may add the set ratio of the area of ​​the drawing image. For example, if the aspect ratio of the drawing image is equal to or greater than 1:2, the electronic device may confirm 30% of the square root of the area of ​​the drawing image as a margin. For example, as illustrated in FIG. 9, the electronic device may confirm a crop area (920) having a length corresponding to 30% of the square root of the area of ​​the drawing image (910) as a margin (914) of each side of the drawing image (910). Accordingly, if the difference between the horizontal and vertical lengths of the drawing image is large, the margins on the top, bottom, left, and right sides of the drawing image may have the same length, and the length of the margins may vary depending on the horizontal and vertical lengths of the drawing image. In this way, when the difference between the horizontal and vertical lengths of a drawing image is large, the deterioration in image creation efficiency and quality can be reduced and deformation of the margins can be reduced by preventing the margins in the long direction from being longer than those in the short direction.

[0162] According to one embodiment, if the aspect ratio of the drawing image is less than the set value (operation 830 - No), in operation 850, the electronic device may add the set ratio of the horizontal length and the vertical length of the drawing image. For example, if the ratio of the length of one side of the drawing image to the image is 10% or more and the aspect ratio of the drawing image is 1:2 or less, the electronic device may determine the left and right margins as 40% of the horizontal length of the drawing image, and the top and bottom margins as 40% of the vertical length of the drawing image. For example, as illustrated in FIG. 9, the electronic device may determine a crop area (921) having a length corresponding to 40% of the horizontal length of the drawing image (910) as the left and right margins (912) of the drawing image (910), and having a length corresponding to 40% of the vertical length of the drawing image (910) as the top and bottom margins (913). Accordingly, if the difference between the horizontal and vertical lengths of the drawing image is not significant, different left and right margins and top and bottom margins can be identified based on the length of the drawing image. In this way, if the difference between the horizontal and vertical lengths of the drawing image is not significant, determining the margins based on the length can improve the efficiency and quality of image generation by including more information necessary for image generation in the direction of the longer length.

[0163] FIG. 10 is a flowchart illustrating an operation of determining the shape of a generation area of ​​an object image corresponding to a sketch input of an electronic device according to one embodiment.

[0164] FIG. 11 is a flowchart illustrating the shape of a generation area of ​​an object image corresponding to a sketch input of an electronic device according to one embodiment.

[0165] Referring to FIG. 10, in operation 1010, an electronic device (e.g., the electronic device (101) of FIG. 1 or the processor (120) of FIG. 1) may determine whether the area of ​​the drawing image relative to the background image is greater than or equal to a first set value. For example, the electronic device may determine whether the area of ​​the drawing image is greater than or equal to 50% of the background image.

[0166] According to one embodiment, if the area of ​​the drawing image for the background image is equal to or greater than the first set value (operation 1010 - Yes), in operation 1020, the electronic device may determine a polygon or a closed curve based on the outermost trajectory of the drawing image as the shape of the object image generation area. For example, if the area of ​​the drawing image is equal to or greater than 50% of the background image, the electronic device may determine the shape of the object image generation area as a polygon (1120) or a closed curve (1130) based on the outermost trajectory included in the drawing image, as illustrated in FIG. 11. In this case, when the area of ​​the drawing image is large, if the shape of the object image generation area is determined as a rectangle, there is a high possibility that a margin other than the drawing image will be included, and in the case of the margin, there is a high possibility that it will be deformed when generating the object image. Therefore, by determining the shape of the object image generation area as a polygon or a closed curve based on the outermost trajectory of the drawing image, the efficiency and quality of object image generation can be increased, and deformation of the background image can be reduced.

[0167] According to one embodiment, if the area of ​​the drawing image for the background image is less than the first set value (operation 1010 - No), in operation 1030, the electronic device may determine a rectangle as the shape of the object image generation area. For example, if the area of ​​the drawing image is less than 50% of the background image, the electronic device may determine the shape of the object image generation area as a rectangle (1110), as illustrated in FIG. 11. In this way, when the size of the drawing image is small, if the object image generation area has a complex shape such as a polygon or a closed curve, the efficiency and quality of object image generation are lowered, and therefore, by determining the object image generation area as a rectangle, the efficiency and quality of object image generation can be increased.

[0168] According to one embodiment, the shape of the generation area of ​​the object image is determined based on the area of ​​the drawing image as described above, but the present invention is not limited thereto, and the electronic device may determine the shape of the generation area of ​​the object image by further considering the ratio of the margin included in the drawing image and / or the complexity of the background image around the drawing image. For example, even if the area of ​​the drawing image is 50% or more of the background image, if the ratio of the margin included in the drawing image is less than a set value, the possibility of deformation of the margin is low, and if the complexity of the background image of the margin within the drawing image (e.g., the shape of the border, the number of borders, the number of colors) is low, even if the margin is deformed, the influence is not great, and therefore the electronic device may confirm the shape of the generation area of ​​the object image as a rectangle.

[0169] FIG. 12A is a drawing for explaining an operation of displaying a drawing image corresponding to a sketch input on an image including a face of a person of an electronic device according to one embodiment.

[0170] Referring to FIG. 12A, an electronic device (e.g., the electronic device (101) of FIG. 1 or the processor (120) of FIG. 1) may display a background image (1210) including a face of a person. According to one embodiment, when a sketch input is received around a face area, the electronic device may display a drawing image (1211) corresponding to the sketch input around the face area.

[0171] According to one embodiment, the electronic device can identify an object image generation area (or mask) including a margin around a drawing image (1211) and generate an object image corresponding to the drawing image (1211).

[0172] According to one embodiment, the electronic device may obtain different output images depending on whether the object image generation area includes a face area, as will be described in more detail below with reference to FIG. 12b.

[0173] FIG. 12b is a drawing for explaining whether a face area of ​​a person is deformed depending on whether the face area of ​​the person is included in an object image generation area according to one embodiment.

[0174] Referring to FIG. 12B, when an electronic device (e.g., the electronic device (101) of FIG. 1 or the processor (120) of FIG. 1) identifies an object image generation area (1220) including a face area, it may obtain an edited image (1230) including a generated object image (1231). According to an embodiment, a face area (1232) surrounding the object image (1231) of the edited image (1230) may be deformed according to the generation of the object image (1231). According to an embodiment, when a bounding box (1221) of the object image generation area (1220) is set to a square, a face area (1232) included within the bounding box (1221) may be deformed when the object image (1231) is generated.

[0175] According to one embodiment, when the electronic device confirms the object image generation area (1240) excluding the face area (1242), the electronic device may obtain an edited image (1250) including the generated object image (1251). According to one embodiment, the face area (1252) surrounding the object image (1251) of the edited image (1530) may not be deformed even when the object image (1251) is generated. According to one embodiment, when an adaptive mask is applied to the object image generation area (1240), and the face area (1252) excluded from the bounding box (1241) of the object image generation area (1240) is excluded, when the object image (1251) is generated, the face area (1252) excluded from the bounding box (1242) may not be deformed.

[0176] According to one embodiment, the electronic device may identify a face area (1242) included in a background image (e.g., background image (1210) of FIG. 12A) through facial recognition technology, or identify an area selected by a user input as a face area (1242), and confirm an object image generation area (1240) excluding the identified face area (1242).

[0177] By performing face filtering to exclude faces from the object image generation area, we can prevent problems such as incorrectly generated faces in the text-to-image model, such as altering the faces of users and acquaintances, which can hinder the user experience, and prevent exploitation by altering the faces of celebrities. Furthermore, if the text-to-image model is stored on an external server, the face area can be excluded from the object image generation area before transmitting the input data to the external server.

[0178] In one embodiment, when a deformation of the face area is required, such as when the user intends to decorate the face area by adding various accessories to the face area, the electronic device can adaptively determine a mask that includes the face area.

[0179] In one embodiment, a segmentation model capable of distinguishing objects as well as a face region of a person may be utilized to identify a mask to exclude a region of a specific object, such as a specific animal. For example, the electronic device may display a plurality of object regions included in a background image through the segmentation model, and identify a mask including or excluding an object region selected by the user among the displayed object regions. In one embodiment, the electronic device may select a region to be included or excluded from the background image by learning a history of mask generation including or excluding a specific object, and identify a mask including or excluding the selected region.

[0180] FIG. 13 is a diagram for explaining the operation of an artificial intelligence model (sketch to text model) according to one embodiment.

[0181] Referring to FIG. 13, the first artificial intelligence model (1320) (e.g., sketch to text model) can use a drawing image (1310), a background image (1311), and template information (1312) as input data, and obtain a prompt (1330) as output data for the input data.

[0182] According to one embodiment, the drawing image (1310) and the background image (1311) may correspond to the entire area of ​​the background image (1311) or may correspond to an area cropped based on the drawing image (1310). According to one embodiment, the drawing image (1310) and the background image (1311) may each be used as input data, or may be used as a single data in which the drawing image (1310) is included in the background image (1311).

[0183] According to one embodiment, the template information (1312) may include information about at least one of the number of words to be included in the prompt (1330), which is output data of the first artificial intelligence model (1320), or an attribute of the words.

[0184] In one embodiment, the input data may further include a prompt describing at least one of a drawing image (1310) and a background image (1311) along with template information (1312).

[0185] For example, a prompt describing at least one of template information (1312) and drawing image (1310) and background image (1311) may be as shown in [Table 1] below.

[0186]

[0187] Referring to [Table 1], prompt 1 can explain that the sketch is drawn on the second image, prompt 2 can describe the second image by focusing on the sketch in the first image, if the second image contains multiple objects, can explain the relative positions of each object, and if the objects represented by the sketch have different colors, can explain the colors of the sketch.

[0188] According to one embodiment, the template information (1312) may include information on the number of words in the prompt (1330), which is output data, and information on the attributes of the words. For example, the template information (1312) may include information on the number of words in the prompt (1330), which is five, and information on the attributes of the words, which include the type of object (object), the direction of the object (landscape), the appearance, and the color of the object.

[0189] In one embodiment, the template information (1312) may further include a prompt to cause an error if the sketch contains handwritten text.

[0190] Text-to-image models can actually reduce image generation efficiency when large amounts of text are used as input data. Therefore, limiting the number of words and their attributes through templates can improve the performance and stability of text-to-image models. For example, a maximum number of tokens (e.g., 100) can be specified through templates to limit the number of words.

[0191] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1 or the processor (120) of FIG. 1) may provide a user with a plurality of recommended template information stored or received from an external server by displaying the information, and may select one template information through user input. According to one embodiment, the selected template information may be used as input data of a first artificial intelligence model (1320).

[0192] In one embodiment, the electronic device may also allow the user to edit template information through user input. For example, the user may input word attributes they wish to include in the prompt into the template, or delete word attributes they wish to exclude from the prompt. In this way, template usage history, such as user selection and modification, may enable the provision of customized templates.

[0193] FIG. 14 is a diagram for explaining a filtering operation performed on output data of an artificial intelligence model according to one embodiment.

[0194] Referring to FIG. 14, an electronic device (e.g., an electronic device (101) of FIG. 1 or a processor (120) of FIG. 1) or a server (e.g., a server (108) of FIG. 1 or an external server (108-1) of FIG. 3) may perform a filtering (refinement) operation (1420) to exclude specific words from the text included in a prompt (1410) obtained as output data of a second artificial intelligence model, thereby obtaining a filtered prompt (1430).

[0195] In one embodiment, the filtering operation (1420) may be configured to exclude certain words including logos (1421), brands (1422), and characters (1423).

[0196] In one embodiment, if it is desired to generate a realistic object image, words such as 'drawing' may be excluded from the prompt (1410) through a filtering operation (1420).

[0197] In one embodiment, the electronic device or server may use the filtered prompt (1430) as input data for the second artificial intelligence model.

[0198] This can prevent legal issues such as copyright issues with object images obtained from the second artificial intelligence model and / or prevent unintended style image output from the second artificial intelligence model.

[0199] FIG. 15 is a flowchart illustrating an operation of personalizing a sketch input of an electronic device according to one embodiment.

[0200] Referring to FIG. 15, an electronic device (e.g., the electronic device (101) of FIG. 1 or the processor (120) of FIG. 1) may receive (1510) a new sketch input and use it as input data (1520) of a first artificial intelligence model. In one embodiment, the electronic device may store the new sketch input in a sketch database (1530) so that it can be reused.

[0201] In one embodiment, the electronic device can use existing sketch inputs (1540) stored in a sketch database (1530) as input data (1520) of the first artificial intelligence model.

[0202] In one embodiment, the electronic device may use an edited sketch input, such as adding a sketch (1550) to an existing sketch input (1540), as input data (1520) of the first artificial intelligence model. In one embodiment, the electronic device may store the edited sketch input, which adds a sketch (1550) to an existing sketch input (1540), in a sketch database (1530) for reuse.

[0203] In one embodiment, the electronic device can model a user-specific sketch style, such as the sketching method, shape, pen type, and drawing style. For example, the electronic device can manage a user-specific sketch model by learning the sketch style and updating the sketch database (1530) whenever a sketch input is received. For example, if modeling is performed to learn the style of a dolphin sketch previously input by the user, the recognition rate of a newly input dolphin sketch can be improved.

[0204] In one embodiment, when dolphin sketch modeling is performed by learning dolphin sketches of other users, the recognition rate of a newly input dolphin sketch can be improved even if modeling is not performed for each user.

[0205] In one embodiment, the electronic device may use a template reflecting the type of pen and / or pen type information as input data for the first artificial intelligence model to generate different object images depending on the type of pen used when inputting a sketch (e.g., pencil, colored pencil, brush, pen, highlighter). In one embodiment, when inputting a sketch, the user may select the resulting style according to the type of pen, thereby utilizing the sketch to create or decorate a poster or invitation.

[0206] According to one embodiment, the electronic device may provide an edited image by overlaying the generated object image with a different layer from the background image, thereby enabling editing of the object image, such as moving, rotating, and resizing, through user input. According to one embodiment, the electronic device may generate the object image to be composited with the background image, or may generate the object image on a transparent background. According to one embodiment, when generating the object image on a transparent background, editing of the generated object image, such as moving, resizing, and / or rotating, may be performed.

[0207] In one embodiment, the electronic device can personalize object image generation based on sketch input by using the received sketch history, the prompt history used for object image generation, the prompt feedback history, and / or the object image feedback history for object image generation. For example, the electronic device can engineer a prompt output from a text analysis engine. In one embodiment, the electronic device can extract metadata of the prompt as feature information with several depths, such as categories, characteristics, and parameters, and map them to the sketch. In one embodiment, the electronic device can reflect user intent and personalize the engine. For example, the electronic device can improve personalization performance by reflecting user intent and / or preferences in the initial output data and improving it with additional feedback, or by learning usage history and / or preferences and utilizing them in a training database and as a parameter improvement guide.

[0208] In one embodiment, the engineering and / or user intent reflection of prompts output from the text analysis engine and engine personalization may be stored in the sketch storage engine or the prompt engineering engine, or may be personalized to enable performance updates. In one embodiment, personalized modeling updates are managed within the electronic device, enabling privacy-free operation. In one embodiment, anonymized (or grouped or clustered) modeling and learning history data may be distributed and processed on a server for continuous engine updates.

[0209] FIG. 16 is a diagram illustrating an operation for reducing latency when displaying an edited image each including a plurality of generated object images of an electronic device according to an embodiment.

[0210] Referring to FIG. 16, an electronic device (e.g., the electronic device (101) of FIG. 1 or the processor (120) of FIG. 1) may input a background image (1610) and a drawing image (1620) corresponding to a sketch input into a generative AI model (1630) to obtain the generated image as output data. For example, the generative AI model may include a sketch-to-text model and a text-to-image model.

[0211] According to one embodiment, when a plurality of images (1640, 1641, 1642, 1643) are acquired as output data, the electronic device can sequentially display the generated images among the plurality of images (1640, 1641, 1642, 1643) on a screen (1650) (e.g., the display module (160) of FIG. 1). For example, the electronic device may display the first image (1640) on the screen (1650) when the generation of the first image (1640) is completed, display the second image (1641) together with the previously displayed first image (1640) on the screen (1650) when the generation of the second image (1641) is completed, display the third image (1642) together with the previously displayed first image (1640) and second image (1641) on the screen (1650) when the generation of the third image (1642) is completed, and display the fourth image (1643) together with the previously displayed first image (1640), second image (1641) and third image (1642) on the screen (1650) when the generation of the fourth image (1643) is completed.

[0212] In this case, when it takes a long time to display multiple images (1640, 1641, 1642, 1643) after they are all generated, the usability degradation due to latency can be reduced by asynchronously outputting the multiple images (1640, 1641, 1642, 1643).

[0213] According to one embodiment, an electronic device may include at least one processor, a display, and a memory that stores instructions that, when individually or collectively executed by the at least one processor, cause the electronic device to perform the following operations:

[0214] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display an image on the display.

[0215] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to, based on receiving a sketch input on the image, display on the image a drawing image corresponding to the sketch input.

[0216] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to provide an edited image including the object image by generating an object image based on the sketch input.

[0217] According to one embodiment, the object image may be obtained as output data from the second artificial intelligence model, with the drawing image corresponding to the sketch input as input data of the first artificial intelligence model, and a prompt obtained as output data from the first artificial intelligence model as input data of the second artificial intelligence model.

[0218] According to one embodiment, the first artificial intelligence model may be trained to obtain a prompt for image generation as output data, using a drawing image as input data.

[0219] According to one embodiment, the second artificial intelligence model may be trained to obtain an object image as output data by using a prompt as input data.

[0220] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to crop a region of the image associated with the sketch input to include a margin identified based on at least one of an aspect ratio of the drawing image or a length ratio of one side of the drawing image to the image.

[0221] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to provide the edited image by editing the image based on the cropped region where the object image was generated.

[0222] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify a margin having a length corresponding to a set number of pixels from each side of the drawing image based on a length ratio of one side of the drawing image to the image being less than a set first value.

[0223] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify a margin having a length corresponding to a set ratio of a product of a horizontal length and a vertical length of the drawing image, based on a length ratio of one side of the drawing image to the image being greater than or equal to the set first value and an aspect ratio of the drawing image being greater than or equal to the set second value.

[0224] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine a horizontal margin corresponding to a set ratio of the horizontal length of the drawing image and a vertical margin corresponding to a set ratio of the vertical length of the drawing image, based on a length ratio of one side of the drawing image to the image being greater than or equal to the set first value and an aspect ratio of the drawing image being less than the set second value.

[0225] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine the shape of the generation area of ​​the object image as a rectangle based on a ratio of a margin included in a rectangle bordering the outline of the drawing image being less than a fourth value while the area ratio of the drawing image to the image is less than a third value or greater than or equal to the third value.

[0226] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine the shape of the generation area of ​​the object image as a polygon or a closed curve based on the outermost trajectory of the drawing image, based on a ratio of an area of ​​the drawing image to the image being equal to or greater than the third value and a ratio of an empty space around an outermost trajectory included in the drawing image being equal to or greater than the fourth value.

[0227] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify a shape of a generated area of ​​the object image as a polygon excluding the face area based on whether the image includes a face area.

[0228] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to use a template for at least one of a number of words or an attribute of a word to be included in the prompt, which is the output data of the first artificial intelligence model, as the input data of the first artificial intelligence model.

[0229] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to filter the prompt, which is the output data of the first artificial intelligence model, to exclude preset words including a brand name or logo.

[0230] According to one embodiment, the first artificial intelligence model and the second artificial intelligence model may be stored in the memory.

[0231] According to one embodiment, the electronic device may further include a communication module.

[0232] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to transmit the drawing image to a first external server where the first artificial intelligence model is stored, via the communication module.

[0233] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive the object image corresponding to the drawing image from a second external server where the second artificial intelligence model is stored through the communication module.

[0234] According to one embodiment, a server may include a communication module, at least one processor, and a memory storing instructions that, when individually or collectively executed by the at least one processor, cause the server to perform the following operations:

[0235] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the server to receive, through the communication module, from an external electronic device a drawing image corresponding to a sketch input received from the external electronic device.

[0236] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the server to transmit the drawing image to a first external server where the first artificial intelligence model is stored, via the communication module.

[0237] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the server to receive the object image corresponding to the drawing image from a second external server where the second artificial intelligence model is stored through the communication module.

[0238] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the server to transmit the object image to the external electronic device via the communication module.

[0239] According to one embodiment, the first artificial intelligence model may be trained to obtain a prompt for image generation as output data, using a drawing image as input data.

[0240] According to one embodiment, the second artificial intelligence model may be trained to obtain an object image as output data by using a prompt as input data.

[0241] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the server to receive, through the communication module, a background image from which the sketch input is received from the external electronic device.

[0242] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the server to transmit the background image to the first external server via the communication module.

[0243] In one embodiment, the background image may be a cropped area related to the sketch input.

[0244] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the server to transmit a template for at least one of a number of words or an attribute of a word to be included in the prompt, which is the output data of the first artificial intelligence model, to the first external server to be used as the input data of the first artificial intelligence model.

[0245] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the server to transmit, via the communication module, information about a plurality of recommendation templates to the external electronic device.

[0246] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the server to receive information about a selected one of the plurality of recommended templates from the external electronic device via the communication module.

[0247] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the server to transmit a template corresponding to information about the selected template to the first external server.

[0248] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the server to filter the prompt, which is the output data of the first artificial intelligence model, to exclude preset words that include a brand name or logo.

[0249] According to one embodiment, a method of controlling an electronic device may include an action of displaying an image on a display of the electronic device.

[0250] According to one embodiment, the control method of the electronic device may include an action of displaying on the image a drawing image corresponding to the sketch input, based on receiving a sketch input on the image.

[0251] According to one embodiment, the control method of the electronic device may include an operation of providing an edited image including the object image by generating an object image based on the sketch input.

[0252] According to one embodiment, the object image may be obtained as output data from the second artificial intelligence model, with the drawing image corresponding to the sketch input as input data of the first artificial intelligence model, and a prompt obtained as output data from the first artificial intelligence model as input data of the second artificial intelligence model.

[0253] According to one embodiment, the first artificial intelligence model may be trained to obtain a prompt for image generation as output data, using a drawing image as input data.

[0254] According to one embodiment, the second artificial intelligence model may be trained to obtain an object image as output data by using a prompt as input data.

[0255] In one embodiment, the control method of the electronic device may further include an action of cropping an area of ​​the image related to the sketch input to include a margin identified based on at least one of an aspect ratio of the drawing image or a length ratio of one side of the drawing image to the image.

[0256] The operation of providing the above-mentioned edited image may be performed by editing the image based on the cropped area where the object image is generated, thereby providing the edited image.

[0257] According to one embodiment, the control method of the electronic device may further include an operation of checking a margin having a length corresponding to a set number of pixels from each side of the drawing image based on a length ratio of one side of the drawing image to the image being less than a set first value.

[0258] According to one embodiment, the control method of the electronic device may further include an operation of checking a margin having a length corresponding to a set ratio of a product of a horizontal length and a vertical length of the drawing image, based on a length ratio of one side of the drawing image to the image being greater than or equal to the set first value and an aspect ratio of the drawing image being greater than or equal to the set second value.

[0259] According to one embodiment, the control method of the electronic device may further include an operation of checking a horizontal margin corresponding to a set ratio of the horizontal length of the drawing image and a vertical margin corresponding to a set ratio of the vertical length of the drawing image, based on a length ratio of one side of the drawing image to the image being greater than or equal to the set first value and an aspect ratio of the drawing image being less than or equal to the set second value.

[0260] According to one embodiment, the control method of the electronic device may further include an operation of confirming the shape of the generation area of ​​the object image as a rectangle based on whether the ratio of the area of ​​the drawing image to the image is less than a third value, or whether the ratio of the area of ​​the drawing image to the image is greater than or equal to the third value, and whether the ratio of the margin included in a rectangle bordering the outer edge of the drawing image is less than a fourth value.

[0261] According to one embodiment, the control method of the electronic device may further include an operation of confirming the shape of the generation area of ​​the object image as a polygon or a closed curve based on the outermost trajectory of the drawing image, based on a ratio of an area of ​​the drawing image to the image being equal to or greater than the third value and a ratio of an empty space around an outermost trajectory included in the drawing image being equal to or greater than the fourth value.

[0262] According to one embodiment, the control method of the electronic device may further include an operation of confirming the shape of a generation area of ​​the object image as a polygon excluding the face area, based on whether the image includes a face area.

[0263] According to one embodiment, a non-transitory computer-readable recording medium storing one or more programs may include instructions that cause an electronic device to display an image on a display of the electronic device.

[0264] In one embodiment, the one or more programs may include instructions that cause the electronic device to, based on receiving a sketch input on the image, display on the image a drawing image corresponding to the sketch input.

[0265] In one embodiment, the one or more programs may include instructions that cause the electronic device to generate an object image based on the sketch input, thereby providing an edited image including the object image.

[0266] According to one embodiment, the object image may be obtained as output data from the second artificial intelligence model, with the drawing image corresponding to the sketch input as input data of the first artificial intelligence model, and a prompt obtained as output data from the first artificial intelligence model as input data of the second artificial intelligence model.

[0267] According to one embodiment, the first artificial intelligence model may be trained to obtain a prompt for image generation as output data, using a drawing image as input data.

[0268] According to one embodiment, the second artificial intelligence model may be trained to obtain an object image as output data by using a prompt as input data.

[0269] In one embodiment, the one or more programs may include instructions that cause the electronic device to crop a region of the image associated with the sketch input to include a margin identified based on at least one of an aspect ratio of the drawing image or a length ratio of one side of the drawing image to the image.

[0270] According to one embodiment, the one or more programs may include instructions that cause the electronic device to provide the edited image by editing the image based on the cropped region where the object image was generated.

[0271] In one embodiment, the one or more programs may include instructions that cause the electronic device to determine a margin having a length corresponding to a set number of pixels from each side of the drawing image based on a length ratio of one side of the drawing image to the image being less than a set first value.

[0272] In one embodiment, the one or more programs may include instructions that cause the electronic device to determine a margin having a length corresponding to a set ratio of a product of a horizontal length and a vertical length of the drawing image, based on a length ratio of one side of the drawing image to the image being greater than or equal to the set first value and an aspect ratio of the drawing image being greater than or equal to the set second value.

[0273] In one embodiment, the one or more programs may include instructions that cause the electronic device to check a horizontal margin corresponding to a set ratio of the horizontal length of the drawing image and a vertical margin corresponding to a set ratio of the vertical length of the drawing image based on a length ratio of one side of the drawing image to the image being greater than or equal to the set first value and an aspect ratio of the drawing image being less than or equal to the set second value.

[0274] According to one embodiment, the one or more programs may include instructions that cause the electronic device to determine the shape of the creation area of ​​the object image as a rectangle based on whether the ratio of the area of ​​the drawing image to the image is less than a third value, or whether the ratio of the area of ​​the drawing image to the image is greater than or equal to the third value, and whether the ratio of the margin included in a rectangle bordering the outer edge of the drawing image is less than a fourth value.

[0275] According to one embodiment, the one or more programs may include instructions that cause the electronic device to determine the shape of the generation area of ​​the object image as a polygon or a closed curve based on the outermost trajectory of the drawing image, based on a ratio of an area of ​​the drawing image to the image being equal to or greater than the third value and a ratio of an empty space around an outermost trajectory included in the drawing image being equal to or greater than the fourth value.

[0276] According to one embodiment, the one or more programs may include instructions that cause the electronic device to determine the shape of a generated area of ​​the object image as a polygon excluding the face area based on whether the image includes a face area.

[0277] In one embodiment, the one or more programs may include instructions that cause the electronic device to use a template for at least one of a number of words or an attribute of a word to be included in the prompt, which is the output data of the first artificial intelligence model, as the input data of the first artificial intelligence model.

[0278] In one embodiment, the one or more programs may include instructions that cause the electronic device to filter the prompt, which is the output data of the first artificial intelligence model, to exclude preset words including a brand name or logo.

[0279] According to one embodiment, the first artificial intelligence model and the second artificial intelligence model may be stored in the memory.

[0280] According to one embodiment, the electronic device may further include a communication module.

[0281] According to one embodiment, the one or more programs may include instructions causing the electronic device to transmit the drawing image to a first external server where the first artificial intelligence model is stored, via the communication module.

[0282] According to one embodiment, the one or more programs may include instructions causing the electronic device to receive the object image corresponding to the drawing image from a second external server where the second artificial intelligence model is stored through the communication module.

[0283] Electronic devices according to the embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments disclosed in this document are not limited to the aforementioned devices.

[0284] The embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0285] The term "module" used in the embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0286] One embodiment of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0287] According to one embodiment, the method according to one embodiment disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0288] According to one embodiment, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to one embodiment, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to one embodiment, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, At least one processor; display; and When performed individually or collectively by at least one processor, the electronic device causes: Display the image on the above display, Based on receiving a sketch input on the image, displaying on the image a drawing image corresponding to the sketch input, A memory storing instructions for providing an edited image including the object image by generating an object image based on the sketch input; The above object image is, The drawing image corresponding to the sketch input is used as input data of the first artificial intelligence model, and the prompt obtained as output data from the first artificial intelligence model is used as input data of the second artificial intelligence model, and is obtained as output data from the second artificial intelligence model. The above first artificial intelligence model is, It is trained to obtain a prompt for image generation as output data by taking a drawing image as input data, The above second artificial intelligence model is, An electronic device that is trained to obtain an object image as output data by using a prompt as input data.

2. In paragraph 1, The above instructions, when executed by the at least one processor, cause the electronic device to: Crop an area of ​​the image related to the sketch input to include a margin identified based on at least one of an aspect ratio of the drawing image or a length ratio of one side of the drawing image to the image; An electronic device that provides an edited image by editing the image based on the cropped area where the object image is generated.

3. In paragraph 2, The above instructions, when executed by the at least one processor, cause the electronic device to: Based on the fact that the length ratio of one side of the drawing image to the image is less than the set first value, a margin having a length corresponding to the set number of pixels from each side of the drawing image is checked, Based on the fact that the length ratio of one side of the drawing image to the image is greater than or equal to the set first value and the aspect ratio of the drawing image is greater than or equal to the set second value, a margin having a length corresponding to the set ratio of the product of the horizontal length and the vertical length of the drawing image is confirmed, An electronic device that checks a horizontal margin corresponding to the set ratio of the horizontal length of the drawing image and a vertical margin corresponding to the set ratio of the vertical length of the drawing image based on the length ratio of one side of the drawing image to the image being greater than or equal to the set first value and the aspect ratio of the drawing image being less than or equal to the set second value.

4. In any one of paragraphs 1 to 3, The above instructions, when executed by the at least one processor, cause the electronic device to: The shape of the creation area of ​​the object image is confirmed as a rectangle based on the ratio of the margin included in the rectangle bordering the drawing image being less than the third value or the ratio of the area of ​​the drawing image to the image being greater than or equal to the third value, An electronic device that determines the shape of the generated area of ​​the object image as a polygon or closed curve based on the outer edge of the drawing image, based on the ratio of the area of ​​the drawing image to the image being equal to or greater than the third value and the ratio of the empty space around the outermost trajectory included in the drawing image being equal to or greater than the fourth value.

5. In any one of paragraphs 1 to 4, The above instructions, when executed by the at least one processor, cause the electronic device to: An electronic device that determines the shape of a generated area of ​​the object image as a polygon excluding the face area based on whether the face area is included in the image.

6. In any one of paragraphs 1 to 5, The above instructions, when executed by the at least one processor, cause the electronic device to: An electronic device that uses a template for at least one of the number of words to be included in the prompt, which is the output data of the first artificial intelligence model, or an attribute of a word, as the input data of the first artificial intelligence model.

7. In any one of paragraphs 1 to 6, The above instructions, when executed by the at least one processor, cause the electronic device to: An electronic device that filters the output data of the first artificial intelligence model, which is the prompt, to exclude preset words including a brand name or logo.

8. In any one of paragraphs 1 to 7, An electronic device, wherein the first artificial intelligence model and the second artificial intelligence model are stored in the memory.

9. In any one of paragraphs 1 to 8, further comprising a communication circuit; The above instructions, when executed by the at least one processor, cause the electronic device to: Through the above communication circuit, the drawing image is transmitted to the first external server where the first artificial intelligence model is stored, An electronic device that receives the object image corresponding to the drawing image from a second external server in which the second artificial intelligence model is stored through the communication circuit.

10. In a method for controlling an electronic device, An act of displaying an image on a display of said electronic device; An operation of displaying on the image a drawing image corresponding to the sketch input based on receiving a sketch input on the image; and An operation for providing an edited image including the object image by generating an object image based on the sketch input; The above object image is, The drawing image corresponding to the sketch input is used as input data of the first artificial intelligence model, and the prompt obtained as output data from the first artificial intelligence model is used as input data of the second artificial intelligence model, and is obtained as output data from the second artificial intelligence model. The above first artificial intelligence model is, It is trained to obtain a prompt for image generation as output data by taking a drawing image as input data, The above second artificial intelligence model is, A control method of an electronic device that is trained to obtain an object image as output data by using a prompt as input data.

11. In paragraph 10, Further comprising an operation of cropping an area of ​​the image related to the sketch input to include a margin identified based on at least one of an aspect ratio of the drawing image or a length ratio of one side of the drawing image to the image; The action that causes the above edited image to be provided is: A control method of an electronic device that provides an edited image by editing the image based on the cropped area where the object image is generated.

12. In paragraph 11, An operation of checking a margin having a length corresponding to a set number of pixels from each side of the drawing image based on a length ratio of one side of the drawing image to the image being less than a set first value; An operation of checking a margin having a length corresponding to a set ratio of the product of the horizontal length and the vertical length of the drawing image based on the length ratio of one side of the drawing image to the image being greater than or equal to the set first value and the aspect ratio of the drawing image being greater than or equal to the set second value; and A control method of an electronic device further comprising: an operation of checking a horizontal margin corresponding to the set ratio of the horizontal length of the drawing image and a vertical margin corresponding to the set ratio of the vertical length of the drawing image based on a length ratio of one side of the drawing image to the image being greater than or equal to the set first value and an aspect ratio of the drawing image being less than or equal to the set second value; 13. In any one of paragraphs 10 to 12, An operation of confirming the shape of the creation area of ​​the object image as a rectangle based on whether the ratio of the area of ​​the drawing image to the image is less than a third value or whether the ratio of the area of ​​the drawing image to the image is greater than or equal to the third value, and wherein the ratio of the margin included in the rectangle bordering the outer edge of the drawing image is less than a fourth value; and A control method of an electronic device further comprising: an operation of confirming the shape of the generation area of ​​the object image as a polygon or closed curve based on the outermost trajectory of the drawing image, based on the ratio of the area of ​​the drawing image to the image being equal to or greater than the third value, and the ratio of the empty space around the outermost trajectory included in the drawing image being equal to or greater than the fourth value; 14. In any one of paragraphs 10 to 13, A control method of an electronic device further comprising: an operation of confirming the shape of a generation area of ​​the object image as a polygon excluding the face area, based on whether the face area is included in the image; 15. A non-transitory computer-readable recording medium storing one or more programs, wherein the one or more programs are: displaying an image on the display of said electronic device; Based on receiving a sketch input on the image, displaying on the image a drawing image corresponding to the sketch input, Including instructions for providing an edited image including the object image by generating an object image based on the sketch input, The above object image is, The drawing image corresponding to the sketch input is used as input data of the first artificial intelligence model, and the prompt obtained as output data from the first artificial intelligence model is used as input data of the second artificial intelligence model, and is obtained as output data from the second artificial intelligence model. The above first artificial intelligence model is, It is trained to obtain a prompt for image generation as output data by taking a drawing image as input data, The above second artificial intelligence model is, A recording medium that is trained to obtain an object image as output data by using a prompt as input data.

Images