Electronic device for providing image completion function and operation method thereof

Abstract

An electronic device according to an embodiment may comprise: a display; a memory storing instructions; and at least one processor. The instructions according to an embodiment may be executed by the at least one processor to enable the electronic device to receive a user input comprising at least one of a line stroke and a color stroke. The instructions according to an embodiment may generate first image data based on at least one of the line stroke and the color stroke. The instructions according to an embodiment may provide the first image data to a first model and acquire description information about the first image data from the first model. The instructions according to an embodiment may provide the description information and the line stroke to a second model and generate, from the second model, second image data based on the line stroke and the description information.

Description

Electronic device providing image completion function and method of operation thereof

[0001] The present disclosure relates to an electronic device that provides an image completion function and a method of operating the same.

[0002] Generative AI is an artificial intelligence technology capable of generating new content by learning from data. Based on existing data, Generative AI can produce creative works in various formats, such as images, videos, music, and code.

[0003] Recently, AI services that generate images based on text input have been provided. In other words, when a user inputs text, an image is provided as a result.

[0004] However, since the data users input to acquire images is text, there may be some discrepancy between the image users expect and the results provided by the AI ​​service. If users could use images instead of text as input data, they would be able to obtain results that are very close to the images they envision.

[0005] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0006] An electronic device according to one embodiment may include a display, a memory for storing instructions, and at least one processor. Instructions according to one embodiment may be executed by at least one processor so that the electronic device may receive user input including at least one of a line stroke or a color stroke. Instructions according to one embodiment may generate first image data based on at least one of a line stroke or a color stroke. Instructions according to one embodiment may provide the first image data to a first model and obtain description information for the first image data from the first model. Instructions according to one embodiment may provide the description information and the line stroke to a second model and cause the second model to generate second image data based on the line stroke and the description information.

[0007] A method of operation of an electronic device according to one embodiment may include an operation of receiving user input comprising at least one of a line stroke or a color stroke. A method of operation of an electronic device according to one embodiment may include an operation of generating first image data based on at least one of a line stroke or a color stroke. A method of operation of an electronic device according to one embodiment may include an operation of providing the first image data to a first model and obtaining description information for the first image data from the first model. It may include an operation of providing the description information and the line stroke to a second model and generating second image data based on the line stroke and the description information from the second model.

[0008] FIG. 1 is a block diagram of an electronic device in a network environment according to one embodiment.

[0009] FIG. 2 is a block diagram between an electronic device and a server according to one embodiment.

[0010] FIG. 3 illustrates a plurality of artificial intelligence models that transmit and receive mutual data with an electronic device according to one embodiment.

[0011] FIG. 4 is a block diagram of an electronic device according to one embodiment.

[0012] FIG. 5 is a diagram showing the hardware configuration of an electronic device according to one embodiment.

[0013] FIG. 6 is a block diagram of an electronic pen, which is an input means according to one embodiment.

[0014] FIG. 7 is a flowchart illustrating a method of operation of an electronic device according to one embodiment.

[0015] FIG. 8 illustrates an example in which an electronic device according to one embodiment receives line strokes and color strokes.

[0016] FIG. 9 illustrates sketch data and coloring data stored in an electronic device according to one embodiment.

[0017] FIG. 10 illustrates an example of first image data provided to a first model according to one embodiment.

[0018] FIG. 11 illustrates an example of descriptive information provided in an electronic device according to one embodiment.

[0019] FIG. 12 illustrates an example of second image data provided from a second model according to one embodiment.

[0020] FIG. 13 illustrates an example of additional information according to one embodiment.

[0021] FIG. 14 illustrates an example of utilizing a deletion tool in first image data according to one embodiment.

[0022] FIG. 15 illustrates an example of generating first image data according to one embodiment.

[0023] FIG. 16 illustrates an example of tool utilization in an application according to one embodiment.

[0024] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Throughout the specification, the same reference numerals refer to the same components.

[0025] Unless otherwise defined, all terms used herein (including technical and scientific terms) may be used in a meaning commonly understood by those skilled in the art to which the present invention pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise. The terms used herein are for describing embodiments and are not intended to limit the present invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text.

[0026] As used in the specification, "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components, steps, actions, and / or elements to the mentioned components, steps, actions, and / or elements.

[0027] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0028] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0029] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0030] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0031] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0032] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0033] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0034] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0035] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0036] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0037] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0038] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0039] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0040] The camera module (180) can capture still images and moving images. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0041] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0042] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0043] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0044] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0045] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0046] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0047] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0048] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0049] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0050] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0051] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0052] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0053] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0054] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0055] FIG. 2 is a block diagram between an electronic device (101) and a server (108) according to one embodiment.

[0056] Referring to FIG. 2, an electronic device (101) according to one embodiment may include a display module (160), a processor (120), and a memory (130). The electronic device (101) may further include a communication module (190).

[0057] The display module (160), processor (120), memory (130), and communication module (190) included in the electronic device (101) of FIG. 2 may correspond to the display module (160), processor (120), memory (130), and communication module (190) of FIG. 1, respectively.

[0058] The electronic device (101) may include additional components in addition to the components shown in FIG. 2. At least one of the components shown in FIG. 2 may be omitted. In some embodiments, some of the components shown in FIG. 2 may be integrated into one, or one component may be separated into two or more.

[0059] The input module (150), display module (160), processor (120), memory (130), and communication module (190) included in the electronic device (101) are electrically and / or operationally connected to each other so as to exchange signals (e.g., commands or data) with one another.

[0060] According to one embodiment, the electronic device (101) can perform an image provision function. For example, the electronic device (101) can perform an image provision function using at least one generative artificial intelligence (AI) model. For example, the electronic device (101) can perform an image creation (or editing) function by running an application (e.g., a drawing application, an image editing application, a note application) and using said application. The application running on the electronic device (101) can operate independently or operate in conjunction with an external electronic device (e.g., the server (108) of FIG. 1).

[0061] The server (108) may include a first model (210) and a second model (220). The first model (210) may be a generative artificial intelligence model trained for the purpose of generating and / or providing descriptive information for a drawing image (or source image, first image data) created by a user. For example, the first model (210) may be a Large Multimodal Model (LMM) artificial intelligence model. The second model (220) may be a generative artificial intelligence model trained for the purpose of generating and / or providing an image (or output image, second image data) corresponding to the drawing image (or source image) created by a user based on natural language descriptions. For example, the second model (220) may be an image generation artificial intelligence model (e.g., a text-to-image diffusion model AI model). In FIG. 2, the first model (210) and the second model (220) are depicted as engines implemented in a server (108) that is implemented externally to the electronic device (101), but according to one embodiment, the first model (210) and the second model (220) may both be implemented in the electronic device (101). Furthermore, according to one embodiment, either the first model (210) or the second model (220) may be implemented in the electronic device (101) and the other in the server (108).

[0062] According to one embodiment, the display module (160) may include a touch panel for providing a touch function and a display panel for providing a display function. The stacked structure of the display module (160) may vary. For example, the touch panel and the display panel may be configured independently and then packaged together. The touch panel may also be implemented as an in-cell type integrated with the pixel layer of the display panel.

[0063] In some embodiments, the touch panel may be composed of a plurality of layers. For example, the touch panel may include a first touch-sensing layer for detecting touch input using a user's hand (e.g., finger, fingernail) and a second touch-sensing layer for detecting touch input using an electronic pen (e.g., the electronic pen (510) of FIG. 5).

[0064] However, the touch detection method and / or the layer structure of the touch panel are not limited thereto and can be modified in various ways. For example, touch input may be detected through a single touch detection layer regardless of the type of input tool (e.g., electronic pen, finger). As another example, the touch panel may include a first touch detection layer and a second touch detection layer, and each layer may be activated alternately depending on the situation. In this case, the second touch detection layer may be activated as the electronic pen mode is turned on or as the approach of the electronic pen is detected.

[0065] According to one embodiment, the touch panel may include one or more touch sensors configured to detect a touch. According to various embodiments, the touch sensor may detect a touch in at least one of a contact-type capacitive method, a pressure-type resistive method, an infrared detection method, a surface-type ultrasonic conduction method, and / or a piezo-effect method. However, the touch detection method is not limited thereto.

[0066] According to one embodiment, the communication module (190) may support a short-range wireless communication connection of the electronic device (101). For example, the communication module (190) may support a short-range wireless communication (e.g., Bluetooth, Bluetooth low energy (BLE), NFC, Wi-Fi direct, or IrDA (infrared data association)) connection between the electronic device (101) and the electronic pen (e.g., the electronic pen (500) of FIG. 5).

[0067] According to one embodiment, the communication module (190) can support a long-distance wireless communication connection of the electronic device (101). For example, the communication module (190) can support a long-distance wireless communication (e.g., cellular communication, network communication such as the Internet) connection between the electronic device (101) and an external server (e.g., server (108) of FIG. 1).

[0068] According to one embodiment, the processor (120) may include at least one processor. The processor (120) may include a processing circuit. The processor (120) may execute various functions supported by the electronic device (101). The processor (120) may control at least some of the display module (160), the processor (120), the memory (130), and the communication module (190). The processor (120) may execute an application and control various hardware by executing code written in a programming language stored in the memory (130) of the electronic device (101). For example, the processor (120) may refer to a plurality of processors that collectively perform a plurality of operations by dividing them among the processors.

[0069] According to one embodiment, the processor (120) may include a driver circuit for touch detection. The driver circuit may be electrically connected to a touch panel. The driver circuit may detect user touch input (e.g., touch down, long touch, swipe, drag, touch release) received through the touch panel.

[0070] According to one embodiment, a driver circuit can identify data related to a user's touch input received on a touch panel using a touch sensor within the touch panel. For example, the driver circuit can identify data related to characteristics related to the user's touch input (e.g., whether a touch occurs, location (coordinates) of the touch point, touch area, touch sensitivity, travel distance, touch duration, and / or whether the touch is released).

[0071] According to one embodiment, a driver circuit drives a touch sensor within a touch panel (e.g., applies power) and can identify an electrical value (e.g., at least one of a voltage value or a current value) or a change in an electrical value generated by the touch sensor based on an input received based on the applied power. The driver circuit can identify data related to a user's touch input based on the identified electrical value or the change in an electrical value.

[0072] In some embodiments, the driver circuit may include a first driver circuit and a second driver circuit. For example, the driver circuit may be composed of a first driver circuit and a second driver circuit for detecting different types of touch inputs. The first driver circuit may detect touch input using a user's hand (e.g., finger, fingernail) in conjunction with a first touch sensing layer of the touch panel. The second driver circuit may detect touch input using an electronic pen (e.g., the electronic pen (510) of FIG. 5) in conjunction with a second touch sensing layer of the touch panel. However, the structure of the driver circuit is not limited thereto. For example, a single driver circuit may detect touch input through a single touch sensing layer regardless of the type of input tool (e.g., finger, electronic pen).

[0073] According to one embodiment, the memory (130) can store instructions that cause the electronic device (101) to perform various operations when executed by the processor (120). For example, the processor (120) of the electronic device (101) can perform an image providing (or editing) function by executing the instructions stored in the memory (130).

[0074] According to one embodiment, the processor (120) can display a user interface for receiving drawings through a display module (160).

[0075] According to one embodiment, the processor (120) may display a user interface for receiving a drawing on the display module (160) as a triggering event occurs. For example, the triggering event may include at least one of a first event detecting a hovering input of an electronic pen (e.g., the electronic pen (510) of FIG. 5), a second event detecting a user input (e.g., touch) selecting a designated visual element within an edge area of ​​the display module (160), or a third event detecting a user input (e.g., touch) selecting a designated visual element within an application execution screen.

[0076] According to one embodiment, the processor (120) can receive (or detect) a drawing by a user according to a first user input (e.g., touch, swipe, drag, gesture, stroke input, drawing input, handwriting input) through a display module (160).

[0077] According to one embodiment, the processor (120) can determine the style of the image (image style) to be generated based on the user's drawing.

[0078] According to one embodiment, the processor (120) may display a first interface element on the display module (160) for selecting the style of the image to be generated from among a plurality of specified styles. The first interface element may be for selecting the style of the image to be generated (image style) as one of a plurality of pre-specified styles (e.g., watercolor, animation, pencil, cartoon, neon, or realistic). The processor (120) may select a specific style (image style) from one of the plurality of specified styles based on user input to the first interface element.

[0079] According to one embodiment, the processor (120) can receive a second user input (e.g., a touch on a designated visual element) for generating an image corresponding to the user's drawing through a display module (160).

[0080] According to one embodiment, the processor (120) can obtain description information about the drawing using the drawing based on the second user input.

[0081] According to one embodiment, the processor (120) can obtain description information for the drawing (first image data) by providing a first prompt containing the drawing to the first model (210) in response to a second user input for image generation.

[0082] According to one embodiment, the processor (120) can acquire an image generated based on a drawing (first image data) and an image style (an image corresponding to the drawing, second image data). Second image data

[0083] According to one embodiment, the processor (120) can acquire (or generate) an image corresponding to the drawing by providing a prompt containing drawing and image style information to a server (a first model (210) and / or a second artificial intelligence model (220)). In one embodiment, the prompt may further include at least some of the following: reference text entered by a user (e.g., text entered through an input window), reference image (e.g., a background image displayed in a drawing area, a captured image of a background application execution screen), form factor information of an electronic device (101), or form factor information of an external electronic device connected to the electronic device (101) via short-range wireless communication.

[0084] According to one embodiment, the processor (120) can display an image (an image corresponding to the drawing, second image data) obtained based on a drawing (first image data) through a display module (160).

[0085] According to one embodiment, the processor (120) may display a user interface containing at least one keyword related to descriptive information about a drawing through a display module (160). The processor (120) may receive user input for the at least one keyword through the user interface. The processor (120) may edit (e.g., delete, modify, add) the descriptive information about the drawing based on the user input. The processor (120) may acquire an image based on at least a portion of the edited descriptive information.

[0086] According to one embodiment, the processor (120) may store an image corresponding to a drawing in memory (130). When storing the image corresponding to the drawing, the processor (120) may store at least one keyword related to the description information of the drawing in the metadata of the image. In one embodiment, the metadata of the image may further include detailed information such as the location (or place), date, and time of the electronic device (101) corresponding to the time of storage of the image.

[0087] According to one embodiment, the processor (120) can provide a search function for a plurality of images corresponding to drawings by using at least one keyword stored in image-specific metadata. The processor (120) can further provide a search function for the plurality of images by using image-specific detailed information.

[0088] According to one embodiment, the electronic device (101) can acquire (or generate) an image corresponding to a drawing using at least one artificial intelligence model (e.g., the first model (210) and / or the second model (220) of FIG. 3).

[0089] According to one embodiment, the first model (210) and / or the second model (220) may be implemented in hardware and / or software to perform a predetermined function.

[0090] According to one embodiment, the electronic device (101) may be configured to include at least some of the first model (210) or the second model (220). The first model (210) and / or the second model (220) may be included in the electronic device (101) (e.g., memory (130)) in the form of an on-device artificial intelligence model, but are not limited thereto. For example, at least some of the first model (210) or the second model (220) may be included in an external server (e.g., server (108) of FIG. 1). In some embodiments, the functions of the first model (210) and the functions of the second model (220) may be provided through a single integrated artificial intelligence model.

[0091] FIG. 3 illustrates a plurality of artificial intelligence models that mutually transmit and receive data with an electronic device according to one embodiment.

[0092] According to one embodiment, the electronic device (101) can perform an image providing function using a first model (210) and / or a second model (220).

[0093] Referring to FIG. 3, the first model (210) may be a generative artificial intelligence model trained for the purpose of generating and / or providing descriptive information about a drawing (or source image, first image data). For example, the first model (210) may be a Large Multimodal Model (LMM) artificial intelligence model.

[0094] According to one embodiment, an electronic device (101) can obtain description information for the first image data by providing the first image data, which includes a drawing as user input, and the first prompt to the first model (210). The first image data may be a drawing image including at least one of a line stroke and a color stroke. Additionally, the first image data may be a stored image output to a display via loading from an image creation (editing) application.

[0095] According to one embodiment, the first model (210) may receive a first prompt along with first image data. The first prompt may further include a text prompt for a drawing (text requesting a brief description and / or detailed description of the drawing, text regarding the length of the description information).

[0096] According to one embodiment, the first model (210) can generate description information for a drawing based on a first prompt together with first image data. The first model (210) can transmit the generated description information to an electronic device (101).

[0097] The second model (220) may be a generative artificial intelligence model trained for the purpose of generating and / or providing an image (or output image, final image, second image data) corresponding to a drawing (or source image). For example, the second model (220) may be an image generation artificial intelligence model (e.g., a text-to-image diffusion model AI model).

[0098] According to one embodiment, the electronic device (101) can obtain second image data corresponding to the first image data by providing description information and a second prompt to the second model (220).

[0099] According to one embodiment, the second model (220) can receive at least one of the description information for the drawing and the second prompt.

[0100] According to one embodiment, the second model (220) can receive information regarding line strokes entered by the user to the electronic device (101). According to one embodiment, the second model (220) can receive information regarding line strokes in addition to description information about the drawing and the second prompt. The electronic device (101) according to one embodiment can store a drawing image generated by line strokes and color strokes from the user, and provide sketch data, which is information regarding line strokes among the drawing images, to the second model (220).

[0101] In one embodiment, the second prompt may further include at least some of image style information, reference text entered by the user, reference image, form factor information of the electronic device (101), or form factor information of an external electronic device connected to the electronic device (101) via short-range wireless communication.

[0102] According to one embodiment, the second model (220) can generate second image data corresponding to the first image data based on the second prompt. The second model (220) can transmit the second image data to the electronic device (101) in response to the second prompt.

[0103] According to one embodiment, the functions of the first model (210) and the second model (220) may be provided through a single integrated artificial intelligence model. In this case, the integrated artificial intelligence model may skip the process of generating descriptive information for the drawing and then transmitting it to an electronic device (101) (e.g., a processor (120)). The integrated artificial intelligence model may immediately generate an image corresponding to the drawing using the generated descriptive information internally. In some embodiments, the drawing may be used as is to generate an image corresponding to the drawing without the process of generating descriptive information for the drawing.

[0104] In one embodiment, the first model (210) can convert the first image data into text-based description information based on the first image data and a first prompt input from the electronic device (101). In one embodiment, the second model (220) can convert the description information into image-based second image data based on the description information and a second prompt input from the electronic device (101).

[0105] FIG. 4 is a block diagram of an electronic device according to one embodiment.

[0106] The electronic device (101) illustrated in FIG. 4 may be a device that provides and / or supports a handwriting function.

[0107] Referring to FIG. 4, the electronic device (101) may include a processor (120), memory (130), and a display module (160).

[0108] The electronic device (101) may include additional components in addition to the components shown in FIG. 4. Alternatively, at least one of the components shown in FIG. 4 may be omitted. In some embodiments, some of the components shown in FIG. 4 may be integrated into one, or one component may be separated into two or more.

[0109] In one embodiment, the processor (120) may include at least one processor. The processor (120) can execute an application and control various hardware by executing code written in a programming language stored in the memory (130) of the electronic device (101). The processor (120) can control the overall operation of each component of the electronic device (101). The processor (120) can control a series of processes to provide and / or support a user's handwriting function. To this end, the processor (120) may be operatively and / or electrically connected to the memory (130) and the display module (160).

[0110] The memory (130) can serve to store instructions required for the operation of the electronic device (101). As the instructions stored in the memory (130) are executed, the operation of the processor (120) within the electronic device (101) can be performed. Alternatively, the processor (120) can execute the instructions stored in the memory (130) to perform a specified function (or logic).

[0111] In one embodiment, the display module (160) may include a touch screen panel (TSP) (420) for providing a touch function and a display panel (430) for providing a display function.

[0112] The stacked structure of the display module (160) can vary. For example, the display panel (430) and the touchscreen panel (420) may be configured independently and then packaged together. The touchscreen panel (420) may also be implemented as an in-cell type integrated with the pixel layer of the display panel (430).

[0113] In some embodiments, the touchscreen panel (420) may be composed of a plurality of layers. For example, the touchscreen panel (420) may include a first touch-sensing layer for detecting touch input using a user's hand (e.g., finger, fingernail) and a second touch-sensing layer for detecting touch input using an electronic pen (e.g., the electronic pen (510) of FIG. 5).

[0114] However, the touch detection method and / or the layer structure of the touchscreen panel (420) is not limited thereto and can be modified in various ways. For example, touch input may be detected through a single touch detection layer regardless of the type of input tool (e.g., finger, electronic pen). As another example, the touchscreen panel (420) may include a first touch detection layer and a second touch detection layer, and each layer may be activated alternately depending on the situation. In this case, the second touch detection layer may be activated as the electronic pen mode is turned on or as the approach of the electronic pen is detected.

[0115] The touchscreen panel (420) may include one or more touch sensors configured to detect a touch. According to various embodiments, the touch sensor may detect a touch in at least one of a contact-type capacitive method, a pressure-type resistive method, an infrared detection method, a surface-type ultrasonic conduction method, and / or a piezo-effect method. The touch detection method may not be limited to any one of these methods.

[0116] In one embodiment, the processor (120) may include a driver circuit (410) for touch detection.

[0117] The driver circuit (410) can be electrically connected to the touchscreen panel (420). The driver circuit (410) can detect user touch input (e.g., normal touch, hand touch (e.g., palm touch), touch release) received through the touchscreen panel (420).

[0118] The driver circuit (410) can identify data related to a user's touch input received on the touchscreen panel (420) using a touch sensor within the touchscreen panel (420). For example, the driver circuit (410) can identify data related to characteristics related to the user's touch input (e.g., whether a touch is made, location (coordinates) of the touch point, touch area, touch sensitivity, distance traveled, touch duration, and / or whether the touch is released). According to various embodiments, the driver circuit (410) can drive the touch sensor (e.g., apply power) and identify an electrical value (e.g., at least one of a voltage value or a current value) or a change in the electrical value generated by the touch sensor based on the input received based on the applied power. The driver circuit (410) can identify the type of touch (e.g., touch down, drag touch, hand touch (e.g., palm touch), touch release) based on the identified electrical value or the change in the electrical value.

[0119] In some embodiments, the driver circuit (410) may include a first driver circuit and a second driver circuit. For example, the driver circuit (410) may be composed of a first driver circuit and a second driver circuit for detecting different types of touch inputs. The first driver circuit may detect touch input using a user's hand (e.g., finger, fingernail) in conjunction with a first touch detection layer of the touchscreen panel (420). The second driver circuit may detect touch input using an electronic pen (e.g., the electronic pen (510) of FIG. 5) in conjunction with a second touch detection layer of the touchscreen panel (420). However, the structure of the driver circuit is not limited thereto. For example, a single driver circuit may detect touch input through a single touch detection layer regardless of the type of input tool (e.g., finger, electronic pen).

[0120] In one embodiment, the processor (120) may display a first user interface for handwriting input (e.g., FIG. 8) through a display module (160). For example, the processor (120) may display a screen of an application (or program) executed by a user. The application may be an application that provides a handwriting function (which may be referred to as an “app” below, e.g., a memo app, an electronic notebook app, a document app, a drawing app). For example, the first user interface may include a handwriting input area (e.g., a memo window, a word input window, a page area, a canvas area) for receiving the user’s handwriting input.

[0121] In one embodiment, the handwriting input state of the electronic device (101) may include a first state and a second state.

[0122] For example, the first state may be a state in which a first user interface for handwriting input (e.g., FIG. 8) is displayed through the display module (160) of the electronic device (101). The second state may be a state in which a second user interface for handwriting support (e.g., an input tool) is displayed through the display module (160) of the electronic device (101).

[0123] As another example, the first state may be a state where the handwriting input function is enabled. The second state may be a state where the keyboard input function is enabled.

[0124] As another example, the first state may be a state where the cursor function showing the current input position is disabled. The second state may be a state where the cursor function is enabled.

[0125] As another example, the first state may be a state where the handwriting support area is not displayed. The second state may be a state where the handwriting support area is displayed.

[0126] In one embodiment, the processor (120) can detect touch and untouch on the display module (160) in a first state in which a first user interface is displayed. The processor (120) can display handwriting data (e.g., text, drawing) entered by the user in response to user input (or touch events, e.g., normal touch, hand touch, or untouch) detected while displaying the screen of the application.

[0127] In one embodiment, the processor (120) may detect a hand touch area after the touch is released. The hand touch area may correspond to an area of ​​the display surface (or touch surface) of the display module (160) that is touched by a body part (e.g., palm, side of hand and / or at least some of fingers) that serves as a primary axis for handwriting. For example, the hand touch area may include a palm touch area.

[0128] In one embodiment, the touch may correspond to a primary touch for handwriting input. The hand touch area may correspond to a touch area of ​​a designated shape (e.g., elliptical) corresponding to a secondary touch that is distinct from the primary touch.

[0129] In one embodiment, the processor (120) may include a first driver and a second driver. A first touch may be detected through the first driver, and a second touch may be detected through the second driver.

[0130] In one embodiment, a first touch may be detected through a first part of the display module (160). A second touch may be detected through a second part of the display module (160) spaced apart from the first part.

[0131] In one embodiment, the processor (120) can detect the touch input by classifying it into one of a primary touch and a secondary touch based on at least one of the type, shape, area, duration, degree of movement, and speed of movement of the touch input input through the display module (160).

[0132] In one embodiment, the processor (120) may transition from a first state to a second state based on a detected hand touch area so that a second user interface is displayed through the display module (160). The second user interface may be for supporting handwriting input. For example, the second user interface may include one or more objects (e.g., a keyboard, a palette, an input symbol collection window) that are not included in the first user interface.

[0133] In one embodiment, the processor (120) may maintain the handwriting input state in a first state if the size of the hand touch area is greater than or equal to a specified reference value. The processor (120) may switch the handwriting input state to a second state if the size of the hand touch area is less than the reference value.

[0134] In one embodiment, the processor (120) can detect hovering after release of touch and detect release of hovering. The processor (120) can switch the handwriting input state to a second state based on release of hovering and reduction of the hand touch area.

[0135] In one embodiment, the processor (120) can record touch events for touch and touch release as hovering is detected. For example, if a user performs handwriting from a first time point (touch start time) to a second time point (touch release time) to write a specific letter, sentence, or picture, the processor (120) can detect handwriting data corresponding to the touch and touch release and store said handwriting data and information related to said handwriting data (e.g., additional information such as handwriting time) in memory (130).

[0136] In one embodiment, the processor (120) can determine whether the hand touch area is reduced to less than a specified threshold value as the release of hovering is detected. Based on the determination result, the processor (120) can switch the handwriting input state to a second state.

[0137] In one embodiment, the processor (120) can detect hovering when the input tool is separated from the display module (160) by a first designated distance or more and less than a second designated distance. The processor (120) can detect the release of the hovering when the input tool is separated by more than the second designated distance.

[0138] In one embodiment, information regarding reference values ​​for hand touch areas for a plurality of handwriting postures may be stored in advance in memory (130). The processor (120) can identify the user's current handwriting posture and a reference value corresponding to the current handwriting posture based on the information regarding reference values ​​for hand touch areas for a plurality of handwriting postures that is stored in advance. The processor (120) can determine whether to switch the handwriting input state based on the reference value. If the size of the hand touch area is greater than or equal to the reference value, the processor (120) maintains the handwriting input state in a first state, and if the size of the hand touch area is less than the reference value, switches the handwriting input state to a second state to provide a new user interface for supporting handwriting input.

[0139] The electronic device (101) according to the various embodiments disclosed in this document may be of various types of devices capable of providing a handwriting function. The electronic device (101) may include, for example, a smartphone, a tablet, a flexible display, or a stretchable display, but is not limited thereto.

[0140] FIG. 5 is a diagram showing the hardware configuration of an electronic device according to one embodiment.

[0141] An electronic device (101) according to one embodiment may be a smartphone-type device. The electronic device (101) may include a display module (160 of FIG. 1).

[0142] In one embodiment, the display module (160) may include a display panel (430), a touchscreen panel (420), and a digitizer module (440). The touchscreen panel (420) is located on the upper surface of the display panel (430) and can detect capacitive touch input. The digitizer module (440) is located on the lower surface of the display panel (160) and can detect EMR pen input.

[0143] If the display panel (430) is configured as a conventional LCD display panel, it may include a liquid crystal panel that forms an image and a backlight assembly that irradiates light onto the back of the liquid crystal panel. If the display panel (430) is configured as a conventional OLED display panel, it may be configured to include an upper electrode, an organic layer, a lower electrode, and a substrate.

[0144] According to one embodiment, an electronic pen (510) may be used as an input tool for handwriting. The electronic device (101) may detect touch input by the electronic pen (510) through a display module (160).

[0145] According to one embodiment, the electronic device (101) may include a structure into which an electronic pen (510) can be inserted. The electronic device (101) may include a housing (500). A hole (520) may be formed in a part of the housing (500), for example, a part of the side of the housing (500). The electronic pen (510) may be designed to be accommodated in the hole (520) within the housing (500) of the electronic device (101). An internal space, which is a storage space connected to the hole (520), may be formed on the housing (500). The electronic pen (510) may be inserted into the internal space of the hole (520) and may be detached from the internal space.

[0146] FIG. 6 is a block diagram of an electronic pen, which is an input means according to one embodiment.

[0147] Referring to FIG. 6, an electronic pen (600, 510) according to one embodiment may include a memory (610), a communication circuit (620), a processor (630) (e.g., the processor (120) of FIG. 1), a resonant circuit (640), and a trigger circuit (650) as a handwriting input means. According to one embodiment, the processor (630) may be electrically connected to the memory (610), the communication circuit (620), the resonant circuit (640), and / or the trigger circuit (650). However, the components constituting the electronic pen (600) are not limited to those shown in FIG. 6, and the electronic pen (600) may include more or fewer other components.

[0148] According to one embodiment, the memory (610) of the electronic pen (600) may store information related to the operation of the electronic pen (600). For example, the information may include information for communication with the electronic device (101) and frequency information related to the input operation of the electronic pen (600). According to one embodiment, the memory (610) may also store the communication stack of the communication circuit (620). Depending on the implementation, the communication circuit (620) and / or the processor (630) may include a dedicated memory.

[0149] According to one embodiment, the communication circuit (620) of the electronic pen (600) may be configured to perform a wireless communication function between the electronic pen (600) and the communication module of the electronic device (101). According to one embodiment, the communication circuit (620) may transmit information related to the status, input information, and / or location of the electronic pen (600) to the electronic device (101) using a short-range communication method. For example, the communication circuit (620) may transmit to the electronic device (101) information regarding the direction information of the electronic pen (600) obtained through the trigger circuit (650) (e.g., motion sensor data), voice information input through the microphone, remaining battery level information, and / or the status of a button (e.g., on / off button) provided on the electronic pen (600). According to one embodiment, the short-range communication method may include at least one of Bluetooth, Bluetooth Low Energy (BLE), NFC, and Wi-Fi Direct, but the type is not limited.

[0150] According to one embodiment, the processor (630) of the electronic pen (600) may be configured to collect various data and calculate a desired output value. According to one embodiment, the processor (630) may support various operations based on at least some of the user input. For example, the processor (630) may include a generic processor (630) configured to execute a customized hardware module or software (e.g., an application). The processor (630) may include, for example, one or more combinations of hardware, software, or firmware.

[0151] According to one embodiment, the processor (630) of the electronic pen (600) may be configured to transmit information indicating the pressed state of a button, sensing information acquired by a sensor, and / or information calculated based on the sensing information (e.g., information associated with the position of the electronic pen (600)) to the electronic device (101) through a communication circuit (620).

[0152] According to one embodiment, the resonance circuit (640) of the electronic pen (600) can be resonated based on an electromagnetic field signal generated from the digitizer module (235) of the electronic device (101) and can radiate an electromagnetic resonance (EMR) input signal through resonance. The electronic device (101) can determine the position of the electronic pen (600) using the electromagnetic resonance input signal. For example, the electronic device (101) can determine the position of the electronic pen (600) based on the magnitude of the induced electromotive force (e.g., output current) generated by the electromagnetic resonance input signal in each of the plurality of channels (e.g., plurality of loop coils) within the digitizer module (235). Meanwhile, although the electronic device (101) and the electronic pen (600) have been described above as operating based on an EMR method, this is merely illustrative, and the electronic device (101) may generate an electric field-based signal based on an ECR (electrically coupled resonance) method. The resonance circuit (640) of the electronic pen (600) may be resonated by an electric field. The electronic device (101) may detect potentials in multiple channels (e.g., electrodes) due to resonance in the electronic pen (600), and may determine the position of the electronic pen (600) based on the potentials. As another example, the electronic pen (600) may be implemented using an AES (active electrostatic) method. As another example, the electronic device (101) may detect the electronic pen (600) based on a change in capacitance associated with at least one electrode of the touch panel. In this case, the electronic pen (600) may not include a resonant circuit (640).

[0153] According to one embodiment, the resonant circuit (640) of the electronic pen (600) may include a coil (or inductor) and / or a capacitor. The resonant circuit (640) may resonate based on an input electric field and / or magnetic field. When the electronic pen (600) transmits a signal by the EMR method, the electronic pen (600) may generate a signal including a resonant frequency based on an electromagnetic field generated from an inductive panel of the electronic device (101). When the electronic pen (600) transmits a signal by the AES method, the electronic pen (600) may generate a signal using capacitance coupling with the electronic device (101). When the electronic pen (600) transmits a signal by the ECR method, the electronic pen (600) can generate a signal including a resonant frequency based on an electric field generated from a capacitive device of the electronic device (101).

[0154] According to one embodiment, the resonant circuit (640) of the electronic pen (600) may be used to change the strength or frequency of the electromagnetic field according to the user's operation state. For example, the resonant circuit (640) may provide various frequencies for recognizing hovering input, drawing input, button input, or erasing input. For example, the resonant circuit (640) may provide various resonant frequencies according to a combination of connections of multiple capacitors, or may provide various resonant frequencies based on a variable inductor and / or a variable capacitor.

[0155] According to one embodiment, the trigger circuit (650) of the electronic pen (600) may include at least one button or sensor circuit. According to one embodiment, the processor (630) may determine the button input method (e.g., touch, press) or type of the electronic pen (600). According to one embodiment, the trigger circuit (650) may transmit a trigger signal to an electronic device (101) connected via wireless communication using the input signal of the button.

[0156] In one embodiment, the electronic pen (600) may be connected to the electronic device (101) via short-range wireless communication. For example, the electronic device (101) and the electronic pen (600) may be connected via short-range communication using at least one of Bluetooth communication, low-power Bluetooth communication, and / or Wi-Fi communication. In this case, the electronic pen (600) may be used in the electronic device (101) for handwriting input. For example, the electronic pen (600) may make contact with the display module (160) of the electronic device (101) or be in non-contact (close) with the electronic device (101) within a certain distance to generate input to the electronic device (101).

[0157] In some embodiments, even if there is no short-range wireless communication connection established between the electronic pen (600) and the electronic device (101), the electronic device (101) can detect touch input by the electronic pen (600) through the display module (160) or provide a handwriting function using the electronic pen (600).

[0158] FIG. 7 is a flowchart illustrating a method of operation of an electronic device according to one embodiment. Some of the illustrated operations may be omitted, some operations may be integrated, some operations may be changed in order, or other operations may be added.

[0159] According to one embodiment, an electronic device (101) may receive user input including at least one of a line stroke or a color stroke (701). The line stroke and the color stroke are included in the user's drawing and may be implemented through a plurality of tools of an application that provides an image creation (or editing) function. For example, referring to FIG. 8, the line stroke may set the shape of an object (O) when a first tool (1610) is selected, and the color stroke may set the color of an object (O) when a second tool (1620) is selected.

[0160] According to one embodiment, the electronic device (101) can generate first image data (703). The first image data may correspond to a drawing image generated by line strokes and / or color strokes input by a user. The first image data may include sketch data by line strokes and coloring data by color strokes. In the first image data, the sketch data and the coloring data may be stored separately from each other. For example, the user may perform saving of the sketch data after completing line stroke input in the application, and perform saving of the coloring data after completing color stroke input.

[0161] Meanwhile, the first image data may be an image created by the user as a drawing input, or it may correspond to an image stored in the electronic device (101) through the application's import function. Additionally, the first image data may correspond to an image downloaded in real time via a network.

[0162] According to one embodiment, the electronic device (101) can provide first image data to the first model (705). According to one embodiment, the electronic device (101) can, along with providing the first image data to the first model, transmit a first prompt entered by a user to the first model. When the first model receives the first image data, it can generate description information for the first image data according to the command included in the first prompt.

[0163] Meanwhile, according to one embodiment, the electronic device (101) may provide various additional information that complements the first image data to the first model in addition to the first prompt and the first image data. For example, the additional information may include the history of images stored by the user in the electronic device (101), the path through which the image was obtained if the first image data is a pre-existing image, the attributes of the tool and / or text included in the image if the first image data is an image created by the user. Accordingly, the first model may generate description information for the first image data and provide the description information to the electronic device (101) by considering the weight that the additional information has for the first image data.

[0164] According to one embodiment, the electronic device (101) may obtain description information from a first model (707). The description information refers to details necessary to specifically describe or represent an object and / or background included in the first image data. For example, the description information may include information regarding the shape, color, size, and location of the object and / or details regarding the location of the background excluding the object in the image, a specific scene, or a situation.

[0165] According to one embodiment, the electronic device (101) may provide description information to the second model (709). According to one embodiment, the electronic device (101) may provide a second prompt to the second model along with the description information. The second prompt corresponds to a command requesting an image generation to a generative AI engine based on the description information. According to one embodiment, the second model may generate second image data based on the description information provided by the electronic device (101). The second image data corresponds to the final image created or edited from the first image data produced or provided by the user.

[0166] According to one embodiment, the second image data may be image data in which the completeness of the objects included in the first image data and the capacity of the image data are increased. The second image data may have higher clarity through digital drawing compared to the first image data, and unnecessary backgrounds may be removed. Additionally, the objects included in the second image data may have improved balance relative to the overall image by adding colors, shadows, highlights, etc., compared to the objects included in the first image data. Furthermore, other objects may be added to the second image data in addition to the objects included in the first image data.

[0167] According to one embodiment, the electronic device (101) can obtain second image data from the second model (711).

[0168] According to one embodiment, the electronic device (101) can output second image data (713). According to one embodiment, the electronic device (101) can output second image data through an application as a result of generative artificial intelligence in response to a user providing description information and a second prompt to a second model and acquiring second image data from the second model. According to one embodiment, the electronic device (101) can store second image data in memory (130 in FIG. 1) along with outputting second image data.

[0169] FIG. 8 illustrates an example in which an electronic device according to one embodiment receives line strokes and color strokes, and FIG. 9 illustrates sketch data and coloring data stored in an electronic device according to one embodiment.

[0170] Referring to FIG. 8, the electronic device (101) can display a user interface such as a first screen (810) and a second screen (820) through a display (160 in FIG. 1).

[0171] The first screen (810) and the second screen (820) may be a user interface for receiving drawings and / or generating images. The first screen (810) and the second screen (820) may display visual elements that enable the execution of a first tool (1610) for inputting line strokes, a second tool (1620) for inputting color strokes, a third tool (1630) which is an eraser function for deleting line strokes and / or color strokes, and a fourth tool (1640) which is a saving function. The first tool (1610) to the fourth tool (1640) correspond to several examples, and the first screen (810) and the second screen (820) may provide at least some of the functions executable in relation to the drawing, such as drawing erase, undo / redo, drawing redraw, drawing save / share, transparency, minimize, copy, settings, or closing the drawing.

[0172] The first screen (810) corresponds to a user interface for receiving line strokes while receiving drawings. An electronic device (101) according to one embodiment can receive (or detect) a drawing by a user based on user input (e.g., touch, swipe, drag, gesture, stroke input, drawing input, handwriting input) while the first tool (1610) is selected, and can output a sketch image corresponding to the drawing to the first screen (810). Additionally, an electronic device (101) according to one embodiment can receive a drawing by an electronic pen (600) while the first tool (1610) is selected.

[0173] In one embodiment, the electronic device (101) can store sketch data (910 in FIG. 9) corresponding to a line stroke when the user's line stroke input ends while the first tool (1610) is selected. The sketch data can be automatically stored if the user's line stroke input does not occur for a certain period of time. Additionally, the sketch data can be stored by the user selecting the fourth tool (1640), which is a storage function.

[0174] The second screen (820) corresponds to a user interface for receiving color strokes while receiving drawings. An electronic device (101) according to one embodiment can receive (or detect) a drawing by a user based on user input (e.g., touch, swipe, drag, gesture, stroke input, drawing input, handwriting input) when the second tool (1620) is selected, and output a color corresponding to the drawing to the second screen (820). That is, the second screen (820) refers to an output screen in which coloring by color strokes is added to the sketch image generated in the first screen (810).

[0175] In one embodiment, the electronic device (101) can store coloring data (920 in FIG. 9) corresponding to the color stroke when the user's color stroke input ends while the second tool (1620) is selected. The coloring data can be automatically stored if the user's color stroke input does not occur for a certain period of time. Additionally, the coloring data can be stored by the user selecting the fourth tool (1640), which is a storage function.

[0176] An electronic device (101) according to one embodiment can store an image created by a user as first image data when color stroke input is finished. Here, the first image data may include sketch data resulting from a line stroke and coloring data resulting from a color stroke, and the sketch data and coloring data may be distinguished in the first image data by metadata. That is, the sketch data and coloring data may be stored separately in different layers among a plurality of layers.

[0177] An electronic device (101) according to one embodiment can store sketch data resulting from a line stroke when the first tool (1610) is selected, and separate and store coloring data resulting from a color stroke when the second tool (1620) is selected.

[0178] An electronic device (101) according to one embodiment may receive a first user input corresponding to a line stroke when a first tool (1610) included in the application is selected while an image editing application is running. Here, the first user input may include a touch by the user's hand and / or a drawing by an electronic pen (600). An electronic device (101) according to one embodiment may receive a second user input corresponding to a color stroke when a second tool (1620) included in the application is selected. Here, the second user input may include a touch by the user's hand and / or a drawing by an electronic pen (600).

[0179] An electronic device (101) according to one embodiment may store sketch data (910) corresponding to a first tool (1610) when a first user input for a line stroke is terminated while an image editing application is running. An electronic device (101) according to one embodiment may store coloring data (920) corresponding to a second tool (1620) and distinct from the sketch data (910) when a second user input for a color stroke is terminated. Here, the sketch data or coloring data may be stored by the user selecting a fourth tool (1640) which is a storage function, or may be automatically stored after a certain amount of time has elapsed since the last input for a line stroke or a color stroke occurred.

[0180] FIG. 10 illustrates an example of first image data provided to a first model according to one embodiment, and FIG. 12 illustrates an example of second image data provided from a second model according to one embodiment.

[0181] Referring to FIG. 10, the first screen (1000) may be a user interface in a state where drawing reception is completed. The first screen (1000) is in a state where the user has completed drawing on an object (O), and the first image data may be displayed on the display (160).

[0182] According to one embodiment, the electronic device (101) may provide a generation icon (1650) that converts the first image data into second image data while a first screen (1000) is provided where the first image data is output. When user input regarding the generation icon (1650) occurs, the electronic device (101) according to one embodiment may acquire second image data (1200, FIG. 12), which is a final image based on the first image data. According to one embodiment, when user input regarding the generation icon (1650) occurs, the electronic device (101) may transmit the first image data to a first model (210 in FIG. 2) and receive the second image data from a second model (220 in FIG. 2). The electronic device (101) according to one embodiment may output the second image data to an application screen.

[0183] According to one embodiment, the electronic device (101) may receive user input based on a line stroke when the first tool (1610) is selected while an image editing application is running. According to one embodiment, the electronic device (101) may output an object (O) created by the line stroke to a display. According to one embodiment, the electronic device (101) may generate the first image data in which coloring is provided within the object through at least one of a touch gesture or a drag gesture when the second tool (1620) is selected. According to one embodiment, the electronic device (101) may transmit the first image data to a first model (210 in FIG. 2) and receive the second image data from a second model (220 in FIG. 2). According to one embodiment, the electronic device (101) may obtain the second image data containing the object with completed coloring from the second model (220). According to one embodiment, the second image data may be provided with a background image added by the second model according to the type of object (O). In one embodiment, when the electronic device (101) is provided with a background image added to the second image data, the background image may be removed according to the user's selection.

[0184] FIG. 11 illustrates an example of descriptive information provided in an electronic device according to one embodiment.

[0185] Referring to FIG. 11, the first screen (1100) is a user interface in a state where drawing reception is complete, and may be a screen in which description information (1110) is temporarily displayed in a state where sketching and coloring of an object (O) by the user is completed. Referring together to FIG. 10, the description information (1110) corresponds to information that specifically expresses the first image data (or object) obtained from the first model (210 in FIG. 2) in text form when the user selects the creation icon (1650). An electronic device (101) according to one embodiment may temporarily provide the description information (1110) for the first image data (or object) to the user before outputting the second image data.

[0186] According to one embodiment, when the electronic device (101) obtains description information from the first model (210), it can control the display (160) so that the description information is output for a certain period of time. According to one embodiment, the electronic device (101) can automatically output the description information (1110) when the last input for the user's line stroke and / or color stroke occurs and a certain period of time elapses. According to one embodiment, the electronic device (101) can temporarily output the description information (1110) before the second image data is output when the user selects a creation icon (1650).

[0187] According to one embodiment, the electronic device (101) can control the display (160) so that the depiction information (1110) is output for a certain period of time in response to the storage of coloring data (coloring data). The storage of coloring data can be achieved through user input to the fourth tool (1640) of the second screen (820) of FIG. 8. For example, when the user completes and saves the input for the color stroke, the specification of the object (O) can be considered finished. Accordingly, the electronic device (101) according to one embodiment can consider the first image data to be completed and visually provide the depiction information (1110) to the user. After checking the depiction information (1110), the user can modify the first image data through at least one tool.

[0188] Meanwhile, the first image data may include text and / or tool information in addition to line strokes and / or color strokes resulting from the user's drawing input. Here, the text may include at least one of handwriting, characters, symbols, or notes entered by the user. An electronic device (101) according to one embodiment may transmit various information in addition to line strokes and / or color strokes to the first model and / or the second model during the process of acquiring the second image data, thereby bringing it closer to the result required by the user. Related embodiments will be described with reference to FIGS. 13 and 14.

[0189] FIG. 13 illustrates an example of additional information according to one embodiment, and FIG. 14 illustrates an example of utilizing a deletion tool in first image data according to one embodiment.

[0190] Referring to FIG. 13, the first screen (1300) is a user interface in a state where drawing reception is complete, and corresponds to a state where the user has added additional information to the first image data in a state where sketching and / or coloring of the object (O) is completed by the user.

[0191] Additional information is data other than the line strokes and color strokes that constitute the object (O), and corresponds to data that supplements and explains the object (O) through at least one of handwriting, text, symbols, or notes when the user feels that the sketch of the object (O) is insufficient. For example, the additional information may be handwriting in which the user sets a leader line for one end of the object (O) and explains the object (O) at the other end of the leader line. As another example, the additional information may correspond to text explaining the object (O) in an area adjacent to the object (O) while the user has executed a text input function.

[0192] According to one embodiment, when the first tool (1610) is selected, the electronic device (101) receives user input based on a line stroke and can output an object created by the line stroke to the display (160). According to one embodiment, the electronic device (101) can receive additional information about the object in an area adjacent to the object through user input. Here, the additional information may be provided to the electronic device (101) along with user input for the line stroke, not only when the object created by the line stroke is output to the display. Additionally, the additional information may be provided to the electronic device (101) after both the input for the line stroke and the color stroke are completed. According to one embodiment, the electronic device (101) can store the additional information in a memory (130). According to one embodiment, the electronic device (101) can transmit first image data to the first model while the additional information is stored and obtain description information from the first model. According to one embodiment, the electronic device (101) transmits additional information to the second model along with description information obtained from the first model, and can obtain second image data based on the description information and additional information from the second model. According to one embodiment, the electronic device (101) obtains second image data including an object reflecting the additional information and can output the second image data to the display (160).

[0193] According to one embodiment, an electronic device (101) can obtain text information by applying Optical Character Recognition (OCR) to first image data containing additional information. According to one embodiment, the text information can be stored together with the first image data. According to one embodiment, the electronic device (101) can provide the description information and the text information to a second model.

[0194] Referring to FIG. 14, the first screen (1400) is a user interface in a state where drawing reception is complete, and reflects an input in which the user deletes part of the object (O) after sketching and / or coloring of the object (O) by the user is complete.

[0195] The user may use a third tool (1630 in FIG. 8), which is an eraser function, to delete a part of an object (O), but may also obtain a result (second image data) with a part deleted through a first tool (1610 in FIG. 8), which is a line stroke. For example, referring to FIG. 14, the user may provide the first image data to the first model in which the object (whiskers) of a cat, which is an object displayed on the first screen (1400), is removed using the third tool, or the user may provide the first image data to the first model with the first tool (1610) selected, a guide line set for one end of the object, and a delete (or similar word) text assigned to the other end of the guide line. In the latter case, when the user obtains the second image data from the second model, the completeness of the result can be visually verified by comparing it with the first image data that reflects the deletion information.

[0196] According to one embodiment, the first image data may be an image resulting from a user's drawing input or a stored image output through a loading function. Even if the first image data is a stored image, the user may apply deletion information to the first image data through the first tool (1610) or the third tool (1630). Additionally, in the embodiment of FIG. 13, the first image data may also be a stored image.

[0197] According to one embodiment, the electronic device (101) can output an image stored in the electronic device (101) to a display through the loading function of the application while an image generation (or editing) application is running. According to one embodiment, the electronic device (101) can generate first image data by adding at least one of a line stroke or a color stroke to the output image. The user can input additional information to delete or modify a part of the object (O) through the line stroke or the color stroke.

[0198] According to one embodiment, the electronic device (101) can output an image stored in the electronic device (101) to the display through the loading function of the application while an image creation (or editing) application is running. According to one embodiment, the electronic device (101) can receive user input to remove at least one partial area from the outputted image through the erasing function (third tool, 1630) of the application. According to one embodiment, the electronic device (101) can provide the first image data with at least a partial area removed to the first model.

[0199] FIG. 15 illustrates an example of generating first image data according to one embodiment.

[0200] The first screen (1510) is a user interface for receiving drawings and / or generating images, and outputs first image data, and the second screen (1520) is a screen that outputs second image data as a result based on description information for the first image data. In the first screen (1510), visual elements can be displayed to execute a first tool (1610) for inputting line strokes, a second tool (1620) for inputting color strokes, a third tool (1630) which is an eraser function for deleting line strokes and / or color strokes, and a fourth tool (1640) which is a saving function, similar to FIG. 8. The first tool (1610) through the fourth tool (1640) correspond to several examples, and the first screen (1510) and the second screen (1520) may provide at least some of the functions executable in relation to drawing, such as drawing erase, undo / redo, drawing redraw, drawing save / share, transparency, minimize, copy, settings, or drawing close.

[0201] Referring to the first screen (1510) of FIG. 15, the electronic device (101) may receive first image data from the user that is relatively less complete than FIG. 13 and FIG. 14. In the first screen (1510), the first object (O1) and the second object (O2) are distinguished simply by a closed curve (through a line stroke) rather than sketching an outline of the object, and the name (or concept) of each object may be expressed as text inside the closed curve of the first object (O1) and inside the closed curve of the second object (O2). Here, the text may include at least one of handwriting, characters, symbols, or memos entered by the user. Meanwhile, if the text corresponds to handwriting entered by the user, the electronic device (101) according to one embodiment may apply OCR (Optical Character Recognition) to acquire text information and store the text information together with the first image data for each object. According to one embodiment, the text information may be provided to the second model.

[0202] According to one embodiment, the electronic device (101) may receive user input based on the line stroke when the first tool is selected while the image generation (or editing) application is running. According to one embodiment, the electronic device (101) may output at least one object generated by the line stroke to a display. At this time, the object output to the display may be included in an image obtained by the user through a loading function. According to one embodiment, the electronic device (101) may generate first image data that further includes text defining at least one object within at least one object through user input. In one embodiment, if the first image data is a stored image obtained through a loading function, the electronic device (101) may process a closed curve for each object included in the image and ensure that text defining the object is included within the closed curve. According to one embodiment, the electronic device (101) may store text information along with the first image data in memory (130). An electronic device (101) according to one embodiment can transmit first image data and text information to a second model. An electronic device (101) according to one embodiment can obtain second image data including at least one completed object corresponding to the text from the second model (second screen, 1520).

[0203] FIG. 16 illustrates an example of tool utilization in an application according to one embodiment.

[0204] The first screen (1610) is a screen that outputs the first image data as a user interface for receiving drawings and / or generating images, and the second screen (1620) corresponds to a screen in which a user interface that provides a plurality of tools (1622) for additionally editing the first image data is additionally executed.

[0205] According to one embodiment, an application of an electronic device (101) may provide a plurality of tools (1622) for further editing the first image data. The plurality of tools (1622) may further include a brush, a fill, a color mixer, etc., in addition to the first tool (1610), the second tool (1620), and the third tool (1630) described in FIG. 8.

[0206] According to one embodiment, an application of an electronic device (101) may provide recommended colors through a plurality of tools (1622) in response to receiving line stroke input from a user. For example, when the electronic device (101) receives line stroke input from a user in real time or periodically, it may provide image data based on the line stroke input to a first model (210 in FIG. 3). At this time, the electronic device (101) may receive information about an object (O) from the first model (210). According to one embodiment, the electronic device (101) may provide recommended colors based on information about the object (O). For example, as shown in FIG. 16, the electronic device (101) may recognize that the information about the object (O) is a cat and provide representative colors of the cat (e.g., white, black, or brown) through a plurality of tools (1622).

[0207] According to one embodiment, when the first image data is generated, the electronic device (101) can store tool metadata together with the first image data, indicating which tool each of the elements constituting the first image data was generated by.

[0208] According to one embodiment, the electronic device (101) may provide information regarding a plurality of tools included in an application that generates first image data along with description information to a second model. For example, a user may create an object (O) through line strokes, and the user may complete coloring of the object (O) through at least one of the plurality of tools. The second image data may have different completed coloring states even for the same object (O) depending on the tool information.

[0209] An electronic device according to one embodiment may include a display (160 in FIG. 1), a memory (130 in FIG. 1) for storing instructions, and at least one processor (120 in FIG. 1). Instructions according to one embodiment may be executed by at least one processor so that the electronic device may receive user input including at least one of a line stroke or a color stroke. Instructions according to one embodiment may generate first image data based on at least one of a line stroke or a color stroke. Instructions according to one embodiment may provide the first image data to a first model and obtain description information for the first image data from the first model. Instructions according to one embodiment may provide the description information and the line stroke to a second model and cause the second model to generate second image data based on the line stroke and the description information.

[0210] According to one embodiment, a first model can convert the first image data into text-based description information based on the first image data and a first prompt input from an electronic device. According to one embodiment, a second model can convert the description information into image-based second image data based on the description information and a second prompt input from an electronic device.

[0211] According to one embodiment, instructions are executed by at least one processor, so that the electronic device can receive user input corresponding to a line stroke when a first tool included in the application is selected while an image editing-related application is running. According to one embodiment, instructions can receive user input corresponding to a color stroke when a second tool included in the application is selected.

[0212] Instructions according to one embodiment are executed by at least one processor, so that the electronic device can: store sketch data corresponding to a first tool when user input for a line stroke ends. Instructions according to one embodiment can store coloring data corresponding to a second tool and distinct from the sketch data when user input for a color stroke ends.

[0213] Instructions according to one embodiment are executed by at least one processor, so that when the electronic device obtains description information from a first model, it can control a display to output the description information for a certain period of time.

[0214] Instructions according to one embodiment are executed by at least one processor, and the electronic device can control a display to output depiction information for a certain period of time in response to the storage of coloring data.

[0215] Instructions according to one embodiment are executed by at least one processor, so that the electronic device can receive user input based on a line stroke when a first tool is selected. Instructions according to one embodiment can output an object generated by the line stroke to the display. Instructions according to one embodiment can generate the first image data in which coloring is provided within the object through at least one of a touch gesture or a drag gesture when a second tool is selected. Instructions according to one embodiment can obtain second image data including an object with completed coloring from a second model.

[0216] Instructions according to one embodiment are executed by at least one processor, so that the electronic device can receive user input based on a line stroke when a first tool is selected. Instructions according to one embodiment can output an object generated by the line stroke to a display. Instructions according to one embodiment can generate first image data including text defining an object within the object through user input. Instructions according to one embodiment can obtain second image data including a completed object corresponding to the text from a second model.

[0217] Instructions according to one embodiment are executed by at least one processor, so that the electronic device can: receive user input based on a line stroke when a first tool is selected. Instructions according to one embodiment can output an object generated by the line stroke to a display. Instructions according to one embodiment can receive additional information about an object in an area adjacent to the object through user input. Instructions according to one embodiment can obtain second image data including an object reflecting additional information from a second model.

[0218] Instructions according to one embodiment are executed by at least one processor, so that the electronic device can obtain text information by applying Optical Character Recognition (OCR) to first image data containing additional information. Instructions according to one embodiment can provide description information and text information to a second model.

[0219] Instructions according to one embodiment are executed by at least one processor, so that the electronic device can output an image stored in the electronic device to a display through a loading function of the application while an image editing-related application is running. Instructions according to one embodiment can generate first image data by adding at least one of a line stroke or a color stroke to the output image.

[0220] Instructions according to one embodiment are executed by at least one processor, so that while an image editing-related application is running, the electronic device can output an image stored in the electronic device to a display through a load function of the application. User input can be received to remove at least one partial region from the outputted image through an erase function of the application. The first image data with at least a partial region removed can be provided to the first model.

[0221] Instructions according to one embodiment are executed by at least one processor, so that the electronic device can provide information regarding a plurality of tools included in an application that constitutes the first image data together with description information to the second model.

[0222] An electronic device according to one embodiment may further include a communication module for transmitting and receiving data with at least one of a first model or a second model. Instructions according to one embodiment are executed by at least one processor, so that the electronic device: at least one of the first model or the second model may be implemented on an external server.

[0223] A method of operation of an electronic device according to one embodiment may include an operation of receiving user input comprising at least one of a line stroke or a color stroke. A method of operation of an electronic device according to one embodiment may include an operation of generating first image data based on at least one of a line stroke or a color stroke. A method of operation of an electronic device according to one embodiment may include an operation of providing the first image data to a first model and obtaining description information for the first image data from the first model. It may include an operation of providing the description information and the line stroke to a second model and generating second image data based on the line stroke and the description information from the second model.

[0224] According to one embodiment, a first model can convert the first image data into text-based description information based on the first image data and a first prompt input from an electronic device. According to one embodiment, a second model can convert the description information into image-based second image data based on the description information and a second prompt input from an electronic device.

[0225] A method of operation of an electronic device according to one embodiment may further include an operation of receiving user input corresponding to a line stroke when a first tool included in an application is selected while an image editing-related application is running. A method of operation of an electronic device according to one embodiment may further include an operation of receiving user input corresponding to a color stroke when a second tool included in an application is selected.

[0226] A method of operation of an electronic device according to one embodiment may further include an operation of storing sketch data corresponding to a first tool when user input for a line stroke ends. A method of operation of an electronic device according to one embodiment may further include an operation of storing coloring data corresponding to a second tool and distinguishable from the sketch data when user input for a color stroke ends.

[0227] A method of operating an electronic device according to one embodiment may further include an operation of controlling a display so that the description information is output for a certain period of time when description information is obtained from a first model.

[0228] A method of operation of an electronic device according to one embodiment may further include an operation of controlling a display so that depiction information is output for a certain period of time in response to the storage of coloring data.

[0229] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains.

[0230] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs.

Claims

1. In an electronic device, display; Memory for storing instructions; and It includes at least one processor, The above instructions are executed by the above at least one processor, and the electronic device: Receiving user input including at least one of a line stroke or a color stroke, and Generates first image data based on at least one of the above line stroke or the above color stroke, and Providing the first image data to the first model, obtaining description information for the first image data from the first model, and An electronic device that provides the above-mentioned description information and the above-mentioned line stroke to a second model, and generates second image data based on the above-mentioned line stroke and the above-mentioned description information from the second model.

2. In Paragraph 1, The first model above converts the first image data into text-based descriptive information based on the first image data and a first prompt input from the electronic device, and The second model above is an electronic device that converts the description information into second image data, which is an image, based on the description information and a second prompt input from the electronic device.

3. In Paragraph 2, The above instructions are executed by the at least one processor, causing the electronic device: While an image editing application is running: When a first tool included in the above application is selected, user input corresponding to the line stroke is received, and An electronic device that receives user input corresponding to the color stroke when a second tool included in the above application is selected.

4. In Paragraph 3, The above instructions are executed by the at least one processor, causing the electronic device: When user input for the above line stroke is finished, sketch data corresponding to the first tool is saved, and An electronic device that stores coloring data corresponding to the second tool and distinct from the sketch data when user input for the above color stroke is finished.

5. In Paragraph 4, The above instructions are executed by the at least one processor, causing the electronic device: An electronic device that controls the display to output the description information for a certain period of time when the description information is obtained from the first model.

6. In Paragraph 5, The above instructions are executed by the at least one processor, causing the electronic device: An electronic device that controls the display so that the depiction information is output for a certain period of time in response to the storage of the above coloring data.

7. In Paragraph 3, The above instructions are executed by the at least one processor, causing the electronic device: When the above first tool is selected, user input based on the line stroke is received, and Output the object generated by the above line stroke to the above display, and When the above second tool is selected, the first image data is generated in which coloring is provided within the object through at least one of a touch gesture or a drag gesture, and An electronic device for acquiring the second image data including the object with the completed coloring from the second model.

8. In Paragraph 3, The above instructions are executed by the at least one processor, causing the electronic device: When the above first tool is selected, user input based on the line stroke is received, and Output the object generated by the above line stroke to the above display, and A first image data is generated through user input, which includes text defining the object within the object, and An electronic device that obtains the second image data including a completed object corresponding to the text from the second model.

9. In Paragraph 3, The above instructions are executed by the at least one processor, causing the electronic device: When the above first tool is selected, user input based on the line stroke is received, and Output the object generated by the above line stroke to the above display, and Receive additional information about the object in an area adjacent to the object through user input, and An electronic device that acquires the second image data including an object reflecting the additional information from the second model.

10. In Paragraph 9, The above instructions are executed by the at least one processor, causing the electronic device: Text information is obtained by applying Optical Character Recognition (OCR) to the first image data including the above additional information, and An electronic device that provides the above description information and the above text information to the above second model.

11. In Paragraph 2, The above instructions are executed by the at least one processor, causing the electronic device: While an image editing application is running: An image stored in the electronic device is output to the display through the loading function of the above application, and An electronic device that generates first image data by adding at least one of the line stroke or the color stroke to the output image.

12. In Paragraph 2, The above instructions are executed by the at least one processor, causing the electronic device: While an image editing application is running: An image stored in the electronic device is output to the display through the loading function of the above application, and Receiving user input to remove at least one partial area from the output image through the erase function of the above application, and An electronic device that provides the first image data, in which at least some regions have been removed, to the first model.

13. In Paragraph 3, The above instructions are executed by the at least one processor, causing the electronic device: An electronic device that provides information regarding a plurality of tools included in the application constituting the first image data, along with the above description information, to the second model.

14. In Paragraph 2, It further includes a communication module for transmitting and receiving data with at least one of the first model or the second model; and At least one of the first model or the second model is an electronic device implemented on an external server.

15. In a method of operating an electronic device, An action of receiving user input including at least one of a line stroke or a color stroke; An operation to generate first image data based on at least one of the line stroke or the color stroke; The operation of providing the first image data to a first model and obtaining description information for the first image data from the first model; and A method of operating an electronic device comprising the operation of providing the above-described information and the above-described line stroke to a second model, and generating second image data based on the above-described line stroke and the above-described information from the second model.

Images