Electronic device for displaying virtual keyboard, and operation method thereof

Abstract

An electronic device according to one embodiment of the present disclosure can: display a virtual keyboard in a first area through a touch screen display; receive a first user input associated with the virtual keyboard through the touch screen display (310); change, on the basis of the first user input, the appearance of the virtual keyboard such that at least a partial area of the virtual keyboard is hidden; receive a drawing input in the hidden area through the touch screen display (310); generate, on the basis of the drawing input, at least one image by using a machine learning model, the machine learning model including a model trained to generate an image on the basis of at least one from among text, an image and a drawing input; and display the at least one image in a second area, which differs from the first area, through the touch screen display (310). Other various embodiments are possible.

Description

Electronic device for displaying a virtual keyboard and method for operating the same

[0001] The present disclosure relates to an electronic device for displaying a virtual keyboard and a method of operating the same.

[0002] Electronic devices can provide intuitive interfaces and flexible functionality based on touchscreen technology. For example, electronic devices can display a virtual keyboard on a touchscreen display, allowing users to control various functions by touching the screen instead of using physical buttons.

[0003] Users can input text using a virtual keyboard or use a separate window to input drawings. Typically, virtual keyboards are used by displaying fixed, designated letter or number keys. Consequently, direct input of special characters, symbols, or mathematical expressions may be impossible, or multiple touch inputs may be required to enter a mode that supports such input. Furthermore, unlike text input, drawing input requires a separate window for drawing, which can be inconvenient for users on small screens.

[0004] The above information may be provided as background information to aid in understanding this document. None of the above is claimed to be prior art related to this document or can be used to determine prior art.

[0005] One embodiment of the present disclosure may provide an electronic device for displaying a virtual keyboard and a method of operating the same.

[0006] One embodiment of the present disclosure may provide an electronic device and a method of operating the same that can generate an image based on at least one of text input, image input, or drawing input using generative artificial intelligence (AI).

[0007] One embodiment of the present disclosure may provide an electronic device and a method of operating the same capable of generating at least one key of a virtual keyboard using at least a portion of content.

[0008] One embodiment of the present disclosure may provide an electronic device and an operating method thereof capable of generating at least one graphic object to be included in a virtual keyboard based on a preview image acquired through an image sensor.

[0009] An electronic device (101) according to one embodiment of the present disclosure may include a touchscreen display (310), a memory (330), and at least one processor (340) including a processing circuit. The memory (330) may store instructions that, when individually or collectively executed by the at least one processor (340), cause the electronic device (101) to display a virtual keyboard in a first area through the touchscreen display (310), receive a first user input associated with the virtual keyboard through the touchscreen display (310), change the appearance of the virtual keyboard in a manner of covering at least a portion of the virtual keyboard based on the first user input, receive a drawing input in the covered area through the touchscreen display (310), and generate at least one first image based on the drawing input using a machine learning model (422), wherein the machine learning model (422) includes a model trained to generate an image based on at least one of text, an image, or a drawing input, and display the at least one image in a second area different from the first area through the touchscreen display (310).

[0010] An electronic device (101) according to one embodiment of the present disclosure may include a touchscreen display (310), an image sensor (320), a memory (330), and at least one processor (340) including a processing circuit. The memory (330) may store instructions that, when individually or collectively executed by the at least one processor (340), cause the electronic device (101) to display a virtual keyboard in a first area through the touchscreen display (310), obtain a preview image (1120) including at least one object through the image sensor (320), display the preview image (1120) in a second area different from the first area through the touchscreen display (310), generate at least one graphic object using a machine learning model (422) based on the at least one object, the machine learning model (422) including a model trained to generate a graphic object based on at least one of a text, an image, or a drawing input, and display the at least one graphic object in a portion of the virtual keyboard through the touchscreen display (310).

[0011] According to one embodiment of the present disclosure, a method of operating an electronic device (101) may include an operation (1402) of displaying a virtual keyboard in a first area through a touchscreen display (310) included in the electronic device (101), an operation (1404) of receiving a first user input associated with the virtual keyboard through the touchscreen display (310), an operation (1406) of changing the appearance of the virtual keyboard in a manner of covering at least a portion of the virtual keyboard based on the first user input, an operation (1408) of receiving a drawing input in the covered area through the touchscreen display (310), an operation (1410) of generating at least one image based on the drawing input using a machine learning model (422), wherein the machine learning model (422) includes a model trained to generate an image based on at least one of text, an image, or a drawing input, and an operation (1412) of displaying the at least one image in a second area different from the first area through the touchscreen display (310). there is.

[0012] According to one embodiment of the present disclosure, a method of operating an electronic device (101) may include an operation (1702) of displaying a virtual keyboard in a first area through a touchscreen display (310) included in the electronic device (101), an operation (1704) of obtaining a preview image (1120) including at least one object through an image sensor (320) included in the electronic device (101), an operation (1706) of displaying the preview image (1120) in a second area different from the first area through the touchscreen display (310), an operation (1708) of generating at least one graphic object using a machine learning model (422) based on the at least one object, the machine learning model (422) including a model trained to generate a graphic object based on at least one of a text, an image, or a drawing input, and an operation (1710) of displaying the at least one graphic object in a portion of the virtual keyboard through the touchscreen display (310).

[0013] In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

[0014] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0015] Figure 2 is an example of a generative artificial intelligence system.

[0016] FIG. 3 is a block diagram schematically illustrating the configuration of an electronic device according to one embodiment.

[0017] FIG. 4 is a diagram illustrating an image generating device according to one embodiment.

[0018] FIG. 5 is a diagram illustrating a screen of an electronic device for explaining an operation of generating an AI image based on text input and drawing input according to one embodiment.

[0019] FIG. 6 is a diagram illustrating a screen of an electronic device for explaining an operation of generating an AI image based on text and drawing input included in content according to one embodiment.

[0020] FIG. 7A is a diagram illustrating a screen of an electronic device for explaining an operation of changing a virtual keyboard based on image analysis according to one embodiment.

[0021] FIG. 7b is a diagram illustrating a screen of an electronic device for explaining an operation of generating an AI image based on image analysis or drawing input according to one embodiment.

[0022] FIG. 7C is a diagram illustrating a screen of an electronic device for explaining an operation of generating multiple AI images based on image analysis or drawing input according to one embodiment.

[0023] FIG. 8A is a top plan view of an exemplary flexible electronic device in a first state according to one embodiment.

[0024] FIG. 8B is a bottom view of an exemplary flexible electronic device in a first state according to one embodiment.

[0025] FIG. 8C is a plan view of an exemplary flexible electronic device in a second state according to one embodiment.

[0026] FIG. 8D is a bottom view of an exemplary flexible electronic device in a second state according to one embodiment.

[0027] FIG. 8e is a diagram for explaining an operation of generating an AI image based on text input and drawing input in a flexible electronic device according to one embodiment.

[0028] FIG. 9A is a front perspective view of a flip-type electronic device according to one embodiment.

[0029] FIG. 9b is a plan view of the rear side of a flip-type electronic device according to one embodiment.

[0030] FIG. 9c is a diagram for explaining an operation of generating an AI image based on text input and drawing input in a flip-type electronic device according to one embodiment.

[0031] FIG. 10A is a front perspective view of a foldable electronic device according to one embodiment.

[0032] FIG. 10b is a front perspective view of a multi-foldable electronic device according to one embodiment.

[0033] FIG. 10c is a drawing for explaining an operation of displaying a virtual keyboard in a foldable electronic device according to one embodiment.

[0034] FIG. 10d is a drawing for explaining an operation of displaying a virtual keyboard in a multi-foldable electronic device according to one embodiment.

[0035] FIG. 10e is a drawing for explaining an operation of displaying a drawing area in a foldable electronic device according to one embodiment.

[0036] FIG. 10f is a drawing for explaining an operation of displaying a drawing area in a multi-foldable electronic device according to one embodiment.

[0037] FIG. 11 is a drawing for explaining an operation of switching a virtual keyboard in an electronic device according to one embodiment.

[0038] FIG. 12 is a flowchart illustrating an operation of an electronic device generating an AI image according to one embodiment.

[0039] FIG. 13 is a flowchart illustrating an operation of an electronic device according to one embodiment of the present invention to create a graphic object corresponding to at least one key to be included in a virtual keyboard.

[0040] FIG. 14A is a flowchart illustrating an operation of an electronic device according to one embodiment of the present invention to generate and display at least one image based on a drawing input.

[0041] FIG. 14b is a flowchart illustrating an operation of an electronic device according to one embodiment of the present invention to select or modify and display at least one of a plurality of images.

[0042] FIG. 15 is a flowchart illustrating an operation of an electronic device according to one embodiment of the present invention to generate and display at least one image based on text input and drawing input.

[0043] FIG. 16 is a flowchart illustrating an operation of an electronic device according to one embodiment of the present invention to generate and display at least one image based on text and drawing input included in content.

[0044] FIG. 17 is a flowchart illustrating an operation of an electronic device according to one embodiment of the present invention to generate and display at least one image based on a preview image.

[0045] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0046] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0047] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments.

[0048] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0049] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0050] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, in the electronic device (101) itself where artificial intelligence is performed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0051] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0052] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0053] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0054] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0055] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0056] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0057] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0058] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0059] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0060] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0061] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0062] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0063] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0064] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0065] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0066] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0067] According to various embodiments, the antenna module (197) may generate a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0068] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0069] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0070] Figure 2 is an example of a generative artificial intelligence system.

[0071] Referring to FIG. 2, the user query / response interface (210) can receive a user's input. The user's input may be in the form of natural language, images, and / or videos. Additionally, context information (212) may also be transmitted when the user's input is transmitted. The context information (212) may include various additional information at the time of the user input. For example, there is additional information such as information on the application currently being used by the user or information on the user's location. Furthermore, the user's input may be in the form of a mixture of the aforementioned natural language, images, sounds, or context information (212). Furthermore, the user's input may also be in a non-natural language form, such as selecting a menu. The user query / response interface (210) can output the results of the generative artificial intelligence system to the user. The output may be in the form of natural language or specific content, and may also be provided in the form of an action requested by the user. The user query / response interface (210) can output the results of the generative artificial intelligence system to the user. The output can be in natural language form, in the form of specific content, or in the form of an action requested by the user.

[0072] The AI ​​(artificial intelligence) framework (220) can receive user input and coordinate and control each component necessary to perform the user's intention based on the user's query.

[0073] User input received from the user query / response interface (210) can be transmitted to the prompt design component (222). The prompt design component (222) can be used to generate a prompt suitable for inputting the user input into a large language model (LLM) or a large multimodal model (LMM). The prompt design component (222) can be an AI component that uses a machine learning algorithm or a neural network to develop better prompts over time. The prompt design component (222) can access user preference data (214), a prompt library (216), and a knowledge component including prompt examples based on the user input to generate a prompt, and transmit the generated prompt to the LLM or LMM.

[0074] The API / plug-in management component (224) can communicate with external information when there is a request for additional information when passing user input as input to the generative model. The API / plug-in management component (224) can establish a channel for communication with the outside of the AI ​​interface through the API, and can enable access to various data sources (e.g., knowledge repositories (240)) through the established channel. In addition, if the API / plug-in management component (224) needs to perform an action that performs the user input as a final result rather than an intermediate result in an application or service, it can request the application / service component (250) through the API to perform the action. Information obtained from the outside can be used to generate a prompt in the prompt design component (222) together with the user input, or can be passed as input to the generative model.

[0075] The refiner component (226) can fine-tune the output from the generative model. For example, the refiner component (226) can verify that the content generated through the LLM and / or LMM is not irrelevant, does not contain biased content, or does not contain harmful content. Furthermore, the refiner component (226) can determine the degree to which the content matches the user's desired result and, if necessary, perform additional processing. The refiner component (226) can additionally configure and provide hints to the user to avoid undesirable output.

[0076] A generative AI model (230) may generally refer to an artificial intelligence neural network that generates new types of data based on user input information. The generative AI model (230) may include an image-generating model and / or a language-generating model. Representative models for generating images include a generative adversarial network (GAN) and a variational autoencoder (VAE), and examples include a diffusion-based generative model that uses a VAE and a transformer structure. A language-generating model is a model trained to statistically output the most appropriate output based on input values, and representative examples include models such as CHAT-GPT 3 and CHAT-GPT 4. In addition, there is also an LMM that can recognize various types of data input, such as text, images, or voice, and generate new data corresponding to them.

[0077] FIG. 3 is a block diagram schematically illustrating the configuration of an electronic device according to one embodiment.

[0078] Referring to FIG. 3, the electronic device (101) may include a touchscreen display (310), an image sensor (320), a memory (330), and a processor (340). According to one embodiment, the electronic device (101) may include additional components (e.g., a communication circuit corresponding to the communication module (190) of FIG. 1) in addition to the illustrated components, or may omit at least one of the illustrated components.

[0079] According to one embodiment, the electronic device (101) may be any one of a mobile device (e.g., a smartphone or tablet), a computing device (e.g., a personal computer (PC) or a laptop), a wearable device (e.g., a smart watch or a head mounted display (HMD)), or a home appliance (e.g., a television (TV)), but is not limited thereto and may be various types of electronic devices.

[0080] According to one embodiment, a touchscreen display (310) may be a device capable of input and output, and may enable intuitive interaction between a user and an electronic device (101). The touchscreen display (310) may be implemented with various touchscreen technologies, such as a pressure-sensitive (or pressure-resistive), electrostatic capacitive, or infrared type. The touchscreen display (310) may include a touch sensor or a pressure sensor, and may support direct manipulation of icons, buttons, or graphical objects displayed on the screen using a user's body (e.g., a hand or a finger) or an input device (e.g., a stylus pen). The touchscreen display (310) may perform various display operations according to the function of the electronic device (101). For example, the touchscreen display (310) may display at least one of various service information, media information, or text information. The touchscreen display (310) may be a single independent display, or may include multiple displays arranged at different locations. According to one embodiment, the touchscreen display (310) may be implemented identically or similarly to the display module (160) of FIG. 1.

[0081] In one embodiment, the image sensor (320) can capture and acquire an image. The image sensor (320) can capture an image by converting an optical signal (or image signal) received through an optical lens into an electrical digital signal. The image captured by the image sensor (320) can be displayed as a preview image on, for example, a touchscreen display (310). In one embodiment, the image sensor (320) can be included in the camera module (180) of FIG. 1 or implemented in a manner similar to the camera module (180).

[0082] According to one embodiment, the memory (330) can store various data used by at least one component (e.g., a touchscreen display (310), an image sensor (320), or a processor (340)) of the electronic device (101). For example, the memory (330) can store at least one program for processing and controlling the processor (340), and can store input and / or output data. The memory (330) can store at least one AI model (e.g., a generative AI model) and can include volatile memory or non-volatile memory. According to one embodiment, the memory (330) can be implemented substantially the same as or similar to the memory (130) of FIG. 1.

[0083] According to one embodiment, the processor (340) can control the overall operation of the electronic device (101). The processor (340) can execute calculations or data processing related to control and / or communication of at least one other component of the electronic device (101). For example, the processor (340) can be electrically connected to the touchscreen display (310), the image sensor (320), and the memory (330), and can execute instructions of a program stored in the memory (330). According to one embodiment, the processor (340) can be implemented substantially the same as or similar to the processor (120) of FIG. 1.

[0084] The processor (340) may include a processing circuit that executes instructions of a program stored in the memory (330). The processor (340) may include at least one of a central processing unit (CPU), a neural processing unit (NPU), a graphics processing unit (GPU), a micro processing unit (MPU), a micro controller unit (MCU), an application processor (AP), a communication processor (CP), a system on chip (SoC), an integrated circuit (IC), a sensor hub, a supplementary processor, an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA), and may have multiple cores.

[0085] The processor (340) can control the operations of the electronic device (101) by executing instructions stored in the memory (330). For example, the processor (340) can correspond to multiple processors that collectively perform multiple operations by dividing them among the processors.

[0086] An electronic device (101) according to one embodiment may include a touchscreen display (310), a memory (330), and at least one processor (340) including a processing circuit. The memory (330) may store instructions that, when individually or collectively executed by the at least one processor (340), cause the electronic device (101) to display a virtual keyboard in a first area through the touchscreen display (310), receive a first user input associated with the virtual keyboard through the touchscreen display (310), change the appearance of the virtual keyboard in a manner of covering at least a portion of the virtual keyboard based on the first user input, receive a drawing input in the covered area through the touchscreen display (310), and generate at least one image based on the drawing input using a machine learning model (422), wherein the machine learning model (422) includes a model trained to generate an image based on at least one of text, an image, or a drawing input, and display the at least one image in a second area different from the first area through the touchscreen display (310).

[0087] According to one embodiment, the electronic device (101) may further include an image sensor (320). The memory (330) may store instructions that, when individually or collectively executed by the at least one processor (340), cause the electronic device (101) to display, through the touchscreen display (310), a preview image (1120) acquired through the image sensor (320) on at least a portion of the second area, detect at least one object included in the preview image (1120), generate at least one key using the machine learning model (422) based on the at least one object, and display, through the touchscreen display (310), the at least one key on at least a portion of the virtual keyboard.

[0088] According to one embodiment, the memory (330) may store instructions that, when individually or collectively executed by the at least one processor (340), cause the electronic device (101) to display an image provided from an external electronic device in the second area through the touchscreen display (310), generate at least one key using the machine learning model (422) based on the image, and display the at least one key in at least a portion of the virtual keyboard through the touchscreen display (310).

[0089] According to one embodiment, the memory (330) may store instructions that, when individually or collectively executed by the at least one processor (340), cause the electronic device (101) to receive a second user input for the virtual keyboard through the touchscreen display (310), display text corresponding to the second user input in a third area through the touchscreen display (310), and generate the at least one image based on the text and the drawing input using the machine learning model (422).

[0090] According to one embodiment, the electronic device (101) may further include a communication circuit. The memory (330) may store instructions that, when individually or collectively executed by the at least one processor (340), cause the electronic device (101) to display, through the touchscreen display (310), at least one graphic object instructing message transmission at a location adjacent to the third area, display the text in the third area and then receive a third user input for the at least one graphic object, transmit a message including the text to an external electronic device through the communication circuit based on at least a portion of the third user input, and display the text in the second area through the touchscreen display (310). A location adjacent to the third area may mean, for example, a location within a first reference distance from the third area.

[0091] According to one embodiment, the memory (330) may store instructions that, when individually or collectively executed by the at least one processor (340), cause the electronic device (101) to receive, through the touchscreen display (310), a fourth user input for the at least one image, and to transmit, in response to the fourth user input, the at least one image.

[0092] According to one embodiment, the memory (330) may store instructions that, when individually or collectively executed by the at least one processor (340), cause the electronic device (101) to transmit the at least one image to the external electronic device via the communication circuit and to display the at least one second image in the second area along with text included in the transmitted message via the touchscreen display (310).

[0093] According to one embodiment, the memory (330) may store instructions that, when individually or collectively executed by the at least one processor (340), cause the electronic device (101) to display another text in the second area through the touchscreen display (310), receive a fifth user input for the another text through the touchscreen display (310), and generate the at least one image based on the another text and the drawing input.

[0094] According to one embodiment, the at least one image may include a first image and a second image. The memory (330) may store instructions that, when individually or collectively executed by the at least one processor (340), cause the electronic device (101) to display the first image and the second image on at least a portion of the first area through the touchscreen display (310), receive a sixth user input for the first image through the touchscreen display (310), and store the first image in the memory (330) based on at least a portion of the sixth user input.

[0095] According to one embodiment, the memory (330) may store instructions that, when individually or collectively executed by the at least one processor (340), cause the electronic device (101) to receive, through the touchscreen display (310), a seventh user input for the at least one image on the second area, receive another drawing input in the obscured area, generate at least one other image using the machine learning model (422) based on at least a portion of the other drawing input, and display, through the touchscreen display (310), the at least one other image in the second area.

[0096] An electronic device (101) according to one embodiment may include a touchscreen display (310), an image sensor (320), a memory (330), and at least one processor (340) including a processing circuit. The memory (330) may store instructions that, when individually or collectively executed by the at least one processor (340), cause the electronic device (101) to display a virtual keyboard in a first area through the touchscreen display (310), obtain a preview image (1120) including at least one object through the image sensor (320), display the preview image (1120) in a second area different from the first area through the touchscreen display (310), generate at least one graphic object based on the at least one object using a machine learning model (422), the machine learning model (422) including a model trained to generate an image based on at least one of text, an image, or a drawing input, and display the at least one graphic object in a portion of the virtual keyboard through the touchscreen display (310).

[0097] According to one embodiment, the memory (330) may store instructions that, when individually or collectively executed by the at least one processor (340), cause the electronic device (101) to receive a user input for at least another area of ​​the virtual keyboard, change the appearance of the virtual keyboard in a manner that obscures at least a portion of the virtual keyboard based on at least a portion of the user input, receive a drawing input through the touchscreen display (310) in the obscured area, generate at least one image using the machine learning model (422) based on at least a portion of the drawing input, and display the at least one image in the first area through the touchscreen display (310).

[0098] FIG. 4 is a diagram illustrating an image generating device according to one embodiment.

[0099] Referring to FIG. 4, the image generation device (401) may be included in the processor (340) of the electronic device (101), may be a component corresponding to the processor (340), or may be an independent component electrically connected to the processor (340) and operating based on the control of the processor (340). In this document, 'image' and 'video' may be substantially the same or similar, and may be used interchangeably with each other.

[0100] According to one embodiment, the image generation device (401) may include an input module (410), an image generation module (420), an image determination module (430), or an output module (440).

[0101] According to one embodiment, the input module (410) can receive at least one of a text input (412), an image input (414), or a drawing input (416). At least one of the text input (412), the image input (414), or the drawing input (416) can be received based on a graphical user interface (GUI) (e.g., a virtual keyboard) on a touchscreen display (310), for example.

[0102] According to one embodiment, the input method of the input module (410) is not limited to the embodiment and may include voice input, gesture input, or multimodal input (e.g., voice-gesture multimodal input, text-gesture multimodal input, or audio-visual multimodal input).

[0103] According to one embodiment, the image generation module (420) can generate at least one image (424) using a generative AI model (422) (e.g., the generative AI model (230) of FIG. 2) based on at least one of a text input (412), an image input (414), or a drawing input (416). The at least one image (424) can include a still image, an animated image (e.g., a graphics interchange format (GIF) image), or a video.

[0104] According to one embodiment, the generative AI model (422) may be an on-device AI model stored in the electronic device (101) or an AI model stored in an external server. The generative AI model (422) may include a machine learning model trained or learned to generate an image based on at least one of text, an image, or a drawing. At least one image (424) generated by the image generation module (420) through the generative AI model (422) may be provided to the image determination module (430).

[0105] According to one embodiment, the image determination module (430) may prioritize (or rank) at least one image (424) based on the set information. For example, the image determination module (430) may set priority information for each of the at least one image (424) based on the set information. The set information may include, for example, preference information. The preference information may be based on usage information (e.g., usage pattern information and / or usage statistical information) for the electronic device (101) or at least one application included in the electronic device (101).

[0106] According to one embodiment, when there are multiple images (424), the image determination module (430) can delete or exclude at least one image among the multiple images (424) for which the lowest priority information is set or for which priority information below a threshold value is set.

[0107] According to one embodiment, the image determination module (430) can provide at least one image with priority information set to the output module (440).

[0108] According to one embodiment, the output module (440) can output at least one image provided from the image determination module (430) to the touchscreen display (310) based on priority information. For example, the output module (440) can output N ranked images (441) (e.g., Prefrerence Image 1 (442), Prefrerence Image 2 (444), ..., Prefrerence Image N (446)) to the touchscreen display (310). The ranked images (411) can be output in order of high or low priority. The number of ranked images (411) can be one or more. In FIG. 4, an example is illustrated where N is a natural number greater than 2 and the number of ranked images (411) is greater than 2.

[0109] FIG. 5 is a diagram illustrating a screen of an electronic device for explaining an operation of generating an AI image based on text input and drawing input according to one embodiment.

[0110] Referring to (a) of FIG. 5, a messaging application may be executed on the electronic device (101). The messaging application may provide a function that allows a first user (e.g., Moon) of the electronic device (101) to exchange messages with a second user (e.g., Cindy) of an external electronic device. Various types of messages, such as text messages, voice messages, images, videos, or files, may be transmitted and received between the electronic device (101) and the external electronic device.

[0111] According to one embodiment, the electronic device (101) may display a display area (510), a text area (or message input window) (520), or an input area (540) on the touchscreen display (310) based on which message application is running.

[0112] According to one embodiment, the display area (510) may display messages exchanged between a first user and a second user, or messages transmitted and received between an electronic device (101) and an external electronic device.

[0113] According to one embodiment, the text area (520) is a window that displays a text message to be transmitted to a second user, and may display text entered by the first user. The text displayed in the text area (520) may be transmitted to an external electronic device based on the selection of a specific graphic object or icon (e.g., a message transmission button). The text transmitted to the external electronic device may be displayed as a message of the first user in the display area (510).

[0114] According to one embodiment, the input area (540) may be an area where a virtual keyboard (542) for message input is displayed, or a drawing area for drawing input is displayed. When the message application is running, the virtual keyboard (542) may be displayed in the input area (540), but depending on the settings, a drawing area may also be displayed.

[0115] According to one embodiment, a first user may perform text input to generate an image (or icon or emoji) (hereinafter referred to as an “AI image”) using a generative AI model (e.g., the generative AI model (422) of FIG. 4). For example, the first user may input text (e.g., “Good morning with sun”) using at least one key included in a virtual keyboard (542), and the text input by the first user may be displayed in a text area (520).

[0116] According to one embodiment, a first graphical object (532) for controlling an input area (540) may be selected while text is displayed in a text area (520). For example, the first graphical object (532) may include an icon, text, or a menu for indicating a drawing input area that provides an interface for a drawing input of a user (e.g., a first user). The selection of the first graphical object (532) may be based on an input of the first user (e.g., a touch input). Based on the selection of the first graphical object (532), the input area (540) may be changed as illustrated in (b) of FIG. 5 .

[0117] Referring to (b) of FIG. 5, the electronic device (101) may, based on the selection of the first graphic object (532), convert at least a portion of the virtual keyboard (542) displayed in the input area (540) (e.g., an area including multiple keys) into a drawing area (544). The drawing area (544) may be an area where a drawing input entered by the first user by drawing a picture is received or acquired, or a drawing input area.

[0118] According to one embodiment, the electronic device (101) may convert at least a portion of the virtual keyboard (542) into a drawing area (544) based on an input method other than the selection of the first graphic object (532). For example, the electronic device (101) may convert at least a portion of the virtual keyboard (542) into a drawing area (544) based on a set key being entered on the virtual keyboard (542), a voice or gesture of the first user being recognized, or a multimodal input being received.

[0119] According to one embodiment, the drawing area (544) may include, but is not limited to, a translucent area in which at least a portion of the virtual keyboard (542) is converted to translucent. For example, the drawing area (544) may include a transparent area in which at least a portion of the virtual keyboard (542) is converted to transparent, or an opaque area in which at least a portion of the virtual keyboard (542) is converted to opaque.

[0120] According to one embodiment, the drawing area (544) may be formed in various forms. For example, the drawing area (544) may be formed on the obscured area created by changing the appearance of the virtual keyboard by covering at least a portion of the virtual keyboard (542) and by covering at least a portion of the virtual keyboard. For example, the drawing area (544) may be formed to overlap at least a portion of the virtual keyboard (542). For example, the drawing area (544) may be formed by reducing the virtual keyboard (542) so that it is displayed in a portion of the input area (540), and then being formed in the remaining area of ​​the input area (540) secured by reducing the virtual keyboard (542). For example, the drawing area (544) may be formed in at least a portion of the display area (510) in a pop-up form, or may be formed over the entire display area (510).

[0121] According to one embodiment, the electronic device (101) can identify that a drawing input from a first user has been received in a drawing area (544) and a second graphical object (534) for generating an AI image has been selected. For example, the second graphical object (534) may include, but is not limited to, an icon, text, or a menu and may be displayed in various forms. The selection of the second graphical object (534) may be based on an input from the first user (e.g., a touch input).

[0122] According to one embodiment, the electronic device (101) may generate at least one AI image using a generative AI model based on the selection of the second graphic object (534). For example, the electronic device (101) may generate at least one AI image using a generative AI model based on text entered in a text area (520) and a drawing input on a drawing area (544). The text and drawing input may be transmitted to the generative AI model as prompts, and the at least one AI image may include a partially moving object.

[0123] According to one embodiment, the electronic device (101) may generate a prompt to be additionally input to the generative AI model, by generating at least a portion of the text of existing input texts (conversation contents) displayed in the display area (510), the text input in the text area (520), or the drawing input on the drawing area (544). For example, the electronic device (101) may provide additional information, such as contextual information, to the prompt so that the context of the existing conversation contents can be understood and at least one AI image appropriate for the context can be generated.

[0124] According to one embodiment, the electronic device (101) may further provide characteristic information (e.g., gender, age, or preference information) or environmental information (e.g., location information) of the first user and the second user (e.g., Cindy) as prompts to generate at least one AI image suitable for the other party and the user's situation through a generative AI model.

[0125] According to one embodiment, the electronic device (101) can generate at least one AI image based on the first information, the second information, or the third information.

[0126] For example, the first information may include text entered in the text area (520) (e.g., Good morning with sun) or existing conversation content on the display area (510).

[0127] For example, the second information may include characteristic information or environmental information of the first user or the second user, such as location information of the first user (e.g., California beach) and age information of the second user (e.g., Cindy's age (child)).

[0128] For example, the third information may be drawing input information, which may include a picture drawn by the first user (e.g., a sun).

[0129] In one embodiment, the electronic device (101) may use the first information, the second information, or the third information as a prompt input to the generative AI model. Based on the prompt, the electronic device (101) may generate at least one AI image through the generative AI model. For example, an image depicting an animated character rising in a California beach where the first user is located, based on the second user's age (child) and drawing input information, may be generated as an AI image by the generative AI.

[0130] Referring to (c) of FIG. 5, at least one AI image (e.g., AI image (512)) generated by the electronic device (101) may be displayed in a display area (510) on a touchscreen display (310). Although not illustrated, at least one AI image displayed in the display area (510) may be transmitted to an external electronic device through selection by the first user. An AI image selected by the first user may also be displayed in the display area (510).

[0131] According to one embodiment, at least one AI image displayed in the display area (510) may be modified through additional input from the first user or the second user. The additional input may include a text message input and / or a drawing input.

[0132] According to one embodiment, a plurality of AI images may be displayed as candidate images in the display area (510), and at least one of the candidate images may be selected by the first user or the second user. At least one selected image may be displayed in the display area (510).

[0133] In one embodiment, the candidate images may be modified by the first user before being displayed in the display area (510). For example, when the candidate images are generated, the generated candidate images may be displayed in the input area (540), the drawing area (544), or a pop-up window. The first user may select at least one of the candidate images and input additional text or an image. The electronic device (101) may generate a prompt based on the at least one selected image and the input additional text or an image. The electronic device (101) may transmit the generated prompt to the generative AI model and obtain at least one modified image generated through the generative AI model. The at least one modified image may be further modified or displayed in the display area (510) according to the first user's selection. In one embodiment, when the at least one modified image is displayed in the display area (510), the at least one modified image may be further modified by the second user on an external electronic device in a manner similar to the above-described manner.

[0134] In one embodiment, in addition to text input and drawing input, voice input may be used to generate AI images. For example, if a first user inputs a drawing while also inputting a voice, the electronic device (101) may generate at least one AI image based on the drawing input and the voice input.

[0135] According to one embodiment, the electronic device (101) may generate at least one AI image based on multimodal input (e.g., audio-visual multimodal input, voice-gesture multimodal input, or text-gesture multimodal input). For example, the electronic device (101) may receive a user's voice and an image and generate at least one AI image.

[0136] According to one embodiment, when generating an AI image using text and drawing input in a message application, the electronic device (101) may transmit information about the second user (e.g., preferences, age, relationship with the first user, or personality) to the generative AI model to generate at least one AI image reflecting information about the second user.

[0137] FIG. 6 is a diagram illustrating a screen of an electronic device for explaining an operation of generating an AI image based on text and drawing input included in content according to one embodiment.

[0138] Referring to (a) of FIG. 6, the electronic device (101) can generate an AI image based on content including text (e.g., a web page, document, or email).

[0139] According to one embodiment, the electronic device (101) can display content (e.g., a web page of the 'Cinderella story') in the display area (510). All or part of the text (620) included in the content can be selected by the first user. The text (620) selected by the first user can include at least one word or sentence, and can be indicated based on visual effects such as underlining, shading, magnification, or highlighting.

[0140] According to one embodiment, the electronic device (101) can identify that a first graphic object (532) is selected when a text (620) selected by a first user is indicated. Based on the selection of the first graphic object (532), the electronic device (101) can change the input area (540) where a virtual keyboard (542) is displayed, as illustrated in (b) of FIG. 6.

[0141] Referring to (b) of FIG. 6, the electronic device (101) may convert at least a portion of a virtual keyboard (542) displayed in an input area (540) (e.g., an area corresponding to multiple keys) into a drawing area (544) based on the selection of a first graphic object (532). The electronic device (101) may receive a drawing input from a first user in the drawing area (544).

[0142] According to one embodiment, the drawing area (544) may be displayed as semi-transparent, but is not limited thereto, and may also be displayed in a transparent or opaque form. According to one embodiment, the drawing area (544) may be formed in various forms. For example, the drawing area (544) may be formed to overlap at least a portion of the virtual keyboard (542). For example, the drawing area (544) may be formed in an area secured by reducing the virtual keyboard (542) so that it is displayed in a portion of the virtual keyboard (542), and the virtual keyboard (542) may be reduced. For example, the drawing area (544) may be formed in at least a portion of the display area (510) in a pop-up form, or may be formed over the entire display area (510).

[0143] According to one embodiment, the drawing area (544) may include a scroll area. The scroll area may indicate an area that can be scrolled up / down or left / right based on a scroll bar. The scroll area may expand the size of the drawing area (544) and enable multiple drawing inputs to be received based on the expanded size.

[0144] According to one embodiment, the electronic device (101) can identify that a drawing input is received and a second graphical object (534) for generating an AI image is selected. Based on the selection of the second graphical object (534), the electronic device (101) can generate at least one AI image using a generative AI model. For example, the electronic device (101) can generate at least one AI image using a generative AI model based on at least a portion of text (620) selected by the first user and a drawing input on a drawing area (544).

[0145] According to one embodiment, when the number of at least one AI image generated is plural, the electronic device (101) may display a plurality of images as candidate images in the drawing area (544) to receive a user's selection. For example, as illustrated in (c) of FIG. 6, the electronic device (101) may perform a screen transition so that the drawing area (544) disappears from the input area (540), or may display the candidate images by overlapping at least a portion of the drawing area (544).

[0146] Referring to (c) of FIG. 6, a first AI image (630) and a second AI image (640) as candidate images may be displayed in the input area (540). Either the first AI image (630) or the second AI image (640) may be selected by the first user, and the selected image may be saved or transmitted as a message. If the first user does not like either the first AI image (630) or the second AI image (640), the first user may perform an operation again to generate a new AI image. For example, the first user may perform an input (e.g., a set graphic object touch input) to return to the screen of FIG. 5 (a) or (b) to reselect text from the content or input a new drawing in the drawing area (544).

[0147] According to one embodiment, the electronic device (101) may generate a new AI image based on a plurality of candidate images (e.g., a first AI image (630) and a second AI image (640)). For example, the electronic device (101) may receive a first user to select at least one candidate image from among the plurality of candidate images, and receive additional text or an image from the first user. The electronic device (101) may generate a prompt based on the selected at least one candidate image and the input additional text or an image. The electronic device (101) may transmit the generated prompt to a generative AI model, and obtain at least one new image generated through the generative AI model.

[0148] According to one embodiment, when at least two candidate images from among a plurality of candidate images are selected by a first user, the electronic device (101) may generate a new image by combining or summing the features of the at least two candidate images through a generative AI model.

[0149] According to one embodiment, the electronic device (101) can generate an AI image based on at least one of text selected by a first user within the content, at least one image selected by the first user within the content, text input by a user via a virtual keyboard (542), or a drawing input on a drawing area (544).

[0150] According to one embodiment, the electronic device (101) can change and display at least one key of the virtual keyboard (542) based on text selected by the first user within the content.

[0151] In one embodiment, the electronic device (101) may change the virtual keyboard (542) to an English keyboard based on the content, or may generate an AI image based on a word or sentence selected by the first user within the content and display the image on at least one key. For example, the electronic device (101) may generate an icon (or symbol) corresponding to "glass slippers" included in the content using a generative AI model, and display the generated icon instead of the number key "1."

[0152] According to one embodiment, the electronic device (101) may extract key words based on the results of analyzing the content. The electronic device (101) may identify text corresponding to the extracted key words or generate icons related to the extracted key words using a generative AI model. The electronic device (101) may display text corresponding to the extracted key words or icons related to the extracted key words in a portion of a virtual keyboard or on at least one key.

[0153] According to one embodiment, when a first graphic object (532) for drawing input is selected, the electronic device (101) may change the remaining portion of the virtual keyboard (542) except for a key on which an icon (e.g., an icon corresponding to "glass slippers") is displayed into a translucent drawing area (544). Based on the change in the drawing area (544), when a key on which an icon is displayed is selected by the first user, the electronic device (101) may generate an AI image (e.g., an image associated with "glass slippers") associated with the corresponding key using a generative AI model. The electronic device (101) may display the generated AI image in the drawing area (544). In this way, the first user may more easily perform drawing input using a key on which an icon is displayed.

[0154] According to one embodiment, when a plurality of words within content are selected by a first user, the electronic device (101) may generate an AI image using the plurality of words as prompts. For example, when a first graphic object (532) for drawing input is selected while a plurality of words are selected, the electronic device (101) may generate graphic objects corresponding to the plurality of words through a generative AI and display the generated graphic objects on a portion of the drawing area (544). The first user may appropriately arrange the generated graphic objects and use them as drawing input.

[0155] According to one embodiment, the electronic device (101) can search for or crawl at least one image or video on an external server using text and drawing input. Based on the searched or crawled at least one image or video, the electronic device (101) can generate at least one AI image and display it in the drawing area (544).

[0156] FIG. 7A is a diagram illustrating a screen of an electronic device for explaining an operation of changing a virtual keyboard based on image analysis according to one embodiment.

[0157] Referring to (a) of FIG. 7A, based on the execution of a message application on the electronic device (101), a message of a first user, who is a user of the electronic device (101), and a message of a second user, who is a user of an external electronic device, may be displayed on the display area (510). According to one embodiment, an image may be displayed on the display area (510). The image displayed on the display area (510) may correspond to an image message transmitted by the first user or the second user, or may be included in a message of the first user or the second user. For example, an image message transmitted by the second user or an image (e.g., a shape image) (710) included in a message of the second user may be displayed on the display area (510).

[0158] According to one embodiment, the electronic device (101) can identify a first user's input (712) with respect to the displayed image (710). For example, the first user's input (712) can include at least one of the first user's touch input, voice input, gesture input, or multimodal input.

[0159] According to one embodiment, the electronic device (101) may analyze the image (710) using a configured AI model based on an input (712) of a first user (e.g., a touch input on a displayed image (710) or a voice input such as 'Identify the image Cindy just sent me'). Based on the analysis result, the electronic device (101) may identify that the image (710) is an image related to a math problem. According to one embodiment, the configured AI model may be a machine learning model trained to analyze an input image when an image is input and output the analysis result as text.

[0160] According to one embodiment, the electronic device (101) may change the virtual keyboard (542) displayed in the input area (540) to a numeric keyboard (730) as illustrated in (b) of FIG. 7a, based on the image (710) being identified as an image associated with a math problem.

[0161] According to one embodiment, the electronic device (101) may analyze text or images within the display area (510) without receiving input (712) from the first user or may automatically analyze the text or image within the display area (510) and change the virtual keyboard (542) to a numeric keyboard (730) based on the analysis result. Before changing the virtual keyboard (542) to a numeric keyboard (730), the electronic device (101) may display a notification message (e.g., “Do you want to change the virtual keyboard (or character keyboard) to a numeric keyboard?”) in a pop-up form or output it by voice, and may change the virtual keyboard (542) to a numeric keyboard (730) based on the user’s response to the notification message.

[0162] Referring to (b) of FIG. 7A, the numeric keyboard (730) may include a plurality of virtual keys for numeric key input, and may not include English keys included in the virtual keyboard (542). According to one embodiment, when at least one key (732) included in the numeric keyboard (730) is selected by the first user, the electronic device (101) may change the numeric keyboard (730) into an engineering keyboard (740) as illustrated in (c) of FIG. 7A. The engineering keyboard (740) may be for facilitating input of a corresponding mathematical problem.

[0163] Referring to (c) of FIG. 7A, the engineering keyboard (740) may include a plurality of virtual keys containing symbols, formulas, or formula names required to solve a mathematical problem associated with an image (710). The symbols, formulas, or formula names included in the plurality of virtual keys may be associated with an image or text generated using a generative AI model based on a result value output by a set AI model.

[0164] According to one embodiment, at least one key included in the engineering keyboard (740) may include an AI image generated based on a drawing input on a drawing area (e.g., the drawing area (544) of FIG. 5 (b)). The AI ​​image included in the at least one key may be selected by the first user from a plurality of candidate images and may include an additional function. For example, the additional function may include a macro function or a shortcut key function. The macro function or the shortcut key function may refer to a function that provides the same effect as selecting multiple keys even when a single key is selected.

[0165] According to one embodiment, at least one first key associated with text (e.g., a formula or symbol) among keys included in an engineering keyboard (740) may be selected by a first user, and at least one second key associated with an image (e.g., a shape image) may be selected. The electronic device (101) may input the text associated with the at least one first key and the image associated with the at least one second key into a generative AI model, thereby generating an AI image or AI video associated with a mathematical equation solution. The generated AI image or AI video may include at least a portion of the text associated with the at least one first key and the image associated with the at least one second key. For example, the AI ​​video may include a scene in which a virtual instructor solves a problem using a formula or symbol included in the text, or at least a portion of the image.

[0166] FIG. 7b is a diagram illustrating a screen of an electronic device for explaining an operation of generating an AI image based on image analysis or drawing input according to one embodiment.

[0167] Referring to (a) of FIG. 7B, based on the execution of a message application on the electronic device (101), a message of a first user who is a user of the electronic device (101) and a message of a second user who is a user of an external electronic device may be displayed on the display area (510). According to one embodiment, a message (e.g., a text message, an image message, or a message including text and an image) transmitted by the first user or the second user may be displayed on the display area (510).

[0168] According to one embodiment, the display area (510) may display an image message transmitted by a second user, for example, a math problem image (750). The math problem image (750) may include, for example, a math problem (or mathematical equation) written in text or expressed as an image, and / or a figure image associated with the math problem.

[0169] According to one embodiment, the electronic device (101) may perform screen analysis using a set AI model or receive an input from a first user based on the display of a math problem image (750) and display a drawing area on the input area (540). For example, the electronic device (101) may analyze text or an image (e.g., a math problem image (750)) on the display area (510), determine that an input from the first user is required to solve the math problem based on the analysis result, and display the drawing area. For example, the electronic device (101) may also display the drawing area based on at least one of a touch input, a voice input, a gesture input, or a multimodal input of the first user associated with the math problem image (750).

[0170] According to one embodiment, the electronic device (101) may display a notification message (e.g., “Should I display the drawing area?”) in a pop-up form or output it by voice before displaying the drawing area, and display the drawing area based on the user’s response to the notification message.

[0171] According to one embodiment, if a virtual keyboard (e.g., the virtual keyboard (542) of FIG. 7A) is displayed in the input area (540) before displaying the drawing area, the electronic device (101) may change at least a portion of the virtual keyboard into a drawing area and display it, or may reduce the virtual keyboard to be displayed in a portion of the input area (540) and display the drawing area in the area secured by reducing the virtual keyboard. According to one embodiment, the electronic device (101) may provide the drawing area in a pop-up form in at least a portion of the display area (510) other than the input area (540), or may provide it over the entire display area (510).

[0172] According to one embodiment, the electronic device (101) may receive a drawing input of a first user associated with a math problem image (750) in a drawing area. For example, the received drawing input may be an input based on the first user's body (e.g., a hand or a finger), a virtual keyboard displayed in a pop-up window or a set location, or an input device (e.g., a stylus pen) (761).

[0173] According to one embodiment, the electronic device (101) may receive a mathematical equation (760) handwritten by a first user using an input device (761) in a drawing area. According to one embodiment, the mathematical equation (760) may be solved and the answer may be calculated, partially solved, not solved, or not calculated.

[0174] According to one embodiment, the electronic device (101) may input a mathematical equation (or mathematical equation solution) (760) input into a drawing area and / or a mathematical problem image (750) on a display area (510) into a generative AI model, and obtain a problem-solving process or answer associated with the mathematical equation (760) and / or the mathematical problem image (750). The generative AI model may include a machine learning model trained or learned to analyze the mathematical equation or mathematical equation solution and generate the problem-solving process or answer in the form of an image (or text). The generative AI model may receive a handwritten mathematical equation (760) as a prompt in the form of an image, and may analyze the input image to generate a problem-solving process or answer reflecting the handwriting of the first user.

[0175] In one embodiment, the electronic device (101) may input a partially solved or unsolved mathematical equation (760) into a generative AI model. Using the generative AI model, the electronic device (101) may generate and display a complete or partial solution or answer for the input mathematical equation (760).

[0176] Referring to (b) of FIG. 7B, the generative AI model can generate an AI image (764) reflecting the handwriting of the first user by combining the generated solution process or answer (762) with a partially solved mathematical equation (760).

[0177] According to one embodiment, the electronic device (101) can display an AI image (764) generated through a generative AI model in a set manner. For example, the electronic device (101) can display the AI ​​image (764) in the input area (540) or in at least a portion of the display area (510) in a pop-up form.

[0178] In one embodiment, multiple images may be generated using a generative AI model. This is described with reference to FIG. 7c.

[0179] FIG. 7C is a diagram illustrating a screen of an electronic device for explaining an operation of generating multiple AI images based on image analysis or drawing input according to one embodiment.

[0180] According to one embodiment, the electronic device (101) can input a partially solved or unanswered mathematical equation (760) and / or a mathematical problem image (750), such as that shown in (a) of FIG. 7b, into a generative AI model to generate a plurality of images.

[0181] According to one embodiment, the plurality of images may include an AI image (764) reflecting the handwriting characteristics (e.g., cursive) of the first user, as shown in (a) of FIG. 7c, and an AI image (770) reflecting the handwriting characteristics of the first user or a font provided by a computing device such as an electronic device (101).

[0182] According to one embodiment, the plurality of images may include an AI image (764) reflecting the handwriting characteristics of the first user, as shown in (b) of FIG. 7c, and an AI image (780) that is a combination of a math problem image (750) displayed in the display area (510) and an image (770) reflecting the font provided by the computing device.

[0183] According to one embodiment, a plurality of images may be used as candidate images, and at least one of the candidate images may be selected by a first user. The electronic device (101) may transmit the at least one selected image to an external electronic device based on an input of the first user (e.g., a touch input, a voice input, or a multimodal input associated with the at least one selected image) so that the at least one selected image may be shared with a second user. The transmitted at least one image may be displayed in a display area (510).

[0184] According to one embodiment, the electronic device (101) can store various types of virtual keyboards (e.g., virtual keyboard (542), numeric keyboard (730), or engineering keyboard (740)) and share the stored various types of virtual keyboards with an external electronic device. For example, the electronic device (101) can transmit information about the engineering keyboard (740) to the external electronic device, thereby allowing a second user to use the engineering keyboard (740) on the external electronic device.

[0185] According to one embodiment, the electronic device (101) may display a virtual keyboard generated for a first application (e.g., a messaging application) when executing a second application (e.g., an educational application). The virtual keyboard used in each application may be varied according to the user's settings.

[0186] According to one embodiment, the electronic device (101) may generate various types of virtual keyboards, such as a chemical keyboard, a shape keyboard, an emoticon keyboard, or a musical note keyboard, in addition to an engineering keyboard (740). For example, the electronic device (101) may generate virtual keys marked with 'quarter note' and 'do' as keys to be included in the musical note keyboard, based on a user's drawing input (e.g., a drawing input of 'quarter note' and 'do'), or a drawing input (e.g., a drawing input of 'quarter note') and a text input (e.g., a text input of 'do').

[0187] According to one embodiment, the electronic device (101) can generate a virtual keyboard reflecting a font. The font of at least one key included in the virtual keyboard may be a font based on a user's drawing input. When text (e.g., a word or sentence) is input into the drawing area using the user's hand or a stylus pen, the electronic device (101) can generate a font corresponding to the shape of the input text. The electronic device (101) can display at least one key reflecting the generated font, thereby enabling the user to perform text input, such as handwriting, using the virtual keyboard.

[0188] According to one embodiment, the electronic device (101) can support text input using a wider variety of font types by acquiring an image captured through an image sensor (320) and generating a font based on text included in the acquired image.

[0189] According to one embodiment, the electronic device (101) may be a bar-type device as illustrated in FIGS. 5 to 7C, but is not limited thereto and may also be other types of electronic devices, such as a flexible electronic device, a flip-type electronic device, or a foldable electronic device. Hereinafter, an operation of generating an AI image using a virtual keyboard in various types of electronic devices will be described with reference to FIGS. 8A to 10F.

[0190] According to one embodiment, a touchscreen display (310) included in an electronic device (101) (e.g., a flexible electronic device) may be flexible. For example, the touchscreen display (310) may provide at least a portion of an outer surface of the electronic device (101) and may include a display area that is visually exposed outside a housing of the electronic device (101). For example, since the touchscreen display (310) has flexibility, at least a portion of the touchscreen display (310) may be rollable into the housing or slidable into the housing. For example, the size of the display area may vary depending on the size of at least a portion of the touchscreen display (310) that is rolled into or slid into the housing. For example, an electronic device (101) including a touchscreen display (310) may be in a plurality of states, including a first state providing a display area having a first size and a second state providing a display area having a second size different from the first size. For example, the first state may be exemplified through the descriptions of FIGS. 8A and 8B .

[0191] FIG. 8A is a top plan view of an exemplary flexible electronic device in a first state according to one embodiment.

[0192] Referring to FIG. 8A, the electronic device (101) may include a first housing (810), a second housing (820) movable relative to the first housing (810) in a first direction (861) parallel to the y-axis or a second direction (862) parallel to the y-axis and opposite to the first direction (861), and a display (830) (e.g., the touchscreen display (310) of FIG. 3). Although the second housing (820) is described as being movable relative to the first housing (810), this is not limiting. For example, the first housing (810) may be movable relative to the second housing (820). For example, depending on a change in the relative positional relationship between the first housing (810) and the second housing (820), the size of the display area of ​​the display (830) visually exposed outside the housing of the electronic device (101) may change.

[0193] For example, within the first state, the second housing (820) may be movable relative to the first housing (810) in a first direction (861) among the first direction (861) and the second direction (862). For example, within the first state, the second housing (820) may not be movable relative to the first housing (810) in the second direction (862).

[0194] For example, within the first state, the display (830) may provide a display area having the smallest size. For example, within the first state, the display area may correspond to the first area (830a). For example, although not illustrated in FIG. 8A, within the first state, the first area (830a), which is the display area, and another area of ​​the display (830) (e.g., the second area (830b) of FIG. 8C) may be positioned within the first housing (810). For example, within the first state, the second area (830b) may be covered by the first housing (810). For example, within the first state, the second area (830b) may be moved into the first housing (810). For example, at least a portion of the second area (830b) may be rolled into the first housing (810). For example, within the first state, the first region (830a) may include a planar portion. For example, within the first state, a portion of the second region (830b) may include a curved portion. However, this is not limited thereto. For example, the first region (830a) may also include a curved portion extending from the planar portion within the first state.

[0195] For example, the first state may be referred to as a slide-in state in that at least a portion of the second housing (820) is positioned within the first housing (810) as the second housing (820) slides toward the first housing (810). For example, the first state may be referred to as a reduced state in that it provides a display area having the smallest size, but is not limited thereto.

[0196] For example, the second housing (820) may include a front camera (850-1) that obtains visual information through a portion of the first region (830a) and faces a third direction (863) parallel to the z-axis. For example, although not illustrated in FIG. 8A, the second housing (820) may include one or more rear cameras (e.g., rear cameras (850-2) of FIG. 8B) that are visually exposed through a portion of the second housing (820) and face a fourth direction (864) parallel to the z-axis and opposite to the third direction (863). For example, one or more rear cameras (850-2) may be exemplified through the description of FIG. 8B.

[0197] FIG. 8B is a bottom view of an exemplary flexible electronic device in a first state according to one embodiment.

[0198] Referring to FIG. 8B, within the first state, one or more rear cameras (850-2) disposed within the second housing (820) may be positioned within a structure disposed within the first housing (810) for the one or more rear cameras (850-2). For example, since the one or more rear cameras (850-2) are positioned within the structure within the first state, the one or more rear cameras (850-2) may be visually exposed through the structure within the first state. The one or more rear cameras (850-2) may obtain visual information through the structure. For example, the structure may be implemented in various ways. For example, the structure may be an opening or a notch. For example, the structure may be an opening (812a) within a first plate (812) of the first housing (810) that surrounds at least a portion of the second housing (820). However, the present invention is not limited thereto.

[0199] In one embodiment, a first state can be changed to a second state. For example, a first state (or a second state) can be changed to a second state (or a first state) through one or more intermediate states between the first state and the second state.

[0200] For example, the first state (or second state) may be changed to the second state (or first state) based on a defined user input. For example, the first state (or second state) may be changed to the second state (or first state) in response to a user input on a physical button visually exposed through a portion of the first housing (810) or a portion of the second housing (820). There is no limitation on the type of user input. For example, the user input may include a user input through a touch screen within the display area of ​​the display (830) or a user input through a microphone of the electronic device (101). For example, the state of the electronic device (101) may be changed to the second state (or first state) by an external force applied to the first housing (810) and / or the second housing (820).

[0201] The second state can be illustrated through the descriptions of FIGS. 8c and 8d.

[0202] FIG. 8C is a plan view of an exemplary flexible electronic device in a second state according to one embodiment.

[0203] Referring to FIG. 8C, within the second state, the second housing (820) may be movable relative to the first housing (810) in a second direction (862) among the first direction (861) and the second direction (862). For example, within the second state, the second housing (820) may not be movable relative to the first housing (810) in the first direction (861).

[0204] For example, within the second state, the display (830) may provide a display area having the largest size. For example, within the second state, the display area may correspond to an area (830c) that includes a first area (830a) and a second area (830b). For example, the second area (830b), which was included within the first housing (810) within the first state, may be visually exposed within the second state. For example, within the second state, the first area (830a) and the second area (830b) may include a planar portion. However, the present invention is not limited thereto. For example, the first area (830a) and / or the second area (830b) may also include a curved portion that extends from the planar portion and is positioned within the edge portion.

[0205] For example, the second state may be referred to as a slide-out state in that at least a portion of the second housing (820) is positioned outside the first housing (810) as the second housing (820) slides out from the first housing (810). For example, the second state may be referred to as an expanded state in that it provides a display area having the largest size, but is not limited thereto.

[0206] For example, the front camera (850-1) facing the third direction (863) may move together with the first region (830a) according to the movement of the second housing (820) in the first direction (861) when the state of the electronic device (101) changes from the first state to the second state. Although not illustrated in FIG. 8C, one or more rear cameras (e.g., the rear cameras (850-2) of FIG. 8D) facing the fourth direction (864) may move together with the second housing (820) according to the movement of the second housing (820) in the first direction (861) when the state of the electronic device (101) changes from the first state to the second state. For example, the relative positional relationship between one or more rear cameras (850-2) and the structure illustrated through the description of FIG. 8B may change according to the movement of one or more rear cameras (850-2). For example, a change in relative positional relationship can be illustrated through Fig. 8d.

[0207] FIG. 8D is a bottom view of an exemplary flexible electronic device in a second state according to one embodiment.

[0208] Referring to FIG. 8D , within the second state, one or more rear cameras (850-2) may be positioned outside the structure. For example, within the second state, one or more rear cameras (850-2) may be positioned outside the opening (812a) in the first plate (812). For example, since the one or more rear cameras (850-2) are positioned outside the opening (812a) within the second state, the one or more rear cameras (850-2) may be visually exposed within the second state. The one or more rear cameras (850-2) positioned outside the structure may acquire visual information. For example, since one or more rear cameras (850-2) are positioned outside the structure within the second state, the relative positional relationship between one or more rear cameras (850-2) and the structure (e.g., the opening (812a)) within the second state may be different from the relative positional relationship between one or more rear cameras (850-2) and the structure (e.g., the opening (812a)) within the first state (e.g., FIG. 8B).

[0209] Although not illustrated in FIGS. 8A, 8B, 8C, and 8D , the electronic device (101) may be in an intermediate state between the first state and the second state. For example, the size of the display area in the intermediate state may be larger than the size of the display area in the first state and smaller than the size of the display area in the second state. For example, the display area in the intermediate state may correspond to an area including a portion of the first area (830a) and the second area (830b). For example, in the intermediate state, a portion of the second area (830b) may be visually exposed, and another portion (or a remaining portion) of the second area (830b) may be covered by the first housing (810) or moved into the first housing (810). However, the present invention is not limited thereto.

[0210] FIG. 8e is a diagram for explaining an operation of generating an AI image based on text input and drawing input in a flexible electronic device according to one embodiment.

[0211] Referring to FIG. 8e, the electronic device (101) may be a flexible display device configured as illustrated in FIGS. 8a to 8d.

[0212] (a) of FIG. 8E illustrates a first state of the electronic device (101) (e.g., a state in which a display area having the smallest size is provided, such as the first area (830a) of FIG. 8A), and (b) and (c) of FIG. 8E illustrate a second state of the electronic device (101) (e.g., a state in which a display area having the largest size is provided, such as the area (830c) including the first area (830a) and the second area (830b) of FIG. 8C).

[0213] Referring to (a) of FIG. 8e, the electronic device (101) may display a screen similar to the screen illustrated in (a) of FIG. 5 in the first state. For example, the electronic device (101) may display a display area (510), a text area (520), or an input area (540) on the touchscreen display (310).

[0214] According to one embodiment, when text is entered based on a virtual keyboard (542) on an input area (540), the electronic device (101) may display the entered text (e.g., Good morning with sun) in the text area (520).

[0215] According to one embodiment, when the first graphic object (532) is selected, the electronic device (101) may display a drawing area (544). For example, the electronic device (101) may change at least a portion of the virtual keyboard (542) displayed in the input area (540) into a drawing area (544) as illustrated in (b) of FIG. 5. According to one embodiment, the drawing area (544) may be formed by making at least a portion of the virtual keyboard (542) translucent, opaque, or transparent. According to one embodiment, the drawing area (544) may be displayed so as to overlap at least a portion of the virtual keyboard (542). According to one embodiment, the drawing area (544) may be displayed so as not to overlap at least a portion of the virtual keyboard (542). For example, the drawing area (544) may be reduced so that the virtual keyboard (542) is displayed in a portion of the input area (540), and the virtual keyboard (542) may be formed in the remaining portion of the input area (540) secured by the reduction.

[0216] According to one embodiment, the electronic device (101) can change from a first state to a second state, and can utilize a larger size display area in the second state.

[0217] Referring to (b) of FIG. 8E, the electronic device (101) can utilize the additional display area (840) (e.g., the second area (830b) of FIG. 8C) in the second state. According to one embodiment, the electronic device (101) can display a drawing area (842) in the additional display area (840). Even when the electronic device (101) changes to the second state, the display of the virtual keyboard (542) on the input area (540) can be maintained. The electronic device (101) can receive text input based on the virtual keyboard (542) in the first state or the second state. The received text can be displayed in the text area (520).

[0218] According to one embodiment, the electronic device (101) can identify that a second graphic object (534) is selected while text is displayed in a text area (520) and a drawing input of a first user is received in a drawing area (842). Based on the selection of the second graphic object (534), the electronic device (101) can generate at least one AI image using a generative AI model based on the text entered in the text area (520) and the drawing input on the drawing area (842). The electronic device (101) can transmit the at least one generated AI image to an external electronic device according to the selection of the first user. For example, the electronic device (101) can transmit the at least one generated AI image to the external electronic device as a message (e.g., an image message) of the first user.

[0219] In one embodiment, the electronic device (101) may modify at least one generated AI image and transmit it to an external electronic device. For example, the electronic device (101) may receive additional input (e.g., text input or drawing input) from a first user, and modify at least one AI image using a generative AI model based on the at least one generated AI image and the additional input.

[0220] According to one embodiment, when there are multiple AI images generated, the electronic device (101) may receive at least one selection from the multiple AI images from the first user and transmit the selected at least one AI image to an external electronic device. Referring to (c) of FIG. 8E, based on the fact that the at least one AI image has been transmitted to the external electronic device, the electronic device (101) may display at least one AI image (e.g., AI image (512)) in the display area (510) and may not display the drawing area (842). Based on the fact that the drawing area (842) has not been displayed, the electronic device (101) may expand the size of the display area (510) and display the virtual keyboard (542) in the expanded input area (850). The expanded input area (850) may be a larger area than the input area (540) illustrated in (b) of FIG. 8E.

[0221] FIG. 9A is a perspective view of a front side of a flip-type electronic device according to one embodiment. FIG. 9B is a plan view of a rear side of the flip-type electronic device according to one embodiment.

[0222] Referring to FIGS. 9A and 9B, the electronic device (101) may include a first housing (910) (e.g., a first housing structure) including a first side member (913) (e.g., a side bezel) and a second housing (920) (e.g., a second housing structure) including a second side member (923) (e.g., a side bezel) that are foldably coupled to each other with respect to a folding axis (F) through at least one hinge device (940, 940-1) (e.g., a hinge module or a hinge structure). For example, the first housing (910) and the second housing (920) may be configured as a foldable housing (e.g., a housing structure). For example, the electronic device (101) may include a first display (930) (e.g., a flexible display, a foldable display, or a main display) arranged to be supported by a first housing (910) and a second housing (920). For example, the first housing (910) may include a first side (911) and a second side (912) facing in an opposite direction (e.g., a -z-axis direction) of the first side (911). For example, the second housing (920) may include a third side (921) and a fourth side (922) facing in an opposite direction (e.g., a -z-axis direction) of the third side (921). For example, the first housing (910) may include a first rear cover (914) coupled with a first side member (913). For example, the second housing (920) may include a second rear cover (924) coupled with a second side member (923). For example, when the electronic device (101) is in a fully unfolded first state (e.g., an unfolded state or an unfolded state), the first side (911) and the third side (921) may be operated so that they face substantially the same direction (e.g., a z-axis direction). For example, when the electronic device (101) is in a fully folded second state (e.g., a folded state or a folded state), the first side (911) and the third side (921) may face each other or face opposite directions.For example, the electronic device (101) may be operated to maintain a third state (e.g., an intermediate state) between the first state and the second state.

[0223] According to one embodiment, the electronic device (101) may include a receiver (901), at least one sensor module (904) (e.g., an ambient light sensor) and / or at least one first camera module (905) (e.g., an under display camera, UDC) disposed on a first side (911) of the first housing (910). For example, the electronic device (101) may include at least one key (906) disposed on a first side member (913). For example, the electronic device (101) may include a second display (931) (e.g., a sub-display), at least one second camera module (908) and / or a flash (909) disposed on a second side (912) of the first housing (910) (e.g., a first rear cover (914)). For example, the second display (931) may be disposed to be visible from the outside through at least a portion of the first rear cover (914). For example, the electronic device (101) may include a speaker (902), a microphone (903), or a connector port (907) arranged on the second side member (923). At least some of the aforementioned components may be arranged in the first housing (910) and / or the second housing (920).

[0224] According to one embodiment, the first display (930) (e.g., a flexible display) may include a first region (930a) (e.g., a first planar portion) corresponding to at least a portion of the first surface (911), a second region (930b) (e.g., a second planar portion) corresponding to at least a portion of the third surface (921), and a third region (930c) (e.g., a flexible portion) connecting the first region (930a) and the second region (930b) and allowing the electronic device (101) to be deformed in a second state (e.g., a folded state) and / or a third state (e.g., an intermediate state). For example, the third region (930c) may be positioned at a position that at least partially overlaps at least one hinge device (940, 940-1) when the first display (930) is viewed from above (e.g., in the z-axis direction). For example, the first display (930) may be arranged so that it is not visible from the outside in the second state by having the first side (911) and the third side (921) face each other (e.g., inward-fold type). For example, the first display (930) may be arranged so that it is visible from the outside in the second state by having the first side (911) and the third side (921) face each other in opposite directions (e.g., outward-fold type).

[0225] According to one embodiment, the electronic device (101) may, in a first state, perform operations based on the screens illustrated in FIGS. 5 to 7c described above using the first display (930).

[0226] According to one embodiment, the electronic device (101) can perform operations based on the screen illustrated in FIG. 9c using the second display (931) in the second state.

[0227] FIG. 9c is a diagram for explaining an operation of generating an AI image based on text input and drawing input in a flip-type electronic device according to one embodiment.

[0228] Referring to (a) of FIG. 9c, when a message is received from an external electronic device, the electronic device (101) may display a display area (952) displaying the received message (e.g., Hi, Moon) on the second display (931). According to one embodiment, the second display (931) may correspond to the touchscreen display (310) of FIG. 3.

[0229] According to one embodiment, the electronic device (101) may receive an input from a first user (e.g., a touch input on a display area (952)) based on a received message. The input from the first user may be an input requesting display of a virtual keyboard for message input. Based on the input from the first user, the electronic device (101) may output a screen, such as that illustrated in (b) of FIG. 9c, to the second display (931).

[0230] Referring to (b) of FIG. 9c, the electronic device (101) may display or not display the display area (952) in a reduced size based on the input of the first user, and may display the input area (960) and the text area (970). A virtual keyboard (962) may be displayed in the input area (960), and text (e.g., Good morning, Cindy) entered through at least one key of the virtual keyboard (962) may be displayed in the text area (970).

[0231] According to one embodiment, when text is input through at least one key of a virtual keyboard (962) and a first graphic object (964) is selected, the electronic device (101) can output a screen as shown in (c) of FIG. 9c to the second display (931).

[0232] Referring to (c) of FIG. 9C, the electronic device (101) may display a drawing area (972) based on the selection of the first graphic object (964). According to one embodiment, the electronic device (101) may change at least a portion of the virtual keyboard (962) to be translucent, opaque, or transparent, and display the drawing area (972) so as to overlap all or part of the changed at least a portion of the virtual keyboard (962). According to one embodiment, the electronic device (101) may also display the drawing area (972) so as not to overlap at least a portion of the virtual keyboard (962). For example, the electronic device (101) may reduce the virtual keyboard (962) so as to be displayed in a portion of the input area (960), and display the drawing area (972) in the remaining area of ​​the input area (960) secured by reducing the virtual keyboard (962).

[0233] According to one embodiment, the electronic device (101) can display the text area (970) in which the text is displayed as is.

[0234] According to one embodiment, the electronic device (101) may receive a drawing input of a first user in a drawing area (972). The electronic device (101) may identify that a second graphic object (974) is selected while text is displayed in a text area (970) and a drawing input of the first user is received in a drawing area (972). The selection of the second graphic object (974) may be based on a user input (e.g., a touch input) for the second graphic object (974). Based on the selection of the second graphic object (974), the electronic device (101) may generate at least one AI image using a generative AI model based on the text input in the text area (970) and the drawing input on the drawing area (972).

[0235] According to one embodiment, the electronic device (101) can transmit at least one generated AI image as a message according to the selection of the first user.

[0236] In one embodiment, the electronic device (101) may modify at least one generated AI image and transmit it to an external electronic device. For example, the electronic device (101) may receive additional input (e.g., text input) from a first user and modify at least one AI image using a generative AI model based on the at least one generated AI image and the additional input.

[0237] According to one embodiment, when there are multiple AI images generated, the electronic device (101) may select at least one of the multiple AI images from the first user and transmit the selected at least one AI image to an external electronic device. Referring to (d) of FIG. 9c, the electronic device (101) may output the display area (952) to the second display (931) based on the transmission of at least one AI image to the external electronic device. At least one AI image (e.g., AI image (980)) transmitted as a message may be displayed in the display area (952). According to one embodiment, the electronic device (101) may not display the drawing area (972) or the input area (960) where the virtual keyboard (962) is displayed on the second display (931).

[0238] FIG. 10A is a front perspective view of a foldable electronic device according to one embodiment.

[0239] Referring to FIG. 10A, the electronic device (101) may include a first housing (1010) (e.g., a first housing structure) including a first side member (1013) (e.g., a side bezel) and a second housing (1020) (e.g., a second housing structure) including a second side member (1023) (e.g., a side bezel) that are foldably coupled to each other with respect to a folding axis (F) through at least one hinge device (1040, 1040-1) (e.g., a hinge module or a hinge structure). For example, the first housing (1010) and the second housing (1020) may be configured as a foldable housing (e.g., a housing structure). For example, the electronic device (101) may include a display (1030) (e.g., a flexible display, a foldable display, or a main display) arranged to be supported by a first housing (1010) and a second housing (1020). For example, the first housing (1010) may include a first side (1011) and a second side facing in an opposite direction (e.g., a -z-axis direction) of the first side (1011). For example, the second housing (1020) may include a third side (1021) and a fourth side facing in an opposite direction (e.g., a -z-axis direction) of the third side (1021). For example, the first housing (1010) may include a first back cover (1014) coupled with a first side member (1013). For example, the second housing (1020) may include a second back cover coupled with a second side member (1023). For example, when the electronic device (101) is in a first fully unfolded state (e.g., an unfolded state or a spread state), the first side (1011) and the third side (1021) may be operated so that they face substantially the same direction (e.g., the z-axis direction). For example, when the electronic device (101) is in a second fully folded state (e.g., a folded state or a folded state), the first side (1011) and the third side (1021) may face each other or face opposite directions.For example, the electronic device (101) may be operated to maintain a third state (e.g., an intermediate state) between the first state and the second state.

[0240] According to one embodiment, the display (1030) (e.g., a flexible display) may include a first region (1030a) (e.g., a first planar portion) corresponding to at least a portion of a first surface (1011), a second region (1030b) (e.g., a second planar portion) corresponding to at least a portion of a third surface (1021), and a third region (1030c) (e.g., a flexible portion) connecting the first region (1030a) and the second region (1030b) and allowing the electronic device (101) to be deformed in a second state (e.g., a folded state) and / or a third state. For example, the third region (1030c) may be positioned to at least partially overlap at least one hinge device (1040, 1040-1) when the display (1030) is viewed from above (e.g., in the z-axis direction). For example, the display (1030) may be arranged so as not to be visible from the outside by having the first side (1011) and the third side (1021) face each other in the second state (e.g., inward-fold type). For example, the display (1030) may be arranged so as to be visible from the outside by having the first side (1011) and the third side (1021) face each other in opposite directions in the second state (e.g., outward-fold type). FIG. 10B is a perspective view of the front direction of a multi-foldable electronic device according to one embodiment.

[0241] Referring to FIG. 10b, the electronic device (101) may include a first housing (1061) (e.g., a first housing structure) and a second housing (1061) (e.g., a second housing structure) that are foldably coupled to each other with respect to a first folding axis (F1) through at least one hinge device (e.g., a hinge module or a hinge structure) and include a first side member.

[0242] According to one embodiment, the electronic device (101) may be configured with a foldable housing (1060) (e.g., a housing structure) including a first housing (1601), a second housing (1602), and a third housing (1603). The first housing (1601) and the second housing (1602) may be foldably coupled to each other through at least one hinge device (e.g., a hinge module or a hinge structure) with respect to a first folding axis (F1). The second housing (1602) and the third housing (1603) may be foldably coupled to each other through at least one hinge device (e.g., a hinge module or a hinge structure) with respect to a second folding axis (F2). The first housing (1601) and the second housing (1603) may each include a first side member (e.g., a side bezel) and a second side member.

[0243] According to one embodiment, the electronic device (101) may include a display (1050) (e.g., a flexible display, a foldable display, or a main display) arranged to be supported by a first housing (1601), a second housing (1602), and a third housing (1603). The display (1050) may correspond to the touchscreen display (310) of FIG. 3 and may be arranged on a first surface of each of the first housing (1601), the second housing (1602), and the third housing (1603).

[0244] According to one embodiment, a display (1050) (e.g., a flexible display) may include a first region (1050c) (e.g., a first planar portion) corresponding to at least a portion of a first surface, a second region (1050b) (e.g., a second planar portion) corresponding to at least a portion of the first surface and different from the first region (1050c), and a third region (1050a) (e.g., a third planar portion) corresponding to at least a portion of the first surface and different from the first region (1050c) or the second region (1050b).

[0245] According to one embodiment, the electronic device (101) may be operated in any one of a first state, a second state, a third state, or a fourth state. For example, the first state may represent a state in which the first housing (1601), the second housing (1602), and the third housing (1603) are fully unfolded in the same direction (e.g., an unfolded state or an unfolded state).

[0246] For example, the second state may represent a state in which the first housing (1601) and the third housing (1603) are folded (e.g., a folded state or a collapsed state) with respect to the second housing (1602). In the second state, at least two of the first housing (1601), the second housing (1602), and the third housing (1603) may face or contact each other.

[0247] For example, the third state may represent a state in which one of the first housing (1601) and the third housing (1603) is folded, and the other of the first housing (1601) and the third housing (1603) is fully unfolded.

[0248] For example, the fourth state may represent a state in which either the first housing (1601) or the third housing (1603) is folded at a set angle (e.g., 90 degrees), and the other of the first housing (1601) or the third housing (1603) is fully unfolded.

[0249] According to one embodiment, the state of the electronic device (101) is not limited to the four states described above and may have various states.

[0250] FIG. 10c is a drawing for explaining an operation of displaying a virtual keyboard in a foldable electronic device according to one embodiment.

[0251] According to one embodiment, when the electronic device (101) is a foldable electronic device as illustrated in FIG. 10a, it may display a screen as illustrated in FIG. 10c based on the message application being executed in a first state (e.g., an unfolded state or an unfolded state).

[0252] Referring to FIG. 10c, the electronic device (101) can display a display area (1070) and an input area (1080) based on which message application is executed.

[0253] According to one embodiment, a virtual keyboard may be displayed in the input area (1080). According to one embodiment, the virtual keyboard may be a split keyboard in which keys are displayed separately. For example, among the keys included in the virtual keyboard, a first group of keys may be displayed in a first area (1030a) within the input area (1080), and among the keys included in the virtual keyboard, a second group of keys may be displayed in a second area (1030b) within the input area (1080).

[0254] According to one embodiment, the display area (1070) may display at least one message transmitted and received between a first user and a second user. According to one embodiment, at least one image within the displayed at least one message may be selected by the first user or the second user. For example, an image (1072) included in the second user's message may be selected by the first user's input (1074) (e.g., a touch input, a voice input, or a multimodal input for the image (1072).

[0255] The electronic device (101) can analyze the image (1072) using a set AI model based on the first user's input (1074). Based on the analysis result, the electronic device (101) can identify that the image (1072) is an image related to a math problem. Based on the identification of the image (1072) as an image related to a math problem, the electronic device (101) can generate an engineering keyboard (1082).

[0256] According to one embodiment, the electronic device (101) may display an engineering keyboard (1082) in the input area (1080). For example, the electronic device (101) may display the engineering keyboard (1082) between keys separated on the left and right of the virtual keyboard. A user may perform text input based on the virtual keyboard and may input mathematical expressions or symbols based on the engineering keyboard (1082).

[0257] FIG. 10d is a drawing for explaining an operation of displaying a virtual keyboard in a multi-foldable electronic device according to one embodiment.

[0258] According to one embodiment, when the electronic device (101) is a multi-foldable electronic device as illustrated in FIG. 10b, it may display a screen as illustrated in FIG. 10d based on whether a message application is executed in a first state (e.g., an unfolded state or an unfolded state).

[0259] Referring to FIG. 10d, the electronic device (101) may display a screen similar to the screen of the foldable electronic device illustrated in FIG. 10c based on the execution of the message application. According to one embodiment, if the electronic device (101) is a multi-foldable device, the display method of the engineering keyboard (1082) and the virtual keyboard may be different from that illustrated in FIG. 10e.

[0260] According to one embodiment, the electronic device (101) may display an engineering keyboard (1082) in the input area (1080). For example, the electronic device (101) may display keys of a virtual keyboard (e.g., keys of a first group and keys of a second group) in the first area (1050c) and the third area (1050a), and display the engineering keyboard (1082) in the second area (1050b). For example, the electronic device (101) may display keys of a virtual keyboard in the first area (1050c) and the second area (1050b), and display the engineering keyboard (1082) in the third area (1050a). For example, the electronic device (101) may display keys of a virtual keyboard in the second area (1050b) and the third area (1050a), and display an engineering keyboard (1082) in the first area (1050c). A user may perform text input based on the virtual keyboard and input mathematical formulas or symbols based on the engineering keyboard (1082) on one screen.

[0261] FIG. 10e is a drawing for explaining an operation of displaying a drawing area in a foldable electronic device according to one embodiment.

[0262] Referring to FIG. 10e, when the electronic device (101) is a foldable electronic device as illustrated in FIG. 10a, a display area (1070), a text area (1090), or an input area (1080) may be displayed based on which message application is executed.

[0263] According to one embodiment, when text is entered based on a virtual keyboard included in an input area (1080), the electronic device (101) can display the entered text in a text area (1090).

[0264] According to one embodiment, the electronic device (101) may display a drawing area (1044) when a set input, such as a selection of a graphic object or a touch input to an input area (1040), occurs. The set input may be performed before text is input, or after text is input and the input text is displayed in the text area (1090).

[0265] According to one embodiment, the electronic device (101) may display the drawing area (1084) in a different area from the area where the virtual keyboard is displayed. For example, when the virtual keyboard is configured as a split keyboard, the electronic device (101) may display the drawing area (1084) between the separated keys of the virtual keyboard. For example, the electronic device (101) may display the drawing area (1084) in an area including an area where the keys of the first group of the virtual keyboard are not displayed in a first area (1030a) included in the input area (1080) and an area where the keys of the second group of the virtual keyboard are not displayed in a second area (1030b) included in the input area (1080). The drawing area (1084) may not overlap with at least one key of the virtual keyboard.

[0266] According to one embodiment, the electronic device (101) can provide convenience for a user to more intuitively perform text input and drawing input through one screen by displaying a virtual keyboard and a drawing area (1084) together.

[0267] FIG. 10f is a drawing for explaining an operation of displaying a drawing area in a multi-foldable electronic device according to one embodiment.

[0268] Referring to FIG. 10f, if the electronic device (101) is a multi-foldable electronic device as illustrated in FIG. 10b, a screen similar to the screen of the foldable electronic device illustrated in FIG. 10e may be displayed based on the execution of the message application. According to one embodiment, if the electronic device (101) is a multi-foldable device, the display method of the drawing area (1084) and the virtual keyboard may be different from FIG. 10e.

[0269] According to one embodiment, the electronic device (101) may display the drawing area (1084) in a different area from the area where the virtual keyboard is displayed. For example, when the virtual keyboard is configured as a split keyboard, the electronic device (101) may display the drawing area (1084) between the separated keys of the virtual keyboard. For example, the electronic device (101) may display the keys of the virtual keyboard in the first area (1050c) and the third area (1050a), and display the drawing area (1084) in the second area (1050b). For example, the electronic device (101) may display the keys of the virtual keyboard in the first area (1050c) and the second area (1050b), and display the drawing area (1084) in the third area (1050a). For example, the electronic device (101) may display keys of a virtual keyboard in the second area (1050b) and the third area (1050a), and display a drawing area (1084) in the first area (1050c). The drawing area (1084) may not overlap with at least one key of the virtual keyboard.

[0270] According to one embodiment, a user can perform text input based on a virtual keyboard on one screen and perform drawing input through a drawing area (1084).

[0271] FIG. 11 is a drawing for explaining an operation of switching a virtual keyboard in an electronic device according to one embodiment.

[0272] Referring to (a) of FIG. 11, the electronic device (101) may display an execution screen of a calculator application based on a user's selection. The execution screen of the calculator application may include an input area (1110), and the input area (1110) may include a virtual keyboard including a plurality of number keys, operation keys, or function keys.

[0273] According to one embodiment, the electronic device (101) may display a preview image based on a user input. The preview image may include an image acquired by a camera or image sensor (320) of the electronic device (101). The preview image may be displayed in a display area (1112) that does not overlap with an input area (1110) on the execution screen of the calculator application. For example, the preview image may be displayed as shown in (b) of FIG. 11 .

[0274] Referring to (b) of FIG. 11, the electronic device (101) may display a preview image (1120) in a display area (1112) where a calculation process or calculation result is output. The electronic device (101) may analyze the preview image (1120) using a set AI model. Based on the analysis result, the electronic device (101) may identify that the preview image (1120) includes an image related to a mathematical problem. According to one embodiment, the set AI model may be a machine learning model that analyzes an input image when an image is input and outputs the analysis result as text. Based on the identification that the preview image (1120) includes an image related to a mathematical problem, the electronic device (101) may change at least one key in the virtual keyboard area (1110) into at least one key (1111) including a symbol, formula, or formula name for solving a mathematical problem, thereby generating an engineering keyboard. Symbols, formulas, or formula names for solving mathematical problems may be associated with images or text generated using a generative AI model based on the output values ​​of a set AI model.

[0275] In one embodiment, when a user performs a calculation on a math problem using an engineering keyboard, the electronic device (101) may use a generative AI model to generate an associated image (e.g., an image including a solution process for the math problem or an image showing a portion of the solution process).

[0276] In one embodiment, the electronic device (101) can generate a video containing at least a portion of the problem-solving process using a generative AI model, using user input and images as input values. For example, the generated video may include a scene of a virtual instructor solving a math problem.

[0277] In one embodiment, the user input may be performed in a calculator application and another application (e.g., a notepad or note application), and the input image may be an image stored in the electronic device (101) or an image transmitted from an external server or external electronic device, as well as a preview image (1120).

[0278] Hereinafter, various operations of the electronic device (101) will be described with reference to FIGS. 12 to 17. According to one embodiment, the operations illustrated in FIGS. 12 to 17 may be understood to be performed by the processor (340) of the electronic device (101). The operations illustrated in each of FIGS. 12 to 17 are not limited to the illustrated order and may be performed in various orders. According to one embodiment, at least some of the operations illustrated in each of FIGS. 12 to 17 may be omitted, or more operations may be performed than the operations illustrated in each of FIGS. 12 to 17.

[0279] The terms mentioned in the descriptions of FIGS. 12 to 17 may correspond to the terms described above, and therefore, a detailed description thereof is omitted.

[0280] According to one embodiment, the message input window or text area is an area where text entered through at least one key of a virtual keyboard is displayed, and may correspond to the text area (520) of FIGS. 5 and 8E, the text area (970) of FIG. 9C, or the text area (1090) of FIGS. 10E and 10F.

[0281] According to one embodiment, the virtual keyboard is an input interface including at least one key (or virtual key) (e.g., a letter key, a number key, or a function key), which may correspond to the virtual keyboard (542) of FIGS. 5 to 7A, FIG. 8E, the virtual keyboard (962) of FIG. 9C, the virtual keyboard (or split keyboard) of FIGS. 10C to 10F, or the virtual keyboard of FIG. 11.

[0282] According to one embodiment, the drawing area or semi-transparent area is an area where a user's drawing input is received, and may correspond to the drawing area (544) of (b) of FIG. 5, the drawing area (544) of (b) of FIG. 6, the drawing area (842) of (b) of FIG. 8e, the drawing area (972) of (c) of FIG. 9c, or the drawing area (1084) of FIGS. 10e and 10f.

[0283] According to one embodiment, the input area is an area where a virtual keyboard and / or a drawing area is displayed, and may correspond to the input area (510) of FIGS. 5 to 8, the input area (960) of (a) and (d) of FIG. 9c, the input area (1080) of FIGS. 10e and 10f, or the input area (1110) of FIG. 11.

[0284] According to one embodiment, the display area may correspond to the display area (510) of FIGS. 5 to 7A, FIG. 8E, the display area (952) of (a) and (d) of FIG. 9C, the display area (1070) of FIGS. 10C to 10F, or the display area (1112) of FIG. 11.

[0285] FIG. 12 is a flowchart illustrating an operation of an electronic device generating an AI image according to one embodiment.

[0286] Referring to FIG. 12, in operation 1202, the electronic device (101) may display a user interface (UI) including a message input window (or text area) and a virtual keyboard. According to one embodiment, the UI may include a UI or GUI associated with application execution. According to one embodiment, the electronic device (101) may display the message input window and the virtual keyboard in non-overlapping areas on the touchscreen display (310). For example, the electronic device (101) may display the virtual keyboard in the lowest area of ​​the touchscreen display (310) and display the message input window in the upper area of ​​the area where the virtual keyboard is displayed.

[0287] In operation 1204, the electronic device (101) may receive text input via a virtual keyboard. In one embodiment, the input text may include text input by a user based on at least one key of the virtual keyboard. In one embodiment, the input text may also include text based on a user's voice or gesture input, or multimodal input.

[0288] In operation 1206, the electronic device (101) may display the entered text in the message input window based on the received entered text.

[0289] In operation 1208, the electronic device (101) may receive a user input for drawing input while the input text is displayed. For example, the user input may include an input for a set graphic object (e.g., a touch input for the first graphic object (532) of FIG. 5 (a), FIG. 6 (a), or FIG. 8e (a), a touch input for the first graphic object (964) of FIG. 9c (b)), or a touch input on an input area where a virtual keyboard is displayed (e.g., a touch input for the input area (1080) of FIG. 10c, FIG. 10d, FIG. 10e, or FIG. 10f). However, the user input is not limited thereto, and may include various user inputs such as an input for at least one key of the virtual keyboard, a voice input, a gesture input, or a multimodal input. In operation 1210, the electronic device (101) may display a drawing area based on the received user input. According to one embodiment, the electronic device (101) may change at least a portion of the virtual keyboard to be translucent, opaque, or transparent, and display the drawing area so as to overlap all or part of the changed at least portion of the area. According to one embodiment, the electronic device (101) may also display the drawing area so as not to overlap at least a portion of the virtual keyboard. For example, the electronic device (101) may reduce the virtual keyboard (962) and display the drawing area in the area secured by the reduction of the virtual keyboard (962).

[0290] In operation 1212, the electronic device (101) may receive a drawing input on a drawing area. According to one embodiment, the drawing input may include a drawing input (or handwriting input) by a user's body (e.g., a hand or a finger) or an input device (e.g., a stylus pen).

[0291] In operation 1214, the electronic device (101) may receive a user input for generating at least one AI image. According to one embodiment, the user input for generating at least one AI image may include an input for a set graphic object (e.g., a touch input for the second graphic object (534) of (b) of FIG. 5, (b) of FIG. 6, or (b) of FIG. 8e, or a touch input for the second graphic object (974) of (c) of FIG. 9c). However, the user input is not limited thereto, and may include various user inputs such as an input for at least one key of a virtual keyboard, a voice input, a gesture input, or a multimodal input.

[0292] In operation 1216, the electronic device (101) may generate at least one AI image using a generative AI model (e.g., the generative AI model (422) of FIG. 4) based on a text input and a drawing input. According to one embodiment, the generative AI model may include a machine learning model trained or learned to generate at least one image based on at least one of text, an image, a drawing, or characteristic information (e.g., gender, age, or preference information) or environmental information (e.g., location information) of at least one user (e.g., a first user who is a user of the electronic device (101) and / or a second user who is a user of an external electronic device). The generative AI model may be an on-device AI model stored in the electronic device (101) or an AI model stored in an external server. The at least one AI image generated via the generative AI model may include a still image, an animated image (e.g., a GIF image), or a video.

[0293] In operation 1218, the electronic device (101) may display at least one AI image in a drawing area or a portion of the UI. According to one embodiment, the portion of the UI may include at least a portion of the display area, or an area in the form of a pop-up, and may or may not overlap with another area, such as the drawing area.

[0294] According to one embodiment, the electronic device (101) may display a plurality of AI images as candidate images when there are at least one AI image and may receive selection of at least one of the candidate images from the first user or the second user.

[0295] According to one embodiment, the electronic device (101) may modify at least one AI image based on additional input from the first user or the second user. For example, the additional input may include, but is not limited to, text input and / or drawing input, and may also include various inputs such as voice input, gesture input, or multimodal input.

[0296] FIG. 13 is a flowchart illustrating an operation of an electronic device according to one embodiment of the present invention to create a graphic object corresponding to at least one key to be included in a virtual keyboard.

[0297] Referring to FIG. 13, in operation 1302, the electronic device (101) may display a UI including text or images (e.g., one or more images). According to one embodiment, the UI may include a UI or GUI associated with application execution.

[0298] In one embodiment, when the application is a calculator application, the UI may include an input UI and an output UI. For example, the input UI of the calculator application may include a virtual keyboard including at least one of number keys, operation keys (e.g., keys for addition (+), subtraction (-), multiplication (x), or division (÷)), or function keys (e.g., keys for outputting a result, clearing, displaying a decimal point, or inputting or capturing an image). For example, the output UI of the calculator application may include a display area in which at least one number is output based on at least one key input of the virtual keyboard, a calculation process or a calculation result is output, or a function execution result is displayed. In one embodiment, the display area may display text (e.g., at least one number) or at least one image (e.g., an image input by a user or captured by an image sensor) based on at least one key input of the virtual keyboard.

[0299] In one embodiment, if the application is a messaging application, the UI may include an input UI and an output UI. For example, the input UI of the messaging application may include a virtual keyboard including at least one of a character key (e.g., a Korean or English key), a special character key, a case switching key, a character deletion key, an emoticon input key, or an image attachment key. For example, the output UI of the messaging application may include a display area on which text messages or image messages exchanged between conversation users are displayed. A text message may include at least one character input by a user based on at least one key of the virtual keyboard, or at least one character received from a counterpart user. An image message may include at least one image input by the user or received from a counterpart user. In operation 1304, the electronic device (101) may analyze text or images using a first AI model. In one embodiment, the first AI model may be a machine learning model trained to analyze input text or images when text or images are input, and output the analysis results as text. Analysis of text or images may be performed based on user input.

[0300] In operation 1306, the electronic device (101) may receive a user input for displaying a virtual keyboard. According to one embodiment, the electronic device (101) may display the virtual keyboard by obtaining a user input, such as a user's voice input or a user input in which the user selects a user-selectable object (e.g., text, an icon, or a button-type menu) provided by the electronic device (101).

[0301] In operation 1308, the electronic device (101) may generate at least one key (e.g., at least one key including a symbol, a formula, or a formula name) of a virtual keyboard using a second AI model based on an analysis result of text or an image (e.g., the text or image is associated with a mathematical formula) through the first AI model. According to one embodiment, the second AI model may be a generative AI model and may be a machine learning model trained to output an image or text using an output result (e.g., a text or image analysis result) of the first AI model as an input value. According to one embodiment, the first AI model or the second AI model may be implemented as one model. According to one embodiment, at least one of the first AI model and the second AI model may be an on-device AI model included in the electronic device (101) or an AI model formed on an external server.

[0302] In operation 1310, the electronic device (101) may display a virtual keyboard including at least one key generated using the second AI model. In one embodiment, the at least one key may include at least one key including at least one image or text.

[0303] In operation 1312, the electronic device (101) may receive a user input associated with at least one key. For example, the at least one key may be at least one key including at least one image (e.g., a shape image or an emoticon image).

[0304] In operation 1314, the electronic device (101) may generate a graphical object (e.g., a shape or emoticon) corresponding to at least one key using the second AI model. For example, the electronic device (101) may generate a graphical object associated with an image included in at least one key.

[0305] In operation 1316, the electronic device (101) may display a graphic object. For example, the electronic device (101) may display the graphic object in a display area, an area that at least partially overlaps or does not overlap an area where an input UI (e.g., a virtual keyboard) is displayed, or an area in a pop-up form.

[0306] According to one embodiment, the electronic device (101) can generate various types of virtual keyboards in addition to a virtual keyboard for displaying graphic objects. For example, the electronic device (101) can generate an engineering keyboard, a chemical formula keyboard, a shape keyboard, a font keyboard, or a musical score keyboard according to the user's selection, thereby enabling accurate, fast, and easy input in various situations.

[0307] FIG. 14A is a flowchart illustrating an operation of an electronic device according to one embodiment of the present invention to generate and display at least one image based on a drawing input.

[0308] Referring to FIG. 14A, in operation 1402, the electronic device (101) may display a virtual keyboard in the input area of ​​the touchscreen display (310).

[0309] In operation 1404, the electronic device (101) may receive a user input associated with the virtual keyboard. According to one embodiment, the user input associated with the virtual keyboard may include a touch input to a graphical object displayed at a location associated with the virtual keyboard (e.g., the first graphical object (532) of (a) of FIG. 5 , (a) of FIG. 6 , or (a) of FIG. 8E , or the first graphical object (964) of (b) of FIG. 9C ). According to one embodiment, the user input may include other inputs in addition to the touch input. For example, the user input may include a set key input on the virtual keyboard, a voice input, a gesture input, or a multimodal input. For example, the user input may include a user input for selecting a user-selectable object (e.g., text, an icon, or a button-type menu) provided by the electronic device (101).

[0310] In operation 1406, the electronic device (101) may change at least a portion of the virtual keyboard into a drawing area and display it based on user input.

[0311] According to one embodiment, the electronic device (101) can change the appearance of the virtual keyboard in a manner that obscures at least a portion of the virtual keyboard, and display the obscured area created by obscuring at least a portion of the virtual keyboard as a drawing area.

[0312] According to one embodiment, the electronic device (101) can change at least a portion of a virtual keyboard into a translucent area, and display at least a portion of the virtual keyboard changed into a translucent area as a drawing area.

[0313] According to one embodiment, the electronic device (101) can change at least a portion of a virtual keyboard into an opaque area or a transparent area, and display at least a portion of the virtual keyboard changed into an opaque area or a transparent area as a drawing area.

[0314] According to one embodiment, the electronic device (101) may display a drawing area by overlapping at least a portion of a virtual keyboard. For example, the electronic device (101) may display a transparent or translucent drawing area by overlapping the opaque area of ​​the virtual keyboard while changing at least a portion of the virtual keyboard to an opaque area. For example, the electronic device (101) may display an opaque or translucent drawing area by overlapping the transparent area or the opaque area of ​​the virtual keyboard while changing at least a portion of the virtual keyboard to a transparent area or an opaque area.

[0315] According to one embodiment, the electronic device (101) can reduce the virtual keyboard and display a drawing area in the area secured by reducing the virtual keyboard.

[0316] According to one embodiment, the electronic device (101) may be formed in at least a portion of the display area in a pop-up form, or may be formed over the entire display area.

[0317] According to one embodiment, the drawing area is not limited to being displayed in the manners described above, but may be displayed in various ways depending on the implementation.

[0318] In operation 1408, the electronic device (101) may receive a drawing input in a drawing area. According to one embodiment, the drawing input may include a drawing input or handwriting input by a user's body (e.g., a hand or finger) or an input device (e.g., a stylus pen). For example, the drawing input may include a drawing image or handwriting drawn by the user.

[0319] In operation 1410, the electronic device (101) may generate at least one image based on a drawing input using a machine learning model. According to one embodiment, the machine learning model may be a generative AI model (e.g., the generative AI model (422) of FIG. 4) and may include a model trained to generate an image based on at least one of text, an image, a drawing input, or characteristic information (e.g., gender, age, or preference information) or environmental information (e.g., location information) of at least one user (e.g., a first user who is a user of the electronic device (101) and / or a second user who is a user of an external electronic device). The at least one image generated using the machine learning model may include a still image, an animated image (e.g., a GIF image), or a video.

[0320] In operation 1412, the electronic device (101) may display at least one image in a display area of ​​the touchscreen display (310). According to one embodiment, the at least one image may be displayed in addition to the display area, at least a portion of the input area, or in a pop-up area.

[0321] FIG. 14b is a flowchart illustrating an operation of an electronic device according to one embodiment of the present invention to select or modify and display at least one of a plurality of images.

[0322] According to one embodiment, the operations illustrated in FIG. 14b may be detailed operations of operations 1410 and 1412 of FIG. 14a. According to one embodiment, operation 1422 of FIG. 14b may be performed after operation 1408 of FIG. 14a or may correspond to operation 1410.

[0323] Referring to FIG. 14B, in operation 1422, the electronic device (101) may generate at least one image based on a drawing input using a machine learning model. According to one embodiment, the machine learning model may be a generative AI model (e.g., the generative AI model (422) of FIG. 4) and may include a model trained to generate an image based on at least one of text, an image, a drawing input, or characteristic information or environmental information of at least one user. The at least one image generated using the machine learning model may include a still image, an animated image, or a video.

[0324] In operation 1424, the electronic device (101) can determine whether multiple images have been generated through the generative AI model.

[0325] According to one embodiment, if multiple images are not generated through the generative AI model or if one image is generated, the electronic device (101) may determine whether to modify the generated image based on a user input in operation 1430. For example, based on the receipt of a user input indicating not to modify the generated image, the electronic device (101) may display the generated image on the touchscreen display (310) in operation 1434. For example, the electronic device (101) may display the generated image in the display area of ​​the touchscreen display (310), at least a portion of the input area, or a pop-up area. For example, the electronic device (101) may perform an image modification operation in operation 1432 based on the receipt of a user input indicating to modify the generated image. For example, the electronic device (101) may modify the generated image using the generative AI model based on an additional user input (e.g., text input or drawing input). The electronic device (101) can display the modified image on the touchscreen display (310) at operation 1434 based on the completion of the modification.

[0326] According to one embodiment, when multiple images are generated through a generative AI model, the electronic device (101) may display the multiple images in operation 1426. For example, the electronic device (101) may display the multiple images in an input area, a drawing area, or a pop-up area.

[0327] In operation 1428, the electronic device (101) may determine whether at least one image among a plurality of images has been selected by the user. For example, the user's input for selecting at least one image may include a touch input, a voice input, a gesture input, or a multimodal input associated with at least one of the plurality of images.

[0328] According to one embodiment, the electronic device (101) may wait for a user input for a set period of time in operation 1428 if at least one image among the plurality of images is not selected by the user. If no user input is received for the set period of time, the electronic device (101) may terminate all operations or randomly select at least one image by considering the user's characteristic information or environmental information.

[0329] According to one embodiment, the electronic device (101) may determine whether to modify at least one image based on a user input in operation 1430 when at least one image among a plurality of images is selected by the user.

[0330] According to one embodiment, the electronic device (101) may display at least one image on the touchscreen display (310) at operation 1434 based on receiving a user input indicating not to modify at least one image.

[0331] According to one embodiment, the electronic device (101) may perform an image modification operation at operation 1432 based on receiving a user input instructing to modify at least one image. For example, the electronic device (101) may modify the at least one image using a generative AI model based on additional user input (e.g., text input or drawing input). Based on completion of modification of the at least one image, the electronic device (101) may display the at least one modified image on the touchscreen display (310) at operation 1434.

[0332] According to one embodiment, the electronic device (101) can further modify at least one image displayed on the touchscreen display (310) according to a user's selection, and at least one image can also be modified by a user of an external electronic device.

[0333] According to one embodiment, the electronic device (101) may generate a new image by combining or summing the features of at least two candidate images through a generative AI model when at least two of the plurality of images are selected.

[0334] FIG. 15 is a flowchart illustrating an operation of an electronic device according to one embodiment of the present invention to generate and display at least one image based on text input and drawing input.

[0335] Referring to FIG. 15, in operation 1502, the electronic device (101) may display a virtual keyboard in the input area of ​​the touchscreen display (310).

[0336] In operation 1504, the electronic device (101) may receive a first user input for at least one key of the virtual keyboard. According to one embodiment, the first user input may be for a user to input text using at least one key of the virtual keyboard, and may include a touch input for at least one key of the virtual keyboard.

[0337] In operation 1506, the electronic device (101) may display text entered through the first user input in a text area of ​​the touchscreen display (310).

[0338] In operation 1508, the electronic device (101) may receive a second user input associated with the virtual keyboard. According to one embodiment, the second user input associated with the virtual keyboard may include a touch input to a graphical object displayed at a location associated with the virtual keyboard (e.g., the first graphical object (532) of FIG. 5(a), FIG. 6(a), or FIG. 8E(a), or the first graphical object (964) of FIG. 9C(b)). According to one embodiment, the second user input may include other inputs in addition to the touch input. For example, the second user input may include a set key input on the virtual keyboard, a voice input, a gesture input, or a multimodal input. For example, the second user input may include a user input for selecting a user-selectable object (e.g., text, an icon, or a button-type menu) provided by the electronic device (101).

[0339] In operation 1510, the electronic device (101) may change at least a portion of the virtual keyboard into a drawing area and display it based on the second user input.

[0340] According to an embodiment, the electronic device (101) can change the appearance of the virtual keyboard in a way that covers at least a portion of the virtual keyboard, and display the covered area created by covering at least a portion of the virtual keyboard as a drawing area.

[0341] According to one embodiment, the electronic device (101) can change at least a portion of a virtual keyboard into a translucent area, and display at least a portion of the virtual keyboard changed into a translucent area as a drawing area.

[0342] According to one embodiment, the electronic device (101) can change at least a portion of a virtual keyboard into an opaque area or a transparent area, and display at least a portion of the virtual keyboard changed into an opaque area or a transparent area as a drawing area.

[0343] According to one embodiment, the electronic device (101) may display a drawing area by overlapping at least a portion of a virtual keyboard. For example, the electronic device (101) may display a transparent or translucent drawing area by overlapping the opaque area of ​​the virtual keyboard while changing at least a portion of the virtual keyboard to an opaque area. For example, the electronic device (101) may display an opaque or translucent drawing area by overlapping the transparent area or the opaque area of ​​the virtual keyboard while changing at least a portion of the virtual keyboard to a transparent area or an opaque area.

[0344] According to one embodiment, the electronic device (101) can reduce the virtual keyboard and display a drawing area in the area secured by reducing the virtual keyboard.

[0345] According to one embodiment, the electronic device (101) may be formed in at least a portion of the display area in a pop-up form, or may be formed over the entire display area.

[0346] According to one embodiment, the drawing area is not limited to being displayed in the manners described above, but may be displayed in various ways depending on the implementation.

[0347] In operation 1512, the electronic device (101) may receive a drawing input in a drawing area. According to one embodiment, the drawing input may include a drawing input or handwriting input by a user's body (e.g., a hand or a finger) or an input device (e.g., a stylus pen). For example, the drawing input may include a drawing image or handwriting drawn by the user.

[0348] In operation 1514, the electronic device (101) may generate at least one image using a machine learning model based on text displayed in a text area and a drawing input. According to one embodiment, the machine learning model may be a generative AI model (e.g., the generative AI model (422) of FIG. 4) and may include a model trained to generate an image based on at least one of text, an image, a drawing input, or characteristic information (e.g., gender, age, or preference information) or environmental information (e.g., location information) of at least one user (e.g., a first user who is a user of the electronic device (101) and / or a second user who is a user of an external electronic device). The at least one image generated using the machine learning model may include a still image, an animated image (e.g., a GIF image), or a video.

[0349] In operation 1516, the electronic device (101) may display at least one image in a display area of ​​the touchscreen display. According to one embodiment, the at least one image may be displayed in addition to the display area, at least a portion of the input area, or in a pop-up area.

[0350] According to one embodiment, when a plurality of images are generated through a generative AI model, the electronic device (101) may, based on operations such as those illustrated in FIG. 14B, allow a user to select at least one of the plurality of images or modify at least one image. For example, the electronic device (101) may modify at least one image using the generative AI model based on additional input from the user (e.g., text input or drawing input).

[0351] FIG. 16 is a flowchart illustrating an operation of an electronic device according to one embodiment of the present invention to generate and display at least one image based on text and drawing input included in content.

[0352] Referring to FIG. 16, in operation 1602, the electronic device (101) may display a virtual keyboard in an input area of ​​a touchscreen display (310) and display text in a display area. According to one embodiment, the text displayed in the display area may include text included in content (e.g., a web page, a document, or an email).

[0353] In operation 1604, the electronic device (101) may receive a first user input for selecting at least a portion of text displayed in a display area. In one embodiment, the first user input may include an input for indicating at least one word or sentence corresponding to at least a portion of the displayed text. For example, the first user input may include a touch or drag input, a voice input, a gesture input, or a multimodal input for indicating at least a portion of the displayed text. In one embodiment, at least a portion of the text selected by the first user input may be indicated based on a visual effect, such as an underline, a shadow, a magnification, or a highlight.

[0354] In operation 1606, the electronic device (101) may receive a second user input associated with the virtual keyboard based on at least a portion of the text being selected by the first user input. According to one embodiment, the second user input associated with the virtual keyboard may include a touch input to a graphical object displayed at a location associated with the virtual keyboard (e.g., the first graphical object (532) of FIG. 5(a), FIG. 6(a), or FIG. 8E(a), or the first graphical object (964) of FIG. 9C(b)). According to one embodiment, the second user input may include other inputs in addition to the touch input. For example, the second user input may include a set key input on the virtual keyboard, a voice input, a gesture input, or a multimodal input. For example, the second user input may include a user input for selecting a user-selectable object (e.g., text, an icon, or a button-type menu) provided by the electronic device (101).

[0355] In operation 1608, the electronic device (101) may change and display at least a portion of the virtual keyboard into a drawing area based on the second user input.

[0356] According to an embodiment, the electronic device (101) can change the appearance of the virtual keyboard in a manner that obscures at least a portion of the virtual keyboard, and display the obscured area created by obscuring at least a portion of the virtual keyboard as a drawing area.

[0357] According to one embodiment, the electronic device (101) can change at least a portion of a virtual keyboard into a translucent area, and display at least a portion of the virtual keyboard changed into a translucent area as a drawing area.

[0358] According to one embodiment, the electronic device (101) can change at least a portion of a virtual keyboard into an opaque area or a transparent area, and display at least a portion of the virtual keyboard changed into an opaque area or a transparent area as a drawing area.

[0359] According to one embodiment, the electronic device (101) may display a drawing area by overlapping at least a portion of a virtual keyboard. For example, the electronic device (101) may display a transparent or translucent drawing area by overlapping the opaque area of ​​the virtual keyboard while changing at least a portion of the virtual keyboard to an opaque area. For example, the electronic device (101) may display an opaque or translucent drawing area by overlapping the transparent area or the opaque area of ​​the virtual keyboard while changing at least a portion of the virtual keyboard to a transparent area or an opaque area.

[0360] According to one embodiment, the electronic device (101) can reduce the virtual keyboard and display a drawing area in the area secured by reducing the virtual keyboard.

[0361] According to one embodiment, the electronic device (101) may be formed in at least a portion of the display area in a pop-up form, or may be formed over the entire display area.

[0362] According to one embodiment, the drawing area is not limited to being displayed in the manners described above, but may be displayed in various ways depending on the implementation.

[0363] In operation 1610, the electronic device (101) may receive a drawing input in a drawing area. According to one embodiment, the drawing input may include a drawing input or handwriting input by a user's body (e.g., a hand or finger) or an input device (e.g., a stylus pen). For example, the drawing input may include a drawing image or handwriting drawn by the user.

[0364] In operation 1612, the electronic device (101) may generate at least one image using a machine learning model based on at least a portion of text and a drawing input. According to one embodiment, the machine learning model may be a generative AI model (e.g., the generative AI model (422) of FIG. 4) and may include a model trained to generate an image based on at least one of text, an image, a drawing input, or characteristic information (e.g., gender, age, or preference information) or environmental information (e.g., location information) of at least one user (e.g., a first user who is a user of the electronic device (101) and / or a second user who is a user of an external electronic device). The at least one image generated using the machine learning model may include a still image, an animated image (e.g., a GIF image), or a video.

[0365] In operation 1614, the electronic device (101) may display at least one image in the display area. According to one embodiment, the at least one image may be displayed in addition to the display area, at least a portion of the input area, or an area in the form of a pop-up.

[0366] According to one embodiment, when a plurality of images are generated through a generative AI model, the electronic device (101) may, based on operations such as those illustrated in FIG. 14B, allow a user to select at least one of the plurality of images or modify at least one image. For example, the electronic device (101) may modify at least one image using the generative AI model based on additional input from the user (e.g., text input or drawing input).

[0367] FIG. 17 is a flowchart illustrating an operation of an electronic device according to one embodiment of the present invention to generate and display at least one image based on a preview image.

[0368] Referring to FIG. 17, in operation 1702, the electronic device (101) may display a virtual keyboard in an input area of ​​the touchscreen display (310). According to one embodiment, the virtual keyboard may be displayed based on the execution of an application and may include at least one key for an input operation associated with the application. For example, when a calculator application is executed, the virtual keyboard may include at least one of number keys, operation keys, or function keys (e.g., keys for a result output function, a delete function, a decimal point display, or keys for image input or shooting). For example, when a message or text editing application is executed, the virtual keyboard may include at least one of a character key (e.g., a Korean or English key), a special character key, a case switching key, a character deletion key, or an image attachment key.

[0369] In operation 1704, the electronic device (101) may obtain a preview image (e.g., the preview image (1120) of (b) of FIG. 11) through the image sensor (310) (or camera). According to one embodiment, the electronic device (101) may also obtain an image received from an external server or an external electronic device as the preview image.

[0370] In operation 1706, the electronic device (101) may display a preview image in a display area of ​​the touchscreen display (310). According to one embodiment, the preview image may be displayed in addition to the display area, at least a portion of the input area, or in a pop-up area.

[0371] In operation 1708, the electronic device (101) may generate at least one graphical object using a machine learning model based on at least one object (e.g., an object, a person, an animal, a shape, a language, or a symbol) included in a preview image. According to one embodiment, the machine learning model may be a generative AI model (e.g., the generative AI model (422) of FIG. 4) and may include a model trained to analyze at least one input image when at least one image is input and generate the analysis result as text or at least one graphical object. For example, the electronic device (101) may generate at least one graphical object (e.g., an icon indicating a formula or a mathematical symbol) using the machine learning model based on a formula or a mathematical symbol included in the preview image.

[0372] In operation 1710, the electronic device (101) may display at least one graphic object in a portion of the virtual keyboard. According to one embodiment, the electronic device (101) may display at least one graphic object in a portion of the virtual keyboard based on a user's selection. For example, the electronic device (101) may display an icon indicating a formula or mathematical symbol in a portion of the virtual keyboard, thereby enabling the user to easily and conveniently input a formula or mathematical symbol by selecting the displayed icon.

[0373] An electronic device according to one embodiment of the present disclosure may include a touchscreen display; at least one processor including a processing circuit; and a memory including at least one storage medium storing instructions.

[0374] According to one embodiment, the instructions, when executed individually or collectively by at least one processor, may cause the electronic device to: display a virtual keyboard in a first area via the touchscreen display (310); receive, via the touchscreen display (310), a first user input associated with the virtual keyboard; change an appearance of the virtual keyboard in a manner that obscures at least a portion of the virtual keyboard based on the first user input; receive, via the touchscreen display (310), a first drawing input in the obscured area; generate, using a machine learning model (422), at least one image based on the first drawing input, the machine learning model (422) comprising a model trained to generate an image based on at least one of text, an image, or a drawing input; and display, via the touchscreen display (310), the at least one image in a second area different from the first area.

[0375] According to one embodiment, the electronic device further includes an image sensor (320), and the memory (330) may cause the electronic device (101) to: display, through the touchscreen display (310), a preview image (1120) acquired through the image sensor (320) on at least a portion of the second area; detect at least one object included in the preview image (1120); generate at least one key using the machine learning model (422) based on the at least one object; and display, through the touchscreen display (310), the at least one key on at least a portion of the virtual keyboard.

[0376] According to one embodiment, the memory (330), when individually or collectively executed by the at least one processor (340), may cause the electronic device (101) to: display an image provided from an external electronic device on at least a portion of the second area through the touchscreen display (310); generate at least one key using the machine learning model (422) based on the image; and display the at least one key on at least a portion of the virtual keyboard through the touchscreen display (310).

[0377] According to one embodiment, the memory (330), when individually or collectively executed by the at least one processor (340), may cause the electronic device (101) to: receive, through the touchscreen display (310), a second user input for the virtual keyboard; display, through the touchscreen display (310), a first text corresponding to the second user input in a third area; and generate, using the machine learning model (422), the at least one image based on the first text and the first drawing input.

[0378] According to one embodiment, the electronic device further includes a communication circuit, and the memory (330) is configured to cause the electronic device (101) to: display, through the touchscreen display (310), at least one graphical object indicating message transmission at a location adjacent to the third area; receive a third user input for the at least one graphical object after displaying the first text in the third area; and transmit, based on at least a portion of the third user input, a message including the first text to an external electronic device through the communication circuit, and display, through the touchscreen display (310), the first text in the second area.

[0379] According to one embodiment, the memory (330), when individually or collectively executed by the at least one processor (340), may cause the electronic device (101) to: receive, via the touchscreen display (310), a fourth user input for the at least one image, and in response to the fourth user input, transmit the at least one image.

[0380] According to one embodiment, the memory (330), when individually or collectively executed by the at least one processor (340), may cause the electronic device (101) to: transmit the at least one image to the external electronic device via the communication circuit, and to display the at least one image together with the first text in the second area via the touchscreen display (310).

[0381] According to one embodiment, the memory (330), when individually or collectively executed by the at least one processor (340), may cause the electronic device (101) to: display another text in the second area via the touchscreen display (310), receive a fifth user input for the second text via the touchscreen display (310), and generate the at least one image based on the second text and the first drawing input.

[0382] According to one embodiment, the at least one image includes a first image and a second image, and the memory (330) can cause the electronic device (101) to: display, through the touchscreen display (310), the first image and the second image on at least a portion of the first area; receive, through the touchscreen display (310), a sixth user input with respect to the first image; and store, based on at least a portion of the sixth user input, the first image in the memory (330).

[0383] According to one embodiment, the memory (330), when individually or collectively executed by the at least one processor (340), may cause the electronic device (101) to: receive, via the touchscreen display (310), a seventh user input for the at least one first image on the second area; receive a second drawing input in the obscured area; generate, based on at least a portion of the second drawing input, at least one third image using the machine learning model (422); and display, via the touchscreen display (310), the at least one third image in the second area.

[0384] According to one embodiment, a method of operating an electronic device (101) comprises: an operation (1402) of displaying a virtual keyboard in a first area through a touchscreen display (310) included in the electronic device (101); an operation (1404) of receiving a first user input associated with the virtual keyboard through the touchscreen display (310); an operation (1406) of changing the appearance of the virtual keyboard in a manner of covering at least a portion of the virtual keyboard based on the first user input; an operation (1408) of receiving a drawing input in the covered area through the touchscreen display (310); an operation (1410) of generating at least one image based on the drawing input using a machine learning model (422); the machine learning model (422) including a model trained to generate an image based on at least one of text, an image, or a drawing input; and displaying the at least one image in a second area of ​​the touchscreen display (310) different from the first area. It may include action (1412).

[0385] According to one embodiment, the method may further include an operation of acquiring a preview image (1120) through an image sensor (320) included in the electronic device (101), and displaying the preview image (1120) on at least a portion of the second area through the touchscreen display (310), an operation of detecting at least one object included in the preview image (1120), an operation of generating at least one key using the machine learning model (422) based on the at least one object, and an operation of displaying the at least one key on at least a portion of the virtual keyboard through the touchscreen display (310).

[0386] According to one embodiment, the method may further include an operation of displaying an image provided from an external electronic device in the second area through the touchscreen display (310), an operation of generating at least one key using the machine learning model based on the image, and an operation of displaying the at least one key in at least a portion of the virtual keyboard through the touchscreen display (310).

[0387] According to one embodiment, the method may further include an operation of receiving a second user input for at least one key of the virtual keyboard through the touchscreen display (310), an operation of displaying text input through the second user input in a third area, and an operation of generating the at least one image based on the text and the drawing input using the machine learning model (422).

[0388] According to one embodiment, the method may further include an operation of displaying, through the touchscreen display (310), at least one graphic object indicating message transmission at a location adjacent to the third area, an operation of receiving a third user input for the at least one graphic object after displaying the text in the third area, an operation of transmitting a message including the text to an external electronic device based on at least a portion of the third user input, and an operation of displaying the text in the second area through the touchscreen display (310). A location adjacent to the third area may mean, for example, a location within a first reference distance from the third area.

[0389] According to one embodiment, the method may further include an action of displaying, via the touchscreen display (310), the at least one second image in the second area together with text included in the transmitted message.

[0390] According to one embodiment, the method may further include an operation of displaying text in the second area through the touchscreen display (310), an operation of receiving a fourth user input for the further text, and an operation of generating the at least one image based on the further text and the drawing input.

[0391] According to one embodiment, the at least one image includes a first image and a second image, and the method may further include: displaying the first image and the second image on at least a portion of the first area through the touchscreen display (310); receiving, through the touchscreen display (310), a fifth user input for the first image; and storing the first image based on at least a portion of the fifth user input.

[0392] According to one embodiment, an operating method of an electronic device (101) may include an operation (1702) of displaying a virtual keyboard in a first area through a touchscreen display (310) included in the electronic device (101), an operation (1704) of obtaining a preview image (1120) including at least one object through an image sensor (320) included in the electronic device (101), an operation (1706) of displaying the preview image (1120) in a second area different from the first area through the touchscreen display (310), an operation (1708) of generating at least one graphic object using a machine learning model (422) based on at least one object included in the preview image (1120), the machine learning model (422) including a model trained to generate an image based on at least one of text, an image, or a drawing input, and an operation (1710) of displaying the at least one graphic object in a portion of the virtual keyboard through the touchscreen display (310). there is.

[0393] According to one embodiment, the method may further include an operation of receiving a user input for the virtual keyboard through the touchscreen display (310), an operation of changing an appearance of the virtual keyboard in a manner of covering at least a portion of the virtual keyboard based on at least a portion of the user input, an operation of receiving a drawing input in the covered area, an operation of generating at least one image based on the drawing input using the machine learning model, and an operation of displaying the at least one image in the first area through the touchscreen display (310).

[0394] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0395] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0396] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0397] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0398] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0399] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (101), Touchscreen display (310); memory (330); and At least one processor (340) comprising a processing circuit, The above memory (330), when individually or collectively executed by the at least one processor (340), causes the electronic device (101) to: Displaying a virtual keyboard in the first area through the above touch screen display (310), Through the above touch screen display (310), a first user input associated with the virtual keyboard is received, Based on the first user input, changing the appearance of the virtual keyboard in a manner that obscures at least a portion of the virtual keyboard, Through the above touch screen display (310), a first drawing input is received in the covered area, Generating at least one image based on the first drawing input using a machine learning model (422), wherein the machine learning model (422) comprises a model trained to generate an image based on at least one of text, image, or drawing input; An electronic device that stores instructions that cause the at least one image to be displayed in a second area different from the first area through the touch screen display (310).

2. In paragraph 1, Further comprising an image sensor (320), The above memory (330), when executed individually or collectively by the at least one processor (340), causes the electronic device (101) to: Through the touch screen display (310), a preview image (1120) obtained through the image sensor (320) is displayed on at least a part of the second area, Detecting at least one object included in the above preview image (1120), Based on at least one object, at least one key is generated using the machine learning model (422), An electronic device that stores commands that cause at least one key to be displayed in at least a portion of the virtual keyboard via the touch screen display (310).

3. In paragraph 1 or 2, The above memory (330), when executed individually or collectively by the at least one processor (340), causes the electronic device (101) to: Through the above touch screen display (310), display an image provided from an external electronic device on at least a part of the second area, Based on the above image, at least one key is generated using the machine learning model (422), An electronic device that stores commands that cause at least one key to be displayed in at least a portion of the virtual keyboard via the touch screen display (310).

4. In any one of paragraphs 1 to 3, The above memory (330), when executed individually or collectively by the at least one processor (340), causes the electronic device (101) to: Through the above touch screen display (310), a second user input for the virtual keyboard is received, Through the above touch screen display (310), the first text corresponding to the second user input is displayed in the third area, An electronic device storing commands that cause the at least one image to be generated based on the first text and the first drawing input using the machine learning model (422).

5. In paragraph 4, It further includes a communication circuit, The above memory (330), when executed individually or collectively by the at least one processor (340), causes the electronic device (101) to: Through the above touch screen display (310), display at least one graphic object that instructs to transmit a message to a location adjacent to the third area, After displaying the first text in the third area, receiving a third user input for the at least one graphic object, transmitting a message including the first text to an external electronic device through the communication circuit based on at least a portion of the third user input; An electronic device that stores commands that cause the first text to be displayed in the second area through the touch screen display (310).

6. In paragraph 5, The above memory (330), when executed individually or collectively by the at least one processor (340), causes the electronic device (101) to: Through the above touch screen display (310), a fourth user input for the at least one image is received, An electronic device storing commands that cause the at least one image to be transmitted in response to the fourth user input.

7. In paragraph 5, The above memory (330), when executed individually or collectively by the at least one processor (340), causes the electronic device (101) to: transmitting at least one image to the external electronic device via the communication circuit; An electronic device storing commands that cause the at least one image to be displayed in the second area together with the first text through the touch screen display (310).

8. In paragraph 1, The above memory (330), when executed individually or collectively by the at least one processor (340), causes the electronic device (101) to: Through the above touch screen display (310), the second text is displayed in the second area, Through the above touch screen display (310), a fifth user input for the second text is received, An electronic device storing commands that cause the device to generate at least one image based on the second text and the first drawing input.

9. In paragraph 7, wherein at least one image comprises a first image and a second image, The above memory (330), when executed individually or collectively by the at least one processor (340), causes the electronic device (101) to: Through the touch screen display (310), the first image and the second image are displayed on at least a part of the first area, Through the above touch screen display (310), a sixth user input for the first image is received, An electronic device storing commands that cause the second image to be stored in the memory (330) based on at least a portion of the sixth user input.

10. In paragraph 1, The above memory (330), when executed individually or collectively by the at least one processor (340), causes the electronic device (101) to: Through the touch screen display (310), a seventh user input for the at least one image on the second area is received, Receive a second drawing input in the above-mentioned covered area, Generating at least one third image using the machine learning model (422) based on at least a portion of the second drawing input; An electronic device storing commands that cause the at least one third image to be displayed in the second area through the touch screen display (310).

11. In the operating method of an electronic device (101), An operation (1402) of displaying a virtual keyboard in a first area through a touch screen display (310) included in the electronic device (101); An operation (1404) of receiving a first user input associated with the virtual keyboard through the touch screen display (310); An action (1406) of changing the appearance of the virtual keyboard in a manner that obscures at least a portion of the virtual keyboard based on the first user input; An operation (1408) of receiving a first drawing input in the covered area through the touch screen display (310); An operation (1410) of generating at least one image based on the first drawing input using a machine learning model (422), wherein the machine learning model (422) comprises a model trained to generate an image based on at least one of text, image, or drawing input; and A method comprising an operation (1412) of displaying at least one image in a second area different from the first area through the touch screen display (310).

12. In paragraph 11, An operation of acquiring a preview image (1120) through an image sensor (320) included in the electronic device (101) and displaying the preview image (1120) on at least a part of the second area through the touch screen display (310); An operation of detecting at least one object included in the above preview image (1120); An operation of generating at least one key using the machine learning model (422) based on at least one object; and A method further comprising an action of displaying the at least one key in at least a portion of the virtual keyboard through the touch screen display (310).

13. In paragraph 11 or 12, An operation of displaying an image provided from an external electronic device on at least a part of the second area through the touch screen display (310); An operation of generating at least one key using the machine learning model based on the image; and A method further comprising an action of displaying the at least one key in at least a portion of the virtual keyboard through the touch screen display (310).

14. In the electronic device (101), Touchscreen display (310); Image sensor (320); memory (330); and At least one processor (340) comprising a processing circuit, The above memory (330), when individually or collectively executed by the at least one processor (340), causes the electronic device (101) to: Through the above touch screen display (310), a virtual keyboard is displayed in the first area, Obtaining a preview image (1120) including at least one object through the image sensor (320), Through the touch screen display (310), the preview image (1120) is displayed in a second area different from the first area, Generating at least one graphic object based on said at least one object using a machine learning model (422), wherein said machine learning model (422) comprises a model trained to generate an image based on at least one of text, image, or drawing input; An electronic device that stores commands that cause at least one graphic object to be displayed in a portion of the virtual keyboard via the touch screen display (310).

15. In paragraph 14, The above memory (330), when executed individually or collectively by the at least one processor (340), causes the electronic device (101) to: Through the above touch screen display (310), user input for at least another area of ​​the virtual keyboard is received, changing the appearance of the virtual keyboard in a manner that obscures at least a portion of the virtual keyboard based on at least a portion of the user input; Through the above touch screen display (310), drawing input is received in the above covered area, generating at least one image using the machine learning model (422) based on at least a portion of the drawing input; An electronic device storing commands that cause the at least one image to be displayed in the first area through the touch screen display (310).

Images