Method for providing images and electronic device performing method

Abstract

An electronic device according to an embodiment generates a prompt at least partially on the basis of a user input to the electronic device, generates at least one first image, which is an on-device generated image, on the basis of at least a portion of the prompt using a generative AI model of the electronic device, transmits an image generation request including at least a portion of the prompt to an external server, receives, from the external server, at least one second image which is a server generated image generated on the basis of at least a portion of the image generation request, and displays at least a portion of the at least one first image or the at least one second image through the display.

Description

Image providing method and electronic device for doing so

[0001] Various embodiments of the present invention disclose a method for providing an image generated based on a generative artificial intelligence (AI) model and an electronic device for performing the same.

[0002] Generative models that can generate media such as text and images in response to prompts are being used in a variety of applications, including art and development.

[0003] Generating high-quality images using generative models can take a long time, resulting in latency when the images cannot be viewed. This can be particularly problematic when images generated by a generative model running on a separate server are received and displayed in response to user-entered prompts, potentially impacting a seamless and continuous user experience.

[0004] In one embodiment, an electronic device includes a display; at least one processor; and a memory storing instructions executable by the at least one processor, wherein the instructions, when executed by the at least one processor, cause the electronic device to at least: generate a prompt based at least in part on a user input to the electronic device; generate a first image that is an on-device generated image corresponding to at least a portion of the prompt using a generative artificial intelligence (AI) model of the electronic device; transmit an image generation request including at least a portion of the prompt to an external server; receive from the external server at least one second image that is a server generated image generated based at least in part on the image generation request, the at least one second image being generated using the generative AI model of the external server; and display at least a portion of the at least one first image or the at least one second image through the display.

[0005] In one embodiment, a method performed by an electronic device including a display; at least one processor; and a memory storing instructions executable by the at least one processor may include: generating a prompt based at least in part on a user input to the electronic device; generating at least one first image, which is an on-device generated image, based at least in part on the prompt using a generative artificial intelligence (AI) model of the electronic device; transmitting an image generation request including at least in part on the prompt to an external server; receiving at least one second image, which is a server-generated image generated based at least in part on the image generation request, from the external server, wherein the at least one second image is generated using the generative AI model of the external server; and displaying at least in part on the display either the at least one first image or the at least one second image.

[0006] According to one embodiment, an electronic device includes a display; at least one processor; and a memory storing instructions executable by the at least one processor, wherein when the instructions are executed by the at least one processor, the electronic device can cause at least: to receive a user input for generating at least one image through the display; to generate at least one first image, which is an on-device generated image, based on at least a portion of the user input using a generative artificial intelligence (AI) model of the electronic device; to transmit a request for generating an image generated based at least in part on the user input to an external server; and to receive from the external server at least one second image, which is a server generated image generated based on at least a portion of the image generation request, wherein the at least one second image is generated using the generative AI model of the external server; and to display at least a portion of the at least one first image and at least a portion of the at least one second image together through the display.

[0007] By generating images based on an external server's generative AI model in parallel with the on-device generative AI model of the electronic device, the user can be presented with the first generated image. This allows users to seamlessly receive images based on the generative AI model, responding to prompts, and adaptively generating and viewing images based on various conditions and states, enhancing the user experience.

[0008] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0009] Figure 2 is a schematic diagram of an image providing method according to one embodiment.

[0010] Figure 3 is a flowchart of an image providing method according to one embodiment.

[0011] FIG. 4 is a flowchart of a method for providing an image using a high-resolution image according to one embodiment.

[0012] FIG. 5 is a flowchart of a method for providing an image using a high-resolution image, according to one embodiment.

[0013] FIG. 6 is a drawing for explaining an operation of displaying a first image and a second image according to one embodiment.

[0014] FIG. 7 is a drawing for explaining an operation of displaying a first image and a second image according to one embodiment.

[0015] FIG. 8 is a drawing for explaining an operation of displaying a high-resolution image according to one embodiment.

[0016] FIG. 9 is a flowchart of a method for displaying an additional image according to one embodiment.

[0017] FIG. 10 is a drawing for explaining an operation of displaying a first image and a second image according to one embodiment.

[0018] FIGS. 11 to 13 are drawings for explaining the order of generating the first image and the second image according to one embodiment.

[0019] Hereinafter, an embodiment of the present document may be described with reference to the attached drawings.

[0020] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0021] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0022] The processor (120) may control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may store a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the command or data stored in the volatile memory (132), and store the resulting data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0023] The auxiliary processor (123) may control at least a part of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, in the electronic device (101) itself where artificial intelligence is performed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0024] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0025] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0026] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0027] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0028] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0029] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0030] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0031] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0032] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0033] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0034] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0035] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0036] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0037] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0038] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0039] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0040] In one embodiment, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0041] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0042] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service by itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0043] Figure 2 is a schematic diagram of an image providing method according to one embodiment.

[0044] According to one embodiment, an image providing system (hereinafter, system) (200) may include an electronic device (201) (e.g., electronic device (101) of FIG. 1) and an external server (203). For example, the electronic device (201) may include at least one processor (e.g., processor (120) of FIG. 1), a communication module (e.g., communication module (190) of FIG. 1), a memory (e.g., memory (130) of FIG. 1), and a display (e.g., display module (160) of FIG. 1).

[0045] According to one embodiment, the system (200) may be a system based on a generative artificial intelligence model. The system (200) may provide images using the generative AI model.

[0046] According to one embodiment, in the system (200), the electronic device (201) and / or the external server (203) may include a user query / response interface (hereinafter, “interface”) as a part thereof or may receive user input through the interface. The user input may be in the form of natural language, images, text, and / or videos, and is not limited to the present disclosure. Along with the user input, context information may be used. The context information may include various additional information at the time of the user input. For example, the context information may include information on the application currently being used by the user, information on the user’s location, or history associated with the system (200). The user input may be in the form of a mixture of at least one of natural language, images, sounds, or context information. The user input may be in a non-natural language form, such as selecting a menu. The electronic device (201) and / or the external server (203) may output to the user a processing result using a generative AI model based on the user input. The results of the processing may be in the form of an image, natural language, other specific content, or a combination thereof, and may also be provided in the form of an action requested by the user.

[0047] The generative AI model of the system (200) can receive user input and coordinate and control each component necessary to carry out the user's intent based on the user's query. Based on the user input received through the interface, the generative AI model can generate prompts suitable for input into a large language model (LLM) or a large multimodal model (LMM). The generative AI model can include an AI component that uses a machine learning algorithm or neural network to develop better prompts over time. User preference data, a prompt library, and prompt examples can be utilized to generate prompts based on user input.

[0048] Generative AI models generally refer to artificial intelligence neural networks that generate new forms of data based on user input. Generative AI models can include models that generate images. Representative examples of these models include generative adversarial networks (GANs) and variational autoencoders (VAEs). Examples include VAEs and diffusion-based generative models that utilize Transformer architectures.

[0049] According to one embodiment, the electronic device (201) may include at least one generative AI model. The generative AI model of the electronic device (201) may be an on-device AI model capable of generating images without communication with an external electronic device (e.g., a server).

[0050] According to one embodiment, the external server (203) may include one or more processors. One or more processors (e.g., a graphics processing unit) of the external server (203) may execute software to control at least one other component connected to the one or more processors and perform various data processing or calculations. The external server (203) may include a communication module capable of supporting the establishment of a communication channel between an external electronic device, such as the electronic device (201), and the external server (203), and the performance of communication through the established communication channel.

[0051] According to one embodiment, the external server (203) may include a generative AI model.

[0052] For example, the external server (203) may include at least one generative AI model.

[0053] For example, the external server (203) may include multiple external servers. Each of the multiple external servers may include at least one generative AI model.

[0054] In S21, the electronic device (201) can generate a prompt based on user input.

[0055] At S22, the electronic device (201) may generate a first image, which is an on-device generated image. The electronic device (201) may generate at least one first image using a generative AI model of the electronic device (201). At S23, the electronic device (201) may transmit an image generation request to an external server (203). S22 and S23 may be performed (or initiated) simultaneously.

[0056] In S24, the external server (203) can generate a second image, which is a server-generated image. The external server (203) can generate at least one second image using the generative AI model of the external server (203).

[0057] S22, S23, and S24 can be performed in parallel. That is, while the electronic device (201) generates the first image (S22), the electronic device (201) can transmit an image generation request to the external server (203) (S23), and the external server (203) can generate the second image (S24).

[0058] In S25, the electronic device (201) can display the first image through the display of the electronic device (201).

[0059] At S26, the electronic device (201) can receive a second image transmitted by an external server (203). At S27, the electronic device (201) can display the second image through the display of the electronic device (201).

[0060] According to one embodiment, the electronic device (201) can display, through the display of the electronic device (201), an image that is generated or received first among the first image or the second image.

[0061] For example, if the performance (e.g., processing speed) of the processor of the electronic device (201) is higher than the performance of the processor of the external server (203), the electronic device (201) can first display (S25) the first image generated (S22) through the display.

[0062] For example, if the performance of the processor of the electronic device (201) is lower than the performance of the processor of the external server (203), the electronic device (201) can first display (S27) the second image that is generated (S24) and received (S26) through the display.

[0063] Figure 3 is a flowchart of an image providing method according to one embodiment.

[0064] According to one embodiment, the operations 310 to 350 below may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2). For example, the electronic device may include at least one processor (e.g., the processor (120) of FIG. 1), a communication module (e.g., the communication module (190) of FIG. 1), a memory (e.g., the memory (130) of FIG. 1), and a display (e.g., the display module (160) of FIG. 1).

[0065] In operation 310, the electronic device may generate a prompt based at least in part on user input to the electronic device. The user input may include at least one of voice data corresponding to the user's utterance, text data typed by the user, or drawing data obtained through a display of the electronic device.

[0066] In one embodiment, the electronic device can receive a voice corresponding to a user's speech. The electronic device can generate a prompt based at least in part on the voice data corresponding to the user's speech.

[0067] According to one embodiment, an electronic device may generate text data by converting voice data corresponding to a user's utterance into speech-to-text (STT). The electronic device may generate a prompt based on at least a portion of the text data.

[0068] In one embodiment, the electronic device may generate a prompt based at least in part on text data typed by the user.

[0069] According to one embodiment, an electronic device may receive a touch action of a pen connected to the electronic device or a body part of a user (e.g., a hand) through a display. The electronic device may generate a prompt based on at least a portion of drawing data obtained by the touch action. According to one embodiment, the electronic device may receive a touch action on a tablet connected to the electronic device and generate a prompt based on at least a portion of drawing data obtained by the touch action. For example, the drawing data may include a picture including at least one object. For example, the drawing data may include text of words or sentences. For example, the electronic device may convert the drawing data into corresponding text data.

[0070] According to one embodiment, operation 320 and operations 330 and 340 may be performed in parallel. Operations 330 and 340 may be performed while operation 320 is being performed.

[0071] In operation 320, the electronic device may generate at least one first image, which is an on-device generated image, based on at least a portion of the prompt, using a generative AI model of the electronic device.

[0072] In one embodiment, the electronic device may generate at least one first image based on at least a portion of a prompt and options associated with image generation using a generative AI model. The options associated with image generation may include information about the generative AI model, or information regarding at least one of the number, size, resolution, or file size of the images to be generated.

[0073] In one embodiment, options associated with image generation may be predetermined.

[0074] In one embodiment, the electronic device may receive user input to determine options associated with image generation. For example, the user may select a generative AI model of the electronic device or a generative AI model of an external server. For example, the user may select the number of first and second images, respectively. For example, the user may select the total number of images to be generated. For example, the user may select the size, resolution, or file size (or a limit thereof) of the images to be generated.

[0075] In one embodiment, options associated with image generation may be determined based on the battery status of the electronic device, the communication status of the electronic device, or the complexity of the prompt.

[0076] In one embodiment, options associated with generating a predetermined image may be changed at least in part based on user input, the battery status of the electronic device, the communication status of the electronic device, or the complexity of the prompt.

[0077] In operation 330, the electronic device may transmit an image generation request including at least a portion of the prompt to an external server (e.g., external server (203) of FIG. 2). The electronic device may transmit the image generation request to the external server simultaneously with initiating (or initiating) generation of at least one first image.

[0078] According to one embodiment, the image creation request may further include options associated with image creation.

[0079] According to one embodiment, the image generation request may include information regarding a generative AI model. For example, if the external server includes multiple external servers, the image generation request may include the generative AI model(s) of at least one external server from which the second image is to be generated. For example, if the external server includes multiple generative AI models, the image generation request may include at least one generative AI model from which the second image is to be generated. For example, when the battery level of the electronic device is low, the image generation request may include a minimum number of generative AI models. For example, when the battery level of the electronic device is low, the image generation request may include a generative AI model that generates an image with a small file size.

[0080] According to one embodiment, the image generation request may include the number of images to be generated. For example, if the battery level of the electronic device is low, the number of second images to be generated may be greater than the number of first images to be generated in order to reduce the power consumption of the electronic device. For example, if the communication network environment of the electronic device is unstable, the number of first images to be generated may be greater than the number of second images because it is difficult to receive second images from an external server. If the electronic device is in airplane mode or communication is blocked, the number of second images may be 0 or the image generation request may not be transmitted to the external server. Thereafter, when the electronic device becomes able to communicate (e.g., by connecting to WiFi or disabling airplane mode), the electronic device may transmit the image generation request to the external server. For example, if the complexity of the prompt is high, such as when the prompt is long or large, the number of first images that require a short time to generate images may be greater than the number of second images. Conversely, if the processing performance of the electronic device is degraded, which requires a long time to generate images, the number of second images may be greater than the number of first images.

[0081] According to one embodiment, the image generation request may include information regarding at least one of the size, resolution, or file size of the image to be generated. For example, if the communication network environment of the electronic device is unstable, the size or file size of the second image to be generated may be small or have a low resolution. On the other hand, if the remaining battery power of the electronic device is low, the size or file size of the first image to be generated may be small or have a low resolution to reduce the power consumption of the electronic device.

[0082] In operation 340, the electronic device may receive, from an external server, at least one second image generated based on at least a portion of the image generation request.

[0083] In operation 350, the electronic device may display at least a portion of at least one first image or at least one second image via a display. A method for displaying the first image and / or the second image is described in detail with reference to FIGS. 6 to 8 and FIG. 10.

[0084] According to one embodiment, the electronic device can display, through a display of the electronic device, at least one first image or at least one second image, whichever image is generated or received first.

[0085] According to one embodiment, the resolution or file size of the at least one first image and the at least one second image may vary depending on the performance of the processors of the electronic device and the external server. For example, the at least one first image may have a lower resolution or a smaller file size than the at least one second image. In this case, the generation of the at least one first image may be faster than the reception of the at least one second image. The electronic device may display at least a portion of the at least one first image through a display, and then display at least a portion of the received at least one second image through the display.

[0086] According to one embodiment, the above-described operation 310 may be omitted. The electronic device may receive a user input for generating at least one image through a display. The electronic device may generate at least one first image, which is an on-device generated image, based on at least a portion of the user input using a generative AI model. The electronic device may transmit a request for generating an image generated based at least a portion of the user input to an external server. The electronic device may receive, from the external server, at least one second image, which is a server-generated image generated based on at least a portion of the image generation request. The at least one second image may be generated using another generative AI model of the external server. The electronic device may display at least a portion of the at least one first image and at least a portion of the at least one second image together through the display.

[0087] FIG. 4 is a flowchart of a method for providing an image using a high-resolution image according to one embodiment.

[0088] According to one embodiment, the operations 410 and 420 below may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2). For example, the electronic device may include at least one processor (e.g., the processor (120) of FIG. 1), a communication module (e.g., the communication module (190) of FIG. 1), a memory (e.g., the memory (130) of FIG. 1), and a display (e.g., the display module (160) of FIG. 1).

[0089] According to one embodiment, operation 320 of FIG. 3 may include operations 410 and 420 below.

[0090] In operation 410, the electronic device may generate at least one first high-resolution image, which is a high-resolution on-device generated image, based on at least a portion of the prompt using a generative AI model of the electronic device. The at least one first high-resolution image may have a higher resolution or larger size than the at least one first image.

[0091] In operation 420, the electronic device may generate at least one first image by scaling or cropping at least one first high-resolution image. The first image may be used for a preview of the first high-resolution image.

[0092] According to one embodiment, operations 320 and 410 of FIG. 3 may be performed sequentially. That is, the electronic device may first generate at least one first image (operation 320), and while displaying the first image through a display (operation 350), generate a first high-resolution image (operation 410). The electronic device may generate the first high-resolution image based on the first image. The electronic device may generate the first high-resolution image corresponding to the first image using a generative AI model. The electronic device may generate the first high-resolution image corresponding to the first image through image processing using quantization, up-sampling, computer vision, or other deep learning-based models.

[0093] According to one embodiment, operations 320 and 410 of FIG. 3 may be performed in parallel. That is, the electronic device may generate at least one first image (operation 320) and, in parallel (or simultaneously), generate at least one first high-resolution image (operation 410). The electronic device may first display (operation 350) the first image, which is generated more quickly due to its relatively low resolution or small size, through the display, and then seamlessly display the first high-resolution image that is generated.

[0094] FIG. 5 is a flowchart of a method for providing an image using a high-resolution image, according to one embodiment.

[0095] According to one embodiment, the operations 510 and 520 below may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2). For example, the electronic device may include at least one processor (e.g., the processor (120) of FIG. 1), a communication module (e.g., the communication module (190) of FIG. 1), a memory (e.g., the memory (130) of FIG. 1), and a display (e.g., the display module (160) of FIG. 1).

[0096] In one embodiment, the image generation request may include a command directing generation of a second image and / or a second high-resolution image.

[0097] According to one embodiment, operation 340 described above with reference to FIG. 3 may be replaced with operations 510 and 520. The electronic device may transmit (operation 330) an image generation request including a command instructing the external server to generate a second high-resolution image. Accordingly, the external server may generate only the second high-resolution image and transmit it to the electronic device.

[0098] In operation 510, the electronic device may receive at least one second high-resolution image, which is a high-resolution server-generated image generated based on at least a portion of the image generation request, from an external server (e.g., external server (203) of FIG. 2). The at least one second high-resolution image may have a higher resolution or larger size than the at least one second image.

[0099] In operation 520, the electronic device may generate at least one second image by scaling or cropping at least one second high-resolution image. The second image may be used for a preview of the second high-resolution image.

[0100] In one embodiment, operation 520 may be omitted. In this case, operations 340 and 510 of FIG. 3 may be performed sequentially. The electronic device may transmit an image generation request including a command to generate a second image and a second high-resolution image to an external server (operation 330). The electronic device may receive the second image, which is generated more quickly due to its relatively low resolution or small size, and display it on a display (operation 350), and then seamlessly display the received second high-resolution image.

[0101] For example, an external server may generate the second image and the second high-resolution image in parallel. That is, the generation of the second image and the second high-resolution image may be initiated simultaneously. The electronic device may receive the second image, which is generated more quickly due to its relatively low resolution or small size (operation 340), and then receive the second high-resolution image corresponding to the generated second image (operation 510).

[0102] For example, an external server may generate a second high-resolution image after generating a second image. The external server may generate a second high-resolution image corresponding to the second image using a generative AI model. The external server may generate the second high-resolution image corresponding to the second image through image processing using quantization, upsampling, computer vision, or other deep learning-based models. The electronic device may first receive the second image (operation 340) and then receive a second high-resolution image corresponding to the generated second image (operation 510).

[0103] In one embodiment, the electronic device, in response to receiving a second high-resolution image corresponding to one of at least one second image from an external server, may display at least a portion of the received second high-resolution image as a replacement for at least a portion of the second image. The electronic device may display at least a portion of the second high-resolution image so that at least a portion of the second high-resolution image sequentially replaces at least a portion of the corresponding second image. Accordingly, the user may be aware that the second high-resolution image is generated or received subsequent to the corresponding second image.

[0104] FIG. 6 is a drawing for explaining an operation of displaying a first image and a second image according to one embodiment.

[0105] FIG. 6 illustrates screens (601, 602, 603) displayed on a display (e.g., a display module (160) of FIG. 1) of an electronic device (e.g., an electronic device (101) of FIG. 1 or an electronic device (201) of FIG. 2). For example, the electronic device may include at least one processor (e.g., a processor (120) of FIG. 1), a communication module (e.g., a communication module (190) of FIG. 1), a memory (e.g., a memory (130) of FIG. 1), and a display.

[0106] On the screen (601), the electronic device can receive user input. The user input may include at least one of voice data corresponding to the user's speech, text data typed by the user, or drawing data acquired through the display of the electronic device. For example, the electronic device may receive 'a snowman under the sunshine' as text data. The electronic device may receive input for generating an image corresponding to the user input using a generative AI model.

[0107] An electronic device can generate a prompt based at least in part on a user input. The electronic device can generate at least one first image, which is an on-device generated image, based at least in part on the prompt, using a generative AI model of the electronic device. The electronic device can transmit an image generation request including at least in part on the prompt to an external server (e.g., the external server (203) of FIG. 2 ). The electronic device can receive, from the external server, at least one second image generated based at least in part on the image generation request. Additionally / alternatively, the electronic device can receive, from the external server, at least one second high-resolution image generated based at least in part on the image generation request. Descriptions of content that overlap with those described with reference to FIGS. 2 to 5 are omitted.

[0108] On the screen (602), the electronic device can display at least a portion (61) of at least one first image through the display of the electronic device.

[0109] In one embodiment, an electronic device may receive a user's transition input. The transition input may represent an input for viewing an image other than the currently displayed image. For example, the transition input may represent an input for viewing an image (e.g., a server-generated image) generated after an image that was previously generated and displayed (e.g., an on-device generated image). The transition input may be a touch input in which the user swipes the display of the electronic device. The transition input may be a touch input for an object, such as a button, for viewing another image (or switching screens), but is not limited thereto.

[0110] On the screen (603), the electronic device may, in response to a user's switching input, display at least a portion (63) of at least one second image, which is a server-generated image, through the display of the electronic device. If there is a server-generated image that has not yet been generated or received, the electronic device may display an additional image (65). For example, the additional image (65) may include current status information (e.g., a loading indicator or progress indicator that may indicate that the image is being generated or received). For example, the additional image (65) may include a visual effect or percentage indicating the degree of generation or reception of the image.

[0111] On screen (603), the electronic device may receive the user's switching input again. In response to the user's switching input, the electronic device may display screen (602) again or display an image other than the currently displayed image. For example, in response to the user's switching input, the electronic device may display an additional on-device generated image or an additional server generated image. Additional images are described in detail with reference to FIG. 9.

[0112] FIG. 7 is a drawing for explaining an operation of displaying a first image and a second image according to one embodiment.

[0113] FIG. 7 illustrates screens (701, 703) displayed on a display (e.g., a display module (160) of FIG. 1) of an electronic device (e.g., an electronic device (101) of FIG. 1 or an electronic device (201) of FIG. 2). For example, the electronic device may include at least one processor (e.g., a processor (120) of FIG. 1), a communication module (e.g., a communication module (190) of FIG. 1), a memory (e.g., a memory (130) of FIG. 1), and a display.

[0114] On the screen (701), the electronic device can receive user input. The electronic device can receive input for generating an image based on the user input using a generative AI model. Any description that overlaps with the above will be omitted.

[0115] On the screen (703), the electronic device can display at least a portion of at least one on-device generated image (hereinafter, a first image) (71) and at least a portion of at least one server-generated image (hereinafter, a second image) (73) through the display of the electronic device. If there is a server-generated image that has not yet been generated or received, the electronic device can display a visual effect or percentage indicating the degree of generation or reception of the server-generated image being generated or received, together with the display. In this case, if the electronic device first receives a server-generated image or a low-resolution server-generated image corresponding to the server-generated image from an external server (e.g., the external server (203) of FIG. 2) and later receives a high-resolution server-generated image, the electronic device can display the visual effect or percentage together with at least a portion of the server-generated image.

[0116] On the screen (703), the electronic device can display the first image (71) and the second image (73) at substantially the same size. For example, the electronic device can display the first image (71) and the second image (73) at the same size.

[0117] FIG. 8 is a drawing for explaining an operation of displaying a high-resolution image according to one embodiment.

[0118] FIG. 8 illustrates screens (801, 803) displayed on a display (e.g., a display module (160) of FIG. 1) of an electronic device (e.g., an electronic device (101) of FIG. 1 or an electronic device (201) of FIG. 2). For example, the electronic device may include at least one processor (e.g., a processor (120) of FIG. 1), a communication module (e.g., a communication module (190) of FIG. 1), a memory (e.g., a memory (130) of FIG. 1), and a display.

[0119] On the screen (801), the electronic device can display at least a portion of at least one first image that is an on-device generated image or at least one second image that is a server generated image.

[0120] In the screen (801), as described with reference to FIG. 6, the electronic device can display at least a portion of at least one first image through the display, or at least a portion of at least one second image through the display.

[0121] In the screen (801), as described with reference to FIG. 7, the electronic device can display at least a portion of at least one first image and at least a portion of at least one second image together through the display.

[0122] On the screen (801), the electronic device can receive an input for selecting a target image (81) from among at least one first image or at least one second image.

[0123] On the screen (803), the electronic device can display a target high-resolution image (83) corresponding to the target image (81) through the display in response to an input for selecting the target image (81). The target high-resolution image (83) can have a higher resolution or larger size than the target image (81).

[0124] For example, if the target image (81) is an on-device generated image, the target high-resolution image (83) may be a high-resolution on-device generated image.

[0125] For example, if the target image (81) is a server-generated image, the target high-resolution image (83) may be a high-resolution server-generated image.

[0126] According to one embodiment, the target image (81) and the target high-resolution image (83) can be generated sequentially.

[0127] For example, the electronic device may first generate a target image (81) and, while displaying the target image (81) through a display, generate a target high-resolution image (83). For example, the electronic device may first generate a target high-resolution image (83) and, by scaling or cropping the target high-resolution image (83), display the target image (81) through a display.

[0128] For example, an external server (e.g., external server (203) of FIG. 2) may first generate a target image (81), and while the electronic device receives the target image (83) and displays it through a display, the target high-resolution image (83) may be generated. For example, the external server may first generate a target high-resolution image (81) and transmit it to the electronic device, and the electronic device may generate the target image (81) by scaling or cropping the target high-resolution image (81).

[0129] According to one embodiment, the target image (81) and the target high-resolution image (83) can be generated in parallel (or simultaneously). That is, the generation of the target image (81) and the target high-resolution image (83) can be initiated simultaneously. On the screen (801), the electronic device can first display the target image (81), which is generated faster due to its relatively low resolution or small size. In response to an input for selecting the target image (81), the electronic device can seamlessly display the target high-resolution image (83) generated subsequent to the target image (81), on the screen (803).

[0130] On the screen (803), the electronic device can display at least a portion of at least one first image or at least one second image on a display smaller than the target high-resolution image (83). For example, the electronic device can display at least a portion of at least one first image or at least one second image in the form of a filmstrip (85). The user can scroll the filmstrip (85) up and down or left and right to check the first image or the second image that is not displayed on the screen (803).

[0131] According to one embodiment, the electronic device may display a high-resolution image corresponding to the other target image by replacing the target high-resolution image (83) when the user selects a target image other than the target image (81) among at least one first image or at least one second image.

[0132] FIG. 9 is a flowchart of a method for displaying an additional image according to one embodiment.

[0133] According to one embodiment, the operations 910 to 940 below may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2). For example, the electronic device may include at least one processor (e.g., the processor (120) of FIG. 1), a communication module (e.g., the communication module (190) of FIG. 1), a memory (e.g., the memory (130) of FIG. 1), and a display (e.g., the display module (160) of FIG. 1).

[0134] In operation 910, the electronic device may display at least a portion of at least one first image or at least one second image via a display. Operation 910 may be at least a portion of operation 350 of FIG. 3.

[0135] In one embodiment, operations 920 and / or 930 may be performed while the electronic device displays at least a portion of at least one first image or at least one second image via a display.

[0136] In operation 920, the electronic device may generate a first additional image, which is an additional on-device generated image, based on at least a portion of the prompt using the electronic device's generative AI model. The first additional image may represent a subsequently generated on-device generated image in addition to an on-device generated image that is already being displayed on the display or that has already been generated but is not being displayed on the display (e.g., when the maximum number of on-device generated images that can be displayed on a single screen is already being displayed). Operation 920 may be a continuation of operation 320 of FIG. 3 .

[0137] In one embodiment, operation 920 may be performed at a predetermined time interval from operation 320. For example, a delay may occur due to a degradation in processing performance or other conflicts during the generation of an on-device generated image of an electronic device.

[0138] At operation 930, the electronic device may receive, from an external server (e.g., external server (203) of FIG. 2), at least one second additional image, which is an additional server-generated image generated based on at least a portion of the image generation request. The second additional image may represent a server-generated image that is already being displayed on the display, received by the electronic device but not yet displayed on the display (e.g., when the maximum number of server-generated images that can be displayed on a single screen is already being displayed), or generated but not yet received by the electronic device, in addition to the server-generated image that has already been generated. Operation 930 may be a continuation of operation 340 of FIG. 3.

[0139] In one embodiment, operation 930 may be performed at a predetermined time interval from operation 340. For example, a delay may occur due to unstable communication conditions of the electronic device, or a decrease in processing performance or other conflicts of an external server during the generation of a server-generated image. For example, if the electronic device is in airplane mode or communication is blocked, when communication becomes possible, the electronic device may transmit an image generation request to the external server, or the transmission of the interrupted server image generation by the external server may be resumed.

[0140] In one embodiment, operations 920 and 930 may be performed in parallel.

[0141] According to one embodiment, operations 920 and 930 may be optionally performed when a problem situation such as that described above occurs (e.g., a degradation in processing performance of an electronic device or an external server, other crashes, an unstable communication state of an electronic device, etc.).

[0142] In operation 940, the electronic device can display at least a portion of at least one first additional image or at least one second additional image via the display.

[0143] According to one embodiment, the electronic device can maintain at least a portion of at least one first additional image or at least one second additional image in a displayable state while displaying a maximum number of on-device generated images and / or server generated images that can be displayed on a single screen.

[0144] According to one embodiment, the electronic device may first display, through the display, an on-device generated image or a server generated image that has already been generated but is not displayed through the display, or a server generated image that has been received from an external server but is not displayed through the display, and then display a first additional image or a second additional image.

[0145] For example, the electronic device may display at least one first additional image or at least a portion of at least one second additional image on another screen in response to a user's switching input. For example, the electronic device may display at least one first additional image or at least a portion of at least one second additional image in response to an input for scrolling a filmstrip (e.g., filmstrip (85) of FIG. 8 ).

[0146] FIG. 10 is a drawing for explaining an operation of displaying a first image and a second image according to one embodiment.

[0147] FIG. 10 illustrates screens (1001, 1003) displayed on a display (e.g., a display module (160) of FIG. 1) of an electronic device (e.g., an electronic device (101) of FIG. 1 or an electronic device (201) of FIG. 2). For example, the electronic device may include at least one processor (e.g., a processor (120) of FIG. 1), a communication module (e.g., a communication module (190) of FIG. 1), a memory (e.g., a memory (130) of FIG. 1), and a display.

[0148] According to one embodiment, the electronic device can store at least a portion of at least one first image that is an on-device generated image or at least one second image that is a server generated image, in association with metadata.

[0149] In one embodiment, the metadata may include at least one of a prompt, information about a generative AI model of an electronic device or an external server (e.g., external server (203) of FIG. 2), information about an object included in at least one first image or at least one second image, or options associated with image generation.

[0150] For example, metadata of at least one first image may include information about a generative AI model of the electronic device. The information about the generative AI model of the electronic device may include information indicating that the generated first image was generated by the electronic device. For example, metadata of at least one second image may include information about a generative AI model of an external server. The information about the generative AI model of the external server may include information indicating that the generated second image was generated by the external server. If the external server includes multiple external servers, the information about the generative AI model of the external server may include information about a specific external server from which the second image was generated. If the external server includes one or more generative AI models, the information about the generative AI model of the external server may include information about a specific generative AI model from which the second image was generated.

[0151] For example, information about an object included in at least one first image or at least one second image may be determined based on user input or a prompt. For example, if an electronic device receives 'a snowman under the sunshine' as text data, as described with reference to FIG. 7, the metadata may include information about the object, such as a snowman or a sun.

[0152] For example, options associated with image generation may include information regarding at least one of the number, size, resolution, or file size of images to be generated (or generated), or information regarding the generative AI model. Options associated with image generation may be determined based on the battery status of the electronic device, the communication status of the electronic device, or the complexity of the prompt.

[0153] On the screen (1001), the electronic device can display at least a portion of at least one first image or at least one second image through the display.

[0154] The electronic device may, in response to an input for selecting a target image from among at least one first image or at least one second image, display a target high-resolution image corresponding to the target image on a screen (1003). The electronic device may display at least a portion of the at least one first image or at least one second image through the display with an image size smaller than that of the target high-resolution image.

[0155] According to one embodiment, the electronic device may display an indicator indicating at least a portion of metadata via a display. For example, the electronic device may display an indicator indicating information about a generative AI model of the electronic device or an external server. The electronic device may display a first indicator (11) indicating that at least a portion of at least one first image was generated using the generative AI model of the electronic device together with (or in proximity to) at least a portion of at least one first image. The electronic device may display a second indicator (13) indicating that at least a portion of at least one second image was generated using the generative AI model of the external server together with at least a portion of the at least one second image. If the external server includes multiple external servers or one or more generative AI models, the second indicator (13) may indicate the type of the external server or the generative AI model. For example, the indicator may include text, an icon, an image, or a graphic object including the same.

[0156] According to one embodiment, the electronic device may display at least one first image or at least one second image in a predetermined display order (or by indicating the display order) based on metadata.

[0157] According to one embodiment, the electronic device may display at least a portion of at least one first image or at least one second image based on metadata, together with a graphical effect (e.g., applying a highlight, a border color) indicating the source of the image.

[0158] FIGS. 11 to 13 are drawings for explaining the order of generating the first image and the second image according to one embodiment.

[0159] FIGS. 11 to 13 illustrate operations of an electronic device (e.g., electronic device (101) of FIG. 1 or electronic device (201) of FIG. 2) and an external server (e.g., external server (203) of FIG. 2) over time. The operations illustrated in FIGS. 11 to 13 may include the operations described above with reference to FIGS. 2 to 10.

[0160] For example, an electronic device may include at least one processor (e.g., processor (120) of FIG. 1), a communication module (e.g., communication module (190) of FIG. 1), a memory (e.g., memory (130) of FIG. 1), and a display (e.g., display module (160) of FIG. 1). For example, an external server may include a processor and a communication module. The electronic device and the external server may include one or more processors. The operations illustrated in FIGS. 11 to 13 may be performed by one of one or more processors of the electronic device, or by any combination of at least some of the one or more processors. The operations illustrated in FIGS. 11 to 13 may be performed by one of one or more processors of the external server, or by any combination of at least some of the one or more processors.

[0161] Referring to FIG. 11, an electronic device can receive user input. The electronic device can display the user input in real time through a display. A processor of the electronic device (hereinafter, referred to as the electronic device) can generate a prompt based at least in part on the user input.

[0162] The electronic device can use a generative AI model of the electronic device to generate at least one first image based on at least a portion of the prompt.

[0163] The electronic device can initiate generation of at least one first image and simultaneously transmit an image generation request to an external server via a communication module. The external server can receive the image generation request via the communication module.

[0164] While the electronic device generates at least one first image, an external server may generate at least one second image based at least in part on the image generation request. The electronic device may display, through the display of the electronic device, the image that is generated or received first among the at least one first image or the at least one second image.

[0165] Referring to FIG. 11, the electronic device may display at least one first image generated first through a display. While displaying the at least one first image through the display, the electronic device may receive at least one second image from an external server through a communication module. The electronic device may display the at least one second image on the same screen as the first image or maintain the second image in a displayable state through the display.

[0166] Referring to FIG. 12, the operations described with reference to FIG. 11 can be similarly performed.

[0167] The electronic device may generate at least one first high-resolution image corresponding to the at least one first image while displaying at least a portion of at least one first image or at least one second image through a display. The external server may generate at least one second high-resolution image corresponding to the at least one second image. The electronic device may receive the at least one second image and the at least one second high-resolution image from the external server through a communication module in the order in which they are generated (or at one time).

[0168] The electronic device may display at least one first image generated first through a display. While displaying the at least one first image through the display, the electronic device may receive at least one second image from an external server through a communication module and display the second image through the display.

[0169] The electronic device can maintain at least one first high-resolution image and at least one second high-resolution image in a displayable state while displaying at least a portion of at least one first image or at least one second image through the display.

[0170] The electronic device may receive a user input for selecting a target image from at least one of the first images or at least a portion of the second images. In response to the user input for selecting the target image, the electronic device may display a target high-resolution image corresponding to the target image through a display. For example, if the target image is an on-device generated image, the target high-resolution image may be a high-resolution on-device generated image. For example, if the target image is a server-generated image, the target high-resolution image may be a high-resolution server-generated image.

[0171] For example, the electronic device may display an additional image (e.g., an additional image (65) of FIG. 6) including a visual effect indicating the degree of generation or reception of an image when a target image is selected from among the second images and a target high-resolution image corresponding to the target image, i.e., a second high-resolution image, has not yet been received.

[0172] Referring to FIG. 13, an electronic device can receive user input. The electronic device can display the user input in real time through a display. The electronic device can generate a prompt based at least in part on the user input.

[0173] The electronic device can use a generative AI model of the electronic device to generate at least one first high-resolution image, which is a high-resolution on-device generated image, based on at least a portion of the prompt.

[0174] The electronic device can initiate generation of at least one first high-resolution image and simultaneously transmit an image generation request to an external server via the communication module. The external server can receive the image generation request via the communication module.

[0175] While the electronic device generates at least one first high-resolution image, the external server can generate at least one second image based at least in part on the image generation request.

[0176] The electronic device can generate at least one first image by scaling or cropping at least one first high-resolution image.

[0177] Typically, generating a high-resolution image can take a long time, and it may be faster for an external server to generate a second image and transmit it to the electronic device than for the first high-resolution image and the second image to be generated sequentially. However, the order in which the first and second images are generated may vary depending on the processor performance of the electronic device and the external server, or the communication network environment.

[0178] For example, referring to FIG. 13, if the electronic device receives the second image before generating the first image, the electronic device can keep the second image in a displayable state and display it on the display together with the first image.

[0179] For example, if the electronic device receives the second image before generating the first image, it may display the second image first.

[0180] According to one embodiment, an electronic device (101, 201) includes a display, at least one processor (120), and a memory (130) storing instructions executable by the at least one processor (120), wherein when the instructions are executed by the at least one processor (120), the electronic device (101, 201) causes at least: to generate a prompt based at least in part on a user input to the electronic device (101, 201); to generate at least one first image, which is an on-device generated image, based at least in part on the prompt using a generative artificial intelligence (AI) model of the electronic device (101, 201); to transmit an image generation request including at least in part on an external server (203); and to receive from the external server (203) at least one second image, which is a server generated image generated based at least in part on the image generation request, wherein the at least one second image is generated using the generative AI model of the external server. -, at least one first image or at least a part of at least one second image can be displayed through the display.

[0181] According to one embodiment, the user input may include at least one of voice data corresponding to the user's speech, text data by the user's typing, or drawing data obtained through a display.

[0182] According to one embodiment, the image generation request further includes options associated with image generation, wherein the options associated with image generation may include information about at least one of the number, size, resolution, or file size of images to be generated, or information about a generative AI model.

[0183] In one embodiment, at least one first image may have a lower resolution or smaller file size than at least one second image.

[0184] According to one embodiment, the instructions, when executed by at least one processor (120), may cause the electronic device (101, 201) to at least: generate, using a generative AI model of the electronic device (101, 201), at least one first high-resolution image that is a high-resolution on-device generated image based on at least a portion of the prompt, wherein the at least one first high-resolution image has a higher resolution or larger size than the at least one first image; and generate the at least one first image by scaling or cropping the at least one first high-resolution image.

[0185] According to one embodiment, when the instructions are executed by at least one processor (120), the electronic device (101, 201) may cause at least: to receive from an external server (203) at least one second high-resolution image, the second high-resolution image being a high-resolution server-generated image corresponding to at least one second image, generated based on at least a portion of the image generation request, wherein the at least one second high-resolution image is generated using a generative AI model of the external server (203) and has a higher resolution or larger size than the at least one second image.

[0186] According to one embodiment, when the instructions are executed by at least one processor (120), the electronic device (101, 201) may cause at least: displaying at least a portion of at least one first image through the display; and, in response to a transition input, displaying at least a portion of at least one second image through the display.

[0187] According to one embodiment, the instructions, when executed by at least one processor (120), may cause the electronic device (101, 201) to: generate, using a generative AI model of the electronic device (101, 201), at least one first additional image, which is an additional on-device generated image based on at least a portion of the prompt; and / or receive, from an external server (203), at least one second additional image, which is an additional server-generated image generated based on at least a portion of the image generation request, wherein the at least one second additional image is generated using the generative AI model of the external server (203); and display, on the display, at least one first additional image or at least a portion of the at least one second additional image.

[0188] According to one embodiment, when the instructions are executed by at least one processor (120), the electronic device (101, 201) may cause at least: at least a portion of at least one first image and at least a portion of at least one second image to be displayed together on a display at substantially the same size.

[0189] According to one embodiment, when the instructions are executed by at least one processor (120), the electronic device (101, 201) may cause at least: in response to an input of a user of the electronic device (101, 201) selecting a target image from among at least one first image or at least one second image, to display a target high-resolution image corresponding to the target image through a display, wherein the target high-resolution image has a higher resolution or a larger size than the target image.

[0190] According to one embodiment, when the instructions are executed by at least one processor (120), the electronic device (101, 201) may cause at least: at least a portion of at least one first image or at least one second image to be displayed on the display at a size smaller than a target high-resolution image.

[0191] According to one embodiment, the instructions, when executed by at least one processor (120), may cause the electronic device (101, 201) to: store at least a portion of at least one first image or at least one second image in association with metadata, wherein the metadata includes at least a portion of a prompt, information about a generative AI model of the electronic device (101, 201) or the external server (203), information about an object included in the at least one first image or the at least one second image, or at least one of options associated with image generation.

[0192] According to one embodiment, when the instructions are executed by at least one processor (120), the electronic device (101, 201) may cause at least: an indicator representing at least a portion of the metadata to be displayed via a display.

[0193] According to one embodiment, when the instructions are executed by at least one processor (120), the electronic device (101, 201) may be caused to: in response to receiving from an external server (203) one second high-resolution image corresponding to one of the at least one second image, display at least a portion of the one second high-resolution image by replacing at least a portion of the one second image.

[0194] In one embodiment, a method performed by an electronic device (101, 201), the electronic device (101, 201) comprises: a display; at least one processor (120); and a memory (130) storing instructions executable by the at least one processor (120), the method comprising: determining a prompt based at least in part on a user input to the electronic device (101, 201); generating at least one first image, which is an on-device generated image, based at least in part on the prompt, using a generative artificial intelligence (AI) model of the electronic device (101, 201); transmitting an image generation request including at least in part on the prompt to an external server (230); and receiving at least one second image, which is a server generated image, based at least in part on the image generation request, from the external server (230), wherein the at least one second image is generated using the generative AI model of the external server (203); and may include an action of displaying at least a portion of at least one first image or at least one second image through a display.

[0195] According to one embodiment, the user input may include at least one of voice data corresponding to speech, text data by typing by the user, or drawing data obtained through the display of the electronic device (101, 201).

[0196] In one embodiment, at least one first image may have a lower resolution or smaller file size than at least one second image.

[0197] According to one embodiment, the operation of generating at least one first image based on at least a portion of the prompt using the generative AI model of the electronic device (101, 201) may include: generating at least one first high-resolution image, which is a high-resolution on-device generated image, based on at least a portion of the prompt using the generative AI model of the electronic device (101, 201), wherein the at least one first high-resolution image has a higher resolution or a larger size than the at least one first image; and generating the at least one first image by scaling or cropping the at least one first high-resolution image.

[0198] According to one embodiment, a computer-readable recording medium may store a program for executing a method performed by an electronic device (101, 201) in combination with hardware.

[0199] According to one embodiment, an electronic device (101, 201) comprises: a display; at least one processor (120); And a memory (130) storing instructions executable by at least one processor (120), wherein when the instructions are executed by the at least one processor (120), the electronic device (101, 201) can cause at least: receive a user input for generating at least one image through a display; generate at least one first image, which is an on-device generated image, based on at least a portion of the user input using a generative artificial intelligence (AI) model of the electronic device (101, 201); transmit a request for generating an image, which is generated at least in part based on the user input, to an external server (203); receive from the external server (203), at least one second image, which is a server generated image, based on at least a portion of the image generation request, wherein the at least one second image is generated using another generative AI model of the external server (203); and display together through the display at least a portion of the at least one first image and at least a portion of the at least one second image.

[0200]

[0201] The embodiments described above may be implemented using hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0202] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0203] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination, and the program commands recorded on the medium may be those specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0204] The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0205] Although the embodiments have been described with limited drawings, those skilled in the art will appreciate that various technical modifications and variations can be applied based on the described embodiments. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0206] Therefore, other implementations, embodiments and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In an electronic device (101, 201), display; at least one processor (120); and It includes a memory (130) that stores instructions executable by at least one processor (120), When the above instructions are executed by the at least one processor (120), the electronic device (101, 201) causes at least: Generating a prompt based at least in part on user input to said electronic device (101, 201); Using a generative artificial intelligence (AI) model of the electronic device (101, 201), at least one first image, which is an on-device generated image, is generated based on at least a part of the prompt, Transmitting an image generation request including at least a part of the above prompt to an external server (203), Receive at least one second image from the external server (203), which is a server generated image generated based on at least a part of the image generation request, wherein the at least one second image is generated using a generative AI model of the external server; Displaying at least a portion of at least one of the first image or at least one of the second image through the display To do, Electronic devices (101, 201).

2. In paragraph 1, The above image generation request is, Includes more options related to image creation, Options associated with the image generation include information about at least one of the number, size, resolution, or file size of images to be generated, or information about the generative AI model. Electronic devices (101, 201).

3. In either of paragraphs 1 and 2, wherein said at least one first image has a lower resolution or smaller file size than said at least one second image; Electronic devices (101, 201).

4. In any one of paragraphs 1 to 3, When the above instructions are executed by the at least one processor (120), the electronic device (101, 201) causes at least: Using the generative AI model of the electronic device (101, 201), generating at least one first high-resolution image, which is a high-resolution on-device generated image, based on at least a portion of the prompt, wherein the at least one first high-resolution image has a higher resolution or larger size than the at least one first image; Generating at least one first image by scaling or cropping at least one first high-resolution image To do, Electronic devices (101, 201).

5. In any one of paragraphs 1 to 4, When the above instructions are executed by the at least one processor (120), the electronic device (101, 201) causes at least: Receive at least one second high-resolution image, which is a high-resolution server-generated image corresponding to the at least one second image, generated based on at least a part of the image generation request from the external server (203), wherein the at least one second high-resolution image is generated using the generative AI model of the external server (203) and has a higher resolution or larger size than the at least one second image; To do, Electronic devices (101, 201).

6. In any one of paragraphs 1 to 5, When the above instructions are executed by the at least one processor (120), the electronic device (101, 201) causes at least: displaying at least a portion of said at least one first image through said display; In response to a transition input, displaying at least a portion of said at least one second image through said display; To do, Electronic devices (101, 201).

7. In any one of paragraphs 1 to 6, When the above instructions are executed by the at least one processor (120), the electronic device (101, 201) causes at least: While displaying at least a portion of said at least one first image or said at least one second image through said display: Using the generative AI model of the electronic device (101, 201), generate at least one first additional image, which is an additional on-device generated image, based on at least a portion of the prompt, and / or Receive from said external server (203) at least one second additional image, which is an additional server-generated image generated based on at least a portion of said image generation request, wherein said at least one second additional image is generated using said generative AI model of said external server (203); Displaying at least a portion of said at least one first additional image or said at least one second additional image through said display; To do, Electronic devices (101, 201).

8. In any one of paragraphs 1 to 7, When the above instructions are executed by the at least one processor (120), the electronic device (101, 201) causes at least: Displaying together through the display at least a portion of said at least one first image and at least a portion of said at least one second image at substantially the same size. To do, Electronic devices (101, 201).

9. In any one of paragraphs 1 to 8, When the above instructions are executed by the at least one processor (120), the electronic device (101, 201) causes at least: In response to an input of the user of the electronic device (101, 201) to select a target image among the at least one first image or the at least one second image, displaying a target high-resolution image corresponding to the target image through the display, wherein the target high-resolution image has a higher resolution or larger size than the target image. To do, Electronic devices (101, 201).

10. In any one of paragraphs 1 to 9, When the above instructions are executed by the at least one processor (120), the electronic device (101, 201) causes at least: Displaying at least a portion of said at least one first image or said at least one second image smaller than said target high-resolution image through said display; To do, Electronic devices (101, 201).

11. In any one of paragraphs 1 to 10, When the above instructions are executed by the at least one processor (120), the electronic device (101, 201) causes at least: storing at least a portion of said at least one first image or said at least one second image in association with metadata, wherein said metadata comprises at least a portion of said prompt, information about a generative AI model of said electronic device (101, 201) or of said external server (203), information about an object included in said at least one first image or said at least one second image, or at least one of options associated with image generation; Displaying an indicator representing at least a portion of the above metadata through the display To do, Electronic devices (101, 201).

12. In any one of paragraphs 1 to 11, When the above instructions are executed by the at least one processor (120), the electronic device (101, 201) causes at least: In response to receiving one second high-resolution image corresponding to one of the at least one second image from the external server (203), displaying at least a portion of the one second high-resolution image by replacing at least a portion of the one second image To do, Electronic devices (101, 201).

13. In a method performed by an electronic device (101, 201), The above electronic device (101, 201) display; at least one processor (120); and It includes a memory (130) that stores instructions executable by at least one processor (120), An action of determining a prompt based at least in part on user input to said electronic device (101, 201); An operation of generating at least one first image, which is an on-device generated image, based on at least a portion of the prompt, using a generative artificial intelligence (AI) model of the electronic device (101, 201); An action of transmitting an image generation request including at least a portion of the above prompt to an external server (203); An operation of receiving at least one second image, which is a server generated image generated based on at least a part of the image generation request, from the external server (203), wherein the at least one second image is generated using a generative AI model of the external server (203); and An action of displaying at least a portion of at least one of the first images or at least one of the second images through the display. Including, method.

14. A computer-readable recording medium storing a program for executing the method of claim 13 in combination with hardware.

15. In electronic devices (101, 201), display; at least one processor (120); and It includes a memory (130) that stores instructions executable by at least one processor (120), When the above instructions are executed by the at least one processor (120), the electronic device (101, 201) causes at least: Receiving user input to generate at least one image via the display; Using a generative artificial intelligence (AI) model of the electronic device (101, 201), at least one first image, which is an on-device generated image, is generated based on at least a portion of the user input, Sending an image generation request generated at least in part based on the above user input to an external server (203), Receive at least one second image from the external server (203), which is a server generated image generated based on at least a portion of the image generation request, wherein the at least one second image is generated using another generative AI model of the external server (203); Displaying together through the display at least a portion of said at least one first image and at least a portion of said at least one second image To do, Electronic devices (101, 201).

Images