Electronic device and method for generating image with changed timeline, and non-transitory computer-readable recording medium

Abstract

An electronic device is disclosed. The electronic device can: select an object from among one or more objects included in an input image; identify indication information about a state change of the selected object; and obtain, on the basis of the indication information, a prompt related to the input image. The prompt can include one or more words for changing the state of the selected object included in the input image. The state can be included in different states of the selected object. The electronic device inputs the input image and the prompt into a generative artificial intelligence model, and thus can generate one or more result images in which the state of the selected object has been changed.

Description

Electronic device, method, and non-transitory computer-readable recording medium for generating an image with a changed timeline

[0001] The following description relates to an electronic device, a method, and a non-transitory computer-readable recording medium for generating an image with a changed timeline.

[0002] Artificial intelligence (hereinafter referred to as AI) is a technology for simulating the neural activity of humans or other living organisms, such as perception and / or inference. AI can be implemented using hardware, software, or a combination of these designed to perform computations to simulate neural activity.

[0003] An electronic device is disclosed. The electronic device may include a processor. The electronic device may include a memory storing instructions. The instructions, when executed by the processor, may cause the electronic device to select an object from among at least one object included in an input image. The instructions, when executed by the processor, may cause the electronic device to check instruction information regarding a change in the state of the selected object. The instructions, when executed by the processor, may cause the electronic device to obtain a prompt related to the input image based on the instruction information. The prompt may include at least one word for changing the state of the selected object included in the input image. The state may be included in different states of the selected object. The instructions, when executed by the processor, may cause the electronic device to generate at least one result image in which the state of the selected object is changed by inputting the input image and the prompt into a generative AI model.

[0004] A method is disclosed. The method can be performed by an electronic device. The method can include an operation of selecting an object from among at least one object included in an input image. The method can include an operation of checking indication information regarding a change in the state of the selected object. The method can include an operation of obtaining a prompt related to the input image based on the indication information. The prompt can include at least one word for changing the state of the selected object included in the input image. The state can be included in different states of the selected object. The method can include an operation of generating at least one result image in which the state of the selected object is changed by inputting the input image and the prompt to a generative AI model.

[0005] A non-transitory computer-readable storage medium is disclosed. The non-transitory computer-readable storage medium may store a program including instructions. The instructions, when executed by a processor of the electronic device, may cause the electronic device to select an object from among at least one object included in an input image. The instructions, when executed by the processor of the electronic device, may cause the electronic device to check instruction information regarding a change in state of the selected object. The instructions, when executed by the processor of the electronic device, may cause the electronic device to obtain a prompt related to the input image based on the instruction information. The prompt may include at least one word for changing a state of the selected object included in the input image. The state may be included in different states of the selected object. The instructions, when executed by a processor of the electronic device, may cause the electronic device to generate at least one result image in which the state of the selected object is changed by inputting the input image and the prompt to a generative AI model.

[0006] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0007] Figure 2 is a block diagram of an electronic device.

[0008] Figure 3 illustrates an example of an operation in which an electronic device generates an image to provide an intermediate experience to a user.

[0009] Figure 4 illustrates an example of an operation in which an electronic device generates an image to provide an intermediate experience to a user.

[0010] FIG. 5 illustrates an example of an operation in which an electronic device generates an image to provide an intermediate experience to a user.

[0011] FIG. 6A illustrates an example of an operation in which an electronic device generates an image to provide an intermediate experience to a user.

[0012] FIG. 6b illustrates an example of an operation in which an electronic device generates an image to provide an intermediate experience to a user.

[0013] Figure 7 illustrates an example of an operation in which an electronic device generates an image to provide an intermediate experience to a user.

[0014] Figure 8 is a flowchart showing the operation of an electronic device.

[0015] Figure 9 is a flowchart showing the operation of an electronic device.

[0016] Figure 10 is a flowchart showing the operation of an electronic device.

[0017] Figure 11 is a flowchart showing the operation of an electronic device.

[0018] Figure 12 is a flowchart showing the operation of an electronic device.

[0019] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments.

[0020] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0021] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0022] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0023] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0024] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0025] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0026] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0027] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0028] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0029] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0030] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0031] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0032] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0033] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0034] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0035] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0036] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0037] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for realizing eMBB, a loss coverage (e.g., 664 dB or less) for realizing mMTC, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 6 ms or less for round trip) for realizing URLLC.

[0038] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0039] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0040] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0041] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0042] Figure 2 is a block diagram of an electronic device (101).

[0043] Referring to FIG. 2, the electronic device (101) includes a processor (120), a memory (130), a display (260), and a communication circuit (290). The electronic device (101) may correspond to the electronic device (101) of FIG. 1. The processor (120) may correspond to the processor (120) of FIG. 1.

[0044] The memory (130) may correspond to the memory (130) of FIG. 1. The memory (130) may include a program (140). The program (140) may correspond to the program (140) of FIG. 1. The display (260) may correspond to the display module (160) of FIG. 1. The communication circuit (290) may correspond to the communication module (190) of FIG. 1.

[0045] The program (140) includes a UI (user interface) manager (210), an image preprocessing unit (220), a crawling module (230), and an image processing unit (240). The UI manager (210), the image preprocessing unit (220), the crawling module (230), and the image processing unit (240) can be executed by the processor (120).

[0046] The UI manager (210) is configured to display selectable images via the display (260). In one exemplary embodiment, the UI manager (210) includes a portion of a gallery application. In one exemplary embodiment, the UI manager (210) may be interconnected via an application programming interface (API) for providing functionality designated for the gallery application. The API may be called from the gallery application to provide the UI manager (210) with information about one or more images selected via the gallery application. The API may be called from the gallery application to display a screen including one or more selectable prompts associated with one or more images selected via the gallery application via the UI manager (210).

[0047] The UI manager (210) is configured to identify an input image. The UI manager (210) can identify an input image from among selectable images based on a user input. The UI manager (210) can identify an input image acquired through an external electronic device (102, 103). The input image may include one image or one or more input images. At least some objects of the one or more input images may differ. At least some meta information (e.g., EXIF ​​(exchangeable image file format)) of the one or more input images may differ. The meta information for each of the one or more input images may differ in at least one of location information (e.g., latitude information and longitude information), camera setting information (e.g., size, pixels, F number, color temperature, shutter speed, flash, ISO (International Standard Organization) sensitivity, magnification, field of view), composition, or camera angle, shooting device, or shooting time.

[0048] The UI manager (210) is configured to obtain a prompt. The UI manager (210) may obtain a prompt related to an input image based on a user input. In one embodiment, the UI manager (210) may obtain instruction information regarding a state change. In one embodiment, the UI manager (210) may obtain a prompt based on instruction information regarding an intended state change. In one embodiment, the UI manager (210) may obtain a prompt related to an input image based on a user input on a screen displayed on a gallery application. Here, the screen may include one or more selectable prompts related to one or more input images. In one embodiment, the UI manager (210) may obtain a prompt related to an input image through a prompt generator. In one embodiment, the prompt generator may be an artificial intelligence (AI) model (e.g., a stable diffusion model) that can convert an input image into text. In one embodiment, the instruction information may be a guide for generating an image based on user input on a screen displayed in the gallery application. For example, the instruction information may include information regarding composition and / or presentation time points. The concept of presentation time points is discussed in more detail below.

[0049] A prompt may include data for guiding the generation of an image based on an input image. The prompt may be instructions, such as work instructions, for a generative AI model included in the image processing unit (240). The work instructions include, for example, "generate an intermediate experience image," "generate an intermediate experience image with specified camera setting information and / or image effects applied." The prompt may be a contextual input (e.g., information about an object included in the input image, information about the arrangement of the object, or information about the composition of the object) input to the generative AI model. For example, the prompt may be data for setting the depiction direction of one or more objects included in an output image generated based on the input image. The prompt may be data for setting camera setting information, the composition, and / or the image effect of an output image generated based on the input image. The prompt may be data including a set of words. For example, the words may describe a state or property of an image to be generated and / or at least one object included in the image.

[0050] A generative AI model may include multiple parameters related to a neural network having a structure based on an encoder and a decoder, such as a transformer. The generative AI model may include a bidirectional model based on learning about an encoder (e.g., bidirectional encoder representations from transformers (BERT)) or an auto-encoding model (e.g., a diffusion model). The generative AI model may include an auto-regressor model based on learning about a decoder (e.g., a generative pre-trained transformer (GPT)). The generative AI model may include a sequence-to-sequence model based on learning about an encoder and a decoder (e.g., stable diffusion, DALL-E 2). The generative AI model may include a large language model (LLM) for processing natural language based on massive parameters. Generative AI models may include parameters for driving neural networks such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), feedforward neural networks (FNNs), and / or long short-term memory (LSTMs).

[0051] The image preprocessing unit (220) is configured to preprocess an input image.

[0052] The image preprocessing unit (220) can perform image processing on the input image. For example, the image processing performed by the image preprocessing unit (220) can include processing (e.g., denoising, deblurring, dehazing, white balance) to improve the image quality of the input image. By the image preprocessing unit (220) performing processing to improve the image quality of the input image, the quality of the image generated by the image processing unit (240) can be improved.

[0053] The image preprocessing unit (220) can detect and classify one or more objects in an input image. For example, the image preprocessing unit (220) can detect the area of ​​one or more objects in the input image. For example, the image preprocessing unit (220) can identify the type (e.g., food, plate, person, animal, plant, object) of one or more objects detected in the input image. For example, if the input image includes a plate containing food, the image preprocessing unit (220) can identify the food and the plate.

[0054] The image preprocessing unit (220) can separate one or more objects identified in the input image from the input image. The image preprocessing unit (220) can segment one or more objects based on the boundaries of each of the one or more objects identified in the input image. For example, if the input image includes a plate containing food, the image preprocessing unit (220) can identify the area of ​​the food and the area of ​​the plate within the input image.

[0055] The image preprocessing unit (220) can estimate the area of ​​each of one or more objects outside the input image based on the area of ​​each of one or more objects identified in the input image. The complete area of ​​an object can include the area of ​​the object identified in the input image and the area of ​​the object outside the input image. For example, if the input image includes a plate containing food, the image preprocessing unit (220) can estimate the area of ​​the food outside the input image and the area of ​​the plate.

[0056] The image preprocessing unit (220) may transmit data related to one or more objects identified in the input image to the image processing unit (240). The data related to one or more objects may include the type of the detected object, the area of ​​the detected object, and / or an object image for the object segmented from the input image.

[0057] The crawling module (230) can obtain data related to an input image. The crawling module (230) can crawl data related to the input image on the Internet (e.g., WWW (World Wide Web)). The crawling module (230) can obtain data related to the input image through crawling based on meta information (e.g., EXIF) included in the input image. For example, crawling based on meta information may include an operation for obtaining one or more data (e.g., images, videos, text) from one or more web pages searched based on location information (e.g., latitude information and longitude information) included in the meta information. The crawling module (230) can obtain data related to the input image obtained through crawling based on one or more objects included in the input image. For example, the data related to the input image may be an image, video, and / or text (e.g., information about the object) that includes the same object as the object included in the input image. For example, data associated with an input image may be an image and / or video taken at the location where the input image was taken.

[0058] The crawling module (230) can input data related to an input image as a prompt to the image processing unit (240). Hereinafter, data related to an input image acquired by the crawling module (230) may be referred to as crawled data.

[0059] The image processing unit (240) can generate a generated image based on the preprocessed input image. The image processing unit (240) can generate a generated image based on the preprocessed input image and crawled data. The image processing unit (240) can generate an image in which at least one of the area, composition, expression time point, or light source of one or more objects included in the input image is changed. The change in the area of ​​the object may mean that the area of ​​the object included in the generated image is changed to include a newly drawn area from the area of ​​the object included in the input image. For example, the newly drawn area may correspond to an area of ​​the object that is out of the input image. The change in the composition of the object may mean that the angle of the camera of the generated image or the line of sight is changed from the angle of the camera of the input image. The change in the composition of the object may mean that the angle of the camera of the generated image or the line of sight is changed from the angle of the camera of the input image while the position of the object is fixed. A change in the composition of an object may mean that the position of the object within the generated image is changed from the position of the object within the input image. A change in the composition of an object may mean that the arrangement of objects within the generated image is changed from the arrangement of objects within the input image. A change in the composition of an object may mean that the scale or size of the object within the generated image is changed from the scale or size of the object within the input image. A change in the presentation time of an object may mean that the presentation time within the life cycle or timeline of the object included in the generated image is different from the presentation time within the life cycle of the object included in the input image. For example, the life cycle or timeline may include a time interval from the start time of an object to the end time. For example, the life cycle may include a time interval from the start time of a specific action or state change of the object to the end time.For example, if an object is food, at the beginning of its life cycle, the object may have a presentation point before eating the food. For example, if an object is food, at the end of its life cycle, the object may have a presentation point after eating the food. For example, if an object is food, at a midpoint in its life cycle, the object may have a presentation point between the presentation point before eating the food and the presentation point after eating the food. For example, if an object is the sun, at the beginning of its life cycle, the object may have a presentation point related to sunrise. For example, if an object is the sun, at the end of its life cycle, the object may have a presentation point related to sunset. For example, if an object is the sun, at a midpoint in its life cycle, the object may have a presentation point between sunrise and sunset. A change in the light source of an object may mean that the position of the light source illuminating the object in the generated image changes from the position of the light source illuminating the object in the input image. A change in the light source of an object may mean that the color temperature of the light source illuminating the object included in the generated image changes from the color temperature of the light source illuminating the object included in the input image.

[0060] The image processing unit (240) may generate an image in which the background of one or more objects included in the input image is changed. The change in the background of the object may mean that an area outside the area of ​​the object included in the generated image is newly drawn. The image processing unit (240) may generate a generated image having the same background as the background of one or more objects included in the input image. The fact that the background of the generated image is the same as the background of one or more objects included in the input image may mean that the texture, color, and / or shape of the backgrounds are the same. For example, if the background of one or more objects included in the input image is a wooden table, the background of the generated image may be a wooden table. The image processing unit (240) may generate a generated image having a background similar to the background of one or more objects included in the input image. The fact that the background of the generated image is similar to the background of one or more objects included in the input image may mean that the texture, color, and / or shape of the backgrounds are similar to each other. The similarity of the background of the generated image to the background of one or more objects included in the input image may mean that the background characteristics of the generated image are similar to the background characteristics of the input image. The background characteristics may refer to the texture, color, and / or shape of the backgrounds.

[0061] The image processing unit (240) may include a composition change module (241), an image filler (243), an expression change module (245), and a light source control module (247).

[0062] Below, the operation of the image processing unit (240) generating a generated image for an input image through a composition change module (241), an image filler (243), an expression change module (245), and a light source control module (247) is described.

[0063] The composition change module (241) can generate one or more first intermediate stage images in which the composition of one or more objects included in an input image is changed. The composition change module (241) can generate one or more first intermediate stage images in which the composition of one or more objects included in an input image is changed by inputting an input image or a preprocessed input image to a generative AI model for composition change. The composition change module (241) can generate one or more first intermediate stage images in which the composition of one or more objects included in an input image is changed by inputting an input image and a prompt to the generative AI model. The composition change module (241) can generate one or more first intermediate stage images in which the composition of one or more objects included in an input image is changed by inputting a prompt related to the composition of one or more objects to the generative AI model. The composition change module (241) can generate one or more first intermediate stage images in which the angle, line of sight, position, shooting position, arrangement, and / or magnification of a camera for one or more objects included in an input image are changed by inputting a prompt related to the composition of one or more objects to the generative AI model. The prompt related to the composition may be words indicating the angle, line of sight, position of an object, or shooting position of a camera), arrangement of an object, and / or magnification of an image. Examples of prompts related to the composition of an object may include 'change the state of the food being photographed from above', 'change the state of the food being photographed from the front', 'change the state of being illuminated by the sun', 'change the arrangement of plates to a specified arrangement', 'change the size of a plate', or 'change the size of an image'. The prompt input to the generative AI model of the composition change module (241) may include a prompt obtained through the UI manager (210) and / or crawling data obtained through the crawling module.A prompt related to a composition may include a word indicating a selected composition from among one or more compositions suggested to the user. The UI manager (210) may identify at least one composition based on an input to a UI including examples of one or more selectable compositions. The composition change module (241) may generate one or more first intermediate stage images in which a camera angle, a line of sight, a location, a shooting location, an arrangement, and / or a magnification of one or more objects included in an input image are changed by inputting a prompt including a word indicating at least one identified composition to a generative AI model. The UI manager (210) may select one or more selectable compositions based on the number of input images, the number of objects included in the input images, and / or characteristics of the objects. The prompt related to a composition may include a word indicating at least one composition selected based on the number of input images, the number of objects included in the input images, and / or characteristics of the objects.

[0064] The image filler (243) is configured to generate an image on which an image filling process (e.g., in-painting or out-painting) is performed based on an area of ​​one or more objects included in an input image. The image filler (243) can generate one or more second intermediate stage images by inputting one or more first intermediate stage images into a generative AI model for the image filling process. The image filler (243) can generate one or more second intermediate stage images by inputting one or more first intermediate stage images and a prompt into the generative AI model. Here, the image filling process may be a process of newly drawing an area other than an area of ​​one or more objects. The prompt for the image filling process may be words indicating the texture of the area to be filled. Examples of prompts for the image filling process may include 'plates of food placed on a table', 'flower bed', or 'street covered with fallen leaves'. The prompt input to the generative AI model of the image filler (243) may include a prompt obtained through the UI manager (210) and / or crawling data obtained through the crawling module.

[0065] The image filler (243) may redraw an area outside the area of ​​one or more objects in each of one or more first intermediate stage images by inputting a prompt to guide the image filling process into the generative AI model for the image filling process. The newly drawn area in each of one or more first intermediate stage images may include an area of ​​an object outside the area of ​​the input image. The newly drawn area in each of one or more first intermediate stage images may include a background area outside the area of ​​the object.

[0066] The expression change module (245) is configured to generate one or more third intermediate stage images in which the expression time points of one or more objects included in the input image are changed.

[0067] The expression change module (245) can generate one or more third intermediate stage images in which the expression time of one or more objects included in the input image is changed by inputting one or more second intermediate stage images into a generative AI model for changing the expression time. The expression change module (245) can generate one or more third intermediate stage images in which the expression time of one or more objects included in the input image is changed by inputting one or more second intermediate stage images and a prompt into the generative AI model. The expression change module (245) can generate one or more third intermediate stage images in which the expression time of one or more objects included in the input image is changed by inputting a prompt related to the expression time of one or more objects into the generative AI model. The expression change module (245) can generate one or more third intermediate stage images in which the time of one or more objects included in the input image is changed according to the life cycle of one or more objects by inputting a prompt related to the expression time of one or more objects into the generative AI model. The prompt related to the expression time may be words related to the time. Examples of prompts related to the viewpoint of an object may include 'change to a state before eating food', 'change to a state after eating food', 'change to a state just before the sun rises', 'change to a different season', 'change to a different time zone', 'change to a state after it rains', or 'change to a state soaked in rain'. The prompts input to the generative AI model of the expression change module (245) may include prompts obtained through the UI manager (210) and / or crawling data obtained through the crawling module.

[0068] The light source control module (247) is configured to generate an image in which the light source of one or more objects included in the input image is changed. The light source control module (247) can generate one or more fourth intermediate stage images in which the light source of one or more objects is changed by inputting one or more third intermediate stage images into a relight model. The relight model may be based on an AI model.

[0069] The UI manager (210) may display at least some of the generated images among the images finally generated by the image processing unit (240) through the display (260). For example, the UI manager (210) may sequentially display at least some of the generated images through the display (260). For example, the UI manager (210) may play an animation in which at least some of the generated images are arranged through the display (260). For example, the UI manager (210) may play at least some of the generated images, which have a composition and expression time selected by a user input, through the display (260). The finally generated images may be referred to as intermediate experience images. The intermediate experience image may be an image having a different composition and / or expression time from the input image. The intermediate experience image may have a different composition from the input image if the prompt includes a word related to composition.

[0070] The UI manager (210) may feed back a prompt based on a user input for the generated image being displayed to the image processing unit (240). The prompt fed back to the image processing unit (240) may be an operation instruction for the AI ​​model of the image processing unit (240). For example, the user input may include an input for selecting a specific object from among at least one object included in the generated image being displayed. For example, the user input may include an input for adjusting the position of a specific object included in the generated image being displayed. For example, the user input may include an input for adjusting the size of a specific object included in the generated image being displayed. For example, the user input may include an input for deleting a specific object included in the generated image being displayed. The user input may include a touch input for an object through the display (260). The user input may include an input for at least one control element (e.g., executable objects) displayed through the display (260). Each of the at least one control element (e.g., executable objects) may be associated with a designated function. For example, each of the at least one control element (e.g., executable objects) may include a function for selecting a specific object, a function for adjusting the position of a specific object, and / or a function for adjusting the size of a specific object. Each of the at least one control element (e.g., executable objects) may be a UI overlaid on the generated image being displayed. Each of the at least one control element (e.g., executable objects) may be a UI positioned around the generated image being displayed.

[0071] The image processing unit (240) can change the finally generated images based on the fed-back prompt. The image processing unit (240) can generate images in which at least one of the area, composition, presentation time, or light source of one or more objects included in the finally generated images is changed based on the fed-back prompt. For example, the composition change module (241) can generate one or more updated first intermediate stage images by inputting the fed-back prompt into a generative AI model for composition change. For example, the image filler (243) can generate one or more updated second intermediate stage images by inputting the fed-back prompt into a generative AI model for the image filling process. For example, the expression change module (245) can generate one or more updated third intermediate stage images by inputting the fed-back prompt into a generative AI model for changing the presentation time.

[0072] In FIG. 2, the composition change module (241), the image filler (243), the expression change module (245), and the light source control module (247) are illustrated as operating sequentially, but this is merely an example. Depending on the embodiment, the operating order of the composition change module (241), the image filler (243), the expression change module (245), and the light source control module (247) may be changed. In order to generate a result image, only at least some components of the composition change module (241), the image filler (243), the expression change module (245), or the light source control module (247) may operate.

[0073] As described above, the electronic device (101) can provide an experience that allows the user to view an image with a composition and / or expression point that was not captured by generating an image in which at least one of the area, composition, expression point, or light source of the input image has been changed.

[0074] FIG. 3 illustrates an example of an operation in which an electronic device (101) generates an image to provide an intermediate experience to a user.

[0075] Figure 3 is described with reference to the contents of Figures 1 and 2.

[0076] Referring to FIG. 3, an input image (310) can be selected. The input image (310) can be selected from among selectable images displayed through the gallery application.

[0077] The UI manager (210) can identify the selected input image (310). The UI manager (210) can obtain a prompt related to the selected input image (310). The prompt related to the input image (310) may be a prompt selected from one or more prompts selectable on a screen displayed on the gallery application. The prompt related to the input image (310) may be a prompt generated through a prompt generator. The prompt may include data for guiding the generation of an image based on the input image. The prompt may be a work instruction for a generative AI model included in the image processing unit (240).

[0078] The UI manager (210) can transmit the selected input image (310) to the image preprocessing unit (220). The UI manager (210) can transmit the selected input image (310) to the crawling module (230). The UI manager (210) can transmit a prompt related to the selected input image (310) to the image processing unit (240).

[0079] The image preprocessing unit (220) can detect and classify one or more objects (321, 325) in the input image (310). For example, the image preprocessing unit (220) can detect an object (321) in the input image (310) and classify the detected object (321) as food. For example, the image preprocessing unit (220) can detect an object (325) in the input image (310) and classify the detected object (325) as a plate.

[0080] The image preprocessing unit (220) can separate one or more objects (321, 325) identified in the input image (310) from the input image (310). The image preprocessing unit (220) can segment one or more objects (321, 325) based on the boundaries of each of the one or more objects (321, 325) identified in the input image (310).

[0081] The image preprocessing unit (220) can estimate the area (329) of each of one or more objects (321, 325) that are outside the input image (310) based on the area of ​​each of one or more objects (321, 325) identified in the input image (310).

[0082] The image preprocessing unit (220) can transmit data related to one or more objects (321, 325) identified in the input image (310) to the image processing unit (240). The data related to one or more objects (321, 325) can include the type of the detected object, the area of ​​the detected object, and / or an object image for the object segmented from the input image (310).

[0083] The crawling module (230) can obtain data (350) related to the input image (310). The crawling module (230) can crawl the data (350) related to the input image (310) on the Internet. The crawling module (230) can obtain the data (350) related to the input image (310) through crawling based on meta information (e.g., EXIF) included in the input image (310). For example, crawling based on meta information can include an operation to obtain one or more data (e.g., images, videos, texts) from one or more web pages searched based on location information (e.g., latitude information and longitude information) included in the meta information. The crawling module (230) can obtain data (350) related to the input image (310) through crawling based on one or more objects (321, 325) included in the input image (310).

[0084] The crawling module (230) can input data related to the input image as a prompt to the image processing unit (240).

[0085] The image processing unit (240) can generate a set of generated images (330) based on the preprocessed input image (320). The image processing unit (240) can generate a set of images (330) in which at least one of the area, composition, expression point of view, or light source of one or more objects (321, 325) included in the preprocessed input image (320) has been changed. For example, referring to FIG. 3, the input image (310) may be an image taken from a high view of food being eaten. The generated images (331, 332, 333) included in the image set (330) may be images taken from a bird's eye view of the food. The generated images (341, 342, 343) included in the image set (330) may be images taken from a high view. The generated images (331, 341) included in the image set (330) may be images of food taken before eating. The generated images (332, 342) included in the image set (330) may be images of food taken during eating. The generated images (333, 343) included in the image set (330) may be images of food taken after eating. For example, the generated images included in the image set (330) may include images taken by various camera angles (e.g., neutral view, low view, worm's eye view). For example, the generated images included in the image set (330) may include images having various expression points according to the life cycle.

[0086] The image processing unit (240) can generate one or more first intermediate stage images in which the composition of one or more objects (321, 325) included in the input image (310) is changed. The image processing unit (240) can generate one or more first intermediate stage images in which the composition of one or more objects (321, 325) included in the input image (310) is changed by inputting a prompt related to the composition of one or more objects (321, 325) into the generative AI model. The prompt related to the composition may be words indicating the angle of the camera, or the line of sight, position, arrangement, and / or magnification. Examples of prompts related to the composition of objects may include 'change the state of the food to a state of being photographed from above', 'change the state of the food to a state of being photographed from the front', 'change the state of the food to a state of being located in the center of the image', 'change the arrangement of plates to a specified arrangement', or 'change the size of the plates'.

[0087] The image processing unit (240) may generate one or more second intermediate stage images on which an image filling process (e.g., in-painting or out-painting) is performed based on an area of ​​one or more objects (321, 325) included in the input image (310). The image processing unit (240) may perform an image filling process on one or more first intermediate stage images by inputting a prompt to the generative AI model. Here, the image filling process may be a process of newly drawing an area other than an area of ​​one or more objects (321, 325). The prompt for the image filling process may be words indicating the texture of the area to be filled. An example of a prompt for the image filling process may be 'a table with plates of food on it.' The newly drawn portion in each of the one or more first intermediate stage images may include an area (329) of an object outside the input image (310). The newly drawn portion in each of the one or more first intermediate stage images may include a background area outside the area of ​​the object.

[0088] The image processing unit (240) can generate one or more third intermediate stage images in which the expression time of one or more objects (321, 325) included in the input image (310) is changed. The image processing unit (240) can generate one or more third intermediate stage images in which the expression time of one or more objects (321, 325) included in the input image (310) is changed by inputting a prompt related to the expression time of one or more objects (321, 325) into the generative AI model. The prompt related to the expression time may be words related to the time point. Examples of prompts related to the viewpoint of an object may include 'change to a state before eating food', 'change to a state after eating food', or 'change to a state of holding a utensil to eat food'.

[0089] The light source control module (247) can generate one or more fourth intermediate stage images in which the light source for one or more objects (321, 325) included in the input image (310) is changed. The light source control module (247) can generate one or more fourth intermediate stage images in which the light source for one or more objects (321, 325) included in the input image (310) is changed. The one or more fourth intermediate stage images can be the final generated images (331, 332, 333, 341, 342, 342).

[0090] The UI manager (210) can display at least some of the generated images (331, 332, 333, 341, 342, 342) finally generated by the image processing unit (240) through the display (260). For example, the UI manager (210) can sequentially display at least some of the generated images through the display (260). For example, the UI manager (210) can play an animation in which at least some of the generated images are arranged through the display (260). For example, the UI manager (210) can play a generated image having a composition and expression point selected by a user input through the display (260) among at least some of the generated images.

[0091] The UI manager (210) may provide a UI for selecting an image to be displayed through the display (260). For example, the UI manager (210) may select an image to be displayed from among the images (331, 332, 333, 341, 342, 342) finally generated by the image processing unit (240) based on an input for a UI (e.g., a slider) for changing an image to be displayed. For example, the UI (e.g., a slider) for changing an image to be displayed may cause the presentation time of the image to be displayed to come closer to the start time as it moves in a first direction. For example, the UI (e.g., a slider) for changing an image to be displayed may cause the presentation time of the image to be displayed to come closer to the end time as it moves in a second direction opposite to the first direction. For example, the UI (e.g., a slider) for changing an image to be displayed may cause the composition of the image to be displayed to change in the first direction as it moves in the first direction. For example, a UI (e.g., a slider) for changing the image to be displayed may change the composition of the image to be displayed in the second direction as it moves in the second direction opposite to the first direction.

[0092] For example, the UI manager (210) may sequentially display images according to the life cycle to provide a time-lapse experience. For example, the UI manager (210) may sequentially display the final generated images (331, 332, 333). For example, the UI manager (210) may sequentially display the final generated images (341, 342, 343).

[0093] For example, the UI manager (210) may sequentially display images arranged in the order in which the camera angles and / or camera positions are moved, in order to provide the user with an experience of recognizing a three-dimensional (3D) space based on two-dimensional (2D) images. For example, the UI manager (210) may sequentially display the final generated images (331, 341). For example, the UI manager (210) may sequentially display the final generated images (332, 342). For example, the UI manager (210) may sequentially display the final generated images (333, 343).

[0094] The UI manager (210) can feed back a prompt based on user input regarding the generated image being displayed to the image processing unit (240). The prompt fed back to the image processing unit (240) may be a work instruction for the AI ​​model of the image processing unit (240).

[0095] The image processing unit (240) can change the final generated images based on the feedback prompt. The image processing unit (240) can create an image in which at least one of the area, composition, expression point of view, or light source of one or more objects (321, 325) included in the final generated images has been changed based on the feedback prompt.

[0096] As described above, the electronic device (101) can provide an experience that allows the user to view an image with a composition and / or a viewpoint that was not captured by generating an image in which at least one of the area, composition, viewpoint, or light source of the input image has been changed. For example, the electronic device (101) can allow the user to view a photo of food before or after the photo has been captured by generating an image in which at least one of the area, composition, viewpoint, or light source of the input image has been changed.

[0097] FIG. 4 illustrates an example of an operation in which an electronic device (101) generates an image to provide an intermediate experience to a user.

[0098] Figure 4 is described with reference to the contents of Figures 1 and 2.

[0099] Referring to FIG. 4, an input image set (410) can be selected. A plurality of input images (310, 411, 415) can be selected from among selectable images displayed through a gallery application. Some of the plurality of input images (310, 411, 415) included in the input image set (410) can be selected through an external electronic device (102, 103) rather than the electronic device (101).

[0100] The UI manager (210) can identify a selected set of input images (410). The UI manager (210) can obtain prompts related to the selected plurality of input images (310, 411, 415). The prompts related to the input images (310, 411, 415) may be prompts selected from one or more prompts selectable on a screen displayed on the gallery application. The prompts related to the input images (310, 411, 415) may be prompts generated through a prompt generator. The prompts may include data for guiding the generation of images based on the input images. The prompts may be work instructions for a generative AI model included in the image processing unit (240).

[0101] The UI manager (210) can transmit the selected input image set (410) to the image preprocessing unit (220). The UI manager (210) can transmit the selected input image set (410) to the crawling module (230). The UI manager (210) can transmit a prompt related to the selected input image set (410) to the image processing unit (240).

[0102] The image preprocessing unit (220) can detect and classify one or more objects (321, 325, 431, 433, 435, 437, 441, 443, 445, 447) in the input images (310, 411, 415). For example, the image preprocessing unit (220) can detect objects (321, 431, 441) in the input images (310, 411, 415) and classify the detected objects (321, 431, 441) as food. For example, the image preprocessing unit (220) can detect objects (435, 445) in input images (310, 411, 415) and classify the detected objects (435, 445) as tableware. For example, the image preprocessing unit (220) can detect objects (325, 433, 437, 443, 447) in input images (310, 411, 415) and classify the detected objects (325, 433, 437, 443, 447) as plates.

[0103] The image preprocessing unit (220) can separate one or more objects (321, 325, 431, 433, 435, 437, 441, 443, 445, 447) identified in the input images (310, 411, 415) from the input images (310, 411, 415). The image preprocessing unit (220) can segment one or more objects (321, 325, 431, 433, 435, 437, 441, 443, 445, 447) based on the boundaries of each of the one or more objects (321, 325, 431, 433, 435, 437, 441, 443, 445, 447) identified in the input images (310, 411, 415).

[0104] The image preprocessing unit (220) can estimate the area (329) of each of one or more objects (321, 325, 431, 433, 435, 437, 441, 443, 445, 447) that are outside the input images (310, 411, 415) based on the area of ​​each of one or more objects (321, 325, 431, 433, 435, 437, 441, 443, 445, 447) identified in the input images (310, 411, 415).

[0105] The image preprocessing unit (220) can transmit data related to one or more objects (321, 325, 431, 433, 435, 437, 441, 443, 445, 447) identified in the input images (310, 411, 415) to the image processing unit (240). The data related to one or more objects (321, 325, 431, 433, 435, 437, 441, 443, 445, 447) can include the type of the detected object, the area of ​​the detected object, and / or an object image for the object segmented from the input images (310, 411, 415).

[0106] The crawling module (230) is configured to obtain data (480) related to input images (310, 411, 415). The crawling module (230) can crawl data (480) related to input images (310, 411, 415) on the Internet. The crawling module (230) can obtain data (480) related to input images (310, 411, 415) through crawling based on meta information (e.g., EXIF) included in the input images (310, 411, 415). Referring to FIG. 4, data (480) may include objects (e.g., food, a set of dishes, a plate) related to objects (321, 325, 431, 433, 435, 437, 441, 443, 445, 447) included in input images (310, 411, 415).

[0107] The crawling module (230) can input data (480) related to input images (310, 411, 415) as a prompt to the image processing unit (240).

[0108] The image processing unit (240) is configured to generate a generated image set (450) based on a preprocessed image set (420). The image processing unit (240) can generate a generated image set (450) in which at least one of the area, composition, expression viewpoint, or light source of objects (321, 325, 431, 433, 435, 437, 441, 443, 445, 447) included in the preprocessed input images (310, 411, 415) has been changed. For example, referring to FIG. 4, the input images (310, 411, 415) may be different images. One or more generated images (451, 453, 455, 457, 459) included in the generated image set (450) can include objects (321, 325, 431, 433, 435, 437, 441, 443, 445, 447) included in the input images (310, 411, 415) within a single image. Each of the one or more generated images (451, 453, 455, 457, 459) included in the generated image set (450) can have objects (321, 325, 431, 433, 435, 437, 441, 443, 445, 447) included in the input images (310, 411, 415) arranged. Referring to FIG. 4, objects (461, 463, 465, 467, 469) may be arranged within the generated image (451). For example, object (461) may correspond to objects (321, 325). For example, object (463) may correspond to objects (431, 433). For example, object (465) may correspond to objects (435, 437). For example, object (467) may correspond to objects (441, 443). For example, object (469) may correspond to objects (445, 447). Each of one or more generated images (451, 453, 455, 457, 459) may have a newly drawn area outside the area of ​​each of the objects (461, 463, 465, 467, 469).

[0109] The UI manager (210) can display at least some of the generated images (451, 453, 455, 457, 459) finally generated by the image processing unit (240) through the display (260). For example, the UI manager (210) can sequentially display at least some of the generated images through the display (260). For example, the UI manager (210) can play an animation in which at least some of the generated images are arranged through the display (260). For example, the UI manager (210) can play a generated image having a composition and expression point selected by a user input through the display (260) among at least some of the generated images.

[0110] For example, the UI manager (210) may sequentially display images according to a life cycle to provide a time-lapse experience. For example, the UI manager (210) may sequentially display images arranged in the order in which the camera angle and / or camera position are sequentially moved to provide a user with an experience of perceiving 3D space based on 2D images.

[0111] The UI manager (210) can generate tag information for objects (321, 325, 431, 433, 435, 437, 441, 443, 445, 447) of generated images (451, 453, 455, 457, 459) included in the generated image set (450) based on information (e.g., type, where sold, where bought, location, URL (uniform resource locator)) about at least one object included in the data (480). The UI manager (210) can provide a search result of the tag information in response to an input for selecting the tag information while the generated images (451, 453, 455, 457, 459) are displayed through the display (260). For example, the UI manager (210) may provide a result recommending nearby restaurants that sell steak corresponding to the object (465). For example, the UI manager (210) may provide a result recommending one or more restaurants that sell steak corresponding to the object (465) in order of distance. For example, the UI manager (210) may display a page based on a URL where the steak corresponding to the object (465) can be purchased.

[0112] The UI manager (210) can feed back a prompt based on user input regarding the generated image being displayed to the image processing unit (240). The prompt fed back to the image processing unit (240) may be a work instruction for the AI ​​model of the image processing unit (240).

[0113] The image processing unit (240) can change the final generated images based on the feedback prompt. The image processing unit (240) can create an image in which at least one of the area, composition, expression point of view, or light source of one or more objects (461, 463, 465, 467, 469) included in the final generated images is changed based on the feedback prompt.

[0114] As described above, the electronic device (101) can provide an experience that allows the user to view an image with a composition and / or expression point that was not captured by generating an image in which at least one of the area, composition, expression point, or light source of the input image has been changed. For example, the electronic device (101) can allow the user to view a photograph of an entire food by taking photographs of each of the multiple foods by expressing objects included in a plurality of fragmented images in a single image.

[0115] FIG. 5 illustrates an example of an operation in which an electronic device (101) generates an image to provide an intermediate experience to a user.

[0116] Figure 5 is described with reference to the contents of Figures 1 and 2.

[0117] The UI manager (210) can identify the selected input image (510). The UI manager (210) can obtain a prompt related to the selected input image (510). The prompt related to the input image (510) may be a prompt selected from one or more prompts selectable on a screen displayed on the gallery application. For example, the selected prompt may include words related to 'changing facial expressions.'

[0118] The UI manager (210) can transmit the selected input image (510) to the image preprocessing unit (220). The UI manager (210) can transmit the selected input image (510) to the crawling module (230). The UI manager (210) can transmit a prompt related to the selected input image (510) to the image processing unit (240).

[0119] The image preprocessing unit (220) can detect and classify one or more objects (511) in an input image (510). For example, the image preprocessing unit (220) can detect an object (511) in an input image (510) and classify the detected object (511) as a person.

[0120] The image preprocessing unit (220) can separate one or more objects (511) identified in the input image (510) from the input image (510). The image preprocessing unit (220) can segment one or more objects (511) based on the boundaries of each of the one or more objects (511) identified in the input image (510).

[0121] The image preprocessing unit (220) may transmit data related to one or more objects (511) identified in the input image (510) to the image processing unit (240). The data related to one or more objects (511) may include the type of the detected object, the area of ​​the detected object, and / or an object image for the object segmented from the input image (510).

[0122] The image processing unit (240) can generate a set of generated images (530) based on the preprocessed input image (510). By inputting a prompt (e.g., 'change facial expression') into the generative AI model, the image processing unit (240) can generate a set of images (530) in which at least one of the area, composition, expression point of view, or light source of one or more objects (511) included in the preprocessed input image (510) has been changed. For example, referring to FIG. 5, the input image (510) may be an image of a person with a smiling expression. The generated images (531, 533, 535, 537) included in the image set (530) may be images of people with neutral expressions, regretful expressions, unpleasant expressions, and frowning expressions, respectively.

[0123] The image processing unit (240) is configured to acquire information (520) regarding a user's input. Here, the user's input may include a gesture input, an input of shaking the electronic device (101), or a touch input to the electronic device (101). The input information (520) may be acquired at substantially the same time as the selection time of the input image (510).

[0124] The image processing unit (240) can identify a number according to the user's input. The image processing unit (240) can identify the number as a first number based on the user's input indicating a first gesture. The image processing unit (240) can identify the number as a second number based on the user's input indicating a second gesture. The image processing unit (240) can identify the number as the number of times the electronic device (101) is shaken according to the user's input. Here, the number may be the number of result images to be generated by the image processing unit (240).

[0125] The image processing unit (240) can generate a number of result images (531, 533, 535, 537) corresponding to the input. The image processing unit (240) can generate a number of result images (531, 533, 535, 537) corresponding to the input by inputting the number corresponding to the input as a prompt into the generative AI model.

[0126] As described above, the electronic device (101) can provide an experience that allows the user to view an image with a composition and / or expression point that was not captured by generating an image in which at least one of the area, composition, expression point, or light source of the input image has been changed. For example, the electronic device (101) can provide the user with a desired number of result images by allowing the user to select the number of result images for the input image.

[0127] FIG. 6a illustrates an example of an operation in which an electronic device (101) generates an image to provide an intermediate experience to a user. FIG. 6b illustrates an example of an operation in which an electronic device (101) generates an image to provide an intermediate experience to a user.

[0128] Figures 6a and 6b are described with reference to the contents of Figures 1 and 2.

[0129] Referring to FIG. 6a, the input image (610) may include multiple objects (611, 613, 615, 617).

[0130] The image preprocessing unit (220) can preprocess the input image (610). The image preprocessing unit (220) can generate a preprocessed input image (620) by preprocessing the input image (610). Referring to FIG. 6A, a plurality of objects (621, 623, 625, 627) can be segmented in the preprocessed input image (620).

[0131] The UI manager (210) can display a preprocessed input image (620) generated by the image preprocessing unit (220). The UI manager (210) can display the preprocessed input image (620) on the display (260) before generating a result image based on the preprocessed input image (620).

[0132] The UI manager (210) may display a screen including one or more buttons for instructing image processing for one or more separated objects (621, 623, 625, 627) along with a preprocessed input image (620). For example, referring to FIG. 6B, an image (601) may include a screen (640) including one or more buttons (641, 643, 645, 647) for instructing image processing for one or more separated objects (631, 633, 635, 637).

[0133] The UI manager (210) may identify an input requesting image processing for one or more objects (621, 623, 625, 627) separated from the preprocessed input image (620). For example, referring to FIG. 6B, in response to a touch input to a button (654), an image (603) may be displayed on the display (260). The image (603) may, in response to a touch input to the button (654), display buttons (e.g., “Remove”, “Body Shape Change”) that enable selection of image processing options for the object (633). For example, in response to pressing the “Remove” button for the object (633), the object (633) may be removed from the image (603). For example, the UI manager (210) can remove an object (633) separated from an input image (620) preprocessed through the image preprocessing unit (220). For example, the UI manager (210) can transmit the preprocessed input image (620) from which the object (633) has been removed to the image preprocessing unit (220).

[0134] The image processing unit (240) can generate a result image (630) based on a preprocessed input image (620) from which an object (633) has been removed.

[0135] The image processing unit (240) can generate a result image (630) based on the preprocessed input image (620) from which the object (633) has been removed. If the prompt obtained through the UI manager (210) is 'change expression', the image processing unit (240) can generate a result image (630) in which the expression of the object (635) has been changed. For example, referring to FIG. 6A, the input image (610) may include an object (615) with a smiling expression, while the result image (630) may include an object (635) with a regretful expression.

[0136] As described above, the electronic device (101) can provide an experience that allows the user to view a result image that includes an object of a desired shape by generating a result image by excluding and / or modifying unnecessary objects from the input image.

[0137] FIG. 7 illustrates an example of an operation in which an electronic device (101) generates an image to provide an intermediate experience to a user.

[0138] Figure 7 is described with reference to the contents of Figures 1 and 2.

[0139] The UI manager (210) can identify the input image (720).

[0140] The UI manager (210) can obtain a prompt. The UI manager (210) can obtain a prompt related to the input image (720) based on user input. For example, the prompt may be 'weather changes and facial expression changes for the input image (720).'

[0141] The image preprocessing unit (220) can preprocess an input image (720). The image preprocessing unit (220) can detect and classify one or more objects (711, 725, 728) in the input image (720). The image preprocessing unit (220) can separate one or more objects (711, 725, 728) from the input image (720).

[0142] The image processing unit (240) can generate generated images based on the preprocessed input image. The image processing unit (240) can generate generated images (710, 730, 740) with changed weather and / or expressions based on the preprocessed input image and a prompt (e.g., 'weather change and expression change for the input image (720)'). Referring to FIG. 7, in the input image (720), the object related to the weather may be a cloud (725), and the expression of a person (711) may be a neutral expression (728) depending on the weather. In the generated image (710), the object related to the weather may change from a cloud (725) to the sun (715). In the generated image (710), the expression of a person (711) may change to a smiling expression (718) depending on the changed weather. In the generated image (730), the object related to the weather may change from a cloud (725) to rain (735). In the generated image (730), the facial expression of a person (711) may change to an unpleasant facial expression (738) depending on the changed weather. In the generated image (740), the object related to the weather may change from a cloud (725) to rain (735). In the generated image (730), the facial expression of a person (711) may change to a frowning facial expression (748) depending on the changed weather. In the generated image (730), the clothes of a person (711) may change to a wet state (741) depending on the changed weather.

[0143] The UI manager (210) can sequentially play images (710, 720, 730, 740) through the display (260).

[0144] In Fig. 7, images (710, 720, 730, 740) are sequentially displayed according to an input image (720), but this is merely an example. The electronic device (101) can generate result images that match the current weather by inputting weather information as a prompt along with the input image (720). The electronic device (101) can sequentially display result images that match the current weather on the display (260).

[0145] As described above, the electronic device (101) can provide a user with an experience of viewing time-lapsed images from an input image by generating time-lapsed images or animations based on an input image.

[0146] Figure 8 is a flowchart showing the operation of an electronic device (101).

[0147] Figure 8 is described with reference to the contents of Figures 1 to 7.

[0148] Referring to FIG. 8, in operation 810, the electronic device (101) identifies an input image (310). The electronic device (101) may identify the input image from among selectable images displayed through a gallery application based on a user input. The input image may include one or more input images. At least some objects of the one or more input images may differ. At least some meta information (e.g., EXIF) of the one or more input images may differ. The meta information for each of the one or more input images may differ in at least one of location information (e.g., latitude information and longitude information), camera setting information (e.g., size, pixels, F number, color temperature, shutter speed, flash, ISO sensitivity, magnification, field of view), composition, or camera angle, shooting device, or shooting time.

[0149] In operation 820, the electronic device (101) identifies an object. The electronic device (101) can detect and classify one or more objects in the input image (310). For example, the electronic device (101) can detect an area of ​​one or more objects in the input image (310). For example, the electronic device (101) can identify a type (e.g., food, plate, person, animal, plant, object) of one or more objects detected in the input image (310). For example, if the input image (310) includes a plate containing food, the electronic device (101) can identify the food and the plate.

[0150] The electronic device (101) can separate one or more objects identified in the input image (310) from the input image (310). The electronic device (101) can segment one or more objects based on the boundaries of each of the one or more objects identified in the input image (310). For example, if the input image (310) includes a plate containing food, the electronic device (101) can identify an area of ​​the food and an area of ​​the plate within the input image (310).

[0151] The electronic device (101) can estimate the area of ​​each of one or more objects outside the input image (310) based on the area of ​​each of one or more objects identified in the input image (310). The complete area of ​​the object can include the area of ​​the object identified in the input image (310) and the area of ​​the object outside the input image (310). For example, if the input image (310) includes a plate containing food, the electronic device (101) can estimate the area of ​​the food outside the input image (310) and the area of ​​the plate.

[0152] In operation 830, the electronic device (101) generates a plurality of result images in which the composition and / or expression point of the identified object is changed.

[0153] The electronic device (101) can generate a result image based on a preprocessed input image (320). The electronic device (101) can generate a result image in which at least one of the area, composition, expression point of view, or light source of one or more objects included in the preprocessed input image (320) has been changed. The electronic device (101) can generate a result image in which at least one of the area, composition, expression point of view, or light source of one or more objects has been changed by inputting the input image (310) or the preprocessed input image (320) into a generative artificial intelligence (AI) model.

[0154] A change in the area of ​​an object may mean that the area of ​​the object included in the generated result image is changed to include a newly drawn area more than the area of ​​the object included in the input image. For example, the newly drawn area may correspond to an area of ​​the object that is out of the input image. A change in the composition of the object may mean that the angle or line of sight of the camera of the generated result image is changed from the angle of the camera of the input image. A change in the composition of the object may mean that the position of the object in the generated result image is changed from the position of the object in the input image. A change in the composition of the object may mean that the arrangement of objects in the generated result image is changed from the arrangement of objects in the input image. A change in the composition of the object may mean that the magnification or size of the object in the generated image is changed from the magnification or size of the object in the input image. A change in the presentation point of an object may mean that the presentation point within the life cycle or timeline of the object included in the generated result image is different from the presentation point within the life cycle of the object included in the input image. For example, the life cycle or timeline may include a time interval from the start point to the end point of the object. For example, the life cycle may include a time interval from the start point to the end point of a specific action or state change of the object. For example, if the object is food, at the start of the life cycle, the object may have a presentation point before eating the food. For example, if the object is food, at the end of the life cycle, the object may have a presentation point after finishing eating the food. For example, if the object is food, at a midpoint in the life cycle, the object may have a presentation point between the presentation point before eating the food and the presentation point after finishing eating the food. For example, if the object is the sun, at the start of the life cycle, the object may have a presentation point related to sunrise.For example, if an object is a sun, at the end of its life cycle, the object may have a presentation point associated with sunset. For example, if an object is a sun, at an intermediate point in its life cycle, the object may have a presentation point between sunrise and sunset. A change in the light source of an object may mean that the position of the light source illuminating the object included in the generated image changes from the position of the light source illuminating the object included in the input image. A change in the light source of an object may mean that the color temperature of the light source illuminating the object included in the generated image changes from the color temperature of the light source illuminating the object included in the input image.

[0155] Figure 9 is a flowchart showing the operation of an electronic device (101).

[0156] Figure 9 is described with reference to the contents of Figures 1 to 8.

[0157] Operation 810 of FIG. 9 may correspond to operation 810 of FIG. 8. Operations 910 and 920 of FIG. 9 may be included in operation 820 of FIG. 8. Operations 930, 940, and 950 of FIG. 9 may be included in operation 830 of FIG. 8.

[0158] Referring to FIG. 9, in operation 810, the electronic device (101) identifies an input image (310).

[0159] In operation 910, the electronic device (101) preprocesses the input image (310). The preprocessing of the input image (310) may include processing (e.g., denoising, deblurring, dehazing, white balance) to improve the image quality of the input image. The preprocessing of the input image (310) may include processing for detecting and classifying one or more objects. The preprocessing of the input image (310) may include processing for segmenting one or more objects.

[0160] In operation 920, the electronic device (101) changes the composition of an object. The electronic device (101) may generate one or more first intermediate stage images in which the composition of one or more objects included in an input image is changed. The electronic device (101) may generate one or more first intermediate stage images in which the composition of one or more objects included in the input image is changed by inputting a prompt to a generative AI model. The electronic device (101) may generate one or more first intermediate stage images in which the composition of one or more objects included in the input image is changed by inputting a prompt related to the composition of one or more objects to the generative AI model. The electronic device (101) may generate one or more first intermediate stage images in which the angle of a camera, or the line of sight of a camera, or the shooting position, arrangement, and / or magnification of a camera for one or more objects included in the input image is changed by inputting a prompt related to the composition of one or more objects to the generative AI model. Composition-related prompts may be words that indicate the camera angle, or the camera's line of sight, the object's position, or the camera's shooting position, the object's placement, and / or the magnification of the image. Examples of object composition-related prompts may include: 'Change the food to a top-down view', 'Change the food to a front-on view', 'Change the sun-facing view', 'Change the arrangement of the plates to a specified arrangement', 'Change the size of the plates', or 'Change the size of the image'.

[0161] In operation 930, the electronic device (101) performs an image filling process. The electronic device (101) may generate an image on which an image filling process (e.g., in-painting, out-painting) has been performed based on an area of ​​one or more objects included in an input image. The electronic device (101) may perform the image filling process on one or more first intermediate stage images by inputting a prompt to the generative AI model. Here, the image filling process may be a process of newly drawing an area other than an area of ​​one or more objects. The prompt for the image filling process may be words indicating the texture of the area to be filled. Examples of prompts for the image filling process may include 'plates of food placed on a table', 'flower bed', or 'street covered with fallen leaves'.

[0162] In operation 940, the electronic device (101) changes the expression time point of an object. The electronic device (101) can generate one or more third intermediate stage images in which the expression time points of one or more objects included in an input image are changed by inputting a prompt to the generative AI model. The electronic device (101) can generate one or more third intermediate stage images in which the expression time points of one or more objects included in the input image are changed by inputting a prompt related to the expression time points of one or more objects to the generative AI model. The electronic device (101) can generate one or more third intermediate stage images in which the expression time points of one or more objects included in the input image are changed according to the life cycle of one or more objects included in the input image by inputting a prompt related to the expression time points of one or more objects to the generative AI model. The prompts related to the expression time points may be words related to the time points. Examples of prompts related to the object's viewpoint might be 'Change to state before eating food', 'Change to state after eating food', 'Change to state just before sunrise', 'Change to a different season', 'Change to a different time of day', 'Change to state after it rained', or 'Change to state soaked in rain'.

[0163] In operation 950, the electronic device (101) changes the light source of the image. The electronic device (101) may generate an image in which the light source of one or more objects included in the input image is changed based on a relight module. The electronic device (101) may generate one or more fourth intermediate stage images in which the light source of one or more objects included in the input image is changed.

[0164] Figure 10 is a flowchart showing the operation of an electronic device (101).

[0165] Figure 10 is described with reference to the contents of Figures 1 to 8.

[0166] Action 1030 of FIG. 10 may be included in action 830 of FIG. 8.

[0167] Referring to FIG. 10, in operation 1010, the electronic device (101) identifies an input. Here, the user's input may include a gesture input, an input of shaking the electronic device (101), or a touch input to the electronic device (101). The input information (520) may be acquired at substantially the same time as the selection time of the input image (510), but is not limited thereto.

[0168] In operation 1020, the electronic device (101) identifies the number of images to be generated based on the input. The electronic device (101) may identify the number as a first number based on the user's input indicating a first gesture. The electronic device (101) may identify the number as a second number based on the user's input indicating a second gesture. The electronic device (101) may identify the number as the number of times the electronic device (101) is shaken based on the user's input. Here, the number may be the number of result images to be generated by the electronic device (101).

[0169] In operation 1030, the electronic device (101) generates a result image according to the identified number. The electronic device (101) can generate result images (531, 533, 535, 537) corresponding to the number of inputs by inputting the number corresponding to the input as a prompt into the generative AI model.

[0170] Figure 11 is a flowchart showing the operation of an electronic device (101).

[0171] Figure 11 is described with reference to the contents of Figures 1 to 8.

[0172] Operations 810, 820, and 830 of FIG. 11 may correspond to operations 810, 820, and 830 of FIG. 8, respectively.

[0173] Referring to FIG. 11, in operation 810, the electronic device (101) identifies an input image. The electronic device (101) can identify the input image from among selectable images displayed through the gallery application based on a user input.

[0174] In operation 820, the electronic device (101) identifies an object. The electronic device (101) can detect and classify one or more objects in an input image (310). The electronic device (101) can separate one or more objects identified in the input image (310) from the input image (310).

[0175] In operation 1110, the electronic device (101) may perform image processing on an object. The electronic device (101) may perform image processing on the object based on an input. The electronic device (101) may perform image processing on the object based on an input obtained while displaying one or more objects separated from an input image (310). The electronic device (101) may perform image processing on the object based on an input for at least one object among the one or more objects separated from the input image (310). The electronic device (101) may perform image processing on the object based on an input on a screen displayed together with the preprocessed input image (320). The screen may include one or more buttons for instructing image processing on the one or more separated objects. For example, the image processing may include deleting an object selected by the input, correcting a shape (e.g., making it fatter, thinner), and / or moving an object selected by the input.

[0176] In operation 830, the electronic device (101) generates a plurality of result images in which the composition and / or expression point of the identified object is changed. The electronic device (101) may generate the result images based on the image-processed object in the preprocessed input image (320). The electronic device (101) may generate the result images in which at least one of the area, composition, expression point of view, or light source of the image-processed object in the preprocessed input image (320) is changed.

[0177] Figure 12 is a flowchart showing the operation of an electronic device (101).

[0178] Figure 12 is described with reference to the contents of Figures 1 to 8.

[0179] Action 830 of FIG. 12 may correspond to action 830 of FIG. 8.

[0180] Referring to FIG. 12, in operation 830, the electronic device (101) generates a plurality of result images in which the composition and / or expression time point of the identified object is changed. The electronic device (101) may generate the result images based on the preprocessed input image (320). The electronic device (101) may generate the result images in which at least one of the area, composition, expression time point, or light source of one or more objects included in the preprocessed input image (320) is changed. The electronic device (101) may generate the result images in which at least one of the area, composition, expression time point, or light source of one or more objects is changed by inputting the input image (310) or the preprocessed input image (320) into a generative artificial intelligence (AI) model.

[0181] In operation 1210, the electronic device (101) determines whether animation generation is required. The electronic device (101) may determine that animation generation is required based on a user input requesting animation generation. The electronic device (101) may determine that animation generation is required if a result image based on the input image is set as a wallpaper. The electronic device (101) may determine that animation generation is required based on a user input selecting a natural or seamless background transition mode. The user input may be obtained upon selection of the input image. For example, the electronic device (101) may obtain the user input through a screen that queries whether animation should be generated based on the selection of the input image through the UI manager (210).

[0182] The electronic device (101) is configured to perform operation 1220 in response to identifying that generation of animation is required. The electronic device (101) is configured to perform operation 1240 in response to identifying that generation of animation is not required.

[0183] In operation 1220, the electronic device (101) generates an animation in which a plurality of result images are arranged according to a specified condition. The specified condition may be related to weather. For example, the electronic device (101) may select some images from among the plurality of result images that match current weather information. The specified condition may be related to a timeline or a life cycle. For example, the electronic device (101) may select some images within the life cycle of an object from among the plurality of result images.

[0184] The electronic device (101) can arrange some selected images in chronological order. The electronic device (101) can generate an animation based on some selected images arranged in chronological order.

[0185] In operation 1230, the electronic device (101) displays an animation. The electronic device (101) may display the animation as a wallpaper on the display (260).

[0186] In operation 1240, the electronic device (101) displays a plurality of result images so that they can slide to a certain extent. The electronic device (101) can display the plurality of result images so that they slide and switch at a specified time interval.

[0187] As described above, the electronic device (101) includes a processor (120). The electronic device (101) includes a memory (130) that stores instructions. The instructions, when executed by the processor (120), may cause the electronic device (101) to select an object from among at least one object (321, 325) included in an input image (310). The instructions, when executed by the processor, may cause the electronic device to check instruction information regarding a change in the state of the selected object. The instructions, when executed by the processor (120), may cause the electronic device (101) to obtain a prompt related to the input image (310) based on the instruction information. The prompt may include at least one word for changing the state of the selected object included in the input image (310). The above state may be included within a range of possible different states of the selected object. The instructions, when executed by the processor (120), may cause the electronic device (101) to generate at least one result image (330) in which the state of the selected object is changed by inputting the input image (310) and the prompt to a generative AI model.

[0188] The input image (310) may include a plurality of input images (310, 411, 415). The selected object may be one or more of the objects (321, 325, 431, 433, 435, 437, 441, 443, 445, 447) included in the plurality of input images (310, 411, 415). The instructions, when executed by the processor (120), may cause the electronic device (101) to obtain the prompt related to the arrangement of the one or more objects selected from the plurality of input images (310, 411, 415). The above instructions, when executed by the processor (120), may cause the electronic device (101) to generate at least one result image (330) in which the one or more objects are arranged within the at least one result image (330) by inputting the plurality of input images (310, 411, 415) and the prompt to the generative AI model.

[0189] The instructions, when executed by the processor (120), may cause the electronic device (101) to identify a background region of each of the plurality of input images (310, 411, 415). The instructions, when executed by the processor (120), may cause the electronic device (101) to obtain a prompt for guiding image filling processing for a background region of the at least one result image (330). The prompt may include at least one word representing the background region of each of the plurality of input images (310, 411, 415). The above instructions, when executed by the processor (120), may cause the electronic device (101) to generate at least one result image (330) in which the image filling process is performed on the background area of ​​the at least one result image (330) by inputting the plurality of input images (310, 411, 415) and the prompt to the generative AI model.

[0190] The instructions, when executed by the processor (120), may cause the electronic device (101) to obtain at least one reference image based on meta information of the plurality of input images (310, 411, 415). The instructions, when executed by the processor (120), may cause the electronic device (101) to obtain the prompt related to the arrangement of the one or more objects selected from the plurality of input images (310, 411, 415) based on the arrangement of the plurality of objects included in the at least one reference image.

[0191] The electronic device may further include a display (160, 260). The instructions, when executed by the processor (120), may cause the electronic device (101) to, in response to an input selecting the input image (310), display a screen on the display (160, 260) including a plurality of selectable prompts related to the input image (310). The instructions, when executed by the processor (120), may cause the electronic device (101) to obtain a prompt related to the input image (310) based on a selection of some of the plurality of prompts included in the screen.

[0192] The selected object of the input image (310) may represent a person. The state of the selected object included in the at least one result image (330) may be different from the state of the object of the input image (310). The state of the selected object included in the at least one result image (330) may be one of a state in which the person exhibits a first facial expression or a state in which the person exhibits a second facial expression different from the first facial expression.

[0193] The object (321, 325) of the input image (310) may represent food. The state of the selected object included in the at least one result image (330) may be different from the state of the selected object of the input image (310). The state of the selected object included in the at least one result image (330) may be one of a state before eating the food, a state while eating the food, or a state after eating the food.

[0194] The instructions, when executed by the processor (120), may cause the electronic device (101) to obtain at least one reference image based on meta information of the input image (310). The instructions, when executed by the processor (120), may cause the electronic device (101) to obtain the prompt including at least one word for setting the composition for the selected object based on the composition of at least one object included in the at least one reference image.

[0195] The electronic device (101) may further include a display (160, 260). The instructions, when executed by the processor (120), may cause the electronic device (101) to generate tag information for the selected object included in the at least one result image (330) based on information about the at least one object included in the at least one reference image. The instructions, when executed by the processor (120), may cause the electronic device (101) to provide a search result of the tag information in response to an input for selecting the tag information while the at least one result image (330) is displayed through the display (160, 260).

[0196] The above instructions, when executed by the processor (120), may cause the electronic device (101) to obtain an input from a user of the electronic device (101). The above instructions, when executed by the processor (120), may cause the electronic device (101) to generate a number of result images (330) corresponding to the input. The number of result images (330) corresponding to the input may have different states of the object.

[0197] The electronic device (101) may further include a display (160, 260). The result image (330) may include a plurality of result images (330) having different compositions of the object. The instructions, when executed by the processor (120), may cause the electronic device (101) to generate an animation in which the plurality of result images (330) are arranged so that the composition continuously changes during playback. The instructions, when executed by the processor (120), may cause the electronic device (101) to display the generated animation through the display (160, 260).

[0198] The electronic device (101) may further include a display (160, 260). The result image (330) may include a plurality of result images (330) having different states of the object. The instructions, when executed by the processor (120), may cause the electronic device (101) to display, through the display (160, 260), the plurality of result images (330) arranged so that the expression time points are continuously changed over time during playback.

[0199] As described above, the method may be performed by the electronic device (101). The method may include an operation of identifying an object among at least one object (321, 325) included in an input image (310). The method may include an operation of checking indicative information regarding a change in state of the selected object. The method may include an operation of obtaining a prompt related to the input image (310) based on the indicative information. The prompt may include at least one word for changing a state of the selected object included in the input image (310). The state may be included within a range of possible different states of the selected object. The method may include an operation of generating at least one result image (330) in which the state of the selected object is changed by inputting the input image (310) and the prompt to a generative artificial intelligence (AI) model.

[0200] The input image (310) may include a plurality of input images (310, 411, 415). The selected object may be one or more of the objects (321, 325, 431, 433, 435, 437, 441, 443, 445, 447) included in the plurality of input images (310, 411, 415). The method may include an operation of obtaining the prompt related to the arrangement of the one or more objects selected from the plurality of input images (310, 411, 415). The method may include an operation of generating at least one result image (330) in which the one or more objects are arranged within the at least one result image (330) by inputting the plurality of input images (310, 411, 415) and the prompt to the generative AI model.

[0201] The method may include an operation of identifying a background area of ​​each of the plurality of input images (310, 411, 415). The method may include an operation of obtaining the prompt for guiding image filling processing for a background area of ​​the at least one result image (330). The prompt may include at least one word representing the background area of ​​each of the plurality of input images (310, 411, 415). The method may include an operation of generating the at least one result image (330) on which the image filling processing is performed for the background area of ​​the at least one result image (330) by inputting the plurality of input images (310, 411, 415) and the prompt to the generative AI model.

[0202] The method may include an operation of obtaining at least one reference image based on meta information of the plurality of input images (310, 411, 415). The method may include an operation of obtaining the prompt related to the arrangement of one or more objects selected from the plurality of input images (310, 411, 415) based on the arrangement of the plurality of objects included in the at least one reference image.

[0203] The method may include, in response to an input for selecting the input image (310), displaying a screen on a display (160, 260) of the electronic device (101) that includes a plurality of selectable prompts related to the input image (310). The method may include, based on a selection of some of the plurality of prompts included in the screen, obtaining a prompt related to the input image (310).

[0204] The method may include an operation of obtaining at least one reference image based on meta information of the input image (310). The method may include an operation of obtaining the prompt including at least one word for setting the composition for the selected object based on the composition of at least one object included in the at least one reference image.

[0205] The method may include an operation of obtaining an input from a user of the electronic device (101). The method may include an operation of generating a number of result images (330) corresponding to the input. The number of result images (330) corresponding to the input may have different states of the object.

[0206] As described above, a non-transitory computer readable storage medium can store a program including instructions. The instructions, when executed by the processor (120) of the electronic device (101), can cause the electronic device (101) to identify an object among at least one object (321, 325) included in an input image (310). The instructions, when executed by the processor of the electronic device, can cause the electronic device to check instruction information regarding a change in the state of the selected object. The instructions, when executed by the processor (120) of the electronic device (101), can cause the electronic device (101) to obtain a prompt related to the input image (310) based on the instruction information. The instructions can include at least one word for changing the state of the selected object included in the input image (310). The timeline may include states within a range of different states of the object. The instructions may cause the generative AI model to generate at least one result image (330) in which the state of the selected object has changed by inputting the input image (310) and the prompt.

[0207] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0208] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0209] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0210] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0211] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., by download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0212] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (101), processor (120), and It includes a memory (130) that stores instructions, and when the instructions are executed by the processor (120), the electronic device (101) Select the object (321, 325) included in the input image (310), Identify the instruction information that forms the command for the intended state change of the selected object, Obtaining a prompt related to the input image (310) based on the above instruction information, wherein the prompt includes at least one word for changing the state of the selected object, and the state is included within a range of possible different states for the selected object, By inputting the input image (310) and the prompt to the generative artificial intelligence (AI) model, at least one result image (330) in which the state of the selected object is changed is generated. Electronic devices.

2. In claim 1, Multiple input images (310, 411, 415) are provided as the input image (310), The above selected object is one or more objects among the plurality of objects (321, 325, 431, 433, 435, 437, 441, 443, 445, 447) included in the plurality of input images (310, 411, 415), The above instructions, when executed by the processor (120), cause the electronic device (101) to: Obtaining the prompt related to the placement of one or more objects selected from the plurality of input images (310, 411, 415), By inputting the plurality of input images (310, 411, 415) and the prompt to the generative AI model, the at least one result image (330) is generated in which the one or more objects are arranged within the at least one result image (330). Electronic devices.

3. In claim 2, The above instructions, when executed by the processor (120), cause the electronic device (101) to: Identify the background area of ​​each of the above multiple input images (310, 411, 415), Obtaining the prompt for guiding the image filling process for the background area of ​​the at least one result image (330), wherein the prompt includes at least one word representing the background area of ​​each of the plurality of input images (310, 411, 415), By inputting the plurality of input images (310, 411, 415) and the prompt to the generative AI model, the at least one result image (330) is generated in which the image filling process is performed on the background area of ​​the at least one result image (330). Electronic devices.

4. In claim 2, The above instructions, when executed by the processor (120), cause the electronic device (101) to: Based on the meta information of the above multiple input images (310, 411, 415), at least one reference image is obtained, Causing to obtain the prompt related to the arrangement of the one or more objects selected from the plurality of input images (310, 411, 415), based on the arrangement of the plurality of objects included in the at least one reference image. Electronic devices.

5. In any preceding claim, Including additional displays (160, 260), The above instructions, when executed by the processor (120), cause the electronic device (101) to: In response to an input for selecting the input image (310), a screen including a plurality of selectable prompts related to the input image (310) is displayed on the display (160, 260), Based on the selection of some of the plurality of prompts included in the above screen, causing a prompt related to the input image (310) to be obtained. Electronic devices.

6. In any preceding claim, The selected object of the above input image (310) represents a person, The state of the selected object included in at least one result image (330) is different from the state of the object of the input image (310), The state of the selected object included in the at least one result image (330) is one of a state in which the person shows a first expression, or a state in which the person shows a second expression different from the first expression. Electronic devices.

7. In any preceding claim, The object (321, 325) of the input image (310) represents food, The state of the selected object included in at least one result image (330) is different from the state of the selected object of the input image (310), The state of the selected object included in at least one of the result images (330) is one of a state before eating the food, a state while eating the food, or a state after eating the food. Electronic devices.

8. In claim 1, The above instructions, when executed by the processor (120), cause the electronic device (101) to: Based on the meta information of the above input image (310), at least one reference image is obtained, causing said prompt to be obtained, said prompt including said at least one word for setting said composition for said selected object, based on the composition of at least one object included in said at least one reference image; Electronic devices.

9. In claim 8, Including additional displays (160, 260), The above instructions, when executed by the processor (120), cause the electronic device (101) to: Generating tag information for the selected object included in the at least one result image (330) based on information about at least one object included in the at least one reference image, While at least one result image (330) is displayed through the display (160, 260), in response to an input for selecting the tag information, a search result of the tag information is provided. Electronic devices.

10. In any preceding claim, The above instructions, when executed by the processor (120), cause the electronic device (101) to: Obtaining input from the user of the above electronic device (101), Causes to generate a number of result images (330) corresponding to the above input, The above result images (330) are different from each other in the state of the object. Electronic devices.

11. In claim 1, Including additional displays (160, 260), The above result image (330) includes multiple result images (330) with different compositions of the object, The above instructions, when executed by the processor (120), cause the electronic device (101) to: An animation is created in which the plurality of result images (330) are arranged so that the composition changes continuously during playback, Causing the generated animation to be displayed through the display (160, 260), Electronic devices.

12. In claim 1, Including additional displays (160, 260), The above result image (330) includes a plurality of result images (330) in which the states of the object are different from each other, The above instructions, when executed by the processor (120), cause the electronic device (101) to: Causing the display (160, 260) to display the plurality of result images (330) arranged so that the state changes sequentially over time during playback. Electronic devices.

13. In a method performed by an electronic device (101), An action to select an object (321, 325) included in an input image (310), An action to verify the instruction information that forms a command for the intended state change of the selected object. An operation of obtaining a prompt related to the input image (310) based on the above instruction information, the prompt including at least one word for changing the state of the selected object included in the input image (310), the state being included within a range of possible different states for the selected object, and An operation of generating at least one result image (330) in which the state of the selected object is changed by inputting the input image (310) and the prompt to a generative artificial intelligence (AI) model. method.

14. In claim 13, Further comprising the operations described for the electronic device of any one of claims 1 to 12; method.

15. In a non-transitory computer readable storage medium, A program comprising instructions, which, when executed by a processor (120) of an electronic device (101), cause the electronic device (101) to operate as described in any one of claims 1 to 12. A non-transitory computer-readable recording medium.

Images