Electronic device for executing restoration function by using metadata of image, and method thereof

Abstract

An electronic device according to an embodiment may display, via a display, an edited image including a modified object, and a visual object for the modified object. The electronic device may receive user input for the visual object while displaying the edited image. The electronic device may obtain, on the basis of the user input, information for replacing the modified object with an artificial intelligence-generated object from metadata corresponding to the edited image. The electronic device may generate an artificial intelligence-generated image by executing an artificial intelligence model, using the edited image and the information. In response to the user input, the electronic device may display the artificial intelligence-generated image via the display.

Description

Electronic device and method for performing a restoration function using image metadata

[0001] The present disclosure relates to an electronic device and method for performing a restoration function using metadata of an image.

[0002] Technologies are being developed to process photos and / or videos using artificial intelligence. For example, technologies are being developed to recognize one or more characters (or strings) associated with photos and / or videos. For example, technologies are being developed to classify subjects (e.g., objects including people, animals, and / or vehicles) captured in photos and / or videos.

[0003] The above information may be provided as background information to aid in understanding the present disclosure. None of the above is claimed to be prior art related to the present disclosure or can be used in making decisions related to prior art.

[0004] According to one embodiment, an electronic device may include a display, at least one processor including a processing circuit, and a memory including one or more storage media storing instructions. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display, through the display, an edited image including a modified object, and a visual object for the modified object. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive a user input for the visual object while displaying the edited image. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain information from metadata corresponding to the edited image to replace the modified object with an AI-generated object based on the user input. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to execute the artificial intelligence model using the edited image and the information to generate an artificial intelligence-generated image. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display the artificial intelligence-generated image through the display in response to the user input.

[0005] In one embodiment, a method of an electronic device including a display may be provided. The method may include an operation of displaying, through the display, an edited image including a modified object and a visual object for the modified object. The method may include an operation of receiving a user input for the visual object while displaying the edited image. The method may include an operation of obtaining metadata corresponding to the edited image to replace the modified object with an AI-generated object based on the user input. The method may include an operation of executing the AI ​​model using the edited image and the information to generate an AI-generated image. The method may include an operation of displaying, through the display, the AI-generated image in response to the user input.

[0006] In one embodiment, a non-transitory computer-readable storage medium comprising instructions may be provided. The instructions, when executed by an electronic device including a display, may cause the electronic device to display, through the display, an edited image including a modified object and a visual object for the modified object. The instructions, when executed by the electronic device, may cause the electronic device to receive user input for the visual object while displaying the edited image. The instructions, when executed by the electronic device, may cause the electronic device to obtain information from metadata corresponding to the edited image to replace the modified object with an AI-generated object based on the user input. The instructions, when executed by the electronic device, may cause the electronic device to execute the AI ​​model using the edited image and the information to generate an AI-generated image. The above instructions, when executed by the electronic device, may cause the electronic device to display the artificial intelligence-generated image through the display in response to the user input.

[0007] In one embodiment, an electronic device may include a display, at least one processor including a processing circuit, and a memory including one or more storage media storing instructions. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display, on the display, a viewer screen including an edited image corresponding to an original image and a visual object for indicating a difference between the original image and the edited image. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive, while displaying the viewer screen, an input related to the visual object. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain, based on the input, information used to restore the original image using an artificial intelligence model from metadata of a file including the edited image. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to execute the artificial intelligence model using the edited image and the information to generate a restored image. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display the restored image on the display in response to the input.

[0008] In one embodiment, a method of an electronic device including a display may be provided. The method may include an operation of displaying a viewer screen on the display, the viewer screen including an edited image corresponding to an original image and a visual object for indicating a difference between the original image and the edited image. The method may include an operation of receiving an input related to the visual object while displaying the viewer screen. The method may include an operation of obtaining information used to restore the original image using an artificial intelligence model from metadata of a file including the edited image based on the input. The operation may include an operation of executing the artificial intelligence model using the edited image and the information to generate a restored image. The operation may include an operation of displaying the restored image on the display in response to the input.

[0009] In one embodiment, a non-transitory computer-readable storage medium comprising instructions may be provided. The instructions, when executed by an electronic device including a display, may cause the electronic device to display, on the display, a viewer screen including an edited image corresponding to an original image and a visual object for indicating a difference between the original image and the edited image. The instructions, when executed by the electronic device, may cause the electronic device to receive, while displaying the viewer screen, an input related to the visual object. The instructions, when executed by the electronic device, may cause the electronic device to obtain, based on the input, information used to restore the original image using an artificial intelligence model from metadata of a file including the edited image. The instructions, when executed by the electronic device, may cause the electronic device to execute the artificial intelligence model using the edited image and the information to generate a restored image. The above instructions, when executed by the electronic device, may cause the electronic device to display the restored image on the display in response to the input.

[0010] In one embodiment, a non-transitory computer-readable storage medium comprising instructions may be provided. The instructions, when executed by an electronic device including a display, may cause the electronic device to display a viewer screen on the display, the viewer screen including an edited image corresponding to an original image and an indicator corresponding to a first visual object area of ​​the edited image. The instructions, when executed by the electronic device, may cause the electronic device to receive an input related to the indicator while displaying the viewer screen. The instructions, when executed by the electronic device, may cause the electronic device to generate a second visual object area using an artificial intelligence model, based on the input, such that the first visual object area within the edited image is replaced with a second visual object area of ​​the original image having a location corresponding to a location of the first visual object area within the edited image.

[0011] In one embodiment, an electronic device may include a display, at least one processor including a processing circuit, and a memory including one or more storage media storing instructions. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display, on the display, a viewer screen including an edited image corresponding to an original image and an indicator corresponding to a first visual object area of ​​the edited image. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive an input related to the indicator while displaying the viewer screen. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to generate, based on the input, the second visual object area using an artificial intelligence model, such that the first visual object area within the edited image is replaced with a second visual object area of ​​the original image having a location corresponding to a location of the first visual object area within the edited image.

[0012] FIG. 1 illustrates an exemplary operation of an electronic device for restoring an original image using a file containing an edited image, according to one embodiment.

[0013] FIGS. 2A and 2B illustrate block diagrams of an electronic device according to one embodiment.

[0014] FIG. 3 illustrates an exemplary operation of an electronic device for generating an edited image for an original image.

[0015] FIGS. 4A and 4B illustrate exemplary operations of an electronic device that generates a file including an edited image generated using a generative artificial intelligence model.

[0016] FIG. 5 illustrates an exemplary operation of an electronic device for generating a restored image corresponding to an original image from an edited image included in a file.

[0017] FIGS. 6A, 6B, 6C, 6D, 6E, and 6F illustrate exemplary structures of an image restoration model executed by an electronic device according to one embodiment.

[0018] Figure 7 illustrates an exemplary operation of an electronic device that restores an original image using pixel information stored in the metadata of a file.

[0019] Figure 8 illustrates an exemplary operation of an electronic device that restores an original image by utilizing pixel differences indicated by metadata of a file.

[0020] FIG. 9 illustrates an exemplary operation of an electronic device for restoring an original image using pixel information and / or one or more prompts contained in the metadata of a file.

[0021] FIG. 10 illustrates an exemplary operation of an electronic device for restoring an original image using feature information included in the metadata of a file.

[0022] FIG. 11 illustrates an exemplary operation of an electronic device for restoring an original image using one or more prompts included in the metadata of a file.

[0023] FIG. 12 illustrates an exemplary operation of an electronic device for restoring an original image using one or more prompts associated with the original image corresponding to an edited image of the file.

[0024] FIG. 13 illustrates an exemplary operation of an electronic device for restoring an original image using one or more prompts and location information included in the metadata of a file.

[0025] Figures 14a and 14b illustrate exemplary states of an electronic device for restoring an original image.

[0026] Figure 15 illustrates exemplary programs executed by an electronic device to simulate a generative artificial intelligence model.

[0027] FIG. 16 is a block diagram of an electronic device within a network environment according to various embodiments.

[0028] Hereinafter, various embodiments of this document are described with reference to the attached drawings.

[0029] The various embodiments of this document and the terminology used therein are not intended to limit the technology described in this document to a specific embodiment, but should be understood to include various modifications, equivalents, and / or substitutes of the embodiment. In connection with the description of the drawings, similar reference numerals may be used for similar components. The singular expression may include plural expressions unless the context clearly indicates otherwise. In this document, expressions such as "A or B", "at least one of A and / or B", "A, B, or C", or "at least one of A, B, and / or C" may include all possible combinations of the items listed together. Expressions such as "first", "second", "first", or "second" may modify the corresponding components regardless of order or importance, and are only used to distinguish one component from another, but do not limit the corresponding components. When it is said that a component (e.g., a first component) is “(functionally or communicatively) connected” or “connected” to another component (e.g., a second component), said component may be directly connected to said other component, or may be connected via another component (e.g., a third component).

[0030] The term "module" as used in this document includes a unit composed of hardware or firmware, and may be used interchangeably with terms such as logic, block, component, or circuit. A module may be an integral component, or a minimal unit or portion thereof that performs one or more functions. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0031] FIG. 1 illustrates an exemplary operation of an electronic device (101) for restoring an original image (110) using a file (124) including an edited image (120), according to one embodiment. Referring to FIG. 1 , an electronic device (101) having the appearance of a mobile phone is illustrated. The electronic device (101) may include a display (130) visible on one side of a housing. Exemplary form factors of the electronic device (101) including the display (130) are described with reference to FIG. 2A and / or FIG. 2B .

[0032] Referring to FIG. 1, exemplary states (191, 192) of an electronic device (101) executing a function related to a file (124) are illustrated. For example, the file (124) may include a JPEG file, a high efficiency image file format (HEIF) file, a high efficiency image container (HEIC) file, a portable network graphic (PNG) file, and / or a graphics interchange format (GIF) file. For example, the file (124) may include an edited image (120) corresponding to an original image (110). For example, the file (124) may include an edited image (120) and metadata (122) that are converted from the original image (110) based on the execution of an artificial intelligence model (e.g., a generative artificial intelligence model). An exemplary operation of an electronic device (101) for generating or storing a file (124) including an edited image (120) and metadata (122) is described with reference to FIG. 3, FIG. 4A and / or FIG. 4B. Referring to FIG. 1, an edited image (120) generated by changing a subject area (111) corresponding to a tree and a subject area (112) corresponding to a cloud in an original image (110) can be stored in a file (124).

[0033] Referring to an exemplary state (191) of FIG. 1, according to one embodiment, the electronic device (101) may display, on the display (130), a viewer screen including an edited image (120) corresponding to an original image (110) and visual objects (e.g., visual objects (141, 142)) for indicating a difference between the original image (110) and the edited image (120). For example, the electronic device (101) may display a visual object (141) having a position overlapping on the edited image (120). The position on the edited image (120) at which the visual object (141) is displayed may be related to the difference between the original image (110) and the edited image (120). For example, the electronic device (101) may display a visual object (141) having a position on the edited image (120) corresponding to a position of at least a portion of the original image (110) that is different from the edited image (120). The embodiment is not limited thereto, and the electronic device (101) may display a visual object (142) in the form of a button, including designated text (e.g., “AI Revert”) indicating restoration of the original image (110), on a portion of the display (130) spaced apart from the edited image (120).

[0034] According to one embodiment, the electronic device (101) may receive an input (e.g., a tap gesture for the visual object (141)) related to a visual object (e.g., visual objects (141, 142)) while displaying a viewer screen. For example, the electronic device (101) may receive an input indicating selection of the visual object (141). Based on the input, the electronic device (101) may obtain information used to restore the original image (110) using an artificial intelligence model from metadata (122) of a file (124) including the edited image (120). The electronic device (101) may execute the artificial intelligence model using the edited image (120) and the information to generate a restored image (150) related to the original image (110). An exemplary operation of the electronic device (101) to generate the restored image (150) is described with reference to FIG. 5 . The above artificial intelligence model may include a computational model executed by the electronic device (101) to simulate the neural activity of a living organism (e.g., reasoning, recognition, and / or creation). The artificial intelligence model is described with reference to FIGS. 6A to 6F.

[0035] Referring to FIG. 1, in response to an input indicating selection of a visual object (e.g., one of the visual objects (141, 142)), the electronic device (101) may display a restored image (150) on the display (130) in response to the input. The state (192) of FIG. 1 may include an exemplary state in which the restored image (150) is displayed on the display (130).

[0036] In state (192), the electronic device (101) displaying the restored image (150) replaced with the edited image (120) stored in the file (124) may display a visual object (144) related to a function of creating and / or storing a file (e.g., file (124) of FIG. 1) containing the restored image (150). For example, a visual object (144) having the form of a button containing designated text such as “Save” is illustrated, but the embodiment is not limited thereto. For example, in response to an input indicating selection of the visual object (144), the electronic device (101) may create or store a file containing the restored image (150). For example, in response to an input indicating selection of the visual object (144), the electronic device (101) may replace the edited image (120) stored in the file (124) with the restored image (150).

[0037] In state (192), the electronic device (101) may display a visual object (143) for switching to a state (191) prior to the state (192) displaying the restored image (150). For example, a visual object (143) in the form of a button including designated text such as “Cancel” is illustrated, but the embodiment is not limited thereto. For example, in response to an input indicating selection of the visual object (143), the electronic device (101) may switch to a state (191) prior to the state (192). For example, in response to an input indicating selection of the visual object (143), the electronic device (101) may stop displaying a viewer screen associated with the restored image (150) (or the edited image (120)). For example, in response to an input indicating a selection of a visual object (143), the electronic device (101) may discard the edited image (120) of the file (124) as a restored image (150).

[0038] According to one embodiment, the electronic device (101) may execute an artificial intelligence model using information included in metadata (122) to restore the original image (110) prior to the edited image (120). An exemplary operation of the electronic device (101) for generating a restored image (150) related to the original image (110) using information included in metadata (122) is described with reference to FIGS. 7 to 13.

[0039] For example, information indicating a history of editing the original image (110) using the original image (110) and / or an artificial intelligence model (e.g., a generative artificial intelligence model) may be stored within the metadata (122). For example, from a file (124) that includes only the edited image (120) without the original image (110), the electronic device (101) may obtain the information from the metadata (122). Using the information, the electronic device (101) may generate or display a restored image (150) corresponding to the original image (110). For example, since reduced-size information related to the original image (110) is stored (within the metadata (122)) in the file (124) that includes only the edited image (120) without the original image (110), the memory capacity of the electronic device (101) may be reduced. An example of a screen displayed by an electronic device (101) to generate a restoration image (150) is described with reference to FIG. 14a and / or FIG. 14b.

[0040] Hereinafter, an exemplary hardware configuration of an electronic device (101) according to one embodiment is described with reference to FIG. 2a and / or FIG. 2b.

[0041] FIGS. 2A and 2B illustrate block diagrams of an electronic device (101) according to one embodiment. The electronic device (101) of FIG. 1 may be an example of the electronic device (101) described with reference to FIGS. 2A and / or 2B . Referring to FIGS. 2A and / or 2B , the electronic device (101) may include a processor (210), a memory (215), a display (130), a camera (220), and a communication circuit (225). The number and / or types of electronic components included in the electronic device (101) are not limited to the embodiments of FIGS. 2A and / or 2B . For example, the electronic device (101) may further include electronic components described with reference to FIG. 16 . For example, some of the electronic components of FIG. 2A (e.g., the camera (220) and / or the communication circuit (225)) may be excluded from the electronic device (101), or electronic components not shown in FIG. 2A and / or FIG. 2B may be further included in the electronic device (101).

[0042] Referring to FIGS. 2A and / or 2B , the electronic device (101) may have forms such as a laptop PC (personal computer) (101-1), smartphones (e.g., a bar-shaped smartphone (101-2), a foldable smartphone (101-3), or a sliderable (or rollable) smartphone (101-4)), a tablet PC (101-5), a head-mounted display (HMD) device (101-6), and other similar computing devices (not shown). The electronic device (101) may include a housing that forms the appearance (or outer appearance) of the electronic device (101). The housing of the electronic device (101) may be referred to as a case and may be formed of plastic, glass, ceramic, fiber composites, metal (e.g., stainless steel, aluminum, and / or titanium), another suitable material, or a combination of two or more of the foregoing materials. For example, the housing may be formed using a unibody configuration in which all or part of the housing is assembled as a single structure (e.g., a bar-shaped smartphone (101-2)), or may be formed using a plurality of structures (e.g., an internal frame structure, one or more structures forming the exterior of the housing surface) (e.g., a foldable smartphone (101-3) having a plurality of parts).

[0043] For example, the processor (210) may be operably or operatively coupled with another electronic component of the electronic device (101) that includes the memory (215). Referring to FIGS. 2A and / or 2B , the processor (210) may be electrically connected to hardware of the electronic device (101) via a communication bus (202). For example, operably or operatively coupled with an electronic component may indicate that the processor (210) is directly connected to another electronic component. For example, operably or operatively coupled with a first electronic component may indicate that the processor (210) is (indirectly) connected to the first electronic component via a second electronic component of the electronic device (101). For example, operably or operatively coupled with an electronic component may indicate that the state of the processor (210) is capable of controlling the electronic component. For example, the fact that the processor (210) is operatively coupled to an electronic component may indicate that the operation of the electronic component is caused based on information, data, signals, or commands provided from the processor (210), but is not limited thereto.

[0044] For example, the processor (210) of the electronic device (101) may include a circuit (e.g., a processing circuit) for processing data based on one or more instructions. The circuit for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), and / or an application processor (AP). For example, the number of processors may be one or more. The processing circuit of the processor that loads (or fetches) instructions and performs calculations corresponding to the loaded instructions may be referred to as or referred to as a core circuit (or core). For example, the processor may have a multi-core processor structure including a plurality of core circuits, such as a dual core, a quad core, a hexa core, or an octa core. The functions and / or operations described with reference to the present disclosure may be performed individually or collectively by one or more processing circuits included in the processor (210).

[0045] For example, the display (130) of the electronic device (101) can output visualized information to the user (e.g., screens displayed in states (191, 192) of FIG. 1). For example, the display (130) can be controlled by a controller (or processor (210)) such as a GPU (graphic processing unit) to output visualized information to the user. The display (130) can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs). The display (130) can include a flat panel display (FPD), electronic paper, and / or a flexible display having at least a partially curved or deformable shape.

[0046] For example, the display (130) of the electronic device (101) may include a sensor (e.g., a touch sensor panel (TSP)) for detecting an external object (e.g., a user's finger) on the display (130). For example, based on the TSP, the processor (210) may detect an external object that is in contact with the display (130) or floating on the display (130). In response to detecting the external object, the processor (210) may execute a function related to a specific visual object corresponding to a location of the external object on the display (130) among visual objects displayed on the display (130).

[0047] For example, the memory (215) of the electronic device (101) may include a circuit and / or a storage medium for storing data and / or instructions input to or output from the processor (210). The memory may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). The non-volatile memory may be referred to as storage. The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, solid state drive (SSD), and embedded multi media card (eMMC). The processor (210) of the electronic device (101) can execute instructions of the memory (215) within the electronic device (101) to perform functions and / or operations indicated by the instructions. For example, when the electronic device (101) includes at least one processor, the at least one processor can be configured to collectively or individually execute the instructions.

[0048] For example, the camera (220) of the electronic device (101) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate an electrical signal representing the color and / or brightness of light. The plurality of optical sensors included in the camera (220) may be arranged in the form of a two-dimensional array. The camera (220) may acquire the electrical signals of each of the plurality of optical sensors substantially simultaneously, and may generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional array. For example, photographic data (e.g., the original image (110) of FIG. 1) captured using the camera (220) may mean one (a) two-dimensional frame data acquired from the camera (220). For example, video data captured using the camera (220) may mean a sequence of a plurality of two-dimensional frame data acquired from the camera (220).

[0049] In one embodiment, the electronic device (101) may include a communication circuit (225) that is hardware for supporting transmission and / or reception of electrical signals between the electronic device (101) and an external electronic device (e.g., the external electronic device (250) of FIG. 2B). The communication circuit (225) may include, for example, at least one of a modem (MODEM), an antenna, and an optical / electronic (O / E) converter. The communication circuit (225) may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), near field communication (NFC), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), fifth generation (5G), new radio (NR), sixth generation (6G), and / or above-6G.

[0050] According to one embodiment, the electronic device (101) can process a file (124) stored in a memory (215) (e.g., one or more storage media within the electronic device (101), such as a volatile memory and / or a non-volatile memory). For example, the processor (210) can control the display (130) to visualize an edited image (120) included in the file (124). The file (124) can include metadata (122) related to the edited image (120). Within the file (124) having a jpeg format, the edited image (120) and metadata (122) can be aligned according to a format of the International Standard Organization (ISO) called EXIF ​​(exchangeable image file) (e.g., an exemplary format illustrated in FIG. 4B). The metadata (122) can be integrally included in the file (124) or can be stored in another file and / or database linked to the file (124).

[0051] In one embodiment, the processor (210) of the electronic device (101) may execute a function of at least partially modifying an edited image (120) using metadata (122). The function may include an operation of generating a restored image (e.g., restored image (150) of FIG. 1) corresponding to the edited image (120) using information included in the metadata (122). The restored image may include content of an original image (e.g., original image (110) of FIG. 1) corresponding to the edited image (120). The function may include an operation of generating an intermediate version of the edited image between the edited image (120) and the original image using information included in the metadata (122).

[0052] In one embodiment, an artificial intelligence model (e.g., an image restoration model (230)) for the above function may be installed in the electronic device (101). Hereinafter, the artificial intelligence model may refer to a computational model that simulates or imitates the neural activity of a living organism and a set of programs for executing the computational model.

[0053] In one embodiment, the installation of an artificial intelligence model in the electronic device (101) may mean that resources related to the artificial intelligence model (e.g., formulas included in the computational model and weights, parameters and / or coefficients related to the formulas) and instructions are stored in the memory (215) for execution of the artificial intelligence model using only the processor (210) of the electronic device (101). When an artificial intelligence model such as an image restoration model (230) is installed in the electronic device (101), the processor (210) can independently execute a function of restoring an original image using an edited image (120) without an external electronic device. An artificial intelligence model installed in the electronic device (101) to be independently executed by the electronic device (101) may be referred to as an on-device model.

[0054] Referring to FIG. 2A, the processor (210) of the electronic device (101) may execute an on-device model (e.g., an image restoration model (230)) stored in the memory (215) to at least partially modify the edited image (120). The embodiment is not limited thereto. Referring to FIG. 2B, the electronic device (101) may communicate with an external electronic device (250), referred to as a server, to execute a function of restoring an original image using a file (124) including the edited image (120). For example, the processor (210) may transmit at least a portion of the file (124) (e.g., the edited image (120) and / or metadata (122)) to the external electronic device (250) via the communication circuit (225). The processor (210) can transmit at least a portion of the file (124) to an external electronic device (250) along with a command (or request) for restoration of the original image.

[0055] Referring to FIG. 2B, the external electronic device (250) may include a processor (255), a memory (260), and a communication circuit (265). The processor (255), the memory (260), and the communication circuit (265) may be electrically and / or operatively coupled via a communication bus (252). The processor (255), the memory (260), and the communication circuit (265) of the external electronic device (250) may correspond to the processor (210), the memory (215), and the communication circuit (225) of the electronic device (101), respectively. At least a portion of the description of the processor (255), the memory (260), and the communication circuit (265) of the external electronic device (250) that overlaps with the description of the processor (210), the memory (215), and the communication circuit (225) of the electronic device (101) may be omitted.

[0056] Referring to FIG. 2B, an embodiment in which an image restoration model (230) is installed in an external electronic device (250) is illustrated. In the embodiment of FIG. 2B, a processor (255) that receives a signal for restoring an original image using a file (124) from an electronic device (101) may execute the image restoration model (230) installed in the external electronic device (250). For example, by executing the image restoration model (230) using metadata (122) included in the signal, the processor (255) may generate a restoration image (e.g., the restoration image (150) of FIG. 1) representing an original image corresponding to an edited image (120). The processor (255) may transmit the generated restoration image (or a signal including the restoration image) to the electronic device (101) via a communication circuit (265). The processor (210) that receives the restored image through the communication circuit (225) can control the display (130) to display the restored image.

[0057] As described above, according to one embodiment, the processor (210) of the electronic device (101) can restore the original image corresponding to the edited image (120) using the image restoration model (230) installed in the electronic device (101) and / or the external electronic device (250). The image restoration model (230) installed in the electronic device (101) and / or the external electronic device (250) can be trained to restore the original image using the edited image (120) of the original image and metadata (122) related to the original image stored in the edited image (120). In one embodiment, when generating a file (124), the processor (210) can train or update an artificial intelligence model (e.g., the image restoration model (230) of FIG. 2A and / or FIG. 2B) related to the restoration of the original image using information related to the original image. The training may be performed so that the AI ​​model generates or outputs a prompt optimized for restoring the original image. For example, the prompt may be generated by a dedicated AI model (e.g., a prompt inference model) for inferring or generating the prompt. For example, the prompt inference model may be trained together with the AI ​​model for modifying and / or restoring the original image. Upon completion of the training, the AI ​​model may be used as an AI model for restoring the original image from the file (124). In order to reduce the size of the file (124), the file (124) including the edited image (120) may not include the original image, and may further include metadata (122) including information required for restoring the original image. The information may be associated (or linked) with the edited image (120).The information stored in the metadata (122) to restore the original image may include at least one of the data exemplified in Table 1.

[0058] Description of the type (or category) of information Pixel information Color, brightness, and / or saturation of at least one pixel of the original image One or more first prompts Prompts that were input when generating an edited image (120) from the original image One or more second prompts Prompts that include one or more words (e.g., keywords) to describe the original image Edge information (or edge image) Information indicating the boundary of at least one subject area of ​​the original image Layout information Information indicating the location, size, and / or shape of at least one subject area of ​​the original image GPS (global positioning system) information Information indicating the point where the original image was acquired Feature information Feature vectors (e.g., latent vectors) and / or feature points (or key points) for at least a portion of the original image

[0059] Referring to Table 1, pixel information may indicate the color, brightness, and / or saturation of at least one pixel of the original image based on a resolution lower than or equal to the resolution of the original image and / or the edited image (120). Hereinafter, with reference to FIG. 3, an exemplary operation of an electronic device (101) for generating a file (124) including the edited image (120) of FIG. 1, FIG. 2A, and / or FIG. 2B is described.

[0060] FIG. 3 illustrates exemplary operations of an electronic device (101) for generating an edited image (331) for an original image (110). The electronic device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIGS. 2A and / or 2B may perform the operations described with reference to FIG. 3. The order of the operations of FIG. 3 is exemplary, and in one embodiment, the electronic device (101) may perform the operations of FIG. 3 in a different order than the order illustrated in FIG. 3. In one embodiment, the electronic device (101) may perform at least two of the operations of FIG. 3 substantially simultaneously (e.g., multitasking and / or multithreading).

[0061] Referring to FIG. 3, the electronic device (101) can display a screen including an original image (110) on a display (130). The screen can be provided by a software application for viewing images and / or videos stored in the electronic device (101), such as a gallery application. The electronic device (101) can display visual objects (312, 313, 314, 315, 316, 317, 318) related to editing of the original image (110) together with the original image (110) on the screen including the original image (110).

[0062] For example, a visual object (312) may be associated with a function to reduce or remove a shadow represented in the original image (110) (or selected by an input associated with the original image (110). For example, a visual object (313) may be associated with a function to reduce or remove a specular reflection represented in the original image (110). For example, a visual object (315) may be associated with a function to sequentially reverse (e.g., undo) editing actions applied to the original image (110) by one or more inputs. For example, a visual object (316) may be associated with a function to redo (e.g., redo) an action undone by the visual object (315). For example, a visual object (314) may be associated with a function to remove a subject from the image (110) by altering at least a portion of the subject represented in the original image (110). For example, a visual object (317) may be associated with a function of saving the result of editing an original image (110) displayed on a screen to a file (e.g., file (124) of FIG. 1). For example, a visual object (318) may be associated with a function of stopping or canceling editing of an original image (110).

[0063] Referring to FIG. 3 , within operation (310), according to an embodiment, a processor of an electronic device (101) may receive an input for editing an original image (110). The input of operation (310) may include a first input indicating a selection of a portion (311) associated with an exemplary subject, such as a tree, as shown in FIG. 3 . The input of operation (310) may include a second input indicating a selection of a visual object (314), received while a line (e.g., a dashed line) representing the portion (311) is displayed on the original image (110) according to the first input. In response to the second input, the electronic device (101) may perform another operation subsequent to operation (310). The embodiment is not limited to the exemplary first and second inputs, and the input of operation (310) may include another input indicating a selection of any one of the visual objects (312, 313).

[0064] Referring to FIG. 3, in operation (320), according to one embodiment, a processor of an electronic device may execute an image editing model using a prompt associated with a received input. In the present disclosure, the prompt may refer to a natural language sentence to be input into the image editing model. Natural language may refer to a language used in human daily life. The prompt may refer to a natural language sentence that has been binarized based on a binary code such as Unicode (or ASCII code).

[0065] Referring to FIG. 3, upon receiving an exemplary second input for selecting a visual object (314), the electronic device (101) may generate a prompt based on the function of the visual object (314). In response to a second input received after the first input indicating selection of the portion (311), the electronic device (101) may generate a prompt (e.g., "Remove the tree") for removing a subject (e.g., a tree) captured in the portion (311). The embodiment is not limited thereto. For example, the electronic device (101) may generate a prompt from a detected utterance using analysis of an audio signal based on speech-to-text (STT). For example, the electronic device (101) may display a visual object, such as a text box, on the display (130) and directly receive a prompt from a user through the text box. An exemplary operation of an electronic device (101) executing an image editing model using a prompt is described with reference to FIG. 4a and / or FIG. 4b.

[0066] Referring to FIG. 3, in operation (330), according to one embodiment, the processor of the electronic device may display an edited image (331) obtained by executing an image editing model. The electronic device (101) may generate or obtain the edited image (331) by performing calculations indicated by the image editing model for which the prompt of operation (320) has been input. The electronic device (101) may display the generated edited image (331) on the display (130).

[0067] Referring to FIG. 3, in operation (340), according to one embodiment, a processor of an electronic device may store a file (e.g., file (124) of FIG. 1) including metadata (e.g., metadata (122) of FIG. 1) including information for restoring an original image (110) based on an input for storing an edited image (331). The input of operation (340) may include an input indicating a selection of a visual object (317). In response to the input of operation (340), the electronic device (101) may generate or store a file including the edited image (331). In response to the input of operation (340), the electronic device (101) may generate metadata including information for restoring an original image (110). The metadata may be included in, or inserted into, the file including the edited image (331). Information for restoring the original image (110) may include information (e.g., information in Table 1) required for executing an image restoration model (e.g., image restoration model (230) of FIG. 2a and / or FIG. 2b).

[0068] As described above, according to one embodiment, the electronic device (101) may store metadata including an edited image (331) for the original image (110) and information for restoring the edited image (331) to the original image (110) after editing the original image (110) using a generative artificial intelligence model such as an image editing model. For example, the electronic device (101) that receives an input of an operation (340) may remove the original image (110). The information stored in the metadata may be related to (or linked to) at least a portion of the original image (110) that has been edited by the user input. For example, the electronic device (101) may restore the original image (110) using a file including an edited image (331) without the original image (110).

[0069] Hereinafter, with reference to FIG. 4a and / or FIG. 4b, an exemplary operation of an electronic device (101) executing an image editing model of operation (320) is described.

[0070] FIGS. 4A and / or 4B illustrate exemplary operations of an electronic device for generating a file (124) including an edited image (331) generated using a generative artificial intelligence model (e.g., an image editing model (420)). The electronic device (101) of FIGS. 1, 2A and 2B and / or the processor (210) of FIGS. 2A and / or 2B may perform the operations described with reference to FIGS. 4A and / or 4B. The operations of the electronic device described with reference to FIGS. 4A and / or 4B may be related to at least one of the operations of FIG. 3 (e.g., operation (320)).

[0071] Referring to FIG. 4A, according to one embodiment, an electronic device may obtain a prompt (410) (e.g., "Remove the tree") associated with an original image (110). For example, the prompt (410) may be generated by the electronic device upon receiving a touch input received via a display (e.g., a touch input on a visual object (317) of FIG. 3 ). For example, the prompt (410) may correspond to a user's utterance detected by an audio signal received via a microphone. For example, the prompt (410) may be received from the user via a keyboard (or a software keyboard and / or virtual keyboard displayed on the display of the electronic device).

[0072] Referring to FIG. 4A, an electronic device that has acquired a prompt (410) can execute an image editing model (420) using the prompt (410) and the original image (110). For example, the image editing model (420) can be installed in the electronic device (101) in the form of an on-device model. For example, the electronic device (101) can communicate with an external electronic device (e.g., the external electronic device (250) of FIG. 2B) and execute the image editing model (420) installed in the external electronic device.

[0073] For example, when a prompt (410) indicating removal of a specific subject (e.g., a tree) included in an image (110) is received, the electronic device may obtain an edited image (331) generated by changing a subject area (111) corresponding to the subject in the original image (110) from the image editing model (420). For example, the image editing model (420) may be trained to generate, in response to the prompt (410), a partial image to be positioned in the subject area (111) (e.g., a partial image related to the background of the original image (110)) and to replace the subject area (111) of the original image (110) with the generated partial image.

[0074] In one embodiment, the electronic device (101) may generate or store a file (124) including the edited image (331) in response to an input for storing the edited image (331). For example, the electronic device (101) may store information (e.g., a prompt (410)) used to generate the edited image (331) from the original image (110) in first metadata (122-1) of the file (124). For example, the electronic device (101) may store information to be used to restore the original image (110) in second metadata (122-2) of the file (124).

[0075] In the present disclosure, metadata terms may be numbered, such as first metadata (122-1) and second metadata (122-2). The numbered metadata terms may be used for logical distinction according to the content, purpose, type, and / or format of the metadata. For example, all numbered metadata may be stored within a logical area set to store metadata within a file (124). The embodiment is not limited thereto, and the numbered metadata may be stored separately within the file (124) (or a storage medium for storing metadata).

[0076] For example, the first metadata (122-1) may include history information about the edited image (331) stored in the file (124), and the second metadata (122-2) may include reduced-size information for restoring the source (e.g., the original image (110)) of the edited image (331). For example, the history information stored in the first metadata (122-1) may include a value (or flag) indicating whether the edited image (331) was generated or edited by a generative artificial intelligence model (e.g., the image editing model (420)). For example, the history information stored in the first metadata (122-1) may include the time (e.g., date and / or timestamp) at which the edited image (331) was generated, and region information indicating at least a portion of the edited image (331) that has been changed from the original image (110).

[0077] In one embodiment, within a file (124), an edited image (331) and metadata (e.g., first metadata (122-1) and / or second metadata (122-2)) may be arranged based on the EXIF ​​format. Referring to FIG. 4B, an exemplary structure of a file (124) based on the EXIF ​​format is illustrated. The file (124) may start from a region in which a designated value indicating the start of the file (124) (e.g., Start Of Image) is stored. After the region, one or more application segments (APPlication segments) (e.g., APP1, APP2) may be formed within the file (124). The application segment of the file (124) may include a marker indicating the start of the application segment (APP1 Marker) and a length of the application segment (APP1 Length). The application segment of the file (124) may include a code for the application segment (EXIF identifier code), a tagged image file format (TIFF) header, and image file directory (IFD) values. For example, the 0th IFD value may include information related to an image (e.g., an edited image (331)) within the file (124) (e.g., the creation time of the file (124)). For example, the 1st IFD value may include a thumbnail image corresponding to the file (124). The metadata of the file (124) may be stored in the application area of ​​the file (124).

[0078] A file (124) based on the EXIF ​​format may further include, after one or more application areas, a (Define-Quantization-Tables) area, a DHT (Define-Huffman-Tables) area, a DRI (Define-Restart-Interval) area, a SOF (Start of Frame) area, and / or an SOS (Start-Of-Scan) area. After the SOS area, the file (124) may include compressed data representing an edited image (331). At the end point of the file (124), a designated value representing the end of the file (124) (End of Image) may be stored.

[0079] As described above, according to one embodiment, when generating an edited image from an original image, the electronic device may generate or store a file (124) including the edited image among the original image or the edited image. The electronic device may add information for restoring the original image to the metadata included in the file (124). The embodiment is not limited thereto, and the electronic device may generate a file including metadata to which information for generating the original image and the edited image is added. The metadata may be stored in a file different from the file (124) including the edited image. For example, when the metadata is stored in a file different from the file (124), information (e.g., link information) indicating a linkage with the other file (or metadata) may be stored in the file (124). The information may be stored in the metadata of the file (124). For example, when the metadata is stored in a file different from the file (124), the electronic device may manage (e.g., establish and / or release a link) between the file (124) and the other file using a database and / or a program.

[0080] Hereinafter, with reference to FIG. 5, an exemplary operation of an electronic device for restoring an original image from a file (124) generated based on the operation of the electronic device of FIG. 3, FIG. 4a and / or FIG. 4b is described.

[0081] FIG. 5 illustrates exemplary operations of an electronic device for generating a restored image (150) corresponding to an original image from an edited image (120) included in a file (124). The electronic device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIGS. 2A and / or 2B may perform the operations described with reference to FIG. 5. The order of the operations of FIG. 5 is exemplary, and in one embodiment, the electronic device may perform the operations of FIG. 5 in a different order than the order illustrated in FIG. 5. In one embodiment, the electronic device may perform at least two of the operations of FIG. 5 substantially simultaneously.

[0082] Referring to FIG. 5, in operation (510), according to one embodiment, a processor of an electronic device may display an edited image (120) corresponding to an original image. The processor may display the edited image (120) identified from a file (e.g., file (124) of FIG. 1) on a display (e.g., display (130) of FIG. 1) of the electronic device. While the edited image (120) is displayed, the processor may determine, using metadata stored in the file (e.g., first metadata (122-1) and / or second metadata (122-2)), whether the edited image (120) was generated or edited using an artificial intelligence model (e.g., image editing model (420) of FIG. 4A). If it is determined that the edited image (120) was generated using the artificial intelligence model, the processor may display, on the display, a visual object for restoring the original image (e.g., at least one of visual objects (141, 142) of FIG. 1).

[0083] Referring to FIG. 5 , in operation (520), a processor of an electronic device according to one embodiment may receive an input for at least partially restoring an original image. The input may include an input indicating a selection of a visual object for restoring the original image, displayed together with the edited image (120), such as a tap gesture on the visual object. Based on the input, the processor may perform operation (530).

[0084] Referring to FIG. 5, in operation (530), according to one embodiment, a processor of an electronic device may obtain metadata (e.g., first metadata (122-1) and / or second metadata (122-2)) for at least partially restoring an original image from a file (124) including an edited image (120). The metadata may be generated by, or may be stored in, the operation described with reference to FIG. 3, FIG. 4A, and / or FIG. 4B. The metadata of operation (530) may include information required to restore an original image using the edited image (120), such as the information in Table 1.

[0085] Referring to FIG. 5, in operation (540), according to one embodiment, the processor of the electronic device may execute the image restoration model (230) using the acquired metadata to generate at least a portion of the restored image (150). In one embodiment where the on-device model is installed in the electronic device, the processor may execute the image restoration model (230) installed in the electronic device (e.g., the image restoration model (230) of FIG. 2a) based on operation (540) to obtain or generate at least a portion of the restored image (150). In one embodiment connected to an external electronic device such as a server (e.g., the external electronic device (250) of FIG. 2b), the processor may communicate with the external electronic device based on operation (540) to cause execution of the image restoration model (230) installed in the external electronic device (e.g., the image restoration model (230) of FIG. 2b). After triggering the execution of the image restoration model (230) installed in the external electronic device, the processor may receive a signal related to at least a portion of the restored image (150) from the external electronic device. From the perspective of generating using an artificial intelligence model such as the image restoration model (230), the restored image (150) may be referred to as an artificial intelligence-generated image. A partial image (or visual object) generated by the artificial intelligence model to generate the restored image (150) may be referred to as an artificial intelligence-generated object.

[0086] Referring to FIG. 5, within operation (550), according to one embodiment, a processor of an electronic device may display at least a portion of a restored image (150). In one embodiment, where an edited image (120) is displayed prior to operation (550), the processor may control the display (e.g., display (130) of FIG. 1) such that the edited image (120) displayed on the display is changed to a restored image (150).

[0087] Hereinafter, with reference to FIGS. 6A to 6F, an image restoration model (230) executed by an electronic device according to one embodiment is exemplarily described.

[0088] FIGS. 6A, 6B, 6C, 6D, 6E, and 6F illustrate exemplary structures of an image restoration model (e.g., the image restoration model (230) of FIGS. 2A and / or 2B) executed by an electronic device according to one embodiment. The electronic device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIGS. 2A and / or 2B can execute or utilize the artificial intelligence model described with reference to FIGS. 6A to 6F as the image restoration model. The image restoration model can be referred to as a super resolution model from the perspective of generating a high-resolution restored image from a file containing an edited image (e.g., the file (124) of FIG. 1). The image restoration model can be referred to as a generative artificial intelligence model from the perspective of generating a restored image. The image restoration model may be referred to as an auto-encoder-based model based on the structure used to implement the model. The image restoration model may also be referred to as a prompt model from the perspective of generating restored images using prompts.

[0089] In various embodiments of the present disclosure, a generative artificial intelligence model may mean an artificial intelligence model trained using a set of multiple images (e.g., a training set and / or a training database), and capable of generating or outputting new images that are approximate to the multiple images.

[0090] Referring to FIG. 6A, artificial intelligence models trained based on generative adversarial networks (GANs) (e.g., a generator model (611) and a discriminator model (612)) are illustrated. The generator model (611) can be trained to output a generated image (614) from input data (e.g., a latent vector (613) in a latent space). The discriminator model (612) can be trained to output a parameter (R) indicating whether an input image (e.g., a generated image (614) and / or a real image (615) generated by the generator model (611)) was generated by the artificial intelligence model (e.g., a probability that the input image is generated by the artificial intelligence model).

[0091] For example, when executing a discriminator model (612) using a real image (615), the electronic device can obtain, from the discriminator model (612), a parameter indicating that the input image of the discriminator model (612) (in the above example, the real image (615)) was not generated by the artificial intelligence model. For example, when executing a discriminator model (612) using a generated image (614), the electronic device can obtain, from the discriminator model (612), a parameter indicating that the input image of the discriminator model (612) (in the above example, the generated image (614)) was generated by the artificial intelligence model.

[0092] In one embodiment, the generator model (611) may be trained such that the generated image (614) generated by the generator model (611) is determined by the discriminator model (612) not to have been generated by the artificial intelligence model. For example, the generator model (611) may be trained such that the probability that the input image output by the discriminator model (612) is generated by the artificial intelligence model is reduced. The meaning of "the input image was determined by the discriminator model (612) to have been generated by the artificial intelligence model" may include the meaning that the input image is similar to the original image. The meaning of "the input image was determined by the discriminator model (612) to not have been generated by the artificial intelligence model" may include the meaning that the input image is different from the original image.

[0093] Training of the generator model (611) and / or the discriminator model (612) may be repeatedly performed on a plurality of generated images including the generated image (614). Training of the generator model (611) and / or the discriminator model (612) may be terminated when the generated image (614) is determined by the discriminator model (612) not to have been generated by the artificial intelligence model. For example, training of the generator model (611) may be terminated when a generated image (614) having content, composition, and / or color distribution similar to that of the actual image (615) is generated. The trained generator model (611) may be used as an image restoration model (e.g., the image restoration model (230) of FIG. 2A and / or FIG. 2B) for restoring the original image.

[0094] Referring to FIG. 6B, an artificial intelligence model having the structure of an auto encoder (AE) is illustrated. The artificial intelligence model may include an encoding model (621) and a decoding model (622). The artificial intelligence model having the structure of an auto encoder, including the encoding model (621) and the decoding model (622), may be trained to reduce the difference between an input image (623) input to the encoding model (621) and a generated image (625) output from the decoding model (622) (e.g., to generate a generated image (625) that is identical to the input image (623)).

[0095] When an artificial intelligence model having an auto-encoder structure is executed, dimensionality reduction based on the encoding model (621) can be performed. For example, feature information (e.g., latent vector (624)) having a size smaller than the size of the input image (623) can be output from the encoding model (621). The latent vector (624) is implicit information output from the encoding model (621), and the encoding model (621) can be trained to output a latent vector (624) having different elements for images input to the encoding model (621).

[0096] When an artificial intelligence model having an auto-encoder structure is executed, dimensionality expansion based on a decoding model (622) can be performed. For example, the decoding model (622) can be trained to output a generated image (625) from feature information such as a latent vector (624). The decoding model (622) can be trained to output a generated image (625) having the same resolution and / or the same size as the input image (623). For example, the artificial intelligence model having an auto-encoder structure can be trained to reduce or minimize the difference between the input image (623) and the generated image (625). After training, at least a portion of the artificial intelligence model including the decoding model (622) can be used as an image restoration model to restore the original image.

[0097] Referring to Fig. 6c, an artificial intelligence model having a variational autoencoder (VAE) structure is illustrated. The artificial intelligence model having a VAE structure may include an encoding model (631) and a decoding model (632). Within the artificial intelligence model having a VAE structure, the encoding model (631) may be trained to output a latent vector (636) including a mean vector (634) and a standard deviation vector (635) for an input image (633) (e.g., at least one image included in a set of images set for training during training).

[0098] In one embodiment, the decoding model (632) can be trained to generate a generated image (637) using a probability distribution that has features of the input image (633) as probability variables based on the latent vector (636). For example, when training using a set of images for training, the decoding model (632) can be trained to perform dimensionality expansion based on a probability distribution of features included in the images. After the training, the decoding model (632) can output a generated image (637) that expresses one or more features represented by the latent vector (636) using the probability distribution represented by the latent vector (636). After the training, at least a portion of the artificial intelligence model having the structure of a VAE including the decoding model (632) can be used as an image restoration model for restoring the original image.

[0099] Referring to FIG. 6D, an artificial intelligence model having a diffusion model (DM) structure is illustrated. The AI ​​model having the DM structure can be trained to compensate for and / or restore distortions (e.g., distortions based on noise and / or masking) of images included in a set of training images when trained using the set. For example, a corrupted version of an image included in the set (e.g., an image (642-1) completely contaminated by noise) can be input to the AI ​​model. Based on the execution of the AI ​​model, a restored image (642-3) can be generated from the corrupted version of the image (642-1). The process of the AI ​​model generating the restored image (642-3) from the corrupted version of the image (642-1) can be referred to as denoising.

[0100] Referring to FIG. 6D, an artificial intelligence model having a DM structure may include an artificial intelligence model for natural language processing (e.g., a transformer (641)). For example, by executing the transformer (641) using a prompt (643) (e.g., a natural language phrase and / or sentence such as "An image of the face of a man"), information (644) that can be input to the artificial intelligence model for denoising may be generated. The information (644) may include vectors (or tokens) generated from each of the words (or morphemes) included in the prompt (643) based on tokenization.

[0101] For example, as information (644) is input to an artificial intelligence model for denoising, the artificial intelligence model can be trained to generate a restored image (642-3) including features indicated by the prompt (643) from a corrupted version of the image (642-1). For example, information (644) can be input to the artificial intelligence model when an intermediate version of the image (642-2) is generated from the corrupted version of the image (642-1). The information (644) input to the artificial intelligence model can function as a condition for generating the intermediate version of the image (642-2). After training is completed, at least a portion of the artificial intelligence model including the transformer (641) can be used as an image restoration model for restoring the original image.

[0102] Referring to Fig. 6e, an artificial intelligence model having a latent diffusion model (LDM) structure is illustrated. The artificial intelligence model having an LDM structure may have a structure in which a contrastive language-image pre-training model (CLIP) and / or VAE-based structure is combined with an artificial intelligence model having a DM structure of Fig. 6d. The artificial intelligence model having an LDM structure may include a text encoding model (652) that performs tokenization of a prompt (651). The text encoding model (652) may have a structure based on CLIP. Information (653) corresponding to the prompt (651) may be generated from the text encoding model (652). The information (653) may include vectors (or tokens) corresponding to each of the words (or morphemes) included in the prompt (651) based on tokenization.

[0103] Referring to FIG. 6e, an artificial intelligence model having the structure of an LDM may include a U-model (654) (or U-Net) and a scheduler (657) for repeatedly performing image transformation based on the U-model (654). The scheduler (657) is a program based on a scheduling algorithm and may be configured to repeatedly input an input image (655) into the U-model (654) and input an output image (656) output from the U-model (654) as an input image (655) for the U-model (654).

[0104] Referring to FIG. 6e, when the U-model (654) is executed, information (653) corresponding to the prompt (651) can be input to the U-model (654) as a condition for generating an output image (656). The U-model (654) can be trained to perform denoising on the input image (655), similar to the artificial intelligence model having the structure of the DM of FIG. 6d. The U-model (654) can be trained to perform denoising based on the condition set by the information (653).

[0105] Referring to Fig. 6e, an artificial intelligence model having the structure of an LDM may include a decoding model (658) for generating a generated image (659) to be provided as a result of generating an image from an output image (656) of a U-model (654). For example, the output image (656) finally output from the U-model (654) by the scheduler (657) may be converted into a generated image (659) by the decoding model (658). For example, based on the execution of the decoding model (658), the resolution of the output image (656) may be increased, or the size of the output image (656) may be enlarged. The artificial intelligence model having the structure of an LDM of Fig. 6e may be used as an image restoration model for restoring an original image.

[0106] Referring to Fig. 6f, an artificial intelligence model having a layout diffusion model structure is illustrated. The artificial intelligence model of Fig. 6f may have a structure modified from the structure of the LDM of Fig. 6e so that layout information (661) can be input. The layout information (661) may be set to indicate the types of subjects associated with each portion of the generated image (659) in order to guide the layout of the generated image (659). For example, the layout information (661) may include data indicating bounding boxes indicating portions of the generated image (659) where the subjects are to be positioned (e.g., coordinates of the vertices of the bounding boxes, width and height of the bounding boxes). For example, the layout information (661) may include pixel-wise information indicating subjects expressed in each portion of the generated image (659). For example, the layout information (661) may include an image in which a shape of a specified color corresponding to a specific subject is drawn.

[0107] The U-model (654) of the artificial intelligence model having the structure of the layout diffusion model can be trained to process layout information (661) based on attention operations. For example, based on a query-key-value (QKV) attention algorithm, the layout information (661) can be synthesized with other information within the U-model (654). Based on the synthesis, when generating an output image (656) from an input image (655), the layout information (661) can function as a condition related to the output image (656). The artificial intelligence model having the structure of the layout diffusion model of FIG. 6f can be used as an image restoration model for restoring an original image.

[0108] As described above, when restoring an original image using a file (e.g., file (124) of FIG. 1) containing an edited image (e.g., edited image (120) of FIG. 1), image restoration models having various structures can be used. Hereinafter, with reference to FIGS. 7 to 13, exemplary operations of an electronic device restoring an original image using the artificial intelligence model described with reference to FIGS. 6A to 6F are described.

[0109] FIG. 7 illustrates an exemplary operation of an electronic device for restoring an original image using pixel information (722) stored in metadata of a file (124) (e.g., second metadata (122-2)). The electronic device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIGS. 2A and / or 2B may perform the operations described with reference to FIG. 7. At least one of the operations of FIG. 7 may be related to, or may be performed similarly to, the operations of FIG. 5. The order of the operations of FIG. 7 is exemplary, and in one embodiment, the electronic device may perform the operations of FIG. 7 in a different order than the order illustrated in FIG. 7. In one embodiment, the electronic device may perform at least two of the operations of FIG. 7 substantially simultaneously.

[0110] Referring to FIG. 7, in operation (710), according to one embodiment, a processor of an electronic device may receive an input for restoring an original image. The processor may receive the input of operation (710) while displaying a viewer screen including an edited image (711), such as state (191) of FIG. 1, on a display (e.g., display (130) of FIG. 1). For example, the input of operation (710) may include the input of operation (520) of FIG. 5.

[0111] Referring to FIG. 7, in operation (720), according to one embodiment, a processor of an electronic device may obtain, from metadata of a file including an edited image (711), a location of at least a portion (e.g., a portion (721)) of an edited image (711) converted from an original image and pixel information (722) of the original image. When a file (124) including an edited image (711) is generated based on the operation described with reference to FIG. 3, FIG. 4A and / or FIG. 4B, the file (124) may include first metadata (122-1) related to a change history generated during a change from an original image to an edited image (711) and second metadata (122-2) storing information to be used for restoring the original image.

[0112] For example, the electronic device can use the first metadata (122-1) to identify a portion (721) of the edited image (711) that has been changed from the original image. For example, the first metadata (122-1) may include data indicating the position, shape, and / or size of the portion (721) within the edited image (711). Referring to FIG. 7, in one embodiment where the edited image (711) was generated by changing the expression of a face captured by the original image, the electronic device can obtain data indicating a portion (721) corresponding to the face within the edited image (711) from the first metadata (122-1).

[0113] For example, the electronic device can obtain pixel information (722) for at least a portion of the original image corresponding to the portion (721) using the second metadata (122-2). The pixel information (722) may include a color distribution and / or a brightness distribution of the portion of the original image corresponding to the portion (721). Referring to FIG. 7, when an edited image (711) is created by changing at least a portion of the original image, the entire original image is not stored in the file (124), but pixel information (722) corresponding to the portion (721) is stored, so that the file (124) can have a relatively small size and further include information for restoring and / or displaying the original image.

[0114] For example, information (e.g., pixel information (722)) for minimizing or preventing distortion that may occur when restoring the original image may be stored in the file (124). The pixel information (722) may be based on the resolution and / or size of the original image. For example, the pixel information (722) may include pixels that are one-to-one matched with pixels corresponding to the portion (721) in the original image. For example, the pixel information (722) may represent the color distribution of pixels of the original image corresponding to the portion (721) using a resolution lower than the resolution of the original image or a size smaller than the size of the portion (721) of the original image.

[0115] Referring to FIG. 7, in operation (730), according to one embodiment, the processor of the electronic device may generate a partial image corresponding to at least a portion (e.g., portion (721)) of the edited image (711) using pixel information (722) obtained by performing operation (720). For example, the processor may obtain, from second metadata (122-2), a partial image for a second portion of the original image corresponding to the portion (721) of the edited image (711). When the pixel information (722) represents pixel values ​​(or pixel colors) of a portion of the original image corresponding to the portion (721) of the edited image (711) based on the resolution and / or size of the original image, the pixel information (722) may be used as the partial image of operation (730).

[0116] In one embodiment, the partial image of the operation (730) may have dimensions (e.g., width, height, and / or size) and / or resolution of the partial image (721), which are indicated by the first metadata (122-1). An electronic device that identifies pixel information (722) based on a lower resolution than the resolution of the original image (or the edited image (711)) from the second metadata (122-2) may execute an artificial intelligence model (e.g., a high-resolution model) to increase the resolution of the pixel information (722), thereby synthesizing or generating a partial image having the same resolution as the resolution of the original image (or the edited image (711)). An operation of generating a partial image having a relatively higher resolution from pixel information (722) having a relatively lower resolution may be referred to as an upscaling operation.

[0117] Referring to FIG. 7, in operation (740), according to one embodiment, a processor of an electronic device may generate a restored image (741) by combining the generated partial image and the edited image (711). For example, the processor may replace a portion (721) of the edited image (711) with the partial image of operation (730) to generate the restored image (741). For example, the processor may generate the restored image (741) by combining, synthesizing, or stitching the edited image (711) and the partial image. For example, the restored image (741) may include the partial image combined at the location of the portion (721) within the edited image (711). For example, within the restored image (741), the edited image (711) may be positioned in a remaining portion that is different from the portion where the partial image is positioned. The embodiment is not limited thereto, and when an image of the full size of the original image is generated using pixel information (722), the processor may determine the generated image as a restored image (741).

[0118] The processor that generated the restoration image (741) of the operation (740) can display the restoration image (741) on a display (e.g., the display (130) of FIG. 1). The processor that was displaying the edited image (711) on the display can change or replace the edited image (711) with the restoration image (741) based on the operation (740).

[0119] As described above, according to one embodiment, the electronic device can directly obtain at least a portion of the original image required to restore the original image by using the pixel information (722) stored in the second metadata (122-2) of the file (124). In one embodiment of FIG. 7, the processor can generate or display the restored image (741) without executing an image restoration model (e.g., the image restoration model (230) of FIG. 2A and / or FIG. 2B).

[0120] In one embodiment, the electronic device can detect corruption of pixel information (722) stored in the file (124). For example, if the electronic device detects pixel information (722) having a size smaller than the size of the portion (721) indicated by the first metadata (122-1) from the file (124), the electronic device can detect corruption of the pixel information (722). The electronic device that detects corruption of the pixel information (722) can execute an image generation model (e.g., an image restoration model (230) of FIGS. 2A and / or 2B) to change at least one pixel of the edited image (711) that does not correspond to the pixel information (722) (e.g., in-painting) or combine at least one pixel of the edited image (711) that does not correspond to the pixel information (722) (e.g., out-painting) to generate or obtain a restored image (741).

[0121] FIG. 8 illustrates an exemplary operation of an electronic device for restoring an original image by using pixel differences indicated by metadata of a file (124) (e.g., second metadata (122-2)). The electronic device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIGS. 2A and / or 2B may perform the operation described with reference to FIG. 8. At least one of the operations of FIG. 8 may be related to, or may be performed similarly to, the operations of FIG. 5. The order of the operations of FIG. 8 is exemplary, and in one embodiment, the electronic device may perform the operations of FIG. 8 in a different order than the order illustrated in FIG. 8. In one embodiment, the electronic device may perform at least two of the operations of FIG. 8 substantially simultaneously.

[0122] Referring to FIG. 8, in operation (810), according to one embodiment, a processor of an electronic device may receive an input for restoring an original image. The processor may receive the input of operation (810) while displaying a viewer screen including an edited image (711) (e.g., state (191) of FIG. 1). For example, the input of operation (810) may include the input of operation (520) of FIG. 5 and / or operation (710) of FIG. 7.

[0123] Referring to FIG. 8, in operation (820), according to an embodiment, a processor of an electronic device may obtain, from metadata of a file (124) including an edited image (711), information (822) indicating a difference between an original image and an edited image (711) of at least a portion (e.g., a portion (721)) of the edited image (711). If the file (124) including the edited image (711) is generated based on the operation described with reference to FIG. 3, FIG. 4A and / or FIG. 4B, the file (124) may include information (822) indicating a pixel difference (e.g., a pixel-wise color difference and / or a pixel-wise value difference) between the edited image (711) and the original image. The information (822) may be stored in the second metadata (122-2) as information for restoring the original image. The processor can identify a portion (721) on the edited image (711) related to the information (822) from the first metadata (122-1) including history information related to the creation of the edited image (711).

[0124] Referring to FIG. 8, in a case where an edited image (711) is generated by editing a portion related to a face in an original image, a file (124) may include first metadata (122-1) indicating a portion (721) of the edited image (711) that has been changed from the original image. In this case, the file (124) may include information (822) including difference values ​​between pixels of the original image corresponding to the portion (721) and pixels of the edited image (711). Since the information (822) included in the second metadata (122-2) stores only the difference values, the entire original image may not be stored to restore the original image, and the capacity of the file (124) may be reduced. Based on the operation (820), the processor that obtains information (822) indicating the pixel difference between the original image and the edited image (711) from metadata (e.g., second metadata (122-2)) can perform the operation (830).

[0125] Referring to FIG. 8, in operation (830), according to an embodiment, a processor of an electronic device may generate a partial image corresponding to at least a portion of an edited image (711) using information (822) obtained based on operation (820). For example, the processor may change colors (or values) of pixels corresponding to a portion (721) in the edited image (711) using information (822). The colors (or values) of the pixels changed using information (822) may correspond to the colors (or values) of the pixels of the original image indicated by information (822). For example, the processor may change the color of at least one pixel of the edited image (711) using a pixel difference indicated by information (822), thereby generating a partial image.

[0126] Referring to FIG. 8, in operation (840), a processor of an electronic device according to an embodiment may generate a restored image (841) by combining a partial image generated based on operation (830) and an edited image (711). For example, the processor may generate the restored image (841) by replacing a portion (721) of the edited image (711) with the partial image of operation (830). For example, the processor may generate the restored image (841) by combining, synthesizing, or stitching the edited image (711) and the partial image. The processor may display the restored image (841) on a display. The restored image (841) may include the edited image (711) and the partial image of operation (830) superimposed on the edited image (711).

[0127] As described above, by using the pixel difference indicated by the information (822), the processor can change the color of at least one pixel of the edited image (711) to generate a restored image (841). In one embodiment of FIG. 8, the processor can generate or display the restored image (841) without executing an image restoration model (e.g., the image restoration model (230) of FIG. 2A and / or FIG. 2B).

[0128] Below, exemplary operations of an electronic device executing an image restoration model using prompts included in metadata (e.g., second metadata (122-2)) are described.

[0129] FIG. 9 illustrates exemplary operations of an electronic device for restoring an original image using pixel information included in metadata of a file (124) and / or one or more prompts (922). The electronic device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIGS. 2A and / or 2B may perform the operations described with reference to FIG. 9. At least one of the operations of FIG. 9 may be related to, or may be performed similarly to, the operations of FIG. 5. The order of the operations of FIG. 9 is exemplary, and in one embodiment, the electronic device may perform the operations of FIG. 9 in a different order than the order illustrated in FIG. 9. In one embodiment, the electronic device may perform at least two of the operations of FIG. 9 substantially simultaneously.

[0130] Referring to FIG. 9, in operation (910), according to one embodiment, a processor of an electronic device may receive an input for restoring an original image. The processor may receive the input of operation (910) through a viewer screen including an edited image (911) displayed on a display (e.g., display (130) of FIG. 1). For example, the input of operation (910) may include the input of operation (520) of FIG. 5, the input of operation (710) of FIG. 7, and the input of operation (810) of FIG. 8.

[0131] Referring to FIG. 9, in operation (920), according to one embodiment, a processor of an electronic device may obtain, from metadata of a file (124) including an edited image (911), a prompt (e.g., one or more prompts (922)) for a first portion (943) of a restored image (941) and pixel information (921) for a second portion (942). In one embodiment, the file (124) including the edited image (911) may include first metadata (122-1) indicating a portion of the edited image (911) that has been changed from the original image.

[0132] In one embodiment, a portion of an edited image (911) that has been changed from an original image may be divided into a main region and a sub-region. In an exemplary case where an edited image (911) is generated by changing a portion related to a person in the original image, a portion related to the person's face (e.g., a portion corresponding to the second portion (942) of the restored image (941)) may be divided into a main region, and another portion related to the person's clothes (or a portion excluding the face) (e.g., a portion corresponding to the first portion (943) of the restored image (941)) may be divided into a sub-region.

[0133] For example, since a user viewing a restored image (941) generated by an input of an action (910) is likely to focus on a restored face, a part related to a human face may be divided into a main region and a different part from the main region may be divided into a sub region. A file (124) including an edited image (911) may include second metadata (122-2) including pixel information (921) corresponding to the main region in the original image. The file (124) may include third metadata (122-3) including one or more prompts (922) describing content (e.g., a person's clothes, a person's body shape, etc.) expressed in the sub region in the original image. The one or more prompts (922) may include one or more words (e.g., keywords) describing the content of the sub region and / or natural language sentences in which the one or more words are combined. Since all pixels corresponding to sub-regions different from the main region, among at least a portion of the edited image (911) changed from the original image, are replaced with one or more prompts (922) within the file (124), the file (124) can represent the sub-regions in a relatively small size.

[0134] A processor that obtains pixel information (921) of a first portion (e.g., main region) of an original image and a prompt for a second portion (e.g., sub region) from metadata of a file (124) (e.g., second metadata (122-2) and / or third metadata (122-3)) can at least partially change an edited image (911) using the pixel information (921) and the prompt.

[0135] Referring to FIG. 9, in operation (930), according to an embodiment, a processor of an electronic device may execute an image restoration model (e.g., the image restoration model (230) of FIGS. 2A and / or 2B) based on a prompt (e.g., one or more prompts (922)) obtained based on operation (930) to generate a first partial image for the first portion (943). The one or more prompts (922) obtained by the processor from the third metadata (122-3) may include natural language sentences expressing the content of the original image corresponding to the sub-region of the edited image (911) (e.g., “A person is wearing a black backpack. A person is wearing a winter jumper with a brown fur hat attached, black training pants. A person’s right arm is bent toward the body, and a person’s right hand is positioned on the chest area. A person is holding a black mobile phone with a right hand. A person’s left hand is positioned in a pants pocket. A person is wearing a fur hat.”).

[0136] In one embodiment, by inputting one or more prompts (922) into an image restoration model, the processor can generate a first partial image of operation (930). In one embodiment, to process one or more prompts (922), the image restoration model can have a structure as described with reference to FIGS. 6D to 6F. The first partial image obtained using operation (930) can represent the content of a sub-region of the original image. The embodiment is not limited thereto, and the processor can execute the image restoration model using one or more prompts (922) and the edited image (911) to obtain or generate an edited image (911) in which the first partial image is combined.

[0137] Referring to FIG. 9, in operation (940), a processor of an electronic device according to an embodiment may combine a first partial image obtained based on operation (930) and a second partial image based on pixel information (921) to generate a restored image (941) in an edited image (911). The second partial image may include content of a main area of ​​the original image, indicated by the pixel information (921). The processor may generate a restored image (941) by synthesizing a first partial image corresponding to the first portion (943), which is generated by executing an image restoration model using one or more prompts (922), and a second partial image corresponding to the second portion (942), which is indicated by the pixel information (921).

[0138] For example, the first partial image and the second partial image may be combined or overlapped on the edited image (911). For example, the first partial image generated based on the operation (930) may be positioned on a sub-region indicated by the metadata of the file (124) (e.g., the first metadata (122-1) and / or the third metadata (122-3)) in the restored image (941). For example, the second partial image indicated by the pixel information (921) of the operation (920) may be positioned on a main region indicated by the metadata of the file (124) (e.g., the first metadata (122-1) and / or the second metadata (122-2)) in the restored image (941).

[0139] As described above, according to one embodiment, the electronic device may obtain or generate regions (e.g., main regions and / or sub regions) of the original image that have been modified to generate the edited image (911) using pixel information (921) and / or one or more prompts (922) stored in the second metadata (122-2) of the file (124).

[0140] FIG. 10 illustrates an exemplary operation of an electronic device for restoring an original image using feature information (1022) included in metadata of a file (124). The electronic device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIGS. 2A and / or 2B may perform the operations described with reference to FIG. 10. At least one of the operations of FIG. 10 may be related to, or may be performed similarly to, the operations of FIG. 5. The order of the operations of FIG. 10 is exemplary, and in one embodiment, the electronic device may perform the operations of FIG. 10 in a different order than the order illustrated in FIG. 10. In one embodiment, the electronic device may perform at least two of the operations of FIG. 10 substantially simultaneously.

[0141] Referring to FIG. 10, within operation (1010), a processor of an electronic device according to an embodiment may receive an input for restoring an original image. Within a state (e.g., state (191) of FIG. 1) in which a file (124) including an edited image (120) corresponding to the original image of operation (1010) is visualized, the processor may receive an input of operation (1010). For example, the input of operation (1010) may include an input of operation (520) of FIG. 5, an input of operation (710) of FIG. 7, an input of operation (810) of FIG. 8, and an input of operation (910) of FIG. 9.

[0142] Referring to FIG. 10, in operation (1020), according to one embodiment, a processor of an electronic device may obtain feature information (1022) for at least a portion of an original image (e.g., a portion of the original image corresponding to portion (1021)) from metadata of a file (124) including an edited image (120). The file (124) including the edited image (120) may include first metadata (122-1) including a change history from the original image to the edited image (120) and second metadata (122-2) storing information to be used for restoring the original image.

[0143] In one embodiment, from the first metadata (122-1), the processor can detect a portion (1021) that has been changed from the original image to generate an edited image (120). From the second metadata (122-2), the processor can obtain feature information (1022) for a portion of the original image corresponding to the portion (1021). For example, the feature information (1022) may include a latent vector (e.g., a latent vector (624) of FIG. 6B) output from the encoding model (621) of FIG. 6B into which the original image (or the portion of the original image) is input. Since the file (124) includes a latent vector generated based on dimensionality reduction, the file (124) may include only a relatively small size of feature information (1022) and support restoration of the original image using the feature information (1022).

[0144] The feature information (1022) may be compressed information representing a portion of the original image corresponding to a portion (1021) of the edited image (120). The feature information (1022) may be intrinsic information that can be input into an image restoration model (e.g., a decoding model (622) of FIG. 6B ). In one embodiment, the processor may obtain, from the metadata of the file (124), data representing a feature vector (e.g., feature information (1022)) associated with the original image and at least a portion (e.g., portion (1021)) to be replaced by the feature vector within the edited image (120).

[0145] Referring to FIG. 10, in operation (1030), according to an embodiment, a processor of an electronic device may execute an image restoration model (e.g., the image restoration model (230) of FIGS. 2A and / or 2B) using feature information (1022) acquired based on operation (1020) to generate a partial image (1031) corresponding to at least a portion of an original image. The image restoration model of operation (1030) may include a decoding model (622) of an auto-encoder of FIG. 6B. As described above with reference to FIG. 6B, the decoding model (622) may generate or output, from the feature information (1022) (e.g., a latent vector), an image (e.g., at least a portion of the original image) that was input to the encoding model (621) for generating the feature information (1022).

[0146] The pair of the encoding model (621) and the decoding model (622) can be trained to minimize the difference between the input image of the encoding model (621) and the output image of the decoding model (622). For example, a partial image (1031) of an operation (1030) generated using an image restoration model including the decoding model (622) can match or be similar to at least a portion of an original image that was input to the encoding model (621) to generate feature information (1022).

[0147] Referring to FIG. 10, in operation (1040), according to one embodiment, a processor of an electronic device may generate a restored image (150) using an edited image (120) and / or a partial image (1031). For example, the processor may generate the restored image (150) by combining, synthesizing, or stitching the partial image (1031) with a portion (1021) indicated by the first metadata (122-1) on the edited image (120). The processor that generated the restored image (140) may display the restored image (140) on a display in response to an input of operation (1010).

[0148] As described above, by executing an image restoration model using the feature vector (e.g., latent vector) indicated by the feature information (1022), the processor can generate or obtain at least a portion (e.g., a partial image (1031)) of the restored image (150). The processor can display the generated at least a portion (e.g., a partial image (1031)) on the portion (1021) indicated by the first metadata (122-1) within the edited image (120) displayed on the viewer screen. The processor can generate the partial image (1031) using a file (124) including the feature information (1022) used for executing the image restoration model based on the auto-encoder.

[0149] FIG. 11 illustrates exemplary operations of an electronic device for restoring an original image using one or more prompts included in metadata of a file (124). The electronic device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIGS. 2A and / or 2B may perform the operations described with reference to FIG. 11. At least one of the operations of FIG. 11 may be related to, or performed similarly to, the operations of FIG. 5. The order of the operations of FIG. 11 is exemplary, and in one embodiment, the electronic device may perform the operations of FIG. 11 in a different order than the order illustrated in FIG. 11. In one embodiment, the electronic device may perform at least two of the operations of FIG. 11 substantially simultaneously.

[0150] Referring to FIG. 11, in operation (1110), a processor of an electronic device according to an embodiment may receive an input for restoring an original image. The processor may receive an input of operation (1110) while displaying an edited image (1111) included in a file (124). The input of operation (1110) may include an input of operation (520) of FIG. 5, an input of operation (710) of FIG. 7, an input of operation (810) of FIG. 8, an input of operation (910) of FIG. 9, and / or an input of operation (1010) of FIG. 10.

[0151] Referring to FIG. 11, in operation (1120), according to one embodiment, a processor of an electronic device may obtain a prompt (e.g., a second prompt (1122)) for converting an edited image into an original image from metadata of a file (124) including an edited image (1111). In one embodiment where an image editing model (e.g., the image editing model (420) of FIG. 4A) is executed to convert the original image, the edited image (1111) converted from the original image and the first prompt (1121) used to execute the image editing model may be stored in the file (124). The first prompt (1121) may be stored in the first metadata (122-1) of the file (124).

[0152] Referring to FIG. 11, an example of an edited image (1111) modified from an original image including a night view of a city is illustrated based on the execution of an image editing model. For example, an electronic device receiving a first prompt (1121) for adding content to the original image (e.g., "Add a night view of the city on top of the image") may execute the image editing model to generate or obtain an edited image (1111) that further includes a portion (1112). In response to an input for saving the edited image (1111), the electronic device may generate a file (124) including the edited image (1111) and the first prompt (1121).

[0153] In one embodiment, the electronic device may store information for restoring the original image using the edited image (1111) in the second metadata (122-2) of the file (124). For example, the electronic device may execute an artificial intelligence model for natural language processing to generate or obtain a second prompt (1122) from the first prompt (1121), which will be used for restoring the original image. For example, the electronic device may store a second prompt (1122) indicating a different editing action opposite to the editing action indicated by the first prompt (1121), in the second metadata (122-2) of the file (124).

[0154] In one embodiment of FIG. 11, a first prompt (1121) for adding content to an original image may be stored in first metadata (122-1), and a second prompt (1122) for removing the content from an edited image (1111) (e.g., “Remove the part above the light bulb”) may be stored in second metadata (122-2). In one embodiment, an artificial intelligence model for natural language processing that is executed to obtain the second prompt (1122) may be trained to generate the second prompt (1122) having a meaning and / or intent opposite to the meaning and / or intent of the first prompt (1121).

[0155] As described above, the file (124) may include a first portion (e.g., first metadata (122-1)) in which a first prompt (1121) is stored and a second portion (e.g., second metadata (122-2)) in which a second prompt (1122) is stored, the second prompt (1122) including a word having an opposite meaning to a word included in the first prompt (1121). In response to input of the operation (1110), the processor may retrieve one or more prompts (e.g., second prompt (1122)) from the file (124) and / or the metadata (e.g., second metadata (122-2)) to be used to transform the edited image (1111) to obtain the original image. The one or more prompts may be obtained from another prompt (e.g., first prompt (1121)) that was used to transform the original image into the edited image (1111).

[0156] Referring to FIG. 11, in operation (1130), according to one embodiment, a processor of an electronic device may execute an image restoration model (e.g., the image restoration model (230) of FIGS. 2A and / or 2B) using a prompt (e.g., a second prompt (1122)) obtained based on operation (1120) to generate a restoration image (1131). For example, the processor may execute an image restoration model having a structure for processing the second prompt (1122) (e.g., the structure described with reference to FIGS. 6D to 6F). The processor may generate the restoration image (1131) from the image restoration model executed using the second prompt (1122) and the edited image (1111).

[0157] In one embodiment of FIG. 11, which obtains the exemplary second prompt (1122), the processor may perform an editing action indicated by the second prompt (1122) to obtain a restored image (1131) in which a portion (1112) of the edited image (1111) is removed. The processor, which obtains the restored image (1131) of the operation (1130), may display the restored image (1131) on a display. The restored image (1131) may be displayed in response to the input of the operation (1110).

[0158] Although exemplary operations of an electronic device based on a second prompt (1122) indicating removal of a portion (1112) of an edited image (1111) have been described, embodiments are not limited thereto. For example, in one embodiment in which an edited image is generated using a third prompt indicating addition of a particular subject, a fourth prompt indicating a different editing action (e.g., removal of the particular subject) than the third prompt may be stored within a file including the edited image. The processor may execute an image restoration model using the fourth prompt and the edited image to alter or remove a portion associated with the particular subject within the edited image.

[0159] FIG. 12 illustrates exemplary operations of an electronic device for restoring an original image corresponding to an edited image (1211) of a file (124) using one or more prompts (e.g., prompt (1222)) associated with the original image. The electronic device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIGS. 2A and / or 2B may perform the operations described with reference to FIG. 12. At least one of the operations of FIG. 12 may be related to, or may be performed similarly to, the operations of FIG. 5. The order of the operations of FIG. 12 is exemplary, and in one embodiment, the electronic device may perform the operations of FIG. 12 in a different order than the order illustrated in FIG. 12. In one embodiment, the electronic device may perform at least two of the operations of FIG. 12 substantially simultaneously.

[0160] Referring to FIG. 12, in operation (1210), according to one embodiment, a processor of an electronic device may receive an input for restoring an original image. The processor may receive an input of operation (1210) through a viewer screen including an edited image (1211). The input of operation (1210) may include an input of operation (520) of FIG. 5, an input of operation (710) of FIG. 7, an input of operation (810) of FIG. 8, an input of operation (910) of FIG. 9, an input of operation (1010) of FIG. 10, and / or an input of operation (1110) of FIG. 11. In response to the input, the processor may perform other operations of FIG. 12.

[0161] Referring to FIG. 12, in operation (1220), according to one embodiment, a processor of an electronic device may obtain a prompt (1222) related to an original image from metadata of a file (124) including an edited image (1211). The file (124) including the edited image (1211) may include first metadata (122-1) including history information for generating the edited image (1211) from the original image, second metadata (122-2) including a prompt (1222) for restoring the original image, and third metadata (122-3) indicating a boundary (1223) of a portion of the original image that is different from the edited image (1211). The third metadata (122-2) may include an image (e.g., an edge image) representing the boundary (1223).

[0162] In one embodiment, the file (124) may include a result of recognizing one or more subjects associated with an original image. The result may include data representing portions of the original image associated with the one or more subjects (e.g., data for forming bounding boxes). The bounding boxes may include information representing the location and / or size of portions associated with the subjects within the original image. The result may include an identifier (e.g., an ID and / or key value) uniquely assigned to the one or more subjects and the data matching the identifier.

[0163] Referring to FIG. 12, the prompt (1222) stored in the second metadata (122-2) may include a natural language sentence (e.g., “The photo includes a palm tree planted on a sandy beach”) that describes at least a portion of the original image before it was changed to the edited image (1211). The prompt (1222) may be generated by executing an artificial intelligence model trained to output one or more natural language sentences representing characteristics of the input image from the input image. For example, the prompt (1222) may include one or more words that represent the type, shape, and / or location of one or more subjects within the original image that were associated with the original image before it was changed to the edited image (1211).

[0164] Referring to FIG. 12, data (e.g., edge image) representing a boundary (1223) of content (e.g., tree) of an original image that is different from an edited image (1211) may be included in the third metadata (122-3). The third metadata (122-3) may be stored in a file (124) to guide the shape of an artificial intelligence-generated object to be generated by an artificial intelligence model executed using the prompt (1222) of the second metadata (122-2). Information related to the boundary (1223) stored in the third metadata (122-3) may be referred to as layout information for an object to be generated by the artificial intelligence model.

[0165] Referring to FIG. 12, in operation (1230), according to one embodiment, a processor of an electronic device may execute an image restoration model (e.g., the image restoration model (230) of FIG. 2A and / or FIG. 2B) using a prompt (1222) obtained based on operation (1220) to generate a partial image corresponding to at least a portion of an original image. The processor may execute the image restoration model using a prompt (1222) included in second metadata (122-2) and a boundary (1223) indicated by third metadata (122-3) to generate the partial image of operation (1230). For example, the partial image may have a shape and / or size of the boundary (1223). For example, the partial image may include content described in the prompt (1222) (e.g., an image of a beach including palm trees).

[0166] Referring to FIG. 12, in operation (1240), according to one embodiment, a processor of an electronic device may generate a restored image (1241) using an edited image (1211) and a partial image of operation (1230). Referring to FIG. 12, the restored image (1241) may be generated by overlapping a partial image of operation (1230) on the edited image (1211). Referring to FIG. 12, a partial image (1221) included in the restored image (1241) may include pixels representing content (e.g., a sandy beach, an ocean, and a sky) to be overlaid on content (e.g., chairs) of the edited image (1211) that is not included in the original image, as well as content (e.g., a palm tree) of the original image that is not included in the edited image (1211), which has a shape of a boundary line (1223). The processor can generate a restored image (1241) by replacing at least a portion of the edited image (1211) with a partial image of the operation (1230). The restored image (1241) can be a combination of the edited image (1211) and a partial image generated based on the prompt (1222). The processor can display the restored image (1241) generated based on the operation (1240) on a display.

[0167] As described above, a prompt (e.g., prompt (1222)) for at least a portion of the content of the original image may be stored within the file (124). The prompt may be used to restore the original image using the edited image (1211) of the file (124). The information stored within the file (124) for restoring the original image is not limited to the prompt.

[0168] Hereinafter, with reference to FIG. 13, an exemplary operation of an electronic device for restoring an original image using information collected (or crawled) from a network including the Internet is described.

[0169] FIG. 13 illustrates exemplary operations of an electronic device for restoring an original image using one or more prompts and location information included in metadata of a file (124). The electronic device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIGS. 2A and / or 2B may perform the operations described with reference to FIG. 13. At least one of the operations of FIG. 13 may be related to, or may be performed similarly to, the operations of FIG. 5. The order of the operations of FIG. 13 is exemplary, and in one embodiment, the electronic device may perform the operations of FIG. 13 in a different order than the order illustrated in FIG. 13. In one embodiment, the electronic device may perform at least two of the operations of FIG. 13 substantially simultaneously.

[0170] Referring to FIG. 13, in operation (1310), according to an embodiment, a processor of an electronic device may receive an input for restoring an original image. Within a viewer screen including an edited image (1311), the processor may display a visual object (e.g., visual objects (141, 142) of FIG. 1) for receiving an input of operation (1310). The input of operation (1310) may include an input indicating selection of the visual object. The input of operation (1310) may include an input of operation (520) of FIG. 5, an input of operation (710) of FIG. 7, an input of operation (810) of FIG. 8, an input of operation (910) of FIG. 9, an input of operation (1010) of FIG. 10, an input of operation (1110) of FIG. 11, and / or an input of operation (1210) of FIG. 12. In response to the above input, the processor may perform other operations of FIG. 13.

[0171] Referring to FIG. 13, in operation (1320), according to one embodiment, a processor of an electronic device may obtain, from metadata of a file (124) including an edited image (1311), a prompt (1322) related to an original image and location information related to the original image. For example, the file (124) may include first metadata (122-1) including history information related to the creation of the edited image (1211). For example, the file (124) may include second metadata (122-2) including a prompt (1322) for restoration of the original image. For example, the file (124) may include third metadata (122-3) indicating a boundary of a portion (1321) of the original image that is different from the edited image (1311). For example, the file (124) may include fourth metadata (122-4) including time and / or location information at which the original image was captured. The above location information may include GPS coordinates.

[0172] Referring to FIG. 13, an edited image (1311) including a night scene and a person is exemplarily illustrated. The file (124) may include a prompt (1322) describing the content of the original image before it was changed to the edited image (1311) (e.g., "The background of the photo includes a public telephone booth, a tree, and a steel fence."). The file (124) may include data representing a portion (1321) of the edited image (1311) that has been changed from the original image (e.g., a border of the portion (1321)).

[0173] Referring to FIG. 13, in operation (1330), according to an embodiment, a processor of an electronic device may execute an image restoration model (e.g., the image restoration model (230) of FIG. 2A and / or FIG. 2B) using the prompt (1322) and location information obtained based on operation (1320) to generate a partial image corresponding to at least a portion of the original image. For example, the processor may obtain information about a real space related to the original image using the location information indicated by the fourth metadata (122-4). The information about the real space may include an image uploaded to the Internet (e.g., a road view image and / or an image uploaded to a social network service (SNS)). The location information is not limited to GPS coordinates, and may include an address (e.g., a uniform resource locator (URL) and / or a uniform resource indicator (URI)) of information about the real space within a network. In one embodiment, the processor may obtain or download information about the real-world space by communicating with a server storing useful images and / or videos for restoring the original image. The information about the real-world space obtained from the Internet may include a prompt used to restore the original image using the image of the real-world space.

[0174] In one embodiment, the processor may acquire information about a real-world space by performing an operation to collect or obtain information from a network, including the Internet, such as crawling. Using the acquired information, the prompt (1322), information representing the portion (1321), and / or the edited image (1311), the processor may execute an image restoration model of operation (1330). For example, the image restoration model may be trained to generate a partial image that includes content removed from the original image (e.g., an appearance of the real-world space) to generate the edited image (1311), using images and / or videos of the real-world space.

[0175] Referring to FIG. 13, in operation (1340), a processor of an electronic device according to an embodiment may generate a restored image (1341) using an edited image (1311) and a partial image of operation (1340). The processor may generate the restored image (1341) by combining, synthesizing, or stitching the partial image generated based on operation (1330) onto a portion (1321) of the edited image (1311). For example, the restored image (1341) may include a partial image representing content for an actual space obtained from a network and at least a portion of the edited image (1311) (e.g., a remaining portion of the edited image (1311) that is different from the portion (1321). The processor may display the restored image (1341) generated based on operation (1340) on a display.

[0176] As described above, the file (124) may further include not only the prompt (1322) as information used to restore the original image, but also other types of information different from the prompt (1322). Hereinafter, with reference to FIG. 14A and / or FIG. 14B, exemplary operations of an electronic device receiving input for restoring the original image are described.

[0177] Figures 14a and 14b illustrate exemplary states (1401, 1402, 1403, 1404, 1405, 1406) of an electronic device (101) for restoring an original image. The electronic device (101) of Figures 1, 2a, and 2b and / or the processor (210) of Figures 2a and / or 2b can perform the operations described with reference to Figures 14a to 14b. At least one of the operations of Figures 14a and / or 14b can be related to the operations of Figures 5, 7 to 13.

[0178] Referring to FIGS. 14A and 14B , exemplary states (1401, 1402, 1403, 1404, 1405, 1406) of an electronic device (101) displaying a viewer screen including an edited image (1411) on a display (130) are illustrated. According to one embodiment, the electronic device (101) may display, on the display (130), a viewer screen including a visual object for indicating a difference between the edited image (1411) corresponding to the original image and the original image. Referring to FIG. 14A, within a state (1401), the electronic device (101) may display a viewer screen on the display (130) that includes an edited image (1411) and an indicator (e.g., indicators (1412, 1413)) corresponding to a visual object area of ​​the edited image (1411) (e.g., a portion of the edited image (1411) that has been changed from the original image).

[0179] In one embodiment, from a file containing an edited image (1411) (e.g., file (124) of FIG. 1), the electronic device (101) can obtain information related to an original image before it was changed to the edited image (1411) and / or other edited images that are intermediate versions between the original image and the edited image (1411). For example, while a user edits an original image, multiple versions of edited images may be generated. The file may include a latest version of the edited image (1411) among the multiple versions. The file may include information for rolling back to the edited images of the multiple versions and / or the original image (e.g., the first metadata (122-1) to the fourth metadata (122-4) described with reference to FIGS. 7 to 13). For example, the file may include information for restoring the original image and / or information for restoring the edited images of the multiple versions.

[0180] Referring to FIG. 14A, the electronic device (101) may display an indicator (1413) corresponding to the original image and an indicator (1412) related to another version different from the latest version among the plurality of versions on the display (130) in a state (1401). The indicators (1412, 1413) may be identified using metadata of a file including the edited image (1411). For example, the electronic device (101) may identify the positions of the indicators (1412, 1413) on the edited image (1411) and / or the shapes of the indicators (1412, 1413) (e.g., text and / or images to be embedded in each of the indicators (1412, 1413)) from the metadata.

[0181] Referring to FIG. 14A, an embodiment is illustrated in which a first version edited image with added parts expressing palm trees and a second version edited image (1411) with added parts expressing parasols and chairs are sequentially generated from an original image expressing an empty beach and a person, and a file including the second version edited image (1411) is stored. In a state (1401) of displaying a viewer screen including the second version edited image (1411), the electronic device (101) may display an indicator (1413) indicating restoration to the original image and an indicator (1412) indicating restoration to the first version edited image. The indicator (1413) may include an image expressing an empty beach. The indicator (1412) may include an image expressing a subject (e.g., a palm tree) that was expressed by the first version edited image. The indicator (1412) may have a position corresponding to the part added to the edited image of the first version.

[0182] In one embodiment, data indicating the shape and / or position of indicators (1412, 1413) may be stored in the first metadata (122-1) described with reference to FIGS. 7 to 13. For example, the first metadata (122-1) may include information about prompts used to generate edited images of each of the plurality of versions, partial images (e.g., low-resolution partial images and / or edge images) for portions of each of the edited images that are different from the original image, feature information (e.g., latent vectors (624) of FIG. 6b), and / or an image editing model used to generate the edited images.

[0183] In one embodiment, based on an input indicating a selection of one of the indicators (1412, 1413), the electronic device (101) may restore or display an edited image of the original image or one of the multiple versions (e.g., an edited image of a different version than the edited image (1411)). For example, based on an input indicating a selection of an indicator (1412) corresponding to a first version of the edited image, the electronic device (101) may restore the first version of the edited image.

[0184] In one embodiment, the electronic device (101) that generated the restored image (1431) corresponding to the first version of the edited image may switch from state (1401) to state (1403). Within state (1403), the electronic device (101) may display the restored image (1431) on the display (130). For example, together with the restored image (1431), the electronic device (101) may display an indicator (1413) for restoration to the original image. Within state (1403) that displays the restored image (1431) corresponding to a different version from the latest version of the edited image (1411), the electronic device (101) may display, on the display (130), a visual object (144) for storing the restored image (1431) and a visual object (143) for stopping display of the restored image (1431). In response to an input related to a visual object (144), the electronic device (101) may generate or store a file containing a restored image (1431). The file may include the restored image (1431) and information for restoring the original image.

[0185] In one embodiment, the electronic device (101) that receives an input related to the indicator (1413) may generate a partial image representing at least a portion of the original image using an artificial intelligence model, based on the input, such that at least a portion of the edited image (1411) (e.g., a visual object area representing a parasol and a chair) is replaced with at least a portion of the original image having a position corresponding to a position within the at least a portion of the edited image (1411). By changing at least a portion of the edited image (1411) using the generated partial image, the electronic device (101) may generate a restored image (1421) corresponding to the original image. An operation of the electronic device (101) generating one of the restored images (1421, 1431) may be performed similarly to the operation of the electronic device described with reference to FIGS. 5 to 13.

[0186] Referring to FIG. 14A, an electronic device (101) that has generated a restoration image (1421) may switch from a state (1401) to a state (1402). Within the state (1402), the electronic device (101) may display the restoration image (1421) on the display (130). Within the state (1402) in which the restoration image (1421) is displayed, the electronic device (101) may display a visual object (144) for creating and / or storing a file including the restoration image (1421) displayed on the display (130). Within the state (1402), the electronic device (101) may display a visual object (143) for stopping an operation related to the restoration image (1421) displayed on the display (130). For example, in response to an input related to a visual object (144), the electronic device (101) may generate or store a file containing a restored image (1421). For example, the electronic device (101) that receives an input related to a visual object (143) may switch to a state (1401) prior to a state (1402).

[0187] Although exemplary operation of an electronic device (101) for displaying indicators (1412, 1413) representing portions of a latest version of an edited image (1411) that have been changed from an original image (or an edited image of a previous version of a latest version) as overlays on an edited image (1411) has been described, the embodiment is not limited thereto.

[0188] Referring to FIG. 14B, an exemplary state (1404) of an electronic device (101) displaying a latest version of an edited image (1411) is illustrated, together with a slider (1441) having a straight shape. The electronic device (101) may display, on the display (130), the slider (1441) and an indicator (1442) aligned with one of the markers (or points) of the slider (1441). The slider (1441) may include a marker corresponding to the latest version of the edited image (1411) (e.g., a marker with the number 2 written on it), a marker corresponding to a version prior to the latest version (e.g., a marker with the number 1 written on it), and a marker corresponding to the original image (e.g., a marker with the number 0 written on it). Within the state (1404) of displaying the latest version of the edited image (1411), the indicator (1442) may be aligned with the marker corresponding to the latest version on the slider (1441).

[0189] For example, as the number of versions of the edited image (1411) increases, the number of markers listed on the slider (1441) may increase. For example, the electronic device (101) may display markers corresponding to each of the versions of the original image and the edited image (1411) on the slider (1441). An embodiment of visualizing the versions of the edited image (1411) using numbers is illustrated, but the embodiment is not limited thereto.

[0190] Within the state (1404) of FIG. 14B, an input may be received to move an indicator (1442) from one marker of the slider (1441) to another marker. For example, within the state (1404) of FIG. 14B, an electronic device (101) that receives an input to move an indicator (1442) toward a marker having the number 1 written on it may switch to the state (1405) and restore an edited image of a version corresponding to the marker. Within the state (1405), the electronic device (101) may display a restored image (1431) corresponding to the first version of the edited image on the display (130). Within the state (1405), the indicator (1442) may be aligned with the marker having the number 1 written on it. Within the state (1405), the electronic device (101) can display a visual object (144) for storing a restored image (1431) displayed on the display (130) and a visual object (143) for stopping the creation and / or editing of the restored image (1431).

[0191] For example, within the state (1404) of FIG. 14B, the electronic device (101) that receives an input to move the indicator (1442) toward a marker having a number 0 written on it may switch to the state (1406) and display a restored image (1421) corresponding to the original image. Within the state (1406), the indicator (1442) may be aligned with the marker having a number 0 written on it. Within the state (1406), the electronic device (101) may display a visual object (144) for storing the restored image (1421) displayed on the display (130) and a visual object (143) for stopping the creation and / or editing of the restored image (1421).

[0192] Although the operation of the electronic device (101) switching from a state (1404) to another state (e.g., states (1405, 1406)) has been described, the embodiment is not limited thereto. For example, within the state (1405), in response to an input to move the indicator (1442), the electronic device (101) may switch from the state (1405) to another state (e.g., states (1404, 1406)). For example, within the state (1406), in response to an input to move the indicator (1442), the electronic device (101) may switch from the state (1406) to another state (e.g., states (1404, 1405)).

[0193] Although exemplary operations of the electronic device (101) for restoring an intermediate version of an edited image and / or an original image based on a slider (1411) and / or an indicator (e.g., indicators (1412, 1413)) have been described, the embodiments are not limited thereto. For example, the electronic device (101) may display, together with the edited image (1411), a thumbnail image (e.g., an image having a smaller size and / or resolution than the edited image (1411)) representing another edited image and / or an original image of a previous version of the edited image (1411). The thumbnail image may be generated or obtained using an artificial intelligence model (e.g., an image restoration model executed on-device) executed based on metadata of a file including the edited image (e.g., the first metadata (122-1) to the fourth metadata (122-4) described with reference to FIGS. 5 to 13).

[0194] In one embodiment of displaying one or more thumbnail images together with an edited image (1411) of a latest version, in response to an input related to a thumbnail image, the electronic device (101) may execute a function for restoring an edited image and / or an original image corresponding to the thumbnail image. The function may be executed using an image restoration model executed in a server connected to the electronic device (101) (e.g., an external electronic device (250) of FIG. 2B ). Although an embodiment has been described in which the image restoration models of the electronic device (101) and the server are selectively executed, the present disclosure is not limited thereto.

[0195] As described above, according to one embodiment, the electronic device (101) can generate or output a restored image (1421) corresponding to the original image by using an edited image (1411) corresponding to the original image and metadata related to the edited image (1411) after the original image has been deleted. Hereinafter, with reference to FIG. 15, one or more programs installed in the electronic device (101) to execute a generative artificial intelligence model, such as an image restoration model, are exemplarily described.

[0196] FIG. 15 illustrates exemplary programs executed by an electronic device to simulate a generative artificial intelligence model (1530). Referring to FIG. 15, a generative artificial intelligence system is illustrated. According to one embodiment, a processor (e.g., processor (210) of FIGS. 2A and / or 2B) of an electronic device (e.g., electronic device (101) of FIG. 1) may execute one or more programs defined by blocks of FIG. 15 to execute functions related to the generative artificial intelligence model (1530).

[0197] According to one embodiment, a user question response interface (1510) executed by an electronic device may be executed by the electronic device to receive a user input. The input may include a non-verbal gesture (e.g., a touch input on a display of the electronic device), natural language sentences such as prompts, images, videos, or a combination thereof. The electronic device executing the user question response interface (1510) may obtain context information at the time of receiving the input. The context information may include, for example, the status of a program (or software application) executed by the electronic device as various information at the time. The context information may include, for example, location information of the electronic device and / or the user.

[0198] An electronic device executing a user query response interface (1510) may output information generated by a generative artificial intelligence model (1530) in response to the input. The information may be output in the form of natural language sentences. The information may be output in the form of a UI displayed on a display. The information generated by the generative artificial intelligence model (1530) may be output in a format selected by the user.

[0199] The AI ​​framework (1520) may be a program for executing or controlling a component (e.g., a prompt design component (1521), an application programming interface (API) / plugin management component (1522), and / or a refinery component (1523)) based on input received through a user question response interface (1510). The input detected by executing the user question response interface (1510) may be processed by the prompt design component (1521). An electronic device executing the prompt design component (1521) may generate one or more prompts to be input into a generative artificial intelligence model (1530), such as a large language model (LLM), from the input. The prompt design component (1521) may include an artificial intelligence model that is trainable to generate improved prompts.

[0200] An electronic device that executes the prompt design component (1521) can access knowledge repositories (1540) to obtain information for generating prompts. The information for generating prompts may include data on preferences of a user of the electronic device, a prompt library, and / or exemplary prompts. One or more prompts generated based on the execution of the prompt design component (1521) may be used for executing a generative artificial intelligence model (1530).

[0201] An electronic device executing an API / plug-in management component (1522) may, when executing a generative artificial intelligence model (1530) based on a user's input, execute a function to obtain additional information required for executing the generative artificial intelligence model (1530). The API provided by the API / plug-in management component (1522) may be used to establish a communication link (e.g., a channel and / or session) between the generative artificial intelligence model (1530) and another software application. Through the communication link, information additionally required for executing the generative artificial intelligence model (1530) may be provided to the generative artificial intelligence model (1530).

[0202] An electronic device executing an API / plug-in management component (1522) can trigger an action to be performed by an application / service component (1550) (e.g., a program and / or software application installed on the electronic device) using an API provided by the API / plug-in management component (1522). Information acquired by executing the application / service component (1550) can be used to generate a prompt based on the execution of the prompt design component (1521) and / or can be input into a generative artificial intelligence model (1530).

[0203] The refinement component (1523) (or output modification component) can change, tamper with, or tune the output data of the generative artificial intelligence model (1530). The electronic device executing the refinement component (1523) can detect relevance, bias, appropriateness, or hallucination of the output data of the generative artificial intelligence model (1530). The electronic device executing the refinement component (1523) can determine whether to re-execute the generative artificial intelligence model (1530) based on the detected relevance, bias, appropriateness, and / or hallucination. The electronic device executing the refinement component (1523) can display hints (e.g., images and / or text) to the user to reduce or prevent unintended output data.

[0204] A generative artificial intelligence model (1530) may refer to an artificial intelligence model that generates new information and / or data based on a user's input. The generative artificial intelligence model (1530) may include an artificial intelligence model that generates images and / or natural language. The artificial intelligence model that generates images may include a diffusion model based on a generative adversarial network (GAN), a variational auto encoder (VAE), and / or a transformer. The artificial intelligence model that generates natural language may include CHAT-GPT 3 and / or CHAT-GPT 4, which are models trained to output statistically appropriate natural language. The embodiment is not limited thereto, and the generative artificial intelligence model (1530) may include a large multimodal model (LMM) that receives various types of input data including text, images, and / or audio signals, and then generates output data related to the input data.

[0205] FIG. 16 is a block diagram of an electronic device (1601) within a network environment (1600) according to various embodiments. Referring to FIG. 16, in the network environment (1600), the electronic device (1601) may communicate with the electronic device (1602) via a first network (1698) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (1604) or the server (1608) via a second network (1699) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (1601) may communicate with the electronic device (1604) via the server (1608). According to one embodiment, the electronic device (1601) may include a processor (1620), a memory (1630), an input module (1650), an audio output module (1655), a display module (1660), an audio module (1670), a sensor module (1676), an interface (1677), a connection terminal (1678), a haptic module (1679), a camera module (1680), a power management module (1688), a battery (1689), a communication module (1690), a subscriber identification module (1696), or an antenna module (1697). In some embodiments, the electronic device (1601) may omit at least one of these components (e.g., the connection terminal (1678)), or may have one or more other components added. In some embodiments, some of these components (e.g., sensor module (1676), camera module (1680), or antenna module (1697)) may be integrated into a single component (e.g., display module (1660)).

[0206] The processor (1620) may control at least one other component (e.g., a hardware or software component) of the electronic device (1601) connected to the processor (1620) by executing, for example, software (e.g., a program (1640)), and may perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (1620) may store commands or data received from other components (e.g., a sensor module (1676) or a communication module (1690)) in a volatile memory (1632), process the commands or data stored in the volatile memory (1632), and store result data in a non-volatile memory (1634). According to one embodiment, the processor (1620) may include a main processor (1621) (e.g., a central processing unit or an application processor) or an auxiliary processor (1623) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (1621). For example, when the electronic device (1601) includes the main processor (1621) and the auxiliary processor (1623), the auxiliary processor (1623) may be configured to use less power than the main processor (1621) or to be specialized for a given function. The auxiliary processor (1623) may be implemented separately from the main processor (1621) or as a part thereof.

[0207] The auxiliary processor (1623) may control at least a portion of functions or states associated with at least one component (e.g., the display module (1660), the sensor module (1676), or the communication module (1690)) of the electronic device (1601), for example, on behalf of the main processor (1621) while the main processor (1621) is in an inactive (e.g., sleep) state, or together with the main processor (1621) while the main processor (1621) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (1623) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (1680) or a communication module (1690)). In one embodiment, the auxiliary processor (1623) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (1601) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (1608)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0208] The memory (1630) can store various data used by at least one component (e.g., the processor (1620) or the sensor module (1676)) of the electronic device (1601). The data can include, for example, software (e.g., the program (1640)) and input data or output data for commands related thereto. The memory (1630) can include volatile memory (1632) or non-volatile memory (1634).

[0209] The program (1640) may be stored as software in memory (1630) and may include, for example, an operating system (1642), middleware (1644), or an application (1646).

[0210] The input module (1650) can receive commands or data to be used in a component of the electronic device (1601) (e.g., a processor (1620)) from an external source (e.g., a user) of the electronic device (1601). The input module (1650) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0211] The audio output module (1655) can output audio signals to the outside of the electronic device (1601). The audio output module (1655) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0212] The display module (1660) can visually provide information to an external party (e.g., a user) of the electronic device (1601). The display module (1660) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (1660) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0213] The audio module (1670) can convert sound into an electrical signal, or vice versa. According to one embodiment, the audio module (1670) can acquire sound through the input module (1650), output sound through the sound output module (1655), or an external electronic device (e.g., electronic device (1602)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (1601).

[0214] The sensor module (1676) can detect the operating status (e.g., power or temperature) of the electronic device (1601) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (1676) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0215] The interface (1677) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (1601) with an external electronic device (e.g., the electronic device (1602)). In one embodiment, the interface (1677) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0216] The connection terminal (1678) may include a connector through which the electronic device (1601) may be physically connected to an external electronic device (e.g., the electronic device (1602)). In one embodiment, the connection terminal (1678) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0217] The haptic module (1679) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (1679) may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0218] The camera module (1680) can capture still images and videos. In one embodiment, the camera module (1680) may include one or more lenses, image sensors, image signal processors, or flashes.

[0219] The power management module (1688) can manage the power supplied to the electronic device (1601). According to one embodiment, the power management module (1688) can be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

[0220] A battery (1689) may power at least one component of the electronic device (1601). In one embodiment, the battery (1689) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0221] The communication module (1690) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (1601) and an external electronic device (e.g., electronic device (1602), electronic device (1604), or server (1608)), and the performance of communication through the established communication channel. The communication module (1690) may operate independently from the processor (1620) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (1690) may include a wireless communication module (1692) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (1694) (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules may communicate with an external electronic device (1604) via a first network (1698) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (1699) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a local area network or a wide area network)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1692) may use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (1696) to identify or authenticate the electronic device (1601) within a communication network such as the first network (1698) or the second network (1699).

[0222] The wireless communication module (1692) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency communications (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (1692) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (1692) may support various technologies for securing performance in high-frequency bands, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (1692) may support various requirements specified in the electronic device (1601), an external electronic device (e.g., the electronic device (1604)), or a network system (e.g., the second network (1699)). According to one embodiment, the wireless communication module (1692) may support a peak data rate (e.g., 20 Gbps or more) for eMBB implementation, a loss coverage (e.g., 164 dB or less) for mMTC implementation, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC implementation.

[0223] The antenna module (1697) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (1697) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (1697) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (1698) or the second network (1699), may be selected from the plurality of antennas by, for example, the communication module (1690). A signal or power may be transmitted or received between the communication module (1690) and the external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (1697).

[0224] According to various embodiments, the antenna module (1697) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0225] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0226] According to one embodiment, commands or data may be transmitted or received between the electronic device (1601) and an external electronic device (1604) via a server (1608) connected to a second network (1699). Each of the external electronic devices (1602 or 704) may be the same or a different type of device as the electronic device (1601). According to one embodiment, all or part of the operations executed in the electronic device (1601) may be executed in one or more of the external electronic devices (1602, 704, or 708). For example, when the electronic device (1601) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (1601) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (1601). The electronic device (1601) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (1601) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (1604) may include an Internet of Things (IoT) device. The server (1608) may be an intelligent server utilizing machine learning and / or a neural network.According to one embodiment, an external electronic device (1604) or server (1608) may be included within the second network (1699). The electronic device (1601) may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology and IoT-related technology.

[0227] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0228] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0229] The term "module" used in various embodiments of this document may include a unit implemented in hardware, and may be used interchangeably with terms such as logic, block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0230] Various embodiments of the present document may be implemented as software (e.g., a program (1640)) including one or more instructions stored in a storage medium (e.g., an internal memory (1636) or an external memory (1638)) readable by a machine (e.g., an electronic device (1601)). For example, a processor (e.g., a processor (1620)) of the machine (e.g., an electronic device (1601)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0231] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0232] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added. The electronic device (1601) of FIG. 16 may be an example of the electronic device (101) of FIG. 1, FIG. 2a, and / or FIG. 2b.

[0233] In one embodiment, a method may be required to restore an original image by using a file including an edited image among an edited image and an original image corresponding to the edited image. In one embodiment, a method may be required to restore an original image from an edited image of a file by using an artificial intelligence model. As described above, according to an embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (1601) of FIG. 16) may include a display (e.g., the display (130) of FIG. 1), at least one processor including a processing circuit (e.g., the processor (210) of FIG. 2A and / or FIG. 2B), and a memory including one or more storage media for storing instructions (e.g., the memory (215) of FIG. 2A and / or FIG. 2B). The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display, through the display, an edited image including a modified object, and a visual object for the modified object. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive user input for the visual object while displaying the edited image. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain information from metadata corresponding to the edited image to replace the modified object with an AI-generated object based on the user input.The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to execute the artificial intelligence model using the edited image and the information to generate an artificial intelligence-generated image. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display the artificial intelligence-generated image through the display in response to the user input.

[0234] For example, the edited image may be generated through an artificial intelligence model such that an object included in the original image is replaced with the modified object, and the information may include one or more prompts describing the object or the original image.

[0235] For example, the information may include one or more other prompts that were used to convert the original image into the edited image.

[0236] For example, the information may include data representing a feature vector associated with the original image and at least a portion of the edited image to be replaced by the feature vector.

[0237] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to execute the artificial intelligence model using the feature vector to generate the artificial intelligence-generated image, at least as a part of generating the artificial intelligence image. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display an indicator representing the generated portion on at least a portion indicated by the metadata within the edited image.

[0238] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain, from the metadata, pixel information of a first portion of the original image and a prompt for a second portion different from the first portion. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to execute the artificial intelligence model to synthesize another object generated using pixel information corresponding to the first portion and a portion generated using the prompt corresponding to the second portion, thereby generating the artificial intelligence-generated image.

[0239] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain information corresponding to a pixel difference between the original image and the edited image from the metadata. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to execute the artificial intelligence model to change a color of at least one pixel of the edited image using the information, thereby generating the artificial intelligence-generated image.

[0240] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain, from the metadata, a partial image of a portion of the original image corresponding to the modified object of the edited image. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to execute the artificial intelligence model to generate the artificial intelligence-generated image using the partial image and the edited image, such that the modified image is replaced with the partial image.

[0241] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display, through the display, the visual object representing the modified object as changed from the original image on at least a portion of the edited image.

[0242] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display a plurality of indicators, including an indicator corresponding to the original image and other indicators representing each of the plurality of versions, based on displaying a latest version of the edited image from among the plurality of versions on the display. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to execute an artificial intelligence model using the edited image and other information corresponding to the selected indicator to generate another artificial intelligence image, based on another user input for selecting one of the plurality of indicators. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display the another artificial intelligence image through the display.

[0243] As described above, in one embodiment, a method of an electronic device including a display may be provided. The method may include an operation of displaying, through the display, an edited image including a modified object and a visual object for the modified object. The method may include an operation of receiving a user input for the visual object while displaying the edited image. The method may include an operation of obtaining, based on the user input, metadata corresponding to the edited image to replace the modified object with an AI-generated object. The method may include an operation of executing the AI ​​model using the edited image and the information to generate an AI-generated image. The method may include an operation of displaying, through the display, the AI-generated image in response to the user input.

[0244] For example, the edited image may be generated through an artificial intelligence model such that an object included in the original image is replaced with the modified object, and the information may include one or more prompts describing the object or the original image.

[0245] For example, the information may include one or more other prompts that were used to convert the original image into the edited image.

[0246] For example, the information may include data representing a feature vector associated with the original image and at least a portion of the edited image to be replaced by the feature vector.

[0247] For example, the generating operation may include an operation of executing the artificial intelligence model using the feature vector to generate the artificial intelligence-generated image. The displaying operation of the restored image may include an operation of displaying an indicator indicating the generated portion on at least a portion indicated by the metadata within the edited image.

[0248] For example, the operation of obtaining the information may include an operation of obtaining, from the metadata, pixel information of a first portion of the original image and a prompt for a second portion different from the first portion. The operation of generating may include an operation of executing the artificial intelligence model to synthesize another object generated using pixel information corresponding to the first portion and a portion generated using a prompt corresponding to the second portion, thereby generating the artificial intelligence-generated image.

[0249] For example, the operation of obtaining the information may include an operation of obtaining information corresponding to the pixel difference between the original image and the edited image from the metadata. The operation of generating the information may include an operation of executing the artificial intelligence model to change the color of at least one pixel of the edited image using the information, thereby generating the artificial intelligence-generated image.

[0250] For example, the generating operation may include an operation of obtaining a partial image of a portion of the original image corresponding to the modified object of the edited image from the metadata. The generating operation may include an operation of executing the artificial intelligence model to generate the artificial intelligence-generated image using the partial image and the edited image, such that the modified image is replaced with the partial image.

[0251] For example, the act of displaying the restored image may include the act of displaying, through the display, the visual object representing the modified object that has been changed from the original image on at least a portion of the edited image.

[0252] In one embodiment, as described above, a non-transitory computer-readable storage medium comprising instructions may be provided. The instructions, when executed by an electronic device including a display, may cause the electronic device to display, through the display, an edited image including a modified object and a visual object for the modified object. The instructions, when executed by the electronic device, may cause the electronic device to receive user input for the visual object while displaying the edited image. The instructions, when executed by the electronic device, may cause the electronic device to obtain information from metadata corresponding to the edited image to replace the modified object with an AI-generated object based on the user input. The instructions, when executed by the electronic device, may cause the electronic device to execute the AI ​​model using the edited image and the information to generate an AI-generated image. The above instructions, when executed by the electronic device, may cause the electronic device to display the artificial intelligence-generated image through the display in response to the user input.

[0253] In one embodiment, an electronic device may include a display, at least one processor including a processing circuit, and a memory including one or more storage media storing instructions. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display, on the display, a viewer screen including an edited image (e.g., an edited image (120) of FIG. 1) corresponding to an original image (e.g., an original image (110) of FIG. 1) and a visual object (e.g., visual objects (141, 142) of FIG. 1) for indicating a difference between the original image and the edited image. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive an input related to the visual object while displaying the viewer screen. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain, based on the input, information used to restore the original image using an artificial intelligence model from metadata (e.g., metadata (122) of FIG. 1) of a file (e.g., file (124) of FIG. 1) containing the edited image. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to execute the artificial intelligence model using the edited image and the information to generate a restored image (e.g., restored image (150) of FIG. 1). The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display, in response to the input, the restored image on the display.According to one embodiment, an electronic device can restore an original image corresponding to an edited image using a file containing the edited image. According to one embodiment, the electronic device can restore an original image from an edited image of the file using an artificial intelligence model.

[0254] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain, from the metadata, information including one or more prompts describing the original image (e.g., prompt (1222) of FIG. 12).

[0255] For example, the one or more prompts may include sentences describing at least a portion of the original image before it was changed to the edited image.

[0256] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to retrieve from the metadata one or more prompts (e.g., prompt (1122) of FIG. 11) that are to be used to convert the edited image to obtain the original image. The one or more prompts stored in the metadata may be obtained from other prompts (e.g., prompt (1121) of FIG. 11) that were used to convert the original image to the edited image.

[0257] For example, the metadata may include a first portion in which the other prompts are stored and a second portion in which one or more prompts are stored, the second portion including words having opposite meanings to words included in the other prompts.

[0258] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain, from the metadata, information including a feature vector associated with the original image and data representing at least a portion of the edited image to be replaced by the feature vector.

[0259] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to execute the artificial intelligence model using the feature vector to generate a portion of the restored image. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display the generated portion on the at least a portion indicated by the metadata within the edited image displayed on the viewer screen.

[0260] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain, from the metadata, pixel information of a first portion of the original image (e.g., information (921) of FIG. 9) and a prompt for a second portion different from the first portion (e.g., one or more prompts (922) of FIG. 9). The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to synthesize the second portion generated by executing the artificial intelligence model using the prompt and the first portion represented by the pixel information to generate the restored image.

[0261] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain information (e.g., information (822) of FIG. 8) representing a pixel difference between the original image and the edited image from the metadata. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to change a color of at least one pixel of the edited image using the pixel difference to generate the restored image.

[0262] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain, from the metadata, a partial image (e.g., partial image (1031) of FIG. 10) of a second portion of the original image corresponding to the first portion of the edited image. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to replace the first portion (e.g., portion (1021) of FIG. 10) of the edited image with the partial image to generate the restored image.

[0263] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display the visual object representing the portion of the edited image that has been changed from the original image as an overlay over the portion on the viewer screen.

[0264] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display a plurality of indicators (e.g., indicators 1412 and 1413 of FIG. 14A) including an indicator corresponding to the original image and other indicators representing each of the plurality of versions, based on displaying a latest version of the edited image from among the plurality of versions on the display. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display the restored image corresponding to the original image or an edited image from among any one of the plurality of versions, based on the input indicating selection of an indicator from among the plurality of indicators.

[0265] As described above, in one embodiment, a method of an electronic device including a display may be provided. The method may include an operation of displaying a viewer screen on the display, the viewer screen including an edited image corresponding to an original image and a visual object for indicating a difference between the original image and the edited image (e.g., operation 510 of FIG. 5 ). The method may include an operation of receiving an input related to the visual object while displaying the viewer screen (e.g., operation 520 of FIG. 5 ). The method may include an operation of obtaining information used to restore the original image using an artificial intelligence model from metadata of a file including the edited image based on the input. The method may include an operation of executing the artificial intelligence model using the edited image and the information to generate a restored image (e.g., operation 540 of FIG. 5 ). The method may include an operation of displaying the restored image on the display in response to the input (e.g., operation 550 of FIG. 5 ).

[0266] For example, the obtaining operation may include obtaining, from the metadata, information including one or more prompts describing the original image.

[0267] For example, the one or more prompts may include sentences describing at least a portion of the edited image before it was changed from the original image.

[0268] For example, the obtaining operation may include retrieving one or more prompts from the metadata to be used to convert the edited image to obtain the original image. The one or more prompts stored in the metadata may be obtained from other prompts that were used to convert the original image to the edited image.

[0269] For example, the metadata may include a first portion in which the other prompts are stored and a second portion in which one or more prompts are stored, the second portion including words having opposite meanings to words included in the other prompts.

[0270] For example, the obtaining operation may include obtaining, from the metadata, information including a feature vector associated with the original image and data representing at least a portion to be replaced by the feature vector within the edited image.

[0271] For example, the generating operation may include an operation of executing the artificial intelligence model using the feature vector to generate a portion of the restored image. The displaying operation of the restored image may include an operation of displaying the generated portion on at least a portion of the edited image displayed on the viewer screen, indicated by the metadata.

[0272] For example, the operation of obtaining the information may include an operation of obtaining, from the metadata, pixel information of a first portion of the original image and a prompt for a second portion different from the first portion. The operation of generating may include an operation of synthesizing the second portion generated by executing the artificial intelligence model using the prompt and the first portion represented by the pixel information to generate the restored image.

[0273] For example, the operation of obtaining the information may include an operation of obtaining information indicating a pixel difference between the original image and the edited image from the metadata. The operation of generating may include an operation of changing the color of at least one pixel of the edited image using the pixel difference to generate the restored image.

[0274] For example, the generating operation may include an operation of obtaining a partial image of a second portion of the original image corresponding to the first portion of the edited image from the metadata. The generating operation may include an operation of generating the restored image by replacing the first portion of the edited image with the partial image.

[0275] For example, the action of displaying the restored image may include an action of displaying the visual object representing the portion of the edited image that has been changed from the original image as an overlay on the portion on the viewer screen.

[0276] For example, the operation of displaying the viewer screen may include an operation of displaying a plurality of indicators including an indicator corresponding to the original image and other indicators representing each of the plurality of versions, based on displaying the latest version of the edited image among the plurality of versions on the display. The operation of displaying the viewer screen may include an operation of displaying the restored image corresponding to the original image or an edited image of any one of the plurality of versions, based on an input indicating selection of one indicator among the plurality of indicators.

[0277] In one embodiment, as described above, a non-transitory computer-readable storage medium comprising instructions may be provided. The instructions, when executed by an electronic device including a display, may cause the electronic device to display, on the display, a viewer screen including an edited image corresponding to an original image and a visual object for indicating a difference between the original image and the edited image. The instructions, when executed by the electronic device, may cause the electronic device to receive an input related to the visual object while displaying the viewer screen. The instructions, when executed by the electronic device, may cause the electronic device to obtain, based on the input, information used to restore the original image using an artificial intelligence model from metadata of a file including the edited image. The instructions, when executed by the electronic device, may cause the electronic device to execute the artificial intelligence model using the edited image and the information to generate a restored image. The above instructions, when executed by the electronic device, may cause the electronic device to display the restored image on the display in response to the input.

[0278] In one embodiment, as described above, a non-transitory computer-readable storage medium comprising instructions may be provided. The instructions, when executed by an electronic device including a display, may cause the electronic device to display, on the display, a viewer screen including an edited image corresponding to an original image and an indicator corresponding to a first visual object area of ​​the edited image. The instructions, when executed by the electronic device, may cause the electronic device to receive an input related to the indicator while displaying the viewer screen. The instructions, when executed by the electronic device, may cause the electronic device to generate, based on the input, a second visual object area using an artificial intelligence model, such that the first visual object area within the edited image is replaced with a second visual object area of ​​the original image having a location corresponding to a location of the first visual object area within the edited image.

[0279] For example, the instructions, when executed by the electronic device, may cause the electronic device to obtain information used to restore the original image using the artificial intelligence model from metadata of a file including the edited image. The instructions, when executed by the electronic device, may cause the electronic device to execute the artificial intelligence model using at least one of the edited image or information to generate the second visual object area.

[0280] For example, the instructions, when executed by the electronic device, may cause the electronic device to obtain, from the metadata, information including one or more prompts describing the original image.

[0281] For example, the one or more prompts may include sentences describing at least a portion of the original image before it was changed to the edited image.

[0282] For example, the instructions, when executed by the electronic device, may cause the electronic device to retrieve from the metadata one or more prompts to be used to convert the edited image to obtain the original image. The one or more prompts stored in the metadata may be obtained from other prompts that were used to convert the original image to the edited image.

[0283] For example, the metadata may include a first portion in which the other prompts are stored and a second portion in which one or more prompts are stored, the second portion including words having opposite meanings to words included in the other prompts.

[0284] As described above, according to one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1 and / or electronic device (1601) of FIG. 16) may include a display (e.g., display (130) of FIG. 1), at least one processor including processing circuitry (e.g., processor (210) of FIG. 2A and / or FIG. 2B), and a memory (e.g., memory (215) of FIG. 2A and / or FIG. 2B) including one or more storage media for storing instructions. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display, on the display, a viewer screen including an edited image (e.g., an edited image (120) of FIG. 1) corresponding to an original image (e.g., an original image (110) of FIG. 1) and an indicator (e.g., an indicator (1412) of FIG. 14A) corresponding to a first visual object area of ​​the edited image. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive an input related to the indicator while displaying the viewer screen. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to generate the second visual object region using an artificial intelligence model, based on the input, such that the first visual object region within the edited image is replaced with a second visual object region of the original image having a location corresponding to a location of the first visual object region within the edited image.

[0285] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain information used to restore the original image using the artificial intelligence model from metadata (e.g., metadata (122) of FIG. 1) of a file (e.g., file (124) of FIG. 1) containing the edited image. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to execute the artificial intelligence model using at least one of the edited image or information to generate the second visual object area.

[0286] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain, from the metadata, the information including one or more prompts describing the original image.

[0287] For example, the one or more prompts may include sentences describing at least a portion of the original image before it was changed to the edited image.

[0288] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to retrieve from the metadata one or more prompts to be used to convert the edited image to obtain the original image. The one or more prompts stored in the metadata may be obtained from other prompts that were used to convert the original image to the edited image.

[0289] For example, the metadata may include a first portion in which the other prompts are stored and a second portion in which one or more prompts are stored, the second portion including words having opposite meanings to words included in the other prompts.

[0290] As used herein, the term "if" will be understood to mean "when, upon," "in response to determining," or "in response to detecting," depending on the context. Similarly, "if it is determined to," or "if [the stated condition or event] is detected," will optionally be understood to mean "upon determining," or "in response to determining," "upon detecting [the stated condition or event]," or "in response to detecting [the stated condition or event]."

[0291] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0292] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0293] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording media or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0294] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0295] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In electronic devices, display; At least one processor comprising a processing circuit; and A memory comprising one or more storage media for storing instructions, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Through the above display: An edited image containing the modified object; and Visual object for the above modified object and display; While displaying the above edited image, receiving user input for the visual object; and Based on the above user input: Obtain information from metadata corresponding to the above edited image to replace the modified object with an artificial intelligence-generated object; Executing the artificial intelligence model using the above edited image and the above information to generate an artificial intelligence-generated image; and In response to the user input, causing the artificial intelligence generated image to be displayed through the display. Electronic devices.

2. In claim 1, the edited image is generated through an artificial intelligence model so that an object included in the original image is replaced with the modified object, and the information includes one or more prompts describing the object or the original image. Electronic devices.

3. In claim 2, the information is: including one or more other prompts that were used to convert the original image into the edited image; Electronic devices.

4. In claim 2, the information is: feature vectors associated with the original image; and Data representing at least a portion of the above edited image to be replaced by the feature vector; including, Electronic devices.

5. In claim 4, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: As at least a part of generating the artificial intelligence image, executing the artificial intelligence model using the feature vector to generate the artificial intelligence generated image; Causing an indicator representing the generated portion to be displayed on at least a portion of the edited image represented by the metadata, Electronic devices.

6. In claim 2, when the instructions are individually or collectively executed by the at least one processor, the electronic device, From the above metadata, pixel information of a first part of the original image and a prompt for a second part different from the first part are obtained; By executing the artificial intelligence model, the artificial intelligence-generated image is generated by synthesizing another object generated using pixel information corresponding to the first part and a part generated using a prompt corresponding to the second part. Electronic devices.

7. In claim 2, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Obtain information corresponding to the pixel difference between the original image and the edited image from the metadata; By executing the artificial intelligence model, the color of at least one pixel of the edited image is changed using the information, thereby generating the artificial intelligence-generated image. Electronic devices.

8. In claim 1, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Obtaining a partial image of a part of the original image corresponding to the modified object of the edited image from the metadata; By executing the artificial intelligence model, the modified image is replaced with the partial image, thereby generating the artificial intelligence-generated image using the partial image and the edited image. Electronic devices.

9. In claim 1, when the instructions are individually or collectively executed by the at least one processor, the electronic device, Causing, through said display, to display said visual object representing said modified object changed from said original image on at least a portion of said edited image; Electronic devices.

10. In claim 2, when the instructions are individually or collectively executed by the at least one processor, the electronic device, Based on displaying the latest version of the above edited image among multiple versions on the above display: an indicator corresponding to the original image; and Other indicators representing each of the above multiple versions Displays multiple indicators including; Based on another user input for selecting one of the plurality of indicators, executing an artificial intelligence model using the edited image and other information corresponding to the selected indicator to generate another artificial intelligence image; and Causing the display to display the other artificial intelligence image through the display, Electronic devices.

11. In a method of an electronic device including a display, Through the above display; An edited image containing the modified object; and Visual object for the above modified object Actions that indicate; An action of receiving user input for the visual object while displaying the above edited image; Based on the above user input: An operation for obtaining metadata corresponding to the above edited image to replace the modified object with an artificial intelligence-generated object; An operation of generating an artificial intelligence-generated image by executing the artificial intelligence model using the above-mentioned edited image and the above-mentioned information; and In response to the user input, the action of displaying the artificial intelligence generated image through the display, method.

12. In claim 11, the edited image is generated through an artificial intelligence model so that an object included in the original image is replaced with the modified object, and the information includes one or more prompts describing the object or the original image. method.

13. In claim 12, the information includes one or more other prompts that were used to convert the original image into the edited image. method.

14. In claim 12, the information is: feature vectors associated with the original image; and Data representing at least a portion of the above edited image to be replaced by the feature vector; including, method.

15. In claim 14, the generating operation comprises: An operation of generating the artificial intelligence generated image by executing the artificial intelligence model using the feature vector is included. The action of displaying the above restored image is: Including an action of displaying an indicator representing the generated portion on at least a portion of the edited image indicated by the metadata, method.

Images