Method for generating partial region of image by using generative model, and electronic device for performing same

Abstract

In an embodiment of the present disclosure, provided are a method for generating a partial region of an image by using a generative model, and an electronic device for performing same. The method for generating a partial region of an image by using a generative model may comprise the steps of: acquiring an image including information on a partial region; acquiring an intermediate generated image including first image information on the partial region by inputting the image into a first generative model; and acquiring a final generated image including second image information, which is at least partially different from the first image information, by inputting the image and the intermediate generated image into a second generative model.

Description

Method for generating a portion of an image using a generative model and an electronic device for performing the same

[0001] The present disclosure relates to a method for processing an image, and more particularly, to a method for generating an image using a generative model and an electronic device for performing the same.

[0002] Generative AI technology learns the patterns and structures of massive training data and, based on that data, generates new data similar to the input data. Generative AI can be used to obtain images that correspond to text or to expand images to areas not included in the original image.

[0003] Generative AI technology can be applied to image processing to support outpainting or inpainting. Outpainting expands an image while preserving its style and content, while inpainting generates an image to fill a specific area within the image.

[0004] Meanwhile, recent advances in image processing technology, which prioritize user experience, have led to some devices and programs supporting features that allow users to change the position and size of objects within an image. However, if the transformed object appears out of place in the resulting image, this can reduce user satisfaction.

[0005] In one embodiment of the present disclosure, a method for generating a partial region of an image using a generative model may be provided. The method may include the steps of: obtaining an image including information of the partial region; inputting the image into a first generative model to obtain an intermediate generated image including first image information for the partial region; and inputting the image and the intermediate generated image into a second generative model to obtain a final generated image including second image information, the second image information differing at least in part from the first image information.

[0006] In one embodiment of the present disclosure, an electronic device may be provided. The electronic device may include a memory storing at least one instruction, and at least one processor, and the at least one processor may execute at least one instruction stored in the memory, whereby the electronic device may obtain an image including information of a portion of the region, input the image into a first generation model, thereby obtaining an intermediate generation image including first image information for the portion of the region, and input the image and the intermediate generation image into a second generation model, thereby obtaining a final generation image including second image information, at least a portion of which is different from the first image information.

[0007] In one embodiment of the present disclosure, a computer-readable recording medium having recorded thereon a program for performing a method of generating a portion of an image using a generative model on a computer may be provided.

[0008] The present disclosure can be readily understood by the following detailed description and its accompanying drawings, wherein reference numerals refer to structural elements.

[0009] FIG. 1 is a conceptual diagram illustrating an electronic device for generating a portion of an image using a generative model according to one embodiment of the present disclosure.

[0010] FIG. 2 is a conceptual diagram illustrating an encoder and a decoder according to one embodiment of the present disclosure.

[0011] FIG. 3 is a conceptual diagram illustrating a mask map according to one embodiment of the present disclosure.

[0012] FIG. 4 is a conceptual diagram for explaining the operation of a second generation model according to one embodiment of the present disclosure.

[0013] FIG. 5A is a conceptual diagram illustrating text guidance input to a second generation model according to one embodiment of the present disclosure.

[0014] FIG. 5b is a conceptual diagram illustrating image guidance input to a second generation model according to one embodiment of the present disclosure.

[0015] FIG. 5c is a conceptual diagram illustrating text guidance and image guidance input to a second generation model according to one embodiment of the present disclosure.

[0016] FIG. 5d is a conceptual diagram illustrating an embodiment in which the output of the first generation model according to one embodiment of the present disclosure is used as image guidance.

[0017] FIG. 5e is a conceptual diagram illustrating an embodiment in which the output of the first generation model according to one embodiment of the present disclosure is used as image guidance.

[0018] FIG. 5f is a conceptual diagram illustrating an embodiment in which the output of the first generation model is used as image guidance together with text guidance according to one embodiment of the present disclosure.

[0019] FIG. 5g is a conceptual diagram illustrating an embodiment in which the output of the first generation model is used as image guidance together with text guidance according to one embodiment of the present disclosure.

[0020] FIG. 6 is a conceptual diagram illustrating a learning method of a first generation model according to one embodiment of the present disclosure.

[0021] FIGS. 7A and 7B are conceptual diagrams illustrating the configuration of a second generation model according to one embodiment of the present disclosure.

[0022] Figure 8 is a conceptual diagram for explaining the configuration of an interpreter according to one embodiment of the present disclosure.

[0023] FIGS. 9A and 9B are conceptual diagrams illustrating an embodiment of adding noise to an intermediate generated image according to one embodiment of the present disclosure.

[0024] FIGS. 10A to 10C are conceptual diagrams illustrating an embodiment of determining a denoising intensity based on the quality of an intermediate generated image according to one embodiment of the present disclosure.

[0025] FIG. 11 is a conceptual diagram illustrating an electronic device for generating a portion of an image using a generative model according to one embodiment of the present disclosure.

[0026] FIGS. 12a and 12b are conceptual diagrams illustrating a learning method of a fifth encoder according to one embodiment of the present disclosure.

[0027] FIGS. 13a and 13b are conceptual diagrams illustrating a learning method of a fifth encoder according to one embodiment of the present disclosure.

[0028] FIG. 14 is a flowchart illustrating a method for generating a portion of an image using a generative model according to one embodiment of the present disclosure.

[0029] Figure 15 is a flowchart for explaining detailed steps of step S1410 of Figure 14.

[0030] Figure 16 is a flowchart for explaining detailed steps of step S1430 of Figure 14.

[0031] Figure 17 is a flowchart for explaining detailed steps of step S1430 of Figure 14.

[0032] Figure 18 is a flowchart for explaining detailed steps of step S1430 of Figure 14.

[0033] Figure 19 is a flowchart for explaining detailed steps of step S1430 of Figure 14.

[0034] FIG. 20 is a flowchart illustrating a method for learning an encoder for encoding an image according to one embodiment of the present disclosure.

[0035] FIG. 21 is a flowchart illustrating a method for generating a portion of an image using a generative model according to one embodiment of the present disclosure.

[0036] FIG. 22 is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present disclosure.

[0037] FIG. 23 is a block diagram illustrating the configuration of an electronic device and a server device according to one embodiment of the present disclosure.

[0038] The terms used in the embodiments of this specification have been selected from widely used, current terms, taking into account the functions of the present disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant embodiments. Therefore, the terms used in this specification should not be defined simply as names of terms, but rather based on their meanings and the overall content of the present disclosure.

[0039] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein.

[0040] Throughout this disclosure, when a part is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "unit," "module," etc., used herein refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0041] As used herein, the expression "configured to" can be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" does not necessarily mean something is "specifically designed to" in terms of hardware. Instead, in some contexts, the expression "a system configured to" can mean that the system is "capable of" in conjunction with other devices or components. For example, the phrase "a processor configured to perform A, B, and C" can mean a dedicated processor for performing the operations (e.g., an embedded processor), or a general-purpose processor (e.g., a CPU or an application processor) that can perform the operations by executing one or more software programs stored in memory.

[0042] Additionally, when a component is referred to as being "connected" or "connected" to another component in the present disclosure, it should be understood that the component may be directly connected or connected to the other component, but may also be connected or connected via another component in between, unless otherwise specifically stated.

[0043] In the present disclosure, functions related to 'artificial intelligence' are operated through a processor and memory. The processor may be composed of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, AP, or DSP (Digital Signal Processor), a graphics-only processor such as a GPU or VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU. One or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if one or more processors are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0044] The predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is trained using a learning algorithm using a plurality of learning data, thereby creating a predefined operation rules or artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed on the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0045] In one embodiment of the present disclosure, the 'artificial intelligence model' may include a neural network model. The neural network model may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and performs neural network operations through operations between the operation results of the previous layer and the plurality of weights. The plurality of weights of the plurality of neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, the plurality of weights may be updated so that a loss value or a cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. The neural network model may include a deep neural network (DNN), and examples thereof include, but are not limited to, a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks.

[0046] In the present disclosure, a "generative model" may refer to a model that generates new data based on given input data. For example, the generative model may receive various types of data (e.g., text, images, videos, sounds, random vector data, etc.) as input. For example, the generative model may generate new text, images, videos, sounds, or combinations thereof. In one embodiment of the present disclosure, the generative model may be an artificial intelligence model. For example, the generative model may include, but is not limited to, a generative adversarial network (GAN) model, a variational autoencoder (VAE) model, a diffusion model, a transformer, etc.

[0047] In this disclosure, 'inpainting' may refer to the task of performing inference on an internal region of an original image.

[0048] In this disclosure, 'outpainting' may refer to the task of performing inference on an external area of ​​an original image.

[0049] In the present disclosure, a "mask" may represent an area (or an unknown area) within the entire area of ​​a specific image that requires inference. For example, a "masked image" may include pixel information corresponding to an unmasked area and boundary information for an area corresponding to the mask. The pixel information may include location information (e.g., coordinate values) and color information (e.g., RGB values) of the pixel.

[0050] In the present disclosure, a "mask map" may represent data that distinguishes between masked and unmasked areas within the entire image area. For example, the mask map may be a binarization map. For example, the masked area may be represented by a first value, and the unmasked area may be represented by a second value.

[0051] In this disclosure, "denoising" may be defined as a single operation that is input to and output from a second generative model. In this disclosure, "denoising order" may be defined as the number of times denoising is repeated. In this disclosure, "total denoising order" may be defined as a hyperparameter indicating how many times denoising is repeated. In one embodiment of this disclosure, the total denoising order may vary depending on user or manufacturer settings.

[0052] In the present disclosure, "guidance" may refer to information indicating an image to be inferred. For example, the information may be any information indicating an image to be inferred. For example, guidance may include text guidance generated from text or image guidance generated from an image, but the type of guidance is not limited thereto. The guidance or guidance information may be input into the second generative model during the denoising operation.

[0053] FIG. 1 is a conceptual diagram illustrating an electronic device that generates a portion of an image using a generative model according to one embodiment of the present disclosure. Referring to FIG. 1, the electronic device (1000) can obtain a final generated image from a masked image using a first generative model (1100) and a second generative model (1200). In one embodiment of the present disclosure, each of the first generative model (1100) and the second generative model (1200) can generate an entire image inferring an unknown region based on an image in which an unknown region exists. For example, the first generative model (1100) can generate a first image based on an input image, and the second generative model (1200) can generate a second image based on the input image. Here, the unknown region may be a region in the input image that does not include image information or does not include image information intended by a user. Each of the first generative model (1100) and the second generative model (1200) can perform inpainting or outpainting on the input image. According to one embodiment of the present disclosure, the speed and / or performance of image inference can be dramatically improved by utilizing the output of the first generative model in the inference operation of the second generative model.

[0054] In one embodiment of the present disclosure, the electronic device (1000) may be implemented in various forms. For example, the electronic device (1000) may include a personal computer (PC), a terminal, a portable telephone, a smart phone, a tablet PC, a handheld device, a wearable device, a server device, and the like.

[0055] The electronic device (1000) may acquire an image including information of a portion of the image. Some portions of the image may be predefined according to user or manufacturer settings. In some embodiments, the portions of the image may be referred to as target portions or candidate portions. In some embodiments, the information of the portions of the image may also be referred to as information about the portions of the image, information related to the portions of the image, or information corresponding to the portions of the image. For example, the information of the portions of the image may include location information corresponding to the portions of the image and / or boundary information between the portions of the image and the area excluding the portions of the image. For example, the information of the portions of the image may include location information of a bounding box on the image. For example, the information of the portions of the image may include pixel coordinate values ​​corresponding to the portions of the image. In one embodiment of the present disclosure, the portions of the image may also be referred to as masks.

[0056] In one embodiment of the present disclosure, an electronic device (100) may include a sensor and a display. The electronic device (1000) may acquire an original image of an object using the sensor. The sensor may include, but is not limited to, a charge-coupled device (CCD) sensor and a complementary metal-oxide-semiconductor (CMOS) sensor. In some embodiments, the sensor may be referred to as a camera. The original image may include, but is not limited to, an RGB image. The electronic device (1000) may display the original image through a display. The electronic device (1000) may acquire a user input for a portion of the original image through a user interface. For example, the portion may include at least one of an internal portion and an external portion of the original image. The electronic device (1000) may generate an image including information of the portion of the original image based on the user input. An image including information of the portion of the original image may also be referred to as a masked image. For example, an image including information of a portion of a region may be configured in the form of an original image with a mask added. In one embodiment of the present disclosure, the electronic device (1000) may generate a mask map based on at least one of an image including information of a portion of a region and a user input. In one embodiment of the present disclosure, the electronic device (1000) may receive at least one of the original image, information of a portion of a region, and a mask map from an external server.

[0057] The electronic device (1000) may input an image including information of a certain region into the first generation model (1100). For example, the first generation model (1100) may include a pre-trained GAN model. For example, the first generation model (1100) may include a generator model among the pre-trained GAN models. In one embodiment of the present disclosure, the electronic device (1000) may input an image pair comprising a masked image and a mask map into the first generation model.

[0058] The electronic device (100) can obtain an intermediate generated image including first image information for a portion of the region from the first generation model (1100). The intermediate generated image can include color information of at least one pixel corresponding to the portion of the region. For example, the intermediate generated image can include color information of pixels corresponding to the entire region, along with color information of at least one pixel corresponding to the portion of the region. However, the present disclosure is not limited thereto, and the intermediate generated image can include color information of at least one pixel corresponding to the portion of the region, and not include color information of pixels excluding the portion of the region.

[0059] The electronic device (1000) can input an image containing information of a certain region and an intermediate generated image to the second generation model (1200). The second generation model (1200) may include an artificial intelligence model that restores the image from noise. For example, the second generation model (1200) may include a pre-trained diffusion model.

[0060] In one embodiment of the present disclosure, the first generation model (1100) may be configured with fewer layers and / or fewer weight values ​​than the second generation model (1200). In one embodiment of the present disclosure, the processing speed of the first generation model (1100) may be faster than the processing speed of the second generation model. In one embodiment of the present disclosure, the memory capacity occupied by the first generation model (1100) may be smaller than the memory capacity occupied by the second generation model (1200).

[0061] In one embodiment of the present disclosure, the intermediate generated image may be preprocessed before being input to the second generation model (1200). For example, the intermediate generated image may be converted into an image embedding and / or a text embedding corresponding to the image. In the present disclosure, an "embedding" may refer to low-dimensional data transformed from high-dimensional data. For example, an embedding may also be referred to as an embedding vector, a feature vector, a feature representation, a latent vector, or a latent representation. However, the present disclosure is not limited thereto, and an embedding may include low-dimensional data that has been transformed from high-dimensional data in another way.

[0062] In one embodiment of the present disclosure, the electronic device (1000) may generate a connected image based on an image including information of a portion of the region and noise information. For example, the electronic device (1000) may concatenate an image including information of a portion of the region with a predefined initial noise. The electronic device (1000) may input the connected image into a second generation model (1200). The electronic device (1000) may concatenate the image including information of a portion of the region to the output of the second generation model (1200). The electronic device (1000) may input the connected image into the second generation model (1200). The electronic device (1000) may repeat the operation of inputting the connected image into the second generation model (1200). For example, the electronic device (1000) may repeatedly perform the operation of inputting the connected image into the second generation model (1200) as many times as the total number of predefined denoising orders.

[0063] The electronic device (1000) can determine whether the operation has been repeated a predefined total number of denoising orders. Based on the determination that the operation has been repeated a predefined total number of denoising orders, the electronic device (1000) can obtain a final generated image from the second generation model (1200).

[0064] In one embodiment of the present disclosure, the second generation model (1200) may include at least one layer. The electronic device (1000) may input an intermediate generation image or image information (e.g., image embedding) corresponding to the intermediate generation image into at least one layer of the second generation model (1200).

[0065] In one embodiment of the present disclosure, the electronic device (1000) may acquire a denoising strength for an intermediate generated image. The denoising strength may represent a numerical value indicating how strongly noise is added to the image. For example, assuming that the denoising strength has a value between 0 and 1, if the denoising strength is 0, no noise is added to the image, and if the denoising strength is 1, the image may be changed to completely random noise. For example, if the denoising strength is 0, no noise may be added to the image. On the other hand, if the denoising strength is 1, the image may be changed to include random noise. The electronic device (1000) may determine the amount of noise to be added to the intermediate generated image based on the denoising strength. The amount of noise may represent the degree of noise to be added to the intermediate generated image. In one embodiment of the present disclosure, the denoising strength may be predefined according to a user or manufacturer's settings. For example, the electronic device (1000) can obtain user input corresponding to the denoising intensity through the user interface. The electronic device (1000) can determine the denoising intensity based on the user input. The electronic device (1000) can add noise to the intermediate generated image based on the denoising intensity.

[0066] In one embodiment of the present disclosure, at least some of the functions of obtaining a denoising intensity for an intermediate generated image and adding noise to the intermediate generated image based on the denoising intensity may be performed by the second generation model (1200). For example, the second generation model (1200) may receive the intermediate generated image. The second generation model (1200) may obtain a denoising intensity from the intermediate generated image. The second generation model (1200) may add noise to the intermediate generated image based on the denoising intensity. The intermediate generated image with added noise may be input to at least one layer of the second generation model (1200).

[0067] In one embodiment of the present disclosure, the electronic device (1000) can measure the quality of an intermediately generated image. For example, the electronic device (1000) can obtain a confidence value based on the intermediately generated image. For example, the electronic device (1000) can obtain a predicted confidence value based on the intermediately generated image. For example, the predicted confidence value can indicate the degree to which the intermediately generated image output (or predicted, inferred) by the first generation model (1100) can be trusted. The electronic device (1000) can determine a denoising intensity based on the predicted confidence value. For example, denoising intensities corresponding to a plurality of threshold ranges can be mapped in advance. For example, a first denoising intensity can be mapped to a first threshold range, and a second denoising intensity can be mapped to a second threshold range. The electronic device (1000) can determine a threshold range corresponding to a predicted confidence value among the plurality of threshold ranges. The electronic device (1000) can add noise to the intermediate generated image with a denoising intensity mapped to a determined threshold range.

[0068] In one embodiment of the present disclosure, the electronic device (1000) can determine a target denoising order corresponding to an intermediate generated image with added noise among a predefined total number of denoising orders based on the denoising intensity. The electronic device (1000) can set the current denoising order as the target denoising order.

[0069] In one embodiment of the present disclosure, the electronic device (1000) can post-process the image output by the second generation model (1200) to obtain a final generated image. For example, the electronic device (1000) can obtain a final generated image based on image information corresponding to a portion of the image output by the second generation model (1200) (i.e., an unknown portion or a target portion) and image information corresponding to a portion of the initially input image (which may also be referred to as an image including information of a portion of the portion) excluding a portion of the image (i.e., a known portion).

[0070] FIG. 2 is a conceptual diagram illustrating an encoder and decoder according to one embodiment of the present disclosure. Any details overlapping with those described in FIG. 1 are omitted.

[0071] Referring to FIG. 2, the electronic device (1000) may include a first encoder (1310) and a decoder (1400). The first encoder (1310) may encode an image. The decoder (1400) may output an image by decoding an image embedding. In one embodiment of the present disclosure, the first encoder (1310) and the decoder (1400) may be pre-trained artificial intelligence models.

[0072] In one embodiment of the present disclosure, the first encoder (1310) and the decoder (1400) may be implemented as a variational autoencoder (VAE) model or a vector quantized generative adversarial network (VQGAN). For example, the first encoder (1310) may be the encoder portion of a variational autoencoder or a vector quantized generative adversarial network. For example, the decoder (1400) may be the decoder portion of a variational autoencoder or a vector quantized generative adversarial network. The first encoder (1310) and the decoder (1400) may be trained using a learning method of a variational autoencoder model or a vector quantized generative adversarial network. For example, the output of the first encoder (1310) may be input to the decoder (1400). For example, the first encoder (1310) and the decoder (1400) may be trained together by comparing the input of the first encoder (1310) and the output of the decoder (1400). However, the present disclosure is not limited thereto, and thus, according to other embodiments, the first encoder (1310) and / or the decoder (1400) may be implemented in other ways.

[0073] In one embodiment of the present disclosure, the first encoder (1310) may output a latent vector corresponding to the input image. For example, the latent vector may represent a probability value based on a Gaussian probability distribution expressed by a mean and variance. The first encoder (1310) may transmit the latent vector to the second generative model.

[0074] In one embodiment of the present disclosure, the second generation model (1200) can output a latent vector (hereinafter, also referred to as final noise). The decoder (1400) can output a final generated image by decoding the latent vector, which is the output of the second generation model (1200).

[0075] FIG. 3 is a conceptual diagram illustrating a mask map according to one embodiment of the present disclosure. Any details overlapping with those described in FIGS. 1 and 2 are omitted.

[0076] Referring to FIG. 3, the electronic device (1000) may obtain a mask map including location information of a portion of an image (e.g., an unknown portion or a target portion to be inferred). For example, the electronic device (1000) may obtain a mask map corresponding to an image from an external server. For example, the electronic device (1000) may generate a mask map based on the image. For example, the electronic device (1000) may generate a mask map in which a portion of an image (e.g., an unknown portion or a target portion) is expressed as a first value (e.g., a white portion in FIG. 3) and an area excluding the portion of the image (e.g., a preservation area) is expressed as a second value (e.g., a black portion in FIG. 3). For example, the mask map may be a binary image composed of the first value or the second value. However, the present disclosure is not limited thereto, and thus the mask map may be defined in another format.

[0077] The electronic device (1000) can concatenate an image and a mask map. For example, the image can be expressed as a three-channel image, with each of the R, G, and B values ​​of the RGB values ​​as one channel. For example, the mask map can be expressed as a one-channel image. The image to which the mask map is connected can have a total of four channels. However, the present disclosure is not limited thereto, and the number of channels of the image or the number of channels of the mask map is not limited thereto. In the present disclosure, a channel can represent one dimension of three-dimensional input data.

[0078] The electronic device (1000) can transmit the connected image to the first generation model (1100). The electronic device (1000) can transmit the connected image to the second generation model (1200).

[0079] FIG. 4 is a conceptual diagram illustrating the operation of a second generation model according to one embodiment of the present disclosure. Any details overlapping with those described in FIGS. 1 to 3 are omitted.

[0080] Referring to FIG. 4, the electronic device (1000) can transmit an image to a first encoder (1310). The first encoder (1310) can encode the image. The electronic device (1000) can output an encoded image (e.g., a latent vector) (Z image ) can be obtained.

[0081] The electronic device (1000) can adjust the mask map. For example, the electronic device (1000) can adjust the mask map to a specific size. The specific size may be a predefined size. For example, the predefined size may be an encoded image (Z image ) may be identical to the encoded image (Z image ) with the number of channels (e.g., 3) and the adjusted mask map (M r ) may have different number of channels (e.g., 1).

[0082] The electronic device (1000) currently has noise information (Z t ) can be obtained. Current noise information (Z t ) number of channels and size of the encoded image (Z image ) may be the same as the number and size of channels. However, the present disclosure is not limited thereto, and the current noise information (Z t ) and the number of channels and size of the encoded image (Z image) may have different channel numbers and sizes. Here, t is defined as the current denoising order of the second generation model (1200). In the present disclosure, the current denoising order may represent the number of times the second generation model (1200) repeats input and output. For example, t may be expressed as an integer greater than or equal to 0 and less than or equal to the total denoising order (e.g., T). For example, Z T can represent initial noise information that is first input to the second generation model (1200). The initial noise information may be composed of random values. For example, the initial noise information may be composed of Gaussian noise following a Gaussian distribution, but the present disclosure is not limited thereto.

[0083] The electronic device (1000) encodes an image (Z image ), adjusted mask map (M r ), and current noise information (Z t ) can be connected. The electronic device (1000) can be connected to an encoded image (Z image ), adjusted mask map (M r ), and current noise information (Z t ) can transmit the connected data (hereinafter, input data) to the second generation model (1200). The second generation model (1200) can perform a denoising operation based on the input data. In the present disclosure, the denoising operation may refer to an operation of removing a predetermined noise from the input noise. The second generation model (1200) performs the denoising operation to generate the following noise information (Z t-1 ) can be created.

[0084] The electronic device (1000) can determine whether the denoising operation has been performed for the total number of denoising orders. For example, the electronic device (1000) may determine whether the following noise information (Z t-1) can determine whether it is the final noise information (Z0). If it is determined that the denoising operation has been performed as many times as the total number of denoising orders, the electronic device (1000) can transmit the final noise information (Z0) to the decoder (1400). The decoder (1400) can generate a final generated image based on the final noise information (Z0). If it is determined that the denoising operation has not been performed as many times as the total number of denoising orders, the electronic device (1000) can transmit the final noise information (Z0) to the decoder (1400). image ), adjusted mask map (M r ), and the following noise information (Z t-1 ) can be connected. The electronic device (1000) can be connected to the encoded image (Z image ), adjusted mask map (M r ), and the following noise information (Z t-1 ) can be input into the second generation model (1200) to repeat the denoising operation.

[0085] In one embodiment of the present disclosure, the first encoder (1310) and the decoder (1400) may be omitted from the electronic device (1000). In this case, the electronic device (1000) may be configured to receive an image, a mask map, and current noise information (Z t ) can be connected. The electronic device (1000) can connect the connected image, mask map, and current noise information (Z t ) can be input into the second generation model (1200) to repeat the denoising operation. The second generation model (1200) can output the final noise information (Z0). The final noise information (Z0) can include the final generated image. For example, one or more processors of the electronic device (1000) can output the image, the mask map, and the current noise information (Z t ) can be repeatedly performed by combining them and inputting them into the second generation model (1200). The second generation model (1200) can output final noise information (Z0). The final noise information (Z0) can include the final generated image.

[0086] Figure 5a is a conceptual diagram illustrating text guidance input to a second generation model according to one embodiment of the present disclosure. Any details overlapping with those described in Figures 1 to 4 are omitted.

[0087] Referring to FIG. 5A, the electronic device (1000) may include a second encoder (1320). The electronic device (1000) may acquire a text input. The text input may represent text corresponding to an image and / or a final generated image. For example, the text input may be expressed as a sentence such as "A table with a tumbler on in a cafeteria." While the text input is described as an English sentence, the present disclosure is not limited thereto, and the text input may be expressed in any language.

[0088] In one embodiment of the present disclosure, the electronic device (1000) can obtain text input from an external server. In one embodiment of the present disclosure, the electronic device (1000) can obtain text input from a user through a user interface. In one embodiment of the present disclosure, the electronic device (1000) can include a sound-to-text converter. The sound-to-text converter can include a speech-to-text converter. The electronic device (1000) can obtain user voice input from a user through the user interface. The electronic device (1000) can convert the user voice input into text input using the sound-to-text converter. In one embodiment of the present disclosure, the electronic device (1000) can include an image-to-text converter. The electronic device (1000) can obtain a text input representing an image by inputting an image into the image-to-text converter.

[0089] The second encoder (1320) can encode text input. In one embodiment of the present disclosure, the second encoder (1320) can be an artificial intelligence model trained to encode text input. The second encoder (1320) can transmit the encoded text input to the second generative model (1200). The encoded text input can also be referred to as text guidance, text embedding, or guidance information. The electronic device (1000) can transmit the encoded text input to at least one layer of the second generative model (1200). The second generative model (1200) can output a final generated image based on the image, the intermediate generated image, and the encoded text input.

[0090] FIG. 5b is a conceptual diagram illustrating image guidance input to a second generation model according to one embodiment of the present disclosure. Any details overlapping with those described in FIGS. 1 to 5a are omitted.

[0091] Referring to FIG. 5B, the electronic device (1000) may include a third encoder (1330). The electronic device (1000) may transmit an image to the third encoder (1330). The third encoder (1330) may encode the image. In one embodiment of the present disclosure, the third encoder (1330) may be an artificial intelligence model trained to encode images. The third encoder (1330) may transmit the encoded image to the second generation model (1200). The encoded image may also be referred to as image guidance, image embedding, or guidance information. The electronic device (1000) may transmit the encoded image to at least one layer of the second generation model (1200). The second generation model (1200) may output a final generated image based on the image, the intermediate generated image, and the encoded image.

[0092] In one embodiment of the present disclosure, the third encoder (1330) can be pre-trained in the same manner as the fifth encoder (1350) of FIGS. 11, 12A, 12B, 13A, and 13b described below.

[0093] FIG. 5c is a conceptual diagram illustrating text guidance and image guidance input to a second generation model according to one embodiment of the present disclosure. Any details overlapping with those described in FIGS. 1 to 5b are omitted.

[0094] Referring to FIG. 5C, the electronic device (1000) may include a second encoder (1320) and a third encoder (1330). The function and operation of the second encoder (1320) correspond to the function and operation of the second encoder (1320) of FIG. 5A, so that redundant descriptions thereof are omitted. The function and operation of the third encoder (1330) correspond to the function and operation of the third encoder (1330) of FIG. 5B, so that redundant descriptions thereof are omitted. The second encoder (1320) may output text guidance information based on text input. The third encoder (1330) may output image guidance information based on an image.

[0095] The second generation model (1200) may include a first neural network (1210), a second neural network (1220), and a noise blender (1230). The first neural network (1210) may output a first noise (N1) based on an image, an intermediate generated image, and image guidance information. The second neural network (1220) may output a second noise (N2) based on the image, the intermediate generated image, and text guidance information. The noise blender (1230) may blend the first noise and the second noise. The noise blender (1230) may output a blended noise by weighting the first noise and the second noise. The second generation model (1200) may output a final generated image based on the blended noise.

[0096] In one embodiment of the present disclosure, the weights between the first noise and the second noise may vary depending on user or manufacturer settings. For example, the electronic device (1000) may obtain the weights from the user through a user interface.

[0097] FIG. 5d is a conceptual diagram illustrating an embodiment in which the output of the first generation model according to one embodiment of the present disclosure is used as image guidance. Any details overlapping with those described in FIGS. 1 to 5c are omitted.

[0098] Referring to FIG. 5D, the electronic device (1000) may include a fourth encoder (1340). The electronic device (1000) may transmit an intermediate generated image to the fourth encoder (1340). The fourth encoder (1340) may encode the intermediate generated image. In one embodiment of the present disclosure, the fourth encoder (1340) may be an artificial intelligence model trained to encode images. In one embodiment of the present disclosure, the configuration, operation, and function of the fourth encoder (1340) may correspond to the configuration, operation, and function of the third encoder (1330).

[0099] The fourth encoder (1340) can transmit the encoded intermediate generated image to the second generation model (1200). The encoded intermediate generated image may also be referred to as image guidance, image embedding, or guidance information. The electronic device (1000) can transmit the encoded intermediate generated image to at least one layer of the second generation model (1200). The second generation model (1200) can output a final generated image based on the image and the encoded intermediate generated image.

[0100] According to one embodiment of the present disclosure, by utilizing the output of a first generation model, which has the property of referencing a relatively large amount of image information of a known area (or may also be referred to as a base area), as guidance information of a second generation model, a second generation model that outputs an appropriate image suitable for the image context can be provided.

[0101] FIG. 5e is a conceptual diagram illustrating an embodiment in which the output of the first generation model according to one embodiment of the present disclosure is used as image guidance. Any details overlapping with those described in FIGS. 1 to 5c are omitted.

[0102] Referring to FIG. 5e, the electronic device (1000) may include a fourth encoder (1340). The functions and operations of the fourth encoder (1340) correspond to those of the fourth encoder (1340) of FIG. 5d, and thus, redundant descriptions thereof will be omitted. The fourth encoder (1340) may output image guidance information based on an intermediately generated image.

[0103] In one embodiment of the present disclosure, the electronic device (1000) can transmit the intermediate generated image, which is the output of the first generation model (1100), to the second generation model (1200). The second generation model (1200) can infer the final generated image based on the intermediate generated image and the image guidance information. A description of an embodiment in which the second generation model (1200) infers the final generated image based on the intermediate generated image is omitted because it overlaps with the contents described in FIGS. 1 to 5C. A description of an embodiment in which the second generation model (1200) infers the final generated image based on the image guidance information is omitted because it overlaps with the contents described in FIG. 5D.

[0104] In one embodiment of the present disclosure, unlike that illustrated in FIG. 5E, the second generation model (1200) may receive an encoded text input. For example, the electronic device (1000) may receive a text input and encode the text input using an encoder, such as the second encoder (1320) of FIG. 5A. The second generation model (1200) may take as input the encoded text input (text guidance), an image, an intermediate generated image, and an encoded intermediate generated image (image guidance), and output a final generated image.

[0105] According to one embodiment of the present disclosure, the second generation model (1200) may improve the performance of inpainting and / or outpainting by not only referring to image guidance, but also inferring the final generated image based on the intermediate generated image.

[0106] FIG. 5f is a conceptual diagram illustrating an embodiment in which the output of the first generation model is used as image guidance along with text guidance according to one embodiment of the present disclosure. Any details overlapping with those described in FIGS. 1 to 5d are omitted.

[0107] Referring to FIG. 5e, the electronic device (1000) may include a second encoder (1320) and a fourth encoder (1340). The function and operation of the second encoder (1320) correspond to the function and operation of the second encoder (1320) of FIG. 5c, so that redundant descriptions thereof are omitted. The function and operation of the fourth encoder (1340) correspond to the function and operation of the fourth encoder (1340) of FIG. 5d, so that redundant descriptions thereof are omitted. The second encoder (1320) may output text guidance information based on text input. The fourth encoder (1340) may output image guidance information based on an intermediately generated image.

[0108] The second generation model (1200) may include a first neural network (1210), a second neural network (1220), and a noise blender (1230). The configuration, function, and operation of the first neural network (1210), the second neural network (1220), and the noise blender (1230) correspond to the configuration, function, and operation of the first neural network (1210), the second neural network (1220), and the noise blender (1230) of FIG. 5C, and therefore, overlapping content is omitted. The electronic device (1000) may transmit image guidance information from the fourth encoder (1340) to the first neural network (1210). The electronic device (1000) may transmit text guidance information from the second encoder (1340) to the second neural network (1220).

[0109] According to one embodiment of the present disclosure, the performance of inpainting and / or outpainting can be improved by inferring the final generated image by referring to text guidance as well as image guidance.

[0110] FIG. 5g is a conceptual diagram illustrating an embodiment in which the output of the first generation model is used as image guidance together with text guidance according to one embodiment of the present disclosure.

[0111] Referring to FIG. 5g, the electronic device (1000) may include a second encoder (1320) and a fourth encoder (1340). The functions and operations of the second encoder (1320) and the fourth encoder (1340) correspond to the functions and operations of the second encoder (1320) and the fourth encoder (1340) of FIG. 5f, and thus, overlapping content is omitted. The second encoder (1320) may output text guidance information based on text input. The fourth encoder (1340) may output image guidance information based on an intermediately generated image.

[0112] The second generation model (1200) may include a first neural network (1210), a second neural network (1220), and a noise blender (1230). The configuration, function, and operation of the first neural network (1210), the second neural network (1220), and the noise blender (1230) correspond to the configuration, function, and operation of the first neural network (1210), the second neural network (1220), and the noise blender (1230) of FIGS. 5C and 5F, and thus, overlapping content is omitted. The electronic device (1000) may transmit image guidance information from the fourth encoder (1340) to the first neural network (1210). The electronic device (1000) may transmit text guidance information from the second encoder (1340) to the second neural network (1220).

[0113] In one embodiment of the present disclosure, the electronic device (1000) can transmit an intermediate generated image, which is an output of the first generation model (1100), to a second generation model (1200). The second generation model (1200) can infer a final generated image based on the intermediate generated image, image guidance information, and text guidance information. A description of an embodiment in which the second generation model (1200) infers a final generated image based on the intermediate generated image is omitted because it overlaps with the contents described in FIGS. 1 to 5c and 5e. A description of an embodiment in which the second generation model (1200) infers a final generated image based on the image guidance information and the text guidance information is omitted because it overlaps with the contents described in FIGS. 5c and 5f.

[0114] According to one embodiment of the present disclosure, the second generation model (1200) may improve the performance of inpainting and / or outpainting by not only referring to image guidance and / or text guidance, but also inferring the final generated image based on the intermediate generated image.

[0115] In FIGS. 5a to 5g, images are shown as being input to the first generation model (1100) and the second generation model (1200), but unlike what is shown, a mask map may be connected to the images and input to the first generation model (1100) and the second generation model (1200).

[0116] In FIGS. 5a to 5g, the first encoder (1310) and decoder (1400) of FIGS. 2 and 4 are omitted, but unlike the drawings, the image and / or mask map, which are inputs of the second generation model (1200), may be encoded by the first encoder (1310), and the final output of the second generation model (1200) may be decoded by the decoder (1400).

[0117] Figure 6 is a conceptual diagram illustrating a learning method for a first generation model according to one embodiment of the present disclosure. Any details overlapping with those described in Figures 1 to 5g are omitted.

[0118] Referring to FIG. 6, the model learning system (10) may include a first generative model (1100), a discriminator (12), and a loss function (13). For example, the model learning system (10) may be a system that learns an adversarial generative neural network model. In some embodiments, as illustrated in FIG. 6, the first generative model (1100) may also be referred to as a generator (1100). In the present disclosure, the adversarial generative neural network model may represent a model in which the generator (1100) and the discriminator (12) compete adversarially by improving each other's performance through learning. Each of the generator (1100) and the discriminator (12) may include at least one layer. The layer may include a filter composed of weight values ​​for extracting features from input data.

[0119] The generator (1100) can be trained to output fake data (FD) by taking a data set (DS) as input. The data set (DS) may be a set of data including multiple images. The data set (DS) may include a mask map for each of the multiple images. The fake data (FD) may represent fake image data. The real data DB (11) may include a set of real data (RD).

[0120] The discriminator (12) can be trained to determine whether the fake data (FD) or real data (RD) is fake by inputting fake data (FD) or real data (RD).

[0121] The loss function (13) can calculate a loss function value based on the discrimination result (DR). The loss function value can be transmitted to the discriminator (12) and the generator (1100) through backpropagation. The weights of at least one layer included in the discriminator (12) and the generator (1100) can be updated based on the loss function value.

[0122] In one embodiment of the present disclosure, at least a portion of the functions of the model learning system (10) may be performed in the electronic device (1000) described in FIGS. 1 to 5g, but the present disclosure is not limited thereto, and may be performed in an external server device other than the electronic device (1000).

[0123] In one embodiment of the present disclosure, the generator (1100) learned through the model learning system (10) may correspond to the first generation model (1100) described in FIGS. 1 to 5g.

[0124] Figures 7a and 7b are conceptual diagrams illustrating the configuration of a second generation model according to one embodiment of the present disclosure. Any details overlapping with those described in Figures 1 to 6 are omitted.

[0125] Referring to FIG. 7A, the second generation model (1200) may include a first neural network (1210). The first neural network (1210) may include at least one layer. For example, at least one layer may perform cross-attention. However, the present disclosure is not limited thereto, and at least one layer may include a self-attention layer or a residual block, etc. Among the at least one layer, a layer that performs cross-attention may also be referred to as a cross-attention layer.

[0126] The electronic device (1000) can transmit guidance information to the cross-attention layer of the first neural network (1210). The first neural network (1210) can reflect weights on the guidance information based on the correlation between the input image and the guidance information.

[0127] In one embodiment of the present disclosure, the first neural network (1210) can perform a cross-attention operation using a query, a key, and a value as operands. For example, the query may include current noise information, and the key and value may include guidance information. However, the present disclosure is not limited thereto, and the query, key, and value may include different information. The cross-attention operation may be performed in the cross-attention layer. The first neural network (1210) can transmit the cross-attention operation result to the next layer.

[0128] In one embodiment of the present disclosure, unlike that illustrated in FIG. 7A, the first neural network (1210) may include a self-attention layer. The query, key, and value may include current noise information. The first neural network (1210) may perform a self-attention operation using the query, key, and value as operands. The self-attention operation may be performed in the self-attention layer. The first neural network (1210) may transmit the result of the self-attention operation to the next layer.

[0129] In one embodiment of the present disclosure, the second generation model (1200) may include an interpreter (1240). For example, the second generation model (1200) may be a pre-trained model that inputs text guidance information obtained from text into a cross-attention layer. The interpreter (1240) may transform the guidance information into having the properties and / or form of text guidance information so that the performance of the second generation model (1200) is maintained even when guidance information obtained from any data format other than text is input into the cross-attention layer. For example, the interpreter (1240) may transform image guidance information into having the properties and / or form of text guidance information.

[0130] In one embodiment of the present disclosure, the interpreter (1240) can convert the image guidance information output by the third encoder (1330) and the fourth encoder (1340) illustrated in FIGS. 5b to 5g into having the properties and / or form of text guidance information.

[0131] According to one embodiment of the present disclosure, various data can be utilized as guidance by simply adding an interpreter configuration without having to newly learn or fine-tune a second generative model with high learning cost.

[0132] In one embodiment of the present disclosure, the interpreter (1240) may be omitted. In this case, the second generation model (1200) may be pre-trained through a process in which arbitrary guidance information (e.g., image guidance information) is input to the cross-attention layer.

[0133] Referring to FIG. 7B, the performance of a task can be improved by fine-tuning the pre-trained first neural network (1210). In one embodiment of the present disclosure, the electronic device (1000) can calculate the weight change (ΔW) of at least some of the pre-defined layers according to the input of additional training data, while keeping all parameters (e.g., weights) of the pre-trained first neural network (1210) fixed.

[0134] The electronic device (1000) can determine new weights by adding a weight change (ΔW) to the fixed weights of at least some of the predefined layers corresponding to fixed parameters. According to one embodiment of the present disclosure, by fine-tuning the first neural network (1210), the performance of a task corresponding to additional learning data input can be improved.

[0135] Figure 8 is a conceptual diagram illustrating the configuration of an interpreter according to one embodiment of the present disclosure. Any details overlapping with those described in Figures 1 to 7b are omitted.

[0136] Referring to FIG. 8, the interpreter (1240) may include at least one single-layer perceptron. In one embodiment of the present disclosure, the interpreter (1240) may include a first single-layer perceptron (1241) and a second single-layer perceptron (1242). The interpreter (1240) may be configured as a multi-layer perceptron including the first single-layer perceptron (1241) and the second single-layer perceptron (1242). In one embodiment of the present disclosure, the first single-layer perceptron (1241) and the second single-layer perceptron (1242) may be connected to each other and trained.

[0137] The first single-layer perceptron (1241) can take a first embedding as input and output a second embedding. The first embedding may also be referred to as guidance information or a target image embedding. The second embedding may also be referred to as an intermediate image embedding. For example, the first embedding may be a matrix having a size of k x m. (k and m are natural numbers.) The first single-layer perceptron (1241) may include a first layer and a second layer. The first layer may include m nodes. The second layer may include n nodes. (n is a natural number.) In one embodiment of the present disclosure, m may be a natural number greater than or equal to n, but the present disclosure is not limited thereto, and m may be less than or equal to n. For example, the second embedding may be a matrix having a size of k x n.

[0138] The second single-layer perceptron (1242) can take the second embedding as input and output the third embedding. The third embedding may also be referred to as the result image embedding. The second single-layer perceptron (1242) may include a third layer and a fourth layer. The third layer may include k nodes. The fourth layer may include l nodes (l is a natural number). In one embodiment of the present disclosure, k may be less than or equal to l, but the present disclosure is not limited thereto, and k may be greater than or equal to l. For example, the third embedding may be a matrix having a size of lxn. The third embedding may be input to at least one layer, such as a cross-attention layer, of the second generative model (7a, 7b, 1200) (or the first neural network (FIGS. 7a, 7b, 1210)).

[0139] According to one embodiment of the present disclosure, the interpreter (1240) may include two or more single-layer perceptrons. Although only two single-layer perceptrons (a first single-layer perceptron (1241) and a second single-layer perceptron (1242)) are illustrated in FIG. 8, the present disclosure is not limited thereto, and the interpreter (1240) may include three or more single-layer perceptrons. The next single-layer perceptron may output the next embedding by taking as input the embedding output by the previous single-layer perceptron. In this case, the embedding output by the last single-layer perceptron may be referred to as a result image embedding. The result image embedding may be input to at least one layer, such as a cross-attention layer, of the second generative model (7a, 7b, 1200) (or the first neural network (FIGS. 7a, 7b, 1210)).

[0140] According to one embodiment of the present disclosure, unlike that illustrated in FIG. 8, the interpreter (1240) may include a single-layer perceptron. In this case, the embedding output by the single-layer perceptron may be referred to as a result image embedding. The result image embedding may be input to at least one layer, such as a cross-attention layer, of the second generative model (7a, 7b, 1200) (or the first neural network (FIGS. 7a, 7b, 1210)).

[0141] According to one embodiment of the present disclosure, the interpreter (1240) can perform a function of converting the dimension of image guidance into the dimension of text guidance. According to one embodiment of the present disclosure, the interpreter (1240) can perform a function of interpreting or converting the difference in properties between image guidance and text guidance. According to one embodiment of the present disclosure, by performing the functions described above, the second generative model, which has been trained to understand only text guidance, can understand various guidance information such as image guidance.

[0142] Figures 9a and 9b are conceptual diagrams illustrating an embodiment of adding noise to an intermediate generated image according to one embodiment of the present disclosure. Any details overlapping with those described in Figures 1 to 8 are omitted.

[0143] Referring to FIG. 9A, the electronic device (1000) can acquire a denoising strength. The denoising strength may correspond to the amount of noise to be added to the intermediate generated image. In one embodiment of the present disclosure, the denoising strength may be a specific value. For example, the specific value may be a preset value based on user or manufacturer settings. For example, the denoising strength may be determined by considering the performance indicators of the first generation model.

[0144] In one embodiment of the present disclosure, the electronic device (1000) may include a noise generator (1500). The noise generator (1500) may add noise to an intermediate generated image based on a denoising intensity. The electronic device (1000) may identify a denoising order mapped to the denoising intensity. The electronic device (1000) may set the denoising order of the current noise information of the second generation model (1200) to the identified denoising order. The intermediate generated image (Z) with added noise n ) can be utilized as current noise information having the identified denoising order (e.g., n) of the second generation model. The second generation model (1200) generates an image (e.g., a masked image), an intermediate generated image (Z) with added noise, n ), and guidance information as input to the following noise information (Z n-1 ) can be obtained.

[0145] Referring to FIG. 9b, the electronic device (1000) can encode an image using the first encoder (1310). The first encoder (1310) encodes the encoded image (Z i2 ) can be output. The electronic device (1000) can adjust the mask map to a predefined size. The electronic device (1000) can encode the intermediate generated image using the first encoder (1310). The first encoder (1310) can output the encoded intermediate generated image (Z i1 ) can be output. The noise generator (1500) generates an encoded intermediate generated image (Z) based on the denoising intensity. i1 ) can add noise to the image. The electronic device (1000) can determine the denoising order (n) corresponding to the denoising intensity. The noise generator (1500) generates an intermediate generated image (Z) with added noise. n ) can be set to the current noise information.

[0146] The electronic device (1000) encodes an image (Z i2), adjusted mask map (M r ), and intermediate generated images with added noise (Z n ) can be connected. Encoded image (Z i2 ), adjusted mask map (M r ), and intermediate generated images with added noise (Z n ) can be arbitrarily determined, but the order determined during learning of the second generation model (1200) and the order during denoising (or inference) may have to be the same. The electronic device (1000) is an encoded image (Z i2 ), adjusted mask map (M r ), and intermediate generated images with added noise (Z n ) can input the connected data (hereinafter, input data) into the second generation model (1200). The second generation model (1200) generates the following noise information (Z) based on the input data. n-1 ) can be output. The electronic device (1000) can output an encoded image (Z i2 ), adjusted mask map (M r ), and the following noise information (Z n-1 ) can be connected and the connected data can be input into the second generation model (1200). The electronic device (1000) can repeat the denoising operation until the output of the second generation model (1200) becomes the final noise information (Z0). The electronic device (1000) can obtain the final generated image by inputting the final noise information into the decoder (1400).

[0147] According to one embodiment of the present disclosure, the first generative model has lower computational costs and faster inference speed than the second generative model. According to one embodiment of the present disclosure, by utilizing data with added noise to the output image of the first generative model as noise of a specific denoising order for the second generative model, the computational costs of the second generative model can be reduced and the inference speed can be improved.

[0148] Figures 10a to 10c are conceptual diagrams illustrating an embodiment of determining denoising intensity based on the quality of an intermediate generated image according to one embodiment of the present disclosure. Any details overlapping with those described in Figures 1 to 9b are omitted.

[0149] In one embodiment of the present disclosure, the electronic device (1000) may include a denoising strength determiner (1550). The denoising strength determiner (1550) may obtain a predicted confidence value of the intermediate generated image based on the intermediate generated image. For example, the predicted confidence value may be determined based on a confidence score value for an image into which the first generation model (1100) is input. However, the present disclosure is not limited thereto, and the predicted confidence value may be determined using any technique for measuring image quality. The denoising strength determiner (1550) may determine the denoising strength based on the predicted confidence value.

[0150] In one embodiment of the present disclosure, the denoising intensity determiner (1550) can identify the size and / or shape of a portion of an image (e.g., an unknown area or a masked area). The denoising intensity determiner (1550) can determine the denoising intensity based on the identified size and / or shape. For example, as the size of the unknown area increases, the prediction performance of the first generation model (1100) may decrease. The denoising intensity determiner (1550) can increase the denoising intensity as the size of the unknown area increases. For example, if the shape of the unknown area is a specific shape, the prediction performance of the first generation model (1100) may decrease. The denoising intensity determiner (1550) can determine the denoising intensity differently depending on the identified shape. In one embodiment of the present disclosure, in order to identify the shape of the unknown area, a pre-trained classification model that classifies the shape of the area can be used.

[0151] Referring to FIG. 10B, the electronic device (1000) may include a fourth encoder (1340). The configuration, function, and operation of the fourth encoder (1340) correspond to the configuration, function, and operation of the fourth encoder (1340) of FIGS. 5D to 5F, and therefore, overlapping details are omitted. The second generation model (1200) may include a first neural network (1210), a second neural network (1220), and a noise blender (1230). The configuration, function, and operation of the first neural network (1210), the second neural network (1220), and the noise blender (1230) correspond to the configuration, function, and operation of the first neural network (1210), the second neural network (1220), and the noise blender (1230) of FIGS. 5F and 5G , and therefore, redundant descriptions thereof will be omitted. The fourth encoder (1340) can encode an intermediate generated image. The encoded intermediate generated image may also be referred to as image guidance information or image embedding. The electronic device (1000) can input the image guidance information into at least one layer of the first neural network (1210). The electronic device (1000) can input text guidance information (which may also be referred to as text embedding) into at least one layer of the second neural network (1220).

[0152] The noise generator (1500) can add noise to the intermediate generated image based on the denoising strength. The intermediate generated image with added noise (Z n ) can be utilized as current noise information having the identified denoising order (e.g., n) of the first neural network (1210) and / or the second neural network (1220). In Fig. 10b, the added intermediate generated image (Z n ) is shown as being input to both the first neural network (1210) and the second neural network (1220), but the present disclosure is not limited thereto. The added intermediate generated image (Z n) can be input to at least one of the first neural network (1210) and the second neural network (1220). The first neural network (1210) and / or the second neural network (1220) may be input to an image (e.g., a masked image), an intermediate generated image (Z) with added noise n ), and guidance information (e.g., image guidance information and / or text guidance information) as input, and then noise information (e.g., Z n-1 ) can be obtained.

[0153] Referring to FIG. 10c, the denoising intensity determiner (1550) can identify the size and / or shape of a masked region in an image from a mask map. The denoising intensity determiner (1550) can determine the denoising intensity based on at least one of a predicted confidence value, the size of a portion of the region, and the shape of a portion of the region.

[0154] FIG. 11 is a conceptual diagram illustrating an electronic device that generates a portion of an image using a generative model according to one embodiment of the present disclosure. Any details that overlap with those described in FIGS. 1 to 10c are omitted.

[0155] In one embodiment of the present disclosure, an electronic device (1000) may include a fifth encoder (1350) and a second generation model (1200). The electronic device (1000) may acquire an image including information of a preset portion of a region. For example, the portion of a region may also be referred to as an unknown region or a masked region. The image may include color information for an area excluding the portion of a region.

[0156] According to one embodiment of the present disclosure, an image may include a mask map as described in FIG. 3. The electronic device (1000) may obtain a mask map that demarcates a portion (or target portion) from the entire area of ​​the image. In this case, the portion may be a preset portion. For example, the electronic device (1000) may generate a mask map based on the image. The electronic device (1000) may link the mask map to the image.

[0157] The electronic device (1000) can transmit an image including information of a certain region to the fifth encoder (1350). The fifth encoder (1350) may be an artificial intelligence model trained to encode images. The encoded image may be used as guidance information for the second generation model (1200). The second generation model (1200) may take as input the image including information of a certain region and the image encoded by the fifth encoder (1350) and output a final generated image.

[0158] In one embodiment of the present disclosure, an electronic device (1000) may include a first encoder (1310) and a decoder (1400). The electronic device (1000) may encode an image including information of a certain region using the first encoder (1310). The electronic device (1000) may decode an output of a second generation model using the decoder (1400).

[0159] Figures 12a and 12b are conceptual diagrams illustrating a learning method of a fifth encoder according to an embodiment of the present disclosure. Figures 13a and 13b are conceptual diagrams illustrating a learning method of a fifth encoder according to an embodiment of the present disclosure. Any content that overlaps with that described in Figures 1 to 11 is omitted.

[0160] Referring to FIGS. 12A and 12B , the encoder learning system (20) may include a fifth encoder (1350), a sixth encoder (21), and a seventh encoder (22). The fifth encoder (1350) may be an artificial intelligence model whose training has not yet been completed. The encoder learning system (20) may train the fifth encoder (1350). In one embodiment of the present disclosure, the fifth encoder (1350) may be a pre-trained artificial intelligence model. The encoder learning system (20) may train the fifth encoder (1350) through additional training techniques such as fine-tuning.

[0161] Referring to FIG. 12A, the encoder learning system (20) can obtain a learning image (also referred to as a first image) that includes information (e.g., image information) for a portion of the entire region. For example, the learning image can include image information for a portion of the entire region excluding a portion of the entire region. The encoder learning system (20) can input the learning image to the fifth encoder (1350). The fifth encoder (1350) can output a first image embedding by encoding the learning image.

[0162] The encoder learning system (20) can obtain a ground truth image (which may also be referred to as a second image) containing image information for the entire area. In the present disclosure, the ground truth image may represent an answer image to be inferred from the learning image. The encoder learning system (20) can input the ground truth image to the sixth encoder (21). The sixth encoder (21) can output a second image embedding by encoding the ground truth image. In one embodiment of the present disclosure, the sixth encoder (21) may be a pre-trained artificial intelligence model. For example, the sixth encoder (21) may be an image encoder part of a pre-trained Contrastive Language-Image Pretraining (CLIP) model.

[0163] In one embodiment of the present disclosure, the sixth encoder (21) may no longer be trained in a pre-trained state. For example, the sixth encoder (21) may have fixed parameters that have been pre-trained. The parameters of the sixth encoder (21) may no longer be updated.

[0164] The encoder learning system (20) can obtain a first loss based on the first image embedding and the second image embedding. For example, the encoder learning system (20) can calculate the similarity between the first image embedding and the second image embedding. The encoder learning system (20) can obtain the first loss based on the similarity. For example, the greater the similarity, the smaller the first loss.

[0165] The encoder learning system (20) can obtain ground truth text representing a ground truth image. In the present disclosure, the ground truth text can represent a sentence describing a correct image to be inferred from a learning image. The encoder learning system (20) can input the ground truth text to the seventh encoder (22). The seventh encoder (22) can output a text embedding by encoding the ground truth text. In one embodiment of the present disclosure, the seventh encoder (22) can be a pre-trained artificial intelligence model. For example, the seventh encoder (22) can be a text encoder part of a pre-trained contrastive language-image pretraining (CLIP) model.

[0166] In one embodiment of the present disclosure, the seventh encoder (22) may no longer be trained after being trained. For example, the seventh encoder (22) may have fixed parameters that have been trained. The parameters of the seventh encoder (22) may no longer be updated.

[0167] The encoder learning system (20) can obtain the second loss based on the first image embedding and the text embedding. For example, the encoder learning system (20) can calculate the similarity between the first image embedding and the text embedding. The encoder learning system (20) can obtain the second loss based on the similarity. For example, the higher the similarity, the smaller the second loss. In one embodiment of the present disclosure, the second loss can be obtained using a contrastive loss technique, but the present disclosure is not limited thereto, and the second loss can use any loss calculation technique (or loss function) that maximizes the similarity between the first image embedding and the text embedding.

[0168] In one embodiment of the present disclosure, the encoder learning system (20) can update at least one parameter (e.g., weights and / or bias) of the fifth encoder (1350) based on the first loss and the second loss.

[0169] In one embodiment of the present disclosure, at least one parameter of the sixth encoder (21) may not be updated, but the present disclosure is not limited thereto. For example, the encoder learning system (20) may update at least one parameter of the sixth encoder (21) based on the first loss and / or the second loss.

[0170] In one embodiment of the present disclosure, at least one parameter of the seventh encoder (22) may not be updated, but the present disclosure is not limited thereto. For example, the encoder learning system (20) may update at least one parameter of the seventh encoder (22) based on the first loss and / or the second loss.

[0171] Referring to FIG. 12B, the fifth encoder (1350) can input data in which a mask map is connected to a training image and output a first image embedding. In one embodiment of the present disclosure, the fifth encoder (1350) can support an input data format of a total of four channels, including three channels corresponding to the training image and one channel corresponding to the mask map. The sixth encoder (21) can support an input data format of a total of three channels corresponding to the ground truth image. However, the present disclosure is not limited thereto, and the fifth encoder (1350) can support an input data format of more channels than the sixth encoder (21) by the number of channels of the mask map.

[0172] Referring to FIGS. 13a and 13b, the encoder learning system (20) may include a fifth encoder (1350) and a seventh encoder (22). In one embodiment of the present disclosure, the sixth encoder (21) of FIGS. 12a and 12b may be the same encoder as the fifth encoder (1350).

[0173] Referring to FIG. 13A, the fifth encoder (1350) can input a training image and output a first image embedding at any learning stage (e.g., iteration). In the same learning stage in which the first image embedding is output, the fifth encoder (1350) can input a ground truth image and output a second image embedding. The encoder learning system (20) can obtain a first loss based on the first image embedding and the second image embedding. The seventh encoder (22) can input a ground truth text and output a text embedding. The encoder learning system (20) can obtain a second loss based on the first image embedding and the text embedding. The encoder learning system (20) can update at least one parameter (e.g., weights and / or bias) of the fifth encoder (1350) based on the first loss and the second loss.

[0174] Referring to FIG. 13B, the fifth encoder (1350) may output a first image embedding by inputting data in which a first mask map is connected to a training image. The first mask map may be composed of binary values ​​and coordinate values ​​that distinguish a portion of the entire region (e.g., an unknown region or a masked region). The fifth encoder (1350) may output a second image embedding by inputting data in which a second mask map is connected to a ground truth image. Since the second mask map does not have an unknown region, the entire region may be composed of a single value.

[0175] Referring to FIG. 11, along with FIGS. 12A to 13B, a fifth encoder (1350) trained by an encoder training system (20) can input and encode a third image (e.g., an image including information of a portion of a region). The trained fifth encoder (1350) can output a first image embedding corresponding to the third image. The first image embedding may also be referred to as guidance information or image guidance information. The first image embedding can be input to at least one layer of a second generation model (1200) that infers a fourth image (e.g., a final generated image) that is different from the third image by at least a portion thereof, using the third image as input.

[0176] According to one embodiment of the present disclosure, by omitting the sixth encoder (21) illustrated in FIGS. 12a and 12b, the memory capacity of the sixth encoder (21) can be secured, and the learning cost of the fifth encoder can be reduced and the learning speed can be improved.

[0177] Figure 14 is a flowchart illustrating a method for generating a portion of an image using a generative model according to one embodiment of the present disclosure. Any details overlapping with those described in Figures 1 to 13b are omitted. For convenience of explanation, Figure 14 will be described with reference to Figure 1.

[0178] Referring to FIG. 14, a method for generating a partial region of an image using a generative model may include steps S1410 to S1430. In one embodiment of the present disclosure, steps S1410 to S1430 may be performed by an electronic device (1000) or a processor (not shown) of the electronic device (1000). However, the present disclosure is not limited thereto, and steps S1410 to S1430 may be performed by any electronic device. A method for generating a partial region of an image using a generative model according to one embodiment of the present disclosure is not limited to that illustrated in FIG. 14, and any one of the steps illustrated in FIG. 14 may be omitted, or steps not illustrated in FIG. 14 may be further included.

[0179] In step S1410, the electronic device (1000) may acquire an image containing information of a portion of the region. In one embodiment of the present disclosure, the electronic device (1000) may acquire a mask map corresponding to the portion of the region. The electronic device (1000) may connect the mask map to the image containing information of the portion of the region.

[0180] In step S1420, the electronic device (1000) may obtain an intermediate generated image including first image information for a partial region (e.g., an unknown region or a masked region) by using a first generation model that inputs an image including information for a partial region. In one embodiment of the present disclosure, the electronic device (1000) may obtain first pixel information from the first generation model (1100). In one embodiment of the present disclosure, the electronic device (1000) may obtain the intermediate generated image by performing a blending operation between the first pixel information and original pixel information of the image for a region other than the partial region.

[0181] In step S1430, the electronic device (1000) may obtain a final generated image including second image information that is at least partially different from the first image information by using a second generation model (1200) that inputs an image including information of a portion of the region and an intermediate generated image. In one embodiment of the present disclosure, the electronic device (1000) may obtain second pixel information from the second generation model (120). The electronic device (1000) may obtain the final generated image by performing a blending operation between the second pixel information and the original pixel information of the image for an area other than the portion of the region. In one embodiment of the present disclosure, the second generation model (1200) may obtain the final generated image by repeatedly performing a denoising operation for a predefined total number of denoising orders.

[0182] Figure 15 is a flowchart illustrating detailed steps of step S1410 of Figure 14. Any content overlapping with that described in Figures 1 to 14 is omitted. For convenience of explanation, Figure 15 will be described with reference to Figure 3.

[0183] Referring to FIG. 15, step S1410 of FIG. 14 may include steps S1510 to S1520. In one embodiment of the present disclosure, steps S1510 to S1520 may be performed by the electronic device (1000) or a processor (not shown) of the electronic device (1000). However, the present disclosure is not limited thereto, and steps S1510 to S1520 may be performed by any electronic device. The detailed steps of step S1410 according to the present disclosure are not limited to those illustrated in FIG. 15, and any one of the steps illustrated in FIG. 15 may be omitted, and steps not illustrated in FIG. 15 may be further included.

[0184] In step S1510, the electronic device (1000) may obtain a mask map that distinguishes a certain region from the entire region of an image including information of a certain region. In one embodiment of the present disclosure, the electronic device (1000) may obtain the mask map from an external server. In one embodiment of the present disclosure, the mask map may be generated by distinguishing a masked region of the image with a first value and an unmasked region of the image with a second value.

[0185] In step S1520, the electronic device (1000) may connect a mask map to an image including information of a portion of the image. In one embodiment of the present disclosure, the electronic device (1000) may encode an image including information of a portion of the image. The electronic device (1000) may connect a mask map to the encoded image. In one embodiment of the present disclosure, the electronic device (1000) may further connect current noise information to the connected image. The electronic device (1000) may input data connected to the current noise information into the second generation model (1200).

[0186] Figure 16 is a flowchart illustrating detailed steps of step S1430 of Figure 14. Any details overlapping with those described in Figures 1 to 15 are omitted. For convenience of explanation, Figure 16 will be described with reference to Figures 5d and 5e.

[0187] Referring to FIG. 16, step S1430 of FIG. 14 may include steps S1610 to S1620. In one embodiment of the present disclosure, steps S1610 to S1620 may be performed by the electronic device (1000) or a processor (not shown) of the electronic device (1000). However, the present disclosure is not limited thereto, and steps S1610 to S1620 may be performed by any electronic device. The detailed steps of step S1430 according to the present disclosure are not limited to those illustrated in FIG. 16, and any one of the steps illustrated in FIG. 16 may be omitted, or steps not illustrated in FIG. 16 may be further included.

[0188] In step S1610, the electronic device (1000) may encode an intermediate generated image. The electronic device (1000) may encode the intermediate generated image using a pre-trained encoder. In one embodiment of the present disclosure, the encoded intermediate generated image may be input to at least one layer of the second generation model (1200). The encoded intermediate generated image may be utilized as guidance information in the second generation model (1200).

[0189] In step S1620, the electronic device (1000) may obtain a final generated image including second image information that is at least partially different from the first image information by using a second generation model that inputs an image including information of a portion of the image and an encoded intermediate generated image. In one embodiment of the present disclosure, the electronic device (1000) may obtain conversion data by inputting the encoded intermediate generated image into an interpreter. The electronic device (1000) may input the conversion data into at least one layer of the second generation model.

[0190] In one embodiment of the present disclosure, the electronic device (1000) can transmit the intermediate generated image to the second generation model (1200). The electronic device (1000) can obtain a final generated image including second image information that is at least partially different from the first image information by using the second generation model that takes as input an image including information of a portion of the region, the intermediate generated image, and the encoded intermediate generated image.

[0191] Figure 17 is a flowchart illustrating detailed steps of step S1430 of Figure 14. Any details overlapping with those described in Figures 1 to 16 are omitted. For convenience of explanation, Figure 17 will be described with reference to Figure 5a.

[0192] Referring to FIG. 17, step S1430 of FIG. 14 may include steps S1710 to S1730. In one embodiment of the present disclosure, steps S1710 to S1730 may be performed by the electronic device (1000) or a processor (not shown) of the electronic device (1000). However, the present disclosure is not limited thereto, and steps S1710 to S1730 may be performed by any electronic device. The detailed steps of step S1430 according to the present disclosure are not limited to those illustrated in FIG. 17, and any one of the steps illustrated in FIG. 17 may be omitted, or steps not illustrated in FIG. 17 may be further included.

[0193] In step S1710, the electronic device (1000) may obtain a text input. In one embodiment of the present disclosure, the electronic device (1000) may obtain the text input from an external server. In one embodiment of the present disclosure, the electronic device (1000) may obtain the text input from a user interface. For example, the text input may include an image including information of a certain area and / or a sentence describing the final generated image.

[0194] In step S1720, the electronic device (1000) may encode a text input. The electronic device (1000) may encode the text input using a pre-trained encoder. In one embodiment of the present disclosure, the encoded text input may be input to at least one layer of the second generation model (1200). The encoded text input may be utilized as guidance information in the second generation model (1200).

[0195] In step S1730, the electronic device (1000) may obtain a final generated image including second image information that is at least partially different from the first image information by using a second generation model (1200) that inputs an encoded text input, an image including information of a portion of the region, and an intermediate generated image. In one embodiment of the present disclosure, the electronic device (1000) may generate current noise information based on the intermediate generated image. The electronic device (1000) may input the image including information of a portion of the region and the current noise information into the second generation model (1200). The electronic device (1000) may input the encoded text input into at least one layer of the second generation model (1200). The electronic device (1000) may obtain the following noise information from the second generation model (1200).

[0196] Figure 18 is a flowchart illustrating detailed steps of step S1430 of Figure 14. Any details overlapping with those described in Figures 1 to 17 are omitted. For convenience of explanation, Figure 18 will be described with reference to Figures 9a and 9b.

[0197] Referring to FIG. 18, step S1430 of FIG. 14 may include steps S1810 to S1830. In one embodiment of the present disclosure, steps S1810 to S1830 may be performed by the electronic device (1000) or a processor (not shown) of the electronic device (1000). However, the present disclosure is not limited thereto, and steps S1810 to S1830 may be performed by any electronic device. The detailed steps of step S1430 according to the present disclosure are not limited to those illustrated in FIG. 18, and any one of the steps illustrated in FIG. 18 may be omitted, or steps not illustrated in FIG. 18 may be further included.

[0198] In step S1810, the electronic device (1000) may obtain a denoising intensity for the intermediate generated image. In one embodiment of the present disclosure, the denoising intensity may correspond to the amount of noise to be added to the intermediate generated image. In one embodiment of the present disclosure, the denoising intensity may be predefined. In one embodiment of the present disclosure, the electronic device (1000) may obtain a predicted confidence value based on the intermediate generated image. The electronic device (1000) may determine the denoising intensity based on the predicted confidence value. The electronic device (1000) may identify the size and / or shape of a portion of the region (e.g., a masked region) based on a mask map. The electronic device (1000) may determine the denoising intensity based on the size and / or shape of the portion of the region (e.g., a masked region).

[0199] In step S1820, the electronic device (1000) may add noise to the intermediate generated image based on the denoising intensity. In one embodiment of the present disclosure, the higher the denoising intensity, the greater the amount of noise added.

[0200] In step S1830, the electronic device (1000) can obtain a final generated image including second image information that is at least partially different from the first image information by using a second generation model (1200) that inputs an image including information of a certain area and an intermediate generated image with added noise.

[0201] Figure 19 is a flowchart illustrating detailed steps of step S1430 of Figure 14. Any details overlapping with those described in Figures 1 to 18 are omitted. For convenience of explanation, Figure 19 will be described with reference to Figures 9a and 9b.

[0202] Referring to FIG. 19, step S1430 of FIG. 14 may include steps S1910 to S1950. In one embodiment of the present disclosure, steps S1910 to S1950 may be performed by the electronic device (1000) or a processor (not shown) of the electronic device (1000). However, the present disclosure is not limited thereto, and steps S1910 to S1950 may be performed by any electronic device. The detailed steps of step S1430 according to the present disclosure are not limited to those illustrated in FIG. 19, and any one of the steps illustrated in FIG. 19 may be omitted, or steps not illustrated in FIG. 19 may be further included.

[0203] In step S1910, the electronic device (1000) may obtain current noise information. In one embodiment of the present disclosure, the electronic device (1000) may obtain current noise information from the output of the second generation model (1200). In one embodiment of the present disclosure, in the case of the initial denoising order, the output of the second generation model (1200) does not exist. In this case, the electronic device (1000) may generate current noise information composed of random values. In one embodiment of the present disclosure, the electronic device (1000) may generate current noise information composed of random values ​​according to Gaussian noise.

[0204] In step S1920, the electronic device (1000) may connect an image (or encoded image) including current noise information and information of a portion of the region. In one embodiment of the present disclosure, the electronic device (1000) may connect a mask map whose size is adjusted to have the position and width of the image including the mask map or information of a portion of the region, to the image including the current noise information and information of a portion of the region.

[0205] In step S1930, the electronic device (1000) may input the connected image into the second generation model (1200). In one embodiment of the present disclosure, the second generation model (1200) may output the following noise information based on the connected image by performing a denoising operation.

[0206] In step S1940, the electronic device (1000) can obtain the following noise information, which is the output of the second generation model (1200).

[0207] In step S1950, the electronic device (1000) may determine whether the denoising operation has been repeated as many times as the predefined total number of denoising orders. Based on determining that the denoising operation has been repeated as many times as the predefined total number of denoising orders (Yes), the electronic device (1000) may generate a final generated image based on the next noise information. Based on determining that the denoising operation has been repeated as many times as the predefined total number of denoising orders (No), the procedure moves to step S1910. The next noise information may be the current noise information of the next denoising order.

[0208] FIG. 20 is a flowchart illustrating a method for learning an encoder for encoding an image according to an embodiment of the present disclosure. Any content overlapping with that described in FIGS. 1 to 19 is omitted. For convenience of explanation, FIG. 20 will be described with reference to FIGS. 12a and 13b. The configuration, function, and operation of the fifth encoder (1350), the sixth encoder (21), and the seventh encoder (22) of FIGS. 12a and 13b may correspond to the configuration, function, and operation of the first encoder, the second encoder, and the third encoder of FIG. 20, respectively.

[0209] Referring to FIG. 20, a method for learning an encoder for encoding an image may include steps S2010 to S2070. In one embodiment of the present disclosure, steps S2010 to S2070 may be performed by an encoder learning system (20), and at least some of the functions of the encoder learning system (20) may be performed by any electronic device or a processor of any electronic device. According to one embodiment of the present disclosure, a method for learning an encoder for encoding an image is not limited to that illustrated in FIG. 20, and any one of the steps illustrated in FIG. 20 may be omitted, and steps not illustrated in FIG. 20 may be further included.

[0210] In step S2010, the encoder learning system (20) may obtain a first image including information about a portion of the entire area, a second image including image information about the entire area, and text representing the second image. In one embodiment of the present disclosure, the encoder learning system (20) may connect a first mask map that distinguishes a portion of the entire area to the first image.

[0211] In step S2020, the encoder learning system (20) can obtain a first image embedding using a first encoder that takes a first image as input.

[0212] In step S2030, the encoder learning system (20) may obtain a second image embedding using a second encoder that takes a second image as input. In one embodiment of the present disclosure, the second encoder may be a pre-trained artificial intelligence model. In one embodiment of the present disclosure, the second encoder may no longer be trained in a pre-trained state. For example, the second encoder may have fixed parameters that may no longer be updated. In one embodiment of the present disclosure, the first encoder and the second encoder may be the same encoder. In this case, the encoders corresponding to the first encoder and the second encoder may be encoders that have not completed training. For example, the encoders corresponding to the first encoder and the second encoder may not have trained, but the present disclosure is not limited thereto, and the encoders may be pre-trained artificial intelligence models. The pre-trained encoder can be further trained using additional training techniques, such as fine-tuning. In one embodiment of the present disclosure, the encoder training system (20) can connect a second mask map, which comprises the entire area as a single value, to the second image.

[0213] In step S2040, the encoder learning system (20) can obtain a text embedding using a third encoder that inputs text. In one embodiment of the present disclosure, the third encoder may be a pre-trained artificial intelligence model. In one embodiment of the present disclosure, the third encoder may no longer be trained after being pre-trained. For example, the third encoder may have fixed parameters that are no longer updated.

[0214] In step S2050, the encoder learning system (20) can obtain a first loss based on the first image embedding and the second image embedding.

[0215] In step S2060, the encoder learning system (20) can obtain a second loss based on the first image embedding and the text embedding.

[0216] In step S2070, the encoder learning system (20) can update at least one parameter of the first encoder based on the first loss and the second loss. In one embodiment of the present disclosure, the encoder learning system (20) can update at least one parameter of the second encoder and / or the third encoder based on the first loss and the second loss. In one embodiment of the present disclosure, the encoder learning system (20) can repeatedly update at least one parameter of the first encoder, the second encoder, and / or the third encoder for a predefined number of learning rounds.

[0217] FIG. 21 is a flowchart illustrating a method for generating a portion of an image using a generative model according to one embodiment of the present disclosure. Any content overlapping with that described in FIGS. 1 to 20 is omitted. For convenience of explanation, FIG. 21 will be described with reference to FIG. 11. The configuration, function, and operation of the fifth encoder (1350) and the second generative model (1200) of FIG. 11 may correspond to the configuration, function, and operation of the first encoder and the generative model of FIG. 21.

[0218] Referring to FIG. 21, a method for generating a partial region of an image using a generative model may include steps S2110 to S2130. In one embodiment of the present disclosure, steps S2110 to S2130 may be performed by an electronic device (1000) or a processor (not shown) of the electronic device (1000). However, the present disclosure is not limited thereto, and steps S2110 to S2130 may be performed by any electronic device. A method for generating a partial region of an image using a generative model according to one embodiment of the present disclosure is not limited to that illustrated in FIG. 21, and any one of the steps illustrated in FIG. 21 may be omitted, or steps not illustrated in FIG. 21 may be further included.

[0219] In step S2110, the electronic device (1000) can obtain an image including information of a certain area.

[0220] In step S2120, the electronic device (1000) may obtain a target image embedding using a first encoder that inputs an image including information of a partial region. In one embodiment of the present disclosure, the first encoder may obtain a first learning image including location information for a partial region among an entire region, a second learning image including image information for the entire region, and learning text representing the second learning image, obtain a first image embedding using a first encoder that inputs the first learning image, obtain a second image embedding using a second encoder that inputs the second learning image, obtain a second text embedding using a third encoder that inputs the learning text, obtain a first loss based on the first image embedding and the second image embedding, obtain a second loss based on the first image embedding and the text embedding, and update at least one parameter of the first encoder based on the first loss and the second loss. In one embodiment of the present disclosure, the electronic device (1000) can update at least one parameter of the second encoder and / or the third encoder based on the first loss and / or the second loss.

[0221] In step S2130, the electronic device (1000) can obtain a final generated image by using an image including information of a certain area and a generation model that takes as input a target image embedding.

[0222] In one embodiment of the present disclosure, the generative model may include a first neural network that outputs a final generated image based on an image including information of a portion of the image. In one embodiment of the present disclosure, the electronic device (1000) may input a target image embedding into at least one layer of the first neural network. In one embodiment of the present disclosure, the generative model may include an interpreter that transforms the target image embedding and transmits it to at least one layer of the first neural network. In one embodiment of the present disclosure, the interpreter of the generative model may include a first single-layer perceptron and a second single-layer perceptron. In one embodiment of the present disclosure, the interpreter of the generative model may include three or more single-layer perceptrons. In one embodiment of the present disclosure, the interpreter of the generative model may include a single single-layer perceptron.

[0223] In one embodiment of the present disclosure, the electronic device (1000) may obtain an intermediate image embedding using a first single-layer perceptron that inputs a target image embedding. The electronic device (1000) may obtain a result image embedding using a second single-layer perceptron that inputs the intermediate image embedding. The electronic device (1000) may input the result image embedding to at least one layer of the first neural network.

[0224] In one embodiment of the present disclosure, the electronic device (1000) can configure multiple next single-layer perceptrons that input the image embedding output from the previous single-layer perceptron, and obtain the resulting image embedding using the last single-layer perceptron. In one embodiment of the present disclosure, the electronic device (1000) can directly obtain the resulting image embedding using the first single-layer perceptron.

[0225] In one embodiment of the present disclosure, the generative model may output a first noise based on an image containing information in a certain region. The generative model may output a second noise based on the image containing information in a certain region. The generative model may include an interpreter that transforms a target image embedding and passes it to at least one layer of the first neural network.

[0226] In one embodiment of the present disclosure, the generative model may include a first neural network and a second neural network. The first neural network may output a first noise using at least one of an image including information of a certain region, an intermediately generated image, and image guidance information. The second neural network may output a second noise using at least one of an image including information of a certain region, an intermediately generated image, and text guidance information.

[0227] In one embodiment of the present disclosure, the electronic device (1000) can obtain a target text corresponding to an image including information of a certain region. The electronic device (1000) can obtain a target text embedding based on the target text. The electronic device (1000) can input the target text embedding into at least one layer of a second neural network.

[0228] In one embodiment of the present disclosure, the electronic device (1000) can obtain a final generated image based on the first noise and the second noise. In one embodiment of the present disclosure, the electronic device (1000) can obtain the following noise information through a weighted sum of the first noise and the second noise.

[0229] FIG. 22 is a block diagram illustrating the configuration of a user device according to one embodiment of the present disclosure. Any details overlapping with those described in FIGS. 1 to 21 are omitted. The configuration, function, and operation of the electronic device (1000) of FIGS. 1 to 11 may correspond to the configuration, function, and operation of the user device (2000) of FIG. 22.

[0230] Referring to FIG. 22, the user device (2000) may include a communication interface (2100), a user interface (2200), a camera (2300), a processor (2400), and a memory (2500). However, not all of the illustrated components are essential components. The user device (2000) may be implemented with more components than the illustrated components, or may be implemented with fewer components.

[0231] The communication interface (2100) may include one or more components that perform communication between the user device (2000) and a server device (not shown), the user device (2000) and any electronic device (not shown), and the user device (2000) and another user device (not shown).

[0232] In one embodiment of the present disclosure, the user device (2000) can receive an image including a portion (or a masked area) from a server device via a communication interface (2100). The user device (2000) can receive a mask map from the server device via the communication interface (2100). The user device (2000) can receive text input from the server device via the communication interface (2100). The user device (2000) can receive various hyperparameters (e.g., total denoising order, etc.) necessary for inferring a final generated image from another electronic device via the communication interface (2100). The user device (2000) can receive a pre-trained generative model and / or a pre-trained encoder from the server device via the communication interface (2100).

[0233] The user interface (2200) may include an input interface and an output interface.

[0234] The input interface is for receiving input from a user (hereinafter, user input). The input interface may be at least one of a key pad, a dome switch, a touch pad (contact capacitance type, pressure resistive film type, infrared detection type, surface ultrasonic conduction type, integral tension measurement type, piezo effect type, etc.), a jog wheel, a jog switch, and a microphone, but is not limited thereto.

[0235] In one embodiment of the present disclosure, the user device (2000) can receive hyperparameters set by the user through an input interface. The user device (2000) can receive images and / or text through the input interface. In one embodiment of the present disclosure, the user device (2000) can obtain an audio signal processed by the user's voice through a microphone. The user device (2000) can convert the audio signal into text.

[0236] The output interface is for outputting audio signals or video signals, and may include, for example, a display or a speaker.

[0237] In one embodiment of the present disclosure, the user device (2000) can display an image through a display. For example, the user device (2000) can display a GUI corresponding to an input interface through the display. For example, the user device (2000) can display an image through the display. The user device (2000) can receive a user input specifying a portion of an image displayed on the display through the input interface. The user device (2000) can mask a portion of the image based on the user input.

[0238] In one embodiment of the present disclosure, the user device (2000) may receive user input for rotating and / or resizing an image displayed on a display via an input interface. Based on the user input, the user device (2000) may mask an area where there is no image information within a predefined image size.

[0239] In one embodiment of the present disclosure, the user device (2000) can receive a user input (e.g., an arbitrary line or shape) drawn on an image displayed on a display through an input interface. The user device (2000) can mask an area corresponding to the user input.

[0240] In one embodiment of the present disclosure, the user device (2000) may segment at least one object area within an image using an artificial intelligence model that performs object segmentation. The artificial intelligence model may be stored in the memory (2500) of the user device (2000). In this case, the processor (2400) of the user device (2000) may input an image into the artificial intelligence model and output a segmentation result. Alternatively, the segmentation result of the artificial intelligence model may be received from a device external to the user device (e.g., a server). The user device (2000) may display the image and the segmentation result together on the display. The user device (2000) may receive a user input for selecting at least one object displayed on the display through an input interface. The user device (2000) may determine an object corresponding to the user input among objects according to the segmentation result. The user device (2000) may mask the area of ​​the determined object. However, the present disclosure is not limited thereto, and the user device (2000) may merge and mask an area of ​​a determined object and an area corresponding to a user input (e.g., an arbitrary line or shape) drawn on an image displayed on a display. In one embodiment of the present disclosure, the user device (2000) may display an image through a display. The user device (2000) may receive a user input corresponding to at least one position value of the image displayed on the display through an input interface. For example, the at least one position value may be composed of coordinate values ​​of an image pixel. For example, the at least one position value may be composed of coordinate values ​​of an image pixel corresponding to a boundary of a specific area within the image.The user device (2000) can obtain a segmentation map that distinguishes an area of ​​an object corresponding to at least one location value from other areas using an artificial intelligence model that inputs at least one location value. The user device (2000) can mask an area of ​​an object corresponding to a user input based on the segmentation map.

[0241] For example, the display may include at least one of a liquid crystal display, a thin film transistor-liquid crystal display, a light-emitting diode (LED), an organic light-emitting diode, a flexible display, a 3D display, and an electrophoretic display. In addition, the display may include two or more displays depending on the implementation type of the user device (2000).

[0242] The speaker can output an audio signal received from a communication interface (2100) or stored in a memory (2500).

[0243] The camera (2300) can capture images of the surrounding space and generate images. The camera (2300) can include an image sensor. In one embodiment of the present disclosure, the user device (2000) can train the first generation model (1100) and / or the second generation model (1200) based on images captured by the camera (2300). In one embodiment of the present disclosure, the user device (2000) can obtain a final generated image by inputting the images captured by the camera (2300) into the first generation model (1100) and / or the second generation model (1200).

[0244] The processor (2400) can control the overall operation of the user device (2000) using a program or information stored in the memory (2500). The processor (2400) may be implemented through a combination of a general-purpose processor, such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU), and software. In the case of a dedicated processor, it may include a memory for implementing an embodiment of the present disclosure or a memory processing unit for utilizing external memory. The processor (2400) may be composed of multiple processors. In this case, it may be implemented through a combination of dedicated processors, or it may be implemented through a combination of multiple general-purpose processors, such as an AP, a CPU, or a GPU, and software.

[0245] In one embodiment of the present disclosure, the processor (2400) may include an artificial intelligence (AI) dedicated processor. The AI ​​dedicated processor may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as a part of an existing general-purpose processor (e.g., CPU or application processor) or a graphics dedicated processor (e.g., GPU) and mounted on the user device (2000). The AI ​​dedicated processor may perform inference and / or learning operations related to at least one of the first generation model (1100), the second generation model (1200), the encoder (1300), and the decoder (1400).

[0246] In one embodiment of the present disclosure, the processor (2400) may infer a designated area of ​​an image based on image information excluding a designated area of ​​the image using the first generation model (1100) and / or the second generation model (1200). Here, the designated area may mean an unknown area.

[0247] In one embodiment of the present disclosure, the processor (2400) can train at least one of the first generation model (1100), the second generation model (1200), the encoder (1300), and the decoder (1400) using a training dataset stored in the memory (2500). The processor (2400) can store the trained first generation model (1100), the second generation model (1200), the encoder (1300), and / or the decoder (1400) in the memory (2500).

[0248] The memory (2500) may store a program for processing by the processor (2400) and may also store input / output data. In one embodiment of the present disclosure, the memory (2500) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk. The programs stored in the memory (2500) may be classified into a plurality of modules according to their functions.

[0249] In one embodiment of the present disclosure, the memory (2500) may include a first generation model (1100), a second generation model (1200), an encoder (1300), a decoder (1400), a noise generator (1500), and a denoising strength determiner (1550).

[0250] The configuration, function, and operation of the first generation model (1100), the second generation model (1200), the decoder (1400), the noise generator (1500), and the denoising strength determiner (1550) may correspond to the configuration, function, and operation of the first generation model (1100), the second generation model (1200), the decoder (1400), the noise generator (1500), and the denoising strength determiner (1550) of FIGS. 1 to 11. The encoder (1300) may include a plurality of encoders. The configuration, function, and operation of the plurality of encoders may correspond to the configuration, function, and operation of the first to seventh encoders (1310, 1320, 1330, 1340, 1350, 21, 22) of FIGS. 1 to 13b. Each of the multiple encoders may be the encoder part of a different autoencoder.

[0251] FIG. 23 is a block diagram illustrating the configuration of a user device and a server device according to one embodiment of the present disclosure. Any overlapping content described in FIGS. 1 to 21 is omitted. The configuration, function, and operation of the electronic device (1000) of FIGS. 1 to 11 may correspond to the configuration, function, and operation of the server device (3000) of FIG. 23. The configuration, function, and operation of the user device (2000) of FIG. 22 may correspond to the configuration, function, and operation of the user device (2000) of FIG. 23.

[0252] Referring to FIG. 23, the server device (3000) may include a communication interface (3100), a processor (3200), and a memory (3300). However, not all of the illustrated components are essential components. The server device (3000) may be implemented with more components than the illustrated components, or may be implemented with fewer components.

[0253] The communication interface (3100) may include one or more components that perform communication between the server device (3000) and the user device (2000), the server device (3000) and any electronic device (not shown), and the server device (3000) and an external server device (not shown).

[0254] In one embodiment of the present disclosure, the server device (3000) can receive an image including a portion (or a masked area) from the user device (2000) via the communication interface (3100). The server device (3000) can receive a mask map from the user device (2000) via the communication interface (3100). The server device (3000) can receive text input from the user device (2000) via the communication interface (3100). The server device (3000) can receive various hyperparameters (e.g., total denoising order, etc.) required to infer the final generated image from the user device (2000) via the communication interface (3100).

[0255] The processor (3200) can control the overall operation of the server device (3000) using a program or information stored in the memory (3300). The processor (3200) may be implemented through a combination of a general-purpose processor, such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU), and software. In the case of a dedicated processor, it may include a memory for implementing an embodiment of the present disclosure or a memory processing unit for utilizing external memory. The processor (3200) may be composed of multiple processors. In this case, it may be implemented through a combination of dedicated processors, or it may be implemented through a combination of multiple general-purpose processors, such as an AP, a CPU, or a GPU, and software.

[0256] In one embodiment of the present disclosure, the processor (3200) may include an artificial intelligence (AI) dedicated processor. The AI ​​dedicated processor may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as a part of an existing general-purpose processor (e.g., CPU or application processor) or a graphics dedicated processor (e.g., GPU) and mounted on the server device (3000). The AI ​​dedicated processor may perform inference and / or learning operations related to at least one of the first generation model (1100), the second generation model (1200), the encoder (1300), and the decoder (1400).

[0257] In one embodiment of the present disclosure, the processor (3200) may receive a request signal for generating an image and a portion of the image from the user device (2000) via the communication interface (3100). In response to the request signal, the processor (3200) may input the image into the learned first generation model (1100) and / or second generation model (1200), thereby inferring a final generated image in which a portion of the image is generated. The processor (3200) may transmit the final generated image to the user device (2000) via the communication interface (3100). The user device (2000) may receive the final generated image. The user device (2000) may display the final generated image via the user interface (2200).

[0258] In one embodiment of the present disclosure, the processor (3200) can infer a designated region of an image based on image information excluding a designated region of the image using the first generation model (1100) and / or the second generation model (1200).

[0259] In one embodiment of the present disclosure, the processor (3200) can train at least one of the first generation model (1100), the second generation model (1200), the encoder (1300), and the decoder (1400) using a training dataset stored in the memory (3300). The processor (2400) can store the trained first generation model (1100), the second generation model (1200), the encoder (1300), and / or the decoder (1400) in the memory (3300).

[0260] The memory (3300) may store a program for processing by the processor (3200) and may also store input / output data. In one embodiment of the present disclosure, the memory (2500) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk. The programs stored in the memory (2500) may be classified into a plurality of modules according to their functions.

[0261] In one embodiment of the present disclosure, the memory (3300) may include a first generation model (1100), a second generation model (1200), an encoder (1300), a decoder (1400), a noise generator (1500), and a denoising strength determiner (1550).

[0262] The configuration, function, and operation of the first generation model (1100), the second generation model (1200), the decoder (1400), the noise generator (1500), and the denoising strength determiner (1550) may correspond to the configuration, function, and operation of the first generation model (1100), the second generation model (1200), the decoder (1400), the noise generator (1500), and the denoising strength determiner (1550) of FIGS. 1 to 11. The encoder (1300) may include a plurality of encoders. The configuration, function, and operation of the plurality of encoders may correspond to the configuration, function, and operation of the first to seventh encoders (1310, 1320, 1330, 1340, 1350, 21, 22) of FIGS. 1 to 13b. Each of the multiple encoders may be the encoder part of a different autoencoder.

[0263] Unlike as illustrated in FIG. 23, at least some of the functions of the server device (3000) may be performed in the user device (2000). For example, the user device (2000) may infer an intermediate generated image using the first generation model (1100) stored in the memory (2500). The server device (3000) may receive the intermediate generated image via the communication interface (3100). The server device (3000) may infer a final generated image based on the intermediate generated image using the second generation model (1200) stored in the memory (3300). However, this is only an example, and at least some of the first generation model (1100), the second generation model (1200), the encoder (1300), the decoder (1400), the noise generator (1500), and the denoising strength determiner (1550) may be executed on the user device (2000) or on the server device (3000).

[0264] In one embodiment of the present disclosure, a method for generating a partial region of an image using a generative model may be provided. The method may include a step of obtaining an image including information of the partial region. The method may include a step of obtaining an intermediate generated image including first image information for the partial region using a first generative model having as input the image including information of the partial region. The method may include a step of obtaining a final generated image including second image information, wherein at least a portion of the information is different from the first image information, using a second generative model having as input the image including information of the partial region and the intermediate generated image.

[0265] In one embodiment of the present disclosure, the step of obtaining an image including information of the partial region may include the step of obtaining a mask map that distinguishes the partial region from the entire region of the image including information of the partial region. The step of obtaining an image including information of the partial region may include the step of concatenating the mask map to the image including information of the partial region.

[0266] In one embodiment of the present disclosure, the step of obtaining the final generated image may include the step of encoding the intermediate generated image. The step of obtaining the final generated image may include the step of obtaining a final generated image including second image information, at least part of which is different from the first image information, by using a second generation model that takes as input an image including information of the partial region and the encoded intermediate generated image.

[0267] In one embodiment of the present disclosure, the step of obtaining the final generated image may include the step of obtaining a final generated image including second image information that is at least partially different from the first image information, using an image including information of the partial region, the encoded intermediate generated image, and a second generation model that takes the intermediate generated image as input.

[0268] In one embodiment of the present disclosure, the step of obtaining the final generated image may include the step of obtaining a text input. The step of obtaining the final generated image may include the step of encoding the text input. The step of obtaining the final generated image may include the step of obtaining a final generated image including second image information, at least a portion of which is different from the first image information, by using a second generation model that takes as input the encoded text input, an image including information of the partial region, and the intermediate generated image.

[0269] In one embodiment of the present disclosure, the step of obtaining the final generated image may include the step of obtaining a final generated image including second image information that is at least partially different from the first image information, by using the encoded text input, an image including information of the partial region, the intermediate generated image, and a second generation model that takes the encoded intermediate generated image as input.

[0270] In one embodiment of the present disclosure, the step of obtaining the final generated image may include the step of obtaining a denoising strength for the intermediate generated image. The step of obtaining the final generated image may include the step of adding noise to the intermediate generated image based on the denoising strength. The step of obtaining the final generated image may include the step of obtaining a final generated image including second image information, at least a portion of which is different from the first image information, by using a second generation model that inputs an image including information of the partial region and the intermediate generated image to which noise has been added.

[0271] In one embodiment of the present disclosure, the step of obtaining a denoising intensity for the intermediate generated image may include the step of obtaining a predicted confidence value based on the intermediate generated image. The step of obtaining the denoising intensity for the intermediate generated image may include the step of determining the denoising intensity based on at least one of the predicted confidence value, the size of the partial region, and the shape of the partial region.

[0272] In one embodiment of the present disclosure, the step of obtaining the final generated image may include a step of obtaining current noise information. The step of obtaining the final generated image may include a step of connecting an image including the current noise information and information of the partial region. The step of obtaining the final generated image may include a step of inputting the connected image into the second generation model. The step of obtaining the final generated image may include a step of obtaining subsequent noise information from the second generation model.

[0273] In one embodiment of the present disclosure, the current noise information may correspond to an intermediate generated image to which the noise has been added.

[0274] In one embodiment of the present disclosure, the step of obtaining the final generated image may include a step of determining a target denoising order corresponding to the intermediate generated image to which the noise has been added among a predefined total number of denoising orders based on the denoising intensity. The step of obtaining the final generated image may include a step of setting the denoising order of the current noise information to the determined target denoising order.

[0275] In one embodiment of the present disclosure, the first generative model may be a generative adversarial network (GAN) model. In one embodiment of the present disclosure, the second generative model may be a diffusion model.

[0276] In one embodiment of the present disclosure, an electronic device may be provided. The electronic device may include a memory that stores at least one instruction. The electronic device may include at least one processor that executes the at least one instruction. The at least one processor may obtain an image including information of a portion of an image. The at least one processor may obtain an intermediately generated image including first image information for the portion of an image using a first generation model that inputs the image including information of the portion of an image. The at least one processor may obtain a finally generated image including second image information, at least a portion of which is different from the first image information, using a second generation model that inputs the image including information of the portion of an image and the intermediately generated image.

[0277] In one embodiment of the present disclosure, a method for training an encoder to encode an image may be provided. The method may include obtaining a first image including information about a portion of an entire region, a second image including image information about the entire region, and text representing the second image. The method may include obtaining a first image embedding using a first encoder having the first image as an input. The method may include obtaining a second image embedding using a second encoder having the second image as an input. The method may include obtaining a text embedding using a third encoder having the text as an input. The method may include obtaining a first loss based on the first image embedding and the second image embedding. The method may include obtaining a second loss based on the first image embedding and the text embedding. The method may include updating at least one parameter of the first encoder based on the first loss and the second loss.

[0278] In one embodiment of the present disclosure, the second encoder and the third encoder may be artificial intelligence models that have been pre-learned and have fixed parameters.

[0279] In one embodiment of the present disclosure, the method may include updating at least one parameter of the second encoder and the third encoder based on the first loss and the second loss.

[0280] In one embodiment of the present disclosure, the method may include a step of connecting a first mask map that separates the partial region from the entire region to the first image.

[0281] In one embodiment of the present disclosure, the first encoder and the second encoder may be the same encoder.

[0282] In one embodiment of the present disclosure, the method may include a step of connecting a second mask map comprising the entire area as a single value to the second image.

[0283] In one embodiment of the present disclosure, a method for generating a portion of an image using a generative model may be provided. The method may include a step of obtaining an image including information of the portion of the image. The method may include a step of obtaining a target image embedding using a first encoder that inputs the image including information of the portion of the image. The method may include a step of obtaining a final generated image using a generative model that inputs the image including information of the portion of the image and the target image embedding.

[0284] In one embodiment of the present disclosure, the first encoder may be trained by obtaining a first training image including location information for a portion of the entire region, a second training image including image information for the entire region, and training text representing the second training image, obtaining a first image embedding using the first encoder having the first training image as an input, obtaining a second image embedding using the second encoder having the second training image as an input, obtaining a text embedding using the third encoder having the training text as an input, obtaining a first loss based on the first image embedding and the second image embedding, obtaining a second loss based on the first image embedding and the text embedding, and updating at least one parameter of the first encoder based on the first loss and the second loss.

[0285] In one embodiment of the present disclosure, the second encoder and the third encoder may be artificial intelligence models that have been pre-learned and have fixed parameters.

[0286] In one embodiment of the present disclosure, the method comprises: wherein the first learning image is connected to a first mask map that separates the partial region from the entire region.

[0287] In one embodiment of the present disclosure, the first encoder and the second encoder may be the same encoder.

[0288] In one embodiment of the present disclosure, the second learning image may be connected to a second mask map that configures the entire area as one value.

[0289] In one embodiment of the present disclosure, the generative model may include a first neural network that outputs the final generated image based on an image including information of the partial region.

[0290] In one embodiment of the present disclosure, the method may include a step of inputting the target image embedding into at least one layer of the first neural network.

[0291] In one embodiment of the present disclosure, the generative model may include an interpreter that transforms the target image embedding and transmits it to at least one layer of the first neural network. The generative model may include an interpreter composed of at least one single-layer perceptron.

[0292] In one embodiment of the present disclosure, at least one single-layer perceptron may include a first single-layer perceptron and a second single-layer perceptron. In one embodiment of the present disclosure, the interpreter may be composed of three or more single-layer perceptrons. In one embodiment of the present disclosure, the interpreter may be composed of a single single-layer perceptron.

[0293] In one embodiment of the present disclosure, the method may include a step of obtaining an intermediate image embedding using the first single-layer perceptron having the target image embedding as an input. The method may include a step of obtaining a result image embedding using the second single-layer perceptron having the intermediate image embedding as an input. The method may include a step of inputting the result image embedding to at least one layer of the first neural network.

[0294] In one embodiment of the present disclosure, the method may include a step of having a next single-layer perceptron, which takes as input the image embedding output by the previous single-layer perceptron, output a next image embedding. The method may repeat the step of having a next single-layer perceptron, which takes as input the image embedding output by the previous single-layer perceptron, output a next image embedding. The method may include a step of obtaining a result image embedding using a last single-layer perceptron among the at least one single-layer perceptron. The method may include a step of inputting the result image embedding to at least one layer of the first neural network.

[0295] In one embodiment of the present disclosure, at least one single-layer perceptron may comprise a unique single-layer perceptron. The method may include obtaining a result image embedding using the unique single-layer perceptron. The method may include inputting the result image embedding to at least one layer of the first neural network.

[0296] In one embodiment of the present disclosure, the generative model may include a first neural network that outputs a first noise based on an image including information of the partial region. The generative model may include a second neural network that outputs a second noise based on an image including information of the partial region. The generative model may include an interpreter that transforms the target image embedding and transmits it to at least one layer of the first neural network.

[0297] In one embodiment of the present disclosure, the first neural network can input an image including information of the above-described partial region and the target image embedding and output a first noise.

[0298] In one embodiment of the present disclosure, the first neural network may output first noise by taking as input an image including information of the above-described partial region, the target image embedding, and an intermediate generated image.

[0299] In one embodiment of the present disclosure, the method may include a step of obtaining target text corresponding to an image including information of the partial region. The method may include a step of obtaining a target text embedding based on the target text. The method may include a step of inputting the target text embedding to at least one layer of the second neural network.

[0300] In one embodiment of the present disclosure, the second neural network may output second noise by taking as input an image including information of the above-described partial region and a target text embedding.

[0301] In one embodiment of the present disclosure, the second neural network may output second noise by taking as input an image including information of the above-described partial region, a target text embedding, and an intermediate generated image.

[0302] In one embodiment of the present disclosure, the method may include a step of obtaining the final generated image based on the first noise and the second noise.

[0303] In one embodiment of the present disclosure, an electronic device may be provided. The electronic device may include a memory storing at least one instruction. The electronic device may include at least one processor executing the at least one instruction. An electronic device wherein the at least one processor obtains a first image including information about a portion of an entire area, a second image including image information about the entire area, and text representing the second image, the at least one processor obtains a first image embedding using a first encoder having the first image as an input, the at least one processor obtains a second image embedding using a second encoder having the second image as an input, the at least one processor obtains a second text embedding using a third encoder having the text as an input, the at least one processor obtains a first loss based on the first image embedding and the second image embedding, the at least one processor obtains a second loss based on the first image embedding and the text embedding, and the at least one processor updates at least one parameter of the first encoder based on the first loss and the second loss.

[0304] A method according to an embodiment of the present disclosure may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the present disclosure or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0305] Some embodiments of the present disclosure may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules, executed by a computer. Computer-readable media may be any available media that can be accessed by a computer, and include both volatile and nonvolatile media, removable and non-removable media. Furthermore, computer-readable media may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Communication media typically includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave, or other transport mechanism, and includes any information delivery media. Furthermore, some embodiments of the present disclosure may also be implemented as a computer program or computer program product containing computer-executable instructions, such as a computer program that is executed by a computer.

[0306] In one embodiment of the present disclosure, the device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently in the storage medium and cases where data is stored temporarily. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0307] According to an embodiment of the present disclosure, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0308] Although the embodiments of the present disclosure have been described in detail above, the scope of the present disclosure is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concepts of the present disclosure defined in the following claims also fall within the scope of the present disclosure.

Claims

1. A method for generating a portion of an image using a generative model, A step of obtaining an image including information of the above-mentioned part of the area; A step of obtaining an intermediate generated image including first image information for the partial region by inputting the image into the first generation model (1100); and A method comprising the step of obtaining a final generated image including second image information that is at least partially different from the first image information by inputting the above image and the intermediate generated image into a second generation model (1200).

2. In paragraph 1, The steps of obtaining an image containing information of the above-mentioned part of the area are: A step of obtaining a mask map that separates the partial area from the entire area of ​​the image; and A method further comprising the step of concatenating said mask map to said image.

3. In either of paragraphs 1 and 2, The steps to obtain the final generated image are: a step of encoding the above intermediate generated image; and A method comprising the step of obtaining the final generated image by inputting the image and the encoded intermediate generated image into the second generating model.

4. In any one of paragraphs 1 to 3, The steps to obtain the final generated image are: Step of obtaining text input; a step of encoding the above text input; and A method comprising the step of obtaining the final generated image by inputting the encoded text input, the image, and the intermediate generated image into the second generating model.

5. In any one of paragraphs 1 to 4, The steps to obtain the final generated image are: A step of obtaining a denoising intensity for the above intermediate generated image; A step of adding noise to the intermediate generated image based on the denoising intensity; and A method comprising the step of obtaining the final generated image by inputting the intermediate generated image to which the image and the noise have been added into the second generating model.

6. In any one of paragraphs 1 to 5, The steps for obtaining the denoising intensity for the above intermediate generated image are: A step of obtaining a prediction confidence value based on the above intermediate generated image; and A method comprising the step of determining the denoising strength based on at least one of the predicted confidence value, the size of the partial region, and the shape of the partial region.

7. In any one of paragraphs 1 to 6, The steps to obtain the final generated image are: Step of obtaining current noise information; A step of connecting the current noise information and the image; a step of inputting the above-mentioned connected image into the second generation model; and A method comprising the step of obtaining the following noise information from the second generation model.

8. In any one of paragraphs 1 to 7, The steps to obtain the final generated image are: A step of determining a target denoising order corresponding to the intermediate generated image to which the noise has been added among the predefined total denoising orders based on the denoising strength; and A method comprising the step of setting the denoising order of the current noise information to the determined target denoising order.

9. In any one of paragraphs 1 to 8, The above first generative model is a GAN (generative adversarial network) model, A method wherein the second generative model is a diffusion model.

10. Memory (2500; 3300) storing at least one instruction; and Containing at least one processor (2400; 3200), The electronic device, by causing at least one processor (2400; 3200) to execute at least one instruction stored in the memory, Obtain an image containing information of some area, By inputting the above image into the first generation model (1100), an intermediate generation image including first image information for the above partial region is obtained, An electronic device that obtains a final generated image including second image information that is at least partially different from the first image information by inputting the above image and the intermediate generated image into a second generation model (1200).

11. In paragraph 10, The electronic device, by causing at least one processor (2400; 3200) to execute at least one instruction stored in the memory: Obtain a mask map that separates the above-mentioned partial area from the entire area of ​​the above-mentioned image, An electronic device concatenating said mask map with said image.

12. In either of paragraphs 10 and 11, The electronic device, by causing at least one processor (2400; 3200) to execute at least one instruction stored in the memory: Encode the above intermediate generated image, An electronic device that obtains the final generated image by inputting the image and the encoded intermediate generated image into the second generating model.

13. In any one of paragraphs 10 to 12, The electronic device, by causing at least one processor (2400; 3200) to execute at least one instruction stored in the memory: Get text input, Encode the above text input, An electronic device that obtains the final generated image by inputting the encoded text input, the image, and the intermediate generated image into the second generating model.

14. In any one of paragraphs 10 to 13, The electronic device, by causing at least one processor (2400; 3200) to execute at least one instruction stored in the memory: Obtain the denoising strength for the above intermediate generated image, Adding noise to the intermediate generated image based on the denoising intensity, An electronic device that obtains the final generated image by inputting the intermediate generated image to which the above image and the noise have been added into the second generating model.

15. A computer-readable recording medium having recorded thereon a program for performing the method of any one of claims 1 to 9 on a computer.

Images