Editing and enhancing facial images using personalized pre-distributions.

Abstract

To provide a system and a method for identifying a personalized prior distribution within a generative model's latent vector space based on a set of images of a given subject.SOLUTION: The present method includes using a personalized prior distribution to confine inputs of a generative model to a latent vector space W (302) associated with a given subject, such that when the model is tasked with editing an image of the subject, to perform inpainting to fill in masked areas, improve resolution, or deblur the image, the subject's identifying features will be reflected in the images the model produces.SELECTED DRAWING: Figure 6

Description

Background Art

[0001] By using a generative model, tasks can be performed ranging from editing and enhancing an image of a given subject to generating a realistic image (or a part of an image) of a given subject or a synthetically generated subject. To adequately train such a model, generally, a large number of sets of images of a large number of subjects are required. As a result, when using a generative model to edit, enhance, or fill in a part of an image of a known subject, an image that looks realistic but resembles a different subject may be created.

Summary of the Invention

[0002] The present technology relates to a system and method for identifying a personalized prior distribution in the latent vector space of a generative model based on a set of images of a given subject. In some aspects, the present technology uses a personalized prior distribution (e.g., a convex hull defined by a set of codes generated based on a set of images of a subject) and may further include restricting the codes input to the generative model so that the identification features of the subject are reflected in the images created by the model. For example, the generative model may be configured to enhance or fill in facial features in an image of a subject where cues related to the identification of the subject are only partially present (e.g., due to motion blur, low illumination, low resolution, occlusion by other subjects). Without the present technology, the model may be able to successfully enhance or fill in such images, but may do so by creating an image that appears to be a different subject. By focusing the generative model using the present technology, the images created by the generative model may become more consistent with the appearance of the subject.

[0003] In one aspect, the disclosure describes a method performed by a computer, which includes (1) testing a plurality of codes for each given image in a set of images of a subject using one or more processors of a processing system to identify an optimization code for the given image, the testing comprising: (a) for each of the plurality of codes, generating a first image using a generative model and the code; using one or more processors, comparing the first image with the given image to generate a first loss value for the code; and (b) using one or more processors, comparing the first loss values ​​generated for each of the plurality of codes to identify the code having the smallest first loss value as the optimization code for the given image; and (2) using one or more processors, generating a personalized prior distribution for the subject based on a convex hull containing each optimization code identified for each given image in the set of images of the subject. In some embodiments, the above method further includes (1) generating a second image using a generative model and an optimization code identified for each given image in a set of subject images, and generating a second loss value by comparing the second image with the given image using one or more processors, and (2) constructing a tuned generative model by modifying one or more parameters of the generative model based at least partially on each generated second loss value using one or more processors. In some embodiments, the above method includes (1) identifying a plurality of coefficient sets using one or more processors, wherein each coefficient set in the plurality of coefficient sets corresponds to a code in the convex hull, and (2) generating a third image using a tuned generative model and a given code corresponding to the given coefficient set for each given coefficient set using one or more processors, and generating a third loss value for the third image by comparing the third image with at least a portion of the subject input images using one or more processors. The method further includes comparing third loss values ​​to identify the third image having the smallest third loss value as the personalized output image. In some embodiments, the above method further includes: (1) identifying a plurality of coefficient sets using one or more processors, wherein each coefficient set in the plurality of coefficient sets corresponds to a code in the convex hull; (2) identifying a plurality of code sets using one or more processors, wherein each code set in the plurality of code sets includes two or more individual codes, wherein each individual code corresponds to a coefficient set among the plurality of coefficient sets; (3) for each given code set in the plurality of code sets, generating a third image using a tuned generative model and the given code set using one or more processors, wherein each individual code in the given code set is provided to a different layer or set of layers of the tuned generative model; generating a third image by comparing the third image with at least a portion of the input image of the subject using one or more processors; and (4) comparing the generated third loss values ​​for each third image using one or more processors to identify the third image having the smallest third loss value as the personalized output image. In some embodiments, the above method further includes: (1) identifying a plurality of coefficient sets using one or more processors, wherein each coefficient set in the plurality of coefficient sets corresponds to a code in the convex hull; (2) for each given coefficient set in the plurality of coefficient sets, generating a third image using one or more processors with a generative model and a given code corresponding to the given coefficient set; generating a third loss value for the third image by comparing the third image with at least a portion of the input image of the subject using one or more processors; and (3) comparing the generated third loss values ​​for each third image using one or more processors to identify the third image having the smallest third loss value as the personalized output image.In some embodiments, the above method further includes: (1) identifying a plurality of coefficient sets using one or more processors, wherein each coefficient set in the plurality of coefficient sets corresponds to a code in the convex hull; (2) identifying a plurality of code sets using one or more processors, wherein each code set in the plurality of code sets includes two or more individual codes, wherein each individual code corresponds to a coefficient set among the plurality of coefficient sets; (3) for each given code set in the plurality of code sets, generating a third image using a generative model and the given code set using one or more processors, wherein each individual code in the given code set is provided to a different layer or set of layers of the generative model; generating a third image by comparing the third image with at least a portion of the input image of the subject using one or more processors; and (4) comparing the generated third loss values ​​for each third image using one or more processors to identify the third image having the smallest third loss value as the personalized output image. In some embodiments, the multiple coefficient sets include a first coefficient set and a set of coefficient sets selected using gradient descent, directly or indirectly based on the first coefficient set. In some embodiments, the input image of a subject includes a first portion of pixels saved from the original image of the subject and a mask replacing a second portion of pixels from the original image of the subject, and generating a third loss value for the third image by comparing the third image to at least a portion of the input image of the subject using one or more processors includes generating a third loss value for the third image by comparing the third image to the first portion of pixels. In some embodiments, the input image has a first resolution, and the personalized output image has a second resolution higher than the first resolution. In some embodiments, the multiple codes include a first code and a set of coefficients selected using gradient descent, directly or indirectly based on the first code.In some embodiments, the first code represents the mean of the latent vector space W, and the latent vector space W represents all possible codes that can be input into the generative model.

[0004] In another aspect, the disclosure describes a processing system including a memory for storing generative models and one or more processors coupled to the memory and configured to perform one of the methods described above.

[0005] In another aspect, the disclosure describes a processing system comprising: (1) a memory for storing generative models; and (2) one or more processors coupled to the memory and configured to generate personalized prior distributions for subjects for use with the generative models, the processors comprising: (a) testing a plurality of codes for each given image in a set of images of a subject to identify an optimization code for a given image, (i) for each of the plurality of codes generating a first image using the generative model and the code, comparing the first image with a given image to generate a first loss value for the code; (ii) comparing the first loss values ​​generated for each of the plurality of codes to identify the code having the smallest first loss value as the optimization code for the given image; and (b) generating a personalized prior distribution for a subject based on a convex hull containing each optimization code identified for each given image in a set of images of a subject. In some embodiments, one or more processors are further configured to tune the generative model, including (1) generating a second image using the generative model and optimization codes for each given image in a set of subject images, comparing the second image with the given image to generate a second loss value using one or more processors, and (2) constructing a tuned generative model by modifying one or more parameters of the generative model based at least in part on each generated second loss value using one or more processors.In some embodiments, one or more processors are further configured to generate personalized output images based on an input image of a subject, and include: (1) identifying a plurality of coefficient sets, each coefficient set of the plurality of coefficient sets corresponding to a code in a convex hull; (2) for each given coefficient set of the plurality of coefficient sets, generating a third image using a tuned generative model and a given code corresponding to the given coefficient set; comparing the third image with at least a portion of the input image of the subject to generate a third loss value for the third image; and (3) comparing the generated third loss values ​​for each third image to identify the third image having the smallest third loss value as the personalized output image. In some embodiments, one or more processors are further configured to generate a personalized output image based on an input image of a subject, and include: (1) identifying a plurality of coefficient sets, each coefficient set of the plurality of coefficient sets corresponding to a code in a convex hull; (2) identifying a plurality of code sets, each code set of the plurality of code sets containing two or more individual codes, each individual code corresponding to a coefficient set among the plurality of coefficient sets; (3) for each given code set of the plurality of code sets, generating a third image using a tuned generative model and the given code set, each individual code of the given code set being provided to a different layer or set of layers of the tuned generative model; generating a third loss value for the third image by comparing the third image with at least a portion of the input image of the subject; and (4) comparing the generated third loss values ​​for each third image to identify the third image having the smallest third loss value as the personalized output image.In some embodiments, one or more processors are further configured to generate personalized output images based on an input image of a subject, comprising: (1) identifying a plurality of coefficient sets, where each coefficient set corresponds to a code in the convex hull; (2) for each given coefficient set of the plurality of coefficient sets, generating a third image using a generative model and a given code corresponding to the given coefficient set; comparing the third image with at least a portion of the input image of the subject to generate a third loss value for the third image; and (3) comparing the generated third loss values ​​for each third image to determine the best. The process includes identifying a third image having a small third loss value as a personalized output image. In some embodiments, one or more processors are further configured to generate a personalized output image based on an input image of a subject, and the process includes: (1) identifying a plurality of coefficient sets, each coefficient set of the plurality of coefficient sets corresponding to a code in a convex hull; (2) identifying a plurality of code sets, each code set of the plurality of code sets containing two or more individual codes, each individual code corresponding to a coefficient set among the plurality of coefficient sets; (3) generating a third image for each given code set of the plurality of code sets using a generative model and the given code set, each individual code of the given code set being provided to a different layer or set of layers of the generative model; generating a third loss value for the third image by comparing the third image with at least a portion of the input image of the subject; and (4) comparing the generated third loss values ​​for each third image to identify the third image having the smallest third loss value as a personalized output image. In some embodiments, the set of coefficients includes a first set of coefficients and a set of multiple coefficient sets, and one or more processors are further configured to select each of the set of coefficient sets using gradient descent, directly or indirectly based on the first set of coefficients. In some embodiments, the input image of a subject includes a first portion of pixels saved from the original image of the subject and a substitute mask for a second portion of pixels from the original image of the subject, and generating a third loss value for the third image by comparing the third image to at least a portion of the input image of the subject includes generating a third loss value for the third image by comparing the third image to the first portion of pixels. In some embodiments, one or more processors are configured to generate a personalized output image based on the input image of the subject, the input image having a first resolution, and the personalized output image having a second resolution higher than the first resolution.In some embodiments, the multiple codes include a first code and a set of multiple codes, and one or more processors are further configured to select each code of the set of multiple codes using gradient descent, directly or indirectly based on the first code. In some embodiments, one or more processors are further configured to select a first code that represents the mean of a latent vector space W, where the latent vector space W represents all possible codes that can be input into the generative model. [Brief explanation of the drawing]

[0006] [Figure 1] This is a functional diagram of an exemplary system in accordance with the aspects of this disclosure. [Figure 2] This is a functional diagram of an exemplary system in accordance with the aspects of this disclosure. [Figure 3] A and B illustrate a mechanism in which images of different subjects can be generated by different codes from within the latent vector space of a generative model, in accordance with aspects of this disclosure. [Figure 4] A and B illustrate, in accordance with aspects of this disclosure, two different personalized prior distributions in the latent vector space of a generative model, and a mechanism by which different images of a given subject can be generated by points within a given personalized prior distribution. [Figure 5] An exemplary method for generating a personalized prior distribution based on a set of subject images, according to aspects of this disclosure, is provided. [Figure 6] In accordance with aspects of this disclosure, we will show how a given set of images may be used to generate the personalized prior distribution shown in Figure 4A. [Figure 7] An exemplary method for tuning a generative model following the identification of an optimization code for each image in a set of images using the method shown in Figure 5, in accordance with aspects of this disclosure, is provided. [Figure 8] An exemplary method for generating a personalized output image based on an input image and a personalized prior distribution generated by the method of Figure 5 or Figure 7 is shown, according to aspects of this disclosure. [Figure 9] An exemplary method for generating a personalized output image based on an input image and a personalized prior distribution generated by the method of Figure 5 or Figure 7 is shown, according to aspects of this disclosure. [Figure 10] This section provides a comparative explanation of how a generative model, in accordance with aspects of this disclosure, can complete four exemplary image enhancement tasks with and without using the individualized prior distribution shown in Figure 4A. [Figure 11] This section provides a comparative explanation of how a generative model, in accordance with aspects of this disclosure, can complete four exemplary image inpainting tasks with and without using the individualized prior distribution shown in Figure 4B. [Modes for carrying out the invention]

[0007] Next, the technology will be described in relation to the following exemplary systems and methods. Common reference numbers among the figures illustrated and described below are intended to identify the same features.

[0008] Exemplary System Figure 1 shows a high-level system diagram 100 of an exemplary processing system 102 for performing the method described herein. The processing system 102 may include one or more processors 104, as well as memory 106 for storing instructions 108 and data 110. The instructions 108 and data 110 may include generative models described herein (e.g., generative models 306 in Figures 3A, 3B, 4A, 4B, 6, 10 and 11). In addition, data 110 may store training examples used to train such generative models, data used by the generative model when generating images, a set of images of a given subject, personalized prior distributions based on a set of images of a given subject (e.g., personalized prior distributions 402 and 403 in Figures 4A, 4B, 6, 10 and 11), and / or images output by the generative model.

[0009] The processing system 102 may reside on a single computing device. For example, the processing system 102 may be a server, a personal computer, or a mobile device, and therefore the generative model and associated data may be local to that single computing device. Similarly, the processing system 102 may reside on a cloud computing system or other distributed system. In such a case, the generative model and / or associated data may be distributed across two or more different physical computing devices. For example, in some aspects of the present technology, the processing system may include a first computing device that stores the generative model, and a second computing device that stores data used by the generative model when generating images, a set of images of a given subject, a personalized pre-distribution based on the set of images of a given subject, and / or images output by the generative model. Similarly, in some aspects of the present technology, the processing system may include a first computing device that stores layers 1 to n of a generative model having m layers, and a second computing device that stores layers n to m of the generative model.

[0010] Furthermore, in this regard, Figure 2 shows a high-level system diagram 200. This diagram shows the exemplary processing system 102 described above communicating with various websites and / or remote storage systems, including websites 210 and 218 and a remote storage system 226, via one or more networks 208. In this example, each website 210 and 218 includes one or more servers 212a-212n and 220a-220n, respectively. Each of the servers 212a-212n and 220a-220n may have one or more processors (e.g., 214 and 222), and associated memory (e.g., 216 and 224) for storing instructions and data, including the content of one or more web pages. Similarly, although not shown, the remote storage system 226 may also include one or more processors and memory for storing instructions and data. In some aspects of this technology, the processing system 102 trains generative models or The system may be configured to retrieve data, training examples, a set of images of a given subject, and / or input images of a given subject from one or more of the websites 210, 218, and / or the remote storage system 226, so that they may be provided to the generative model for tuning and / or used when generating images.

[0011] The processing systems described herein may be implemented on any type of computing device(s), such as any type of general-purpose computing device, server, or set thereof, and may also include other components typically found in general-purpose computing devices or servers. Similarly, the memory of such a processing system may be of any non-temporary type capable of storing information accessible by the processor(s) of the processing system. For example, memory may include non-temporary media such as hard drives, memory cards, optical discs, solid-state drives, and tape memory. Computing devices suitable for the roles described herein may include different combinations of the foregoing, thereby storing different parts of instructions and data in different types of media.

[0012] In any case, the computing devices described herein may further include any other components commonly used in connection with computing devices, such as a user interface subsystem. A user interface subsystem may include one or more user inputs (e.g., a mouse, keyboard, touchscreen, and / or microphone), and one or more electronic displays (e.g., a monitor with a screen, or any other electrical device capable of displaying information). Output devices other than electronic displays, such as speakers, lights, and vibration elements, pulse elements, or tactile elements, may also be included in the computing devices described herein.

[0013] Each computing device may contain one or more processors, which may be any conventional processor, such as a commercially available central processing unit ("CPU"), graphics processing unit ("GPU"), or tensor processing unit ("TPU"). Alternatively, one or more processors may be dedicated devices, such as ASICs or other hardware-based processors. Each processor may have multiple cores capable of operating in parallel. The processors, memory, and other elements of a single computing device may be housed in a single physical housing or distributed between two or more housings. Similarly, the memory of a computing device may include hard drives or other storage media located in a different housing from the processor(s) housing, such as in an external database or networked storage device. Thus, references to processors or computing devices will be understood to include references to a collection of processors or computing devices or memories, which may or may not operate in parallel, and to one or more servers in a load-balanced server farm or cloud-based system.

[0014] The computing devices described herein may store instructions that can be executed directly by a processor(s) (such as machine code) or instructions that can be executed indirectly by a processor(s) (such as scripts). The computing devices may also store data that can be retrieved, stored, or modified by one or more processors according to the instructions. Instructions may be stored as computing device code on a computing device-readable medium. In this regard, the terms “instruction” and “program” may be used interchangeably herein. Instructions may also be stored in object code form for direct processing by a processor(s), or in any other computing device language, including scripts or sets of independent source code modules that are interpreted on demand or pre-compiled. Example The programming language may be C#, C++, Java®, or another computer programming language. Similarly, any instruction or component of a program may be implemented in a computer scripting language such as JavaScript®, PHP, ASP, or any other computer scripting language. Furthermore, any one of these components may be implemented using a combination of a computer programming language and a computer scripting language.

[0015] Exemplary Method Figures 3A and 3B illustrate how images of different subjects (308, 309) can be generated by different codes from within the latent vector space W(302) of the generative model 306, according to aspects of this disclosure. In these examples, the generative model 306 may be any suitable model configured to generate output images based on input codes, and the latent vector space W(302) represents the range of all possible input codes that can be provided to the generative model 306. In this regard, for the sake of simplicity, the latent vector space W(302) is shown as a two-dimensional space in Figures 3A and 3B (as well as Figures 4A, 4B, 6, 10, and 11). However, the technique may be applied to any suitable generative model (e.g., a GAN or a bidirectional GAN ​​("BiGAN")) configured to generate images based on input vectors of any suitable number of dimensions. For example, in some aspects of this technology, the generative model 306 may be a generative adversarial network (e.g., StyleGAN, StarGAN) configured to use a latent vector space W having 512 dimensions.

[0016] As shown in FIG. 3A, the code at point 304 in the latent vector space W(302) is supplied to the generative model 306, thereby creating an image 308. To explain the effect of the present technology, in the examples shown in FIGS. 3A, 3B, 4A, 4B, 10, and 11, respectively, images of subjects whose faces can be recognized by many people are used. Therefore, in the example of FIG. 3A, it is assumed that an image 308 of Barack Obama (the 44th President of the United States) is created by the code at point 304. Similarly, in the example of FIG. 3B, it is assumed that an image 309 of Lady Gaga (an American singer-songwriter and actress, also known as Stefani Joanne Angelina Germanotta) is created by the code at point 305.

[0017] FIGS. 4A and 4B show two different personalized prior distributions 402, 403 within the latent vector space W(302) of the generative model 306 according to aspects of the present disclosure, and further show how different images of a given subject can result from points (e.g., 404 for 304, 405 for 305) within a given personalized prior distribution. The examples of FIGS. 4A and 4B both assume the use of the same latent vector space W(302) and the same generative model 306 as used in FIGS. 3A and 3B, but further illustrate the personalized prior distributions 402, 403 for individuals within the latent vector space W(302).

[0018] The personalized prior distributions 402 and 403 each represent vector spaces within the latent vector space W(302) that contain subsets of input codes capable of creating images similar to a given subject. In this case, the personalized prior distribution 402 is assumed to represent the range of codes that create images similar to Barack Obama when supplied to the generative model 306. Therefore, the personalized prior distribution 402 includes the point 304 that represents the code that created the image 308 in FIG. 3A, and another point 404 that represents the code that created a different image 406 of Barack Obama.

[0019] Similarly, the per - individual prior distribution 403 is assumed to represent a range of codes that create an image similar to Lady Gaga when provided to the generative model 306. Thus, the per - individual prior distribution 403 includes a point 304 that represents the code to create the image 309 of FIG. 3B, and another point 405 that represents the code to create a different image 407 of Lady Gaga. It includes another point 405 that represents the code to create a different image 407 of Lady Gaga.

[0020] Again, for the purpose of simply simplifying the explanation, the per - individual prior distributions 402 and 403 are shown as being two - dimensional spaces in FIGS. 4A and 4B (as well as FIGS. 6, 10, and 11). However, since the present technology can be applied to any suitable generative model configured to generate an image based on an input vector of any suitable number of dimensions, the per - individual prior distributions 402 and 403 may similarly be vector spaces of any number of dimensions less than or equal to the number of dimensions of the latent vector space W(302). Thus, for example, if the generative model 306 is an adversarial generative network configured to use a latent vector space W having 512 dimensions, the per - individual prior distributions 402 and 403 may each be vector spaces of up to 512 dimensions. Additionally, for ease of explanation, the exemplary vector spaces 402 and 403 are shown in a completely separated state. However, in reality, the per - individual prior distributions for different subjects may intersect each other.

[0021] FIG. 5 shows an exemplary method 500 for generating a personalized prior distribution based on a set of images of a subject, according to an aspect of the present disclosure.

[0022] In step 502, the processing system (e.g., processing system 102) selects a given image from a set of images of the subject. It will be understood that, generally, a larger number of images will result in a more typical personal prior distribution than a smaller number of images. In this regard, it has been found that generally, with 100 to 200 images, a generative model can create an image that is realistic and appears to be consistent with the subject's identity. However, other aspects can also affect how well the personal prior distribution represents the subject's appearance. For example, if the subject's appearance has changed significantly (e.g., a change in hairstyle, hair color, addition or removal of facial hair, or due to the passage of time), it may be useful to tailor the set of images to a specific aspect so that the personal prior distribution reflects a single "appearance" and the image created by the generative model is consistent with that appearance. Similarly, for the same reason, it may be useful to limit the set of images to a single stage of life (e.g., infancy, childhood, adolescence, adulthood, etc.). On the other hand, diversity in the set of images may also be important. For example, a set of images showing a subject's face only from the front may not be as useful as a set of images showing the subject's face from various different angles and under various lighting conditions. Generating a personalized prior distribution from images that are too similar can over-limit the generative model, potentially leading to the creation of images that resemble the subject but are not necessarily photorealistic.

[0023] As shown in step 504, after selecting a given image, the processing system tests multiple codes by repeatedly running steps 506-514 to identify an optimization code for the given image. In this regard, in step 506, the processing system identifies the code under test in the current pass. This code may be identified based on any appropriate selection criteria. For example, in some embodiments of the art, the processing system may be configured to select a first code blindly (e.g., using a random selection process or a pre-selected value such as the mean of a latent vector space W), and then, using an appropriate optimization regime, select each set of codes (in each set of passes through step 506) directly or indirectly based on that first code. Thus, in some embodiments, the processing system may be configured to select each set of codes using gradient descent based on a preceding code and an evaluation of how well the image generated based on the preceding code matched the given image (e.g., a first loss value generated in the most recent pass through step 510).

[0024] In step 508, the processing system generates a generative model (e.g., generative model 306) and ( The first image is generated using the code under test (identified in this path through step 506). The generation model may be configured to generate the first image using any suitable method, including those described above with respect to Figures 3A, 3B, 4A, and 4B.

[0025] In step 510, the processing system compares the first image (generated in this pass through step 508) with a given image (selected in step 502) to generate a first loss value for the code under test. The first loss value can be generated using any suitable function and any suitable method. For example, in some aspects of the art, the first loss value may be based on a comparison of the first image with the given image using a heuristic or trained similarity metric (e.g., a trained perceptual image patch similarity score ("LPIPS"), peak signal-to-noise ratio ("PSNR"), structural similarity index ("SSIM"), L1 or L2 loss, etc.).

[0026] In step 512, the processing system determines whether another code should be tested. Since it is assumed that there are multiple codes to be tested, when the processing system first reaches step 512, it automatically follows the "yes" arrow back to step 506. This identifies and tests a second code. However, each subsequent return to step 512 may determine whether to test another code based on any appropriate criteria. Thus, as described above, in some aspects of the technology, the processing system may determine when to stop testing another code based on an appropriate optimization regime, such as gradient descent. In such cases, the determination in step 512 may be based on a comparison between a first loss value generated in the current pass through step 510 (or some other evaluation of how well the first image matches a given image) and one or more first loss values ​​generated in preceding passes. For example, the processing system may be configured to stop testing a series of codes if the first loss value generated in the current pass through step 510 is greater than or equal to a first loss value generated in a pass prior to step 510.

[0027] Therefore, the processing system repeats steps 506-512 with each subsequent code until it is determined in step 512 that a sufficient number of codes have been tested. Once it is determined that a sufficient number of codes have been tested, the processing system proceeds to step 514 by following the "No" arrow, where it compares the first loss values ​​generated (in step 510) for each of the multiple codes to identify the code with the smallest first loss value. The code with the smallest first loss value is selected as the optimization code for the given image.

[0028] Next, in step 516, the processing system determines whether there are any further images in the set of subject images. If there are, the processing system proceeds to step 518 following the "yes" arrow. There, the processing system selects the next given image to be tested. The processing system then returns to step 504 and tests multiple codes to identify an optimization code for this new given image. Steps 504-518 are repeated in this manner until an optimization code is identified for every image in the set of subject images. Once an optimization code has been selected for the last image in the set of subject images (in step 514), the processing system determines in step 516 that there are no further images in the set and therefore proceeds to step 520 following the "no" arrow.

[0029] In step 520, the processing system generates a personalized prior distribution for each given image in the set of images of the subject based on a convex hull containing each optimization code identified (in step 514). In this regard, in some aspects of the art, the personalized prior distribution is simply each optimization code identified in step 514. It may also be a convex hull defined by . In such a case, n optimization codes {x1, x2, ..., x nAssuming a set of} exists, the personalized prior distribution contains any code c generated by linearly combining the optimization codes by the following equations 1-3. In the equations, the coefficients (alpha values ​​α1-α) n Each of the coefficients is greater than or equal to 0, and the sum of all coefficients is 1.

[0030]

number

[0031] Similarly, in some embodiments of the Art, the personalized prior distribution may be a set of a predetermined number of codes or coefficient sets (e.g., 100, 500, 1,000, 10,000, 100,000, 1,000,000) corresponding to a subset of the convex hull defined by each optimization code identified in step 514, for example, a set of sampled points within the convex hull. Furthermore, in some embodiments of the Art, the personalized prior distribution may be a simpler hull (e.g., a hull with fewer vertices) that fits within or substantially overlaps with the actual convex hull defined by each optimization code identified in step 514. In addition, in some embodiments of the Art, the personalized prior distribution may be based on the convex hull defined by equations 1-3 above by encompassing a broader set of codes than those defined by equations 1-3 above. For example, the personalized prior distribution may be a set of coefficients (α values ​​α1-α n ) may include any code c generated by equations 1 and 3 above, wherein the value is greater than or equal to a predetermined negative value (e.g., -0.01, -0.05, -0.1).

[0032] Figure 6 illustrates how, according to aspects of this disclosure, a given set of n images 602a–612a may be used to generate the personalized prior distribution 402 in Figure 4A. In this regard, each image 602a–612a is assumed to be one of a set of n images of a subject, and the illustrated lines show how these six selected images correspond to different optimization codes 602b–612b in the latent vector space W(302) of a generative model (e.g., generative model 306). Each of these optimization codes 602b–612b may be found by steps 502–518 of the exemplary method in Figure 5, as further described above. In addition, exemplary Figure 600 in Figure 6 illustrates how each of these optimization codes 602b–612b (along with the optimization codes for the rest of the set of n images, which are not illustrated) may be used to define the underlying convex hull of the personalized prior distribution 402, as further described with respect to step 520 in Figure 5. In this case as well, for the sake of simplicity, the individual prior distribution 403 in Figure 6 is shown as a two-dimensional space, and only six sample images out of a set of n images are shown. However, the individual prior distribution 403 may be based on any appropriate number of points, and as explained above, it may be a vector space of any appropriate number of dimensions less than or equal to the number of dimensions of the latent vector space W(302).

[0033] Figure 7 shows an exemplary method 700 for tuning a generative model following the identification of an optimization code for each image in a set of images by the method of Figure 5, in accordance with an aspect of the present disclosure. In this respect, exemplary method 700 identifies an optimization code for each image in a set of images ( This represents a process that can be optionally executed after being identified (in step 514 of Figure 5).

[0034] Therefore, in step 702, it is assumed that the processing system (e.g., processing system 102) performs at least steps 502–518 of the exemplary method of Figure 5 for each image in the set of images. Although the exemplary method of Figure 7 shows that steps 704–720 are performed after step 702, it will be understood that steps 704–720 can be performed in any suitable order with respect to steps 502–518 of the exemplary method of Figure 5. For example, in some aspects of the art, the processing system may be configured to identify an optimization code for a given image in the set of images (in step 514 of Figure 5), and then perform steps 704–712 for that given image, either before selecting the next given image (in step 518 of Figure 5), or in parallel with testing multiple codes to identify an optimization code for the next given image (e.g., steps 504–514 of Figure 5). In such cases, the processing system may further be configured to periodically update the parameters of the generative model while continuously identifying the optimization code (in a series of passes through step 518) by executing step 714 in parallel with steps 504-518 in Figure 5 (after each batch of images has been processed by steps 704-712).

[0035] Regardless of timing, in step 704, the processing system selects a given image from a set of images of the subject. This set of images may be the entire set of images used in Figure 5, or any suitable subset thereof. For example, in some aspects of the art, the processing system may be configured to identify an optimization code for a set of 200 images of the subject by steps 502-518 of Figure 5, but then to adjust the generative model (e.g., generative model 306) based on only 100 of those images.

[0036] In step 706, the processing system generates a second image using the generative model and the optimization code identified for the given image (in step 514 in Figure 5). Here again, the generative model may be configured to create the second image by any suitable method, including those described above with respect to Figures 3A, 3B, 4A, and 4B. In addition, although step 706 refers to a “second image,” it will be understood that this second image may, in some cases, be a copy of one of the “first images” generated in step 508 in Figure 5. Thus, in some aspects of the art, for any “given image” in the first batch of images, the processing system may be configured to use the “first image” (or a copy thereof) associated with the optimization code identified in step 514 in Figure 5 as the “second image” in step 706, rather than generating the image again in step 706. However, if the processing system modifies one or more parameters of the generative model in step 714 (described later), the generative model is modified so that each subsequent optimization code produces a "second image" that is different from the "second image" that the code is thought to have previously produced. Therefore, if method 700 is executed after method 500 (but not in parallel with some or all of the steps of method 500), the processing system may be configured to generate a new "second image" in each set of passes through step 706 (rather than using a copy of the "first image" identified in step 508 of Figure 5).

[0037] In step 708, the processing system generates a second loss value by comparing the second image (generated in step 706) with a given image (selected in step 704). Here again, the second loss value can be generated using any suitable function and any suitable method. For example, in some aspects of the art, the second loss value is generated using heuristic or trained similarity metrics (e.g., trained perceptual image patch similarity ("LPIPS"), peak signal-to-noise ratio ("PSNR"), structural similarity index ("SSIM")). The process may also be based on a comparison between the second image and the given image. In addition, although step 708 refers to a “second loss value,” it will be understood that this second loss value may, in some cases, be a copy of the “minimum first loss value” identified in step 514 of Figure 5 for that image. Thus, in some aspects of this technology, for any “given image” in the first batch of images, the processing system may be configured to use the “minimum first loss value” identified in step 514 of Figure 5 for that given image as the “second loss value” in step 708, rather than generating a new loss value in step 708. However, as already mentioned, if the processing system modifies one or more parameters of the generative model in step 714 (described later), the generative model is modified so that each subsequent optimization code produces a “second image” that is different from the “second image” that the code is thought to have previously produced, and therefore produces a different “second loss value” when compared to the “given image.” Therefore, if method 700 is performed after method 500 (but not in parallel with some or all of the steps of method 500), the processing system may be configured to generate a new "second loss value" in each set of passes through step 708 (rather than using a copy of the "minimum first loss value" identified in step 514 of Figure 5).

[0038] In step 710, the processing system determines whether there are any further images in the batch. In this regard, the set of images may be kept as a whole or divided into any appropriate number of batches. If the set of images is not divided and therefore there is a single "batch" containing all the images in the "set of images" of the subject, the processing system proceeds to step 712 following the "yes" arrow to select the next given image from the set of images of the subject and repeats steps 706-710 for the newly selected image. This process is repeated until there are no more images in the set of images. At this point, the processing system proceeds to step 714 following the "no" arrow. On the other hand, if the set of images is divided into two or more batches (for example, a set of 200 images may be divided into two batches of 100 images each, four batches of 50 images each, ten batches of 20 images each, two batches of 20 single images each, etc.), steps 704-712 are repeated for each image until the end of the batch is reached.

[0039] As shown in step 714, after a “second loss value” has been generated for every image in a batch (in step 708), the processing system modifies one or more parameters of the generative model based at least in part on each generated second loss value. The processing system may be configured to modify one or more parameters based on these generated second loss values ​​by any suitable method and at any suitable interval. Thus, in some aspects of the art, each “batch” may contain a single image such that the processing system performs a backpropagation step to modify one or more parameters of the generative model each time a second loss value is generated. Similarly, if each “batch” contains two or more images, the processing system may be configured to sum each of the “second loss values” generated for each image in that batch (in step 708) to a total loss value (for example, by summing or averaging multiple second loss values) and modify one or more parameters of the generative model based on that total loss value.

[0040] In step 716, the processing system determines whether there are further batches within the set of images of the subject. If the set of images is not divided and therefore there is a single batch containing all the images in the "set of images" of the subject, the determination in step 716 is automatically "no", and then method 700 terminates as shown in step 720. However, if the set of images is divided into two or more batches, the processing system proceeds to step 718 following the "yes" arrow to determine whether there are further batches within the set of images of the subject. Select the next given image from the set. This starts another set of passes through steps 706-714 for each image in the next batch of images, and the process continues until there are no more batches. When there are no more batches, the processing system proceeds to step 720 following the "No" arrow.

[0041] Although Method 700 is shown to terminate in step 720 once all images have been used to tune the generative model, it will be understood that Method 700 can be repeated any appropriate number of times using the same set of images until the output of the generative model for each optimization code produces an image that is sufficiently close to each given image. In this regard, in some embodiments of the Art, the processing system may be configured to aggregate all the second loss values ​​generated during a given pass through Method 700 and determine whether to repeat Method 700 for a set of images based on the aggregated loss values. For example, in some embodiments of the Art, the processing system may be configured to repeat Method 700 for a set of images if the total loss value for the most recent pass through Method 700 is greater than a certain threshold. Similarly, in some embodiments, the processing system may use gradient descent to make this determination and thus be configured to repeat Method 700 for a set of images until the total loss value in a given pass through Method 700 is greater than or equal to the total loss value from the previous pass.

[0042] Figure 8 shows an exemplary method 800 for generating a personalized output image based on an input image and a personalized prior distribution generated by the method of Figure 5 or Figure 7, according to an aspect of the present disclosure. In this regard, exemplary method 800 represents a process that may be optionally performed after at least a personalized prior distribution has been generated (in step 520 of Figure 5) and after the generative model has been further refined by the method of Figure 7. Thus, in step 802, it is assumed that a processing system (e.g., processing system 102) performs at least the method 500 of Figure 5 for each image in the set of images and optionally performs steps 704-720 of Figure 7.

[0043] As described above, the processing system may generate a personalized prior distribution (using method 500 in Figure 5), and optionally tune the generation model (using method 700 in Figure 7), and then use the personalized prior distribution to generate different candidate coefficient sets, and then use the codes corresponding to those candidate coefficient sets to generate candidate images for a given image enhancement task. Thus, as shown in step 804, for a particular input image of a subject, the processing system tests multiple coefficient sets by repeatedly performing steps 806-812 to identify a personalized output image, where each coefficient set in the multiple coefficient sets corresponds to a code in the convex hull (identified in step 520 in Figure 5).

[0044] In each pass through step 806, the processing system identifies a set of coefficients from among several sets of coefficients and uses it to generate a given code. This set of coefficients may be identified based on any appropriate selection criteria. For example, in some embodiments of the art, the processing system may be configured to blindly select a first set of coefficients (e.g., using a random selection process or pre-selected values ​​such as the mean of a vector space represented by a personalized prior distribution), and then, using an appropriate optimization regime, select each set of coefficients (in each set of passes through step 806) directly or indirectly based on that first set of coefficients. Thus, in some embodiments, the processing system may be configured to use gradient descent to select each set of coefficients based on a preceding set of coefficients and an evaluation of how well the image generated based on the preceding set of coefficients matched the input image (e.g., a third loss value generated in the most recent pass through step 810).

[0045] In step 808, the processing system generates a third image using a generative model (e.g., generative model 306, or a refined generative model obtained from one or more passes through steps 704-720 in Figure 7) and a given code (generated in this pass through step 806). Again, the generative model may be configured to produce the third image by any suitable method, including those described above with respect to Figures 3A, 3B, 4A, and 4B.

[0046] In step 810, the processing system compares the third image (generated in this pass through step 808) with the input image of the subject to generate a third loss value for the third image. The third loss value can be generated using any suitable function and any suitable method. For example, in some aspects of the art, the third loss value may be based on a comparison of the third image with the input image using a heuristic or trained similarity metric (e.g., trained perceptual image patch similarity ("LPIPS"), peak signal-to-noise ratio ("PSNR"), structural similarity index ("SSIM").

[0047] In step 812, the processing system determines whether to test another set of coefficients. Since it is assumed that there are multiple sets of coefficients to be tested, when the processing system first reaches step 812, it automatically follows the "yes" arrow back to step 806. This identifies and tests a second set of coefficients. However, each subsequent return to step 812 may determine whether to test another set of coefficients based on any appropriate criteria. Thus, as described above, in some aspects of the art, the processing system may determine when to stop testing another set of coefficients based on an appropriate optimization regime such as gradient descent. In such cases, the decision in step 812 may be based on a comparison of a third loss value generated in the current pass through step 810 (or some other evaluation of how well the third image matches the input image) with one or more third loss values ​​generated in preceding passes. For example, the processing system may be configured to stop testing a set of coefficients if the third loss value generated in the current path passing through step 810 is greater than or equal to the third loss value generated in a previous path passing through step 810.

[0048] Therefore, the processing system repeats steps 806-812 with each of the next sets of coefficients until it is determined in step 812 that a sufficient set of coefficients has been tested. Once it is determined that a sufficient set of coefficients has been tested, the processing system proceeds to step 814 following the "No" arrow. There, the processing system compares the third loss values ​​generated (in step 810) for each third image to identify the third image with the smallest third loss value. The third image with the smallest third loss value is used as the personalized output image. Thus, a personalized output image can be created using method 800. This personalized output image is generated using codes within the personalized prior distribution (and therefore the output image is more likely to resemble the subject than an output image if the codes were not so limited) and is optimized to closely match the input image (and therefore, when performing an image editing or enhancement task, it is ensured that the image created by the model is still consistent with the image that could have been obtained from the input image).

[0049] Figure 9 shows another exemplary method 900 for generating a personalized output image based on an input image and a personalized prior distribution generated by the method of Figure 5 or Figure 7, according to an aspect of the present disclosure. In this respect, exemplary method 900 also includes at least the generation of a personalized prior distribution (in step 520 of Figure 5) This represents a process that can be performed at will and may also be performed after the generative model has been further refined by the method in Figure 7. The only difference between method 800 in Figure 8 and method 900 in Figure 9 is that a given set of codes is tested in each pass through steps 906-912. Here, each individual code in the given set of codes lies in the convex hull identified in step 520 of Figure 5.

[0050] Therefore, as described above, in step 902, it is assumed that the processing system (e.g., processing system 102) performs at least method 500 in Figure 5 for each image in the set of images, and optionally performs steps 704-720 in Figure 7. Similarly, for a particular input image of a subject, as shown in step 904, the processing system tests multiple code sets by repeatedly performing steps 906-912 to identify a personalized output image, where each code in the code set is in the convex hull (identified in step 520 in Figure 5).

[0051] In each pass through step 906, the processing system identifies a given set of codes containing two or more individual codes. Again, this given set of codes may be identified based on any appropriate selection criteria. For example, in some embodiments of the art, the processing system may be configured to blindly select a first given set of codes (e.g., by using a random selection process or by assigning a pre-selected value, such as the mean of a vector space represented by a personalized prior distribution, to each individual code), and then, using an appropriate optimization regime, select each set of codes (in each set of passes through step 906) directly or indirectly based on that first given set of codes. Thus, in some embodiments, the processing system may be configured to use gradient descent to select each set of codes based on a preceding set of codes and an evaluation of how well the image generated based on the preceding set of codes matched the input image (e.g., a third loss value generated in the most recent pass through step 910).

[0052] In step 908, the processing system generates a third image using a generative model (e.g., generative model 306, or a refined generative model obtained from one or more passes through steps 704-720 in Figure 7) and a given set of codes (generated in this pass through step 906). In this regard, in step 908, the processing system provides each individual code of the given set of codes to a different layer or set of layers of the generative model. Again, the generative model may be configured to produce the third image by any suitable method, including those described above with respect to Figures 3A, 3B, 4A, and 4B.

[0053] In step 910, the processing system compares the third image (generated in this pass through step 908) with the input image of the subject to generate a third loss value for the third image. Again, the third loss value can be generated using any suitable function and any suitable method. For example, in some aspects of the art, the third loss value may be based on a comparison of the third image with the input image using a heuristic or trained similarity metric (e.g., trained perceptual image patch similarity ("LPIPS"), peak signal-to-noise ratio ("PSNR"), structural similarity index ("SSIM").

[0054] In step 912, the processing system determines whether another set of code should be tested. Since it is assumed that there are multiple sets of code to be tested, the processing system automatically returns to step 906 by following the "yes" arrow the first time it reaches step 912. This identifies and tests a second set of code. However, each subsequent return to step 912 may determine whether to test another set of code based on any appropriate criteria. Thus, as described above... In some aspects of this technology, the processing system may determine when to stop testing another set of codes based on an appropriate optimization regime, such as gradient descent. In such cases, the determination in step 912 may be based on a comparison between a third loss value generated in the current pass through step 910 (or some other evaluation of how well the third image matches the input image) and one or more third loss values ​​generated in preceding passes. For example, the processing system may be configured to stop testing a set of codes if the third loss value generated in the current pass through step 910 is greater than or equal to a third loss value generated in a pass prior to step 910.

[0055] Therefore, similar to method 800, the processing system repeats steps 906-912 with each of the next code sets until it is determined in step 912 that a sufficient number of code sets have been tested. Once it is determined that a sufficient number of code sets have been tested, the processing system proceeds to step 914 following the "no" arrow. There, the processing system compares the third loss values ​​generated (in step 910) for each third image to identify the third image having the smallest third loss value. Again, the third image identified as having the smallest third loss value is used as the personalized output image.

[0056] Figure 10 is a comparative illustration showing how a generative model (e.g., generative model 306) according to an aspect of this disclosure can complete four exemplary image enhancement tasks with and without using the personalized prior distribution 402 of Figure 4A.

[0057] In Figure 10, a row of four input images 1002a to 1002d is shown in the center of Figure 1000. Since the generative model is assumed to be tasked with creating output images that are similar to the input images but with improved resolution, each of the input images 1002a to 1002d is blurry.

[0058] Images 1004a–1004d in the left column show the potential outputs of the generative model when it is not limited to selecting a code within any particular part of the latent vector space W(302). In this regard, the dashed lines connect each input image to a point in the latent vector space W(302) that represents the code ultimately selected by the generative model (after a selection process such as the process described above in Method 800 in Figure 8 or Method 900 in Figure 9), and the arrows connect that point to the corresponding output image created by the generative model based on that code. As can be seen, although output images 1004a–1004d are visually consistent with their respective input images 1002a–1002d, these output images do not all appear to be of the same subject.

[0059] In contrast, images 1006a–1006d in the right column show the potential output of the generative model when limited to selecting codes within the individual-specific prior distribution 402 in Figure 4A. Here again, the dashed lines connect each input image to a point in the individual-specific prior distribution 402 that represents the code ultimately selected by the generative model (after a selection process such as the process described above for method 800 in Figure 8 or method 900 in Figure 9), and the arrows connect that point to the corresponding output image created by the generative model based on that code. As can be seen, this ultimately produces the output images 1006a–1006d. These output images are visually consistent with each input image 1002a–1002d and all appear to be of the same subject. Specifically, since the personalized prior distribution 402 in Figure 4A represents the range of code that, when provided to the generative model 306 (as described above), creates an image similar to Barack Obama, each of the output images 1006a to 1006d appears to show an image of Barack Obama that is visually consistent with the input images 1002a to 1002d.

[0060] Figure 11 is a comparative illustration showing how a generative model (e.g., generative model 306) according to an aspect of this disclosure may complete four exemplary image inpainting tasks with and without using the personalized prior distribution 403 of Figure 4B.

[0061] Here again, a row of four input images 1102a to 1102d is shown in the center of Figure 1000. The generative model is assumed to be tasked with creating an output image that is similar to the input image but fills in the masked areas; therefore, each of the input images 1102a to 1102d contains a black mask 1103a to 1103d representing the pixels to be replaced.

[0062] In Figure 1100, images 1104a–1104d in the left column show the potential outputs of the generative model when it is not limited to selecting a code within any particular portion of the latent vector space W(302). Here again, the dashed lines connect each input image to a point in the latent vector space W(302) that represents the code ultimately selected by the generative model (after a selection process such as the process described above for method 800 in Figure 8 or method 900 in Figure 9), and the arrows connect that point to the corresponding output image created by the generative model based on that code. As can be seen, although the output images 1104a–1104d are visually consistent with the unmasked portion of each input image 1102a–1102d, these output images do not all appear to be of the same subject.

[0063] In contrast, images 1106a–1106d in the right column show the potential output of the generative model when limited to selecting codes within the individual-specific prior distribution 403 in Figure 4B. Here again, the dashed lines connect each input image to a point in the individual-specific prior distribution 403 that represents the code ultimately selected by the generative model (after a selection process such as the process described above for method 800 in Figure 8 or method 900 in Figure 9), and the arrows connect that point to the corresponding output image that the generative model creates based on that code. As can be seen, this ultimately produces the output images 1106a–1106d. These output images are visually consistent with each input image 1102a–1102d and all appear to be of the same subject. Specifically, since the personalized prior distribution 403 in Figure 4B represents the range of code that, when provided to the generative model 306 (as described above), creates an image similar to Lady Gaga, each of the output images 1106a to 1106d appears to show an image of Lady Gaga that is visually consistent with the unmasked portion of the input images 1102a to 1102d.

[0064] Therefore, Figures 1000 and 1100 both focus on the code used by the generative model and illustrate how a personalized prior distribution can be used to enable the generative model to produce an output image that is visually consistent with both the input image and the identity of a particular subject. Thus, if the subject of the input image is already known, a personalized prior distribution may be selected and used so that the generative model is biased towards producing a more typical, and therefore more appropriate, output image.

[0065] Unless otherwise stated, the aforementioned alternatives are not mutually exclusive and may be implemented in various combinations to achieve specific advantages. Since these and other variations and combinations of the features described above can be used without departing from the subject matter defined by the claims, the foregoing descriptions of exemplary systems and methods should be considered illustrative rather than limiting the subject matter defined by the claims. Furthermore, the provision of examples described herein, and phrases such as “~etc,” “including,” and “comprising,” should not be interpreted as limiting the subject matter of the claims to specific examples; rather, the examples are intended to illustrate only a portion of the many possible embodiments. Additionally, the same reference numeral in different drawings may be used for... This allows for the identification of identical or similar elements.

Claims

1. A method performed by a computer, For each given image in the set of subject images, The method includes generating a loss value for each code of a plurality of codes associated with a given image using one or more processors, the generation of the loss value includes comparing each given image with its respective generated image using each code, and the method further includes Using one or more processors, compare the loss values ​​generated for each of the plurality of codes to identify the optimization code for the given image that has the smallest loss value. A method comprising using one or more processors to generate a personalized prior distribution for the subject, which includes each of the optimization codes identified for each given image in the set of images of the subject.

2. For each given image in the set of images of the subject, the optimization code identified: Using the generative model and the optimization code, generate the selected image, The method according to claim 1, further comprising using one or more processors to compare the selected image with a given image to generate a second loss value.

3. The method according to claim 2, further comprising using one or more processors to modify one or more parameters of the generative model based at least partially on each of the generated second loss values ​​to construct an adjusted generative model.

4. The method further includes identifying a plurality of coefficient sets using one or more processors, wherein each coefficient set of the plurality of coefficient sets corresponds to a code in the convex hull, and the method For each given set of coefficients in the plurality of coefficient sets, Using one or more of the aforementioned processors, generate another image using the adjusted generative model and a given code corresponding to the given set of coefficients, Using one or more processors, the process involves comparing the other image with at least a portion of the input image of the subject to generate a different loss value for the other image, The method according to claim 3, further comprising using one or more processors to compare the other loss values ​​generated for another image and identifying the other image having the smallest loss value as a personalized output image.

5. The method according to claim 4, wherein the input image has a first resolution, and the personalized output image has a second resolution higher than the first resolution.

6. The method further includes identifying a plurality of coefficient sets using one or more processors, wherein each coefficient set of the plurality of coefficient sets corresponds to a code in the convex hull, and the method For each given set of coefficients in the plurality of coefficient sets, Using one or more of the aforementioned processors, generate a new image according to a given code corresponding to the given set of coefficients, Using one or more of the aforementioned processors, the new image is compared with at least a portion of the input image of the subject to generate a loss value for the new image, The method according to any one of claims 1 to 3, further comprising identifying one of the new images having the smallest loss value as a personalized output image.

7. The method further includes identifying a plurality of code sets using one or more processors, each code set of the plurality of code sets comprising two or more individual codes, each individual code corresponding to a set of coefficients among a plurality of coefficient sets, and the method The method according to any one of claims 1 to 3, further comprising identifying the generated image having the minimum loss value as a personalized output image.

8. Before generating the loss value for each of the aforementioned multiple codes, The process further includes generating a first image using the generative model and the code, The method according to claim 1, wherein generating the loss value includes comparing the first image with a given image using one or more processors.

9. Memory for storing the generative model, The system comprises one or more processors coupled to the memory and configured to generate personalized prior distributions for subjects for use with the generative model, wherein generating the personalized prior distributions is: For each given image in the set of images of the subject, The process includes generating a loss value for each of the multiple codes associated with the given image, the generation of the loss value includes comparing each given image with its respective generated image using each code, and the generation of the personalized prior distribution further includes The process involves comparing the loss values ​​generated for each of the aforementioned plurality of codes to identify the optimization code for the given image that has the smallest loss value, A processing system that generates the personalized prior distribution for the subject, which includes each of the optimization codes identified for each of the set of images of the subject.

10. For each given image in the set of images of the subject, the optimization code identified is used by one or more processors: Using the aforementioned generation model and optimization code, the selected image is generated. The processing system according to claim 9, further configured to generate a second loss value by comparing the selected image with the given image.

11. The processing system according to claim 10, wherein the one or more processors are further configured to construct a modified generative model by modifying one or more parameters of the generative model based at least in part on each second loss value generated.

12. The one or more processors described above are: Further configured to identify multiple sets of coefficients, each of the multiple sets of coefficients corresponds to a code in the convex hull, and the one or more processors further, For each given set of coefficients in the plurality of coefficient sets, Another image is generated using the adjusted generative model and the given code corresponding to the given set of coefficients. The aforementioned alternative image is compared with at least a portion of the input image of the subject to generate another loss value for the alternative image. The processing system according to claim 11, configured to compare the other loss values ​​generated for another image with the other image having the smallest loss value and to identify the other image having the smallest loss value as a personalized output image.

13. The processing system according to claim 12, wherein the input image of the subject includes a first portion of pixels saved from the original image of the subject and a mask that replaces the second portion of pixels from the original image of the subject.

14. A method performed by a computer, The method includes, for each image in a set of subject images, identifying a given code from a set of codes in a vector space using one or more processors, wherein the identification of the given code includes comparing each image in the set of images with its respective generated image using each code in the set of codes, wherein the given code has the smallest loss value, and each code in the set of codes corresponds to one of the subject images, and the method further includes, A method comprising using one or more processors to generate a personalized prior distribution for the subject, wherein the personalized prior distribution corresponds to the given code for each image in the set of images.

15. The method according to claim 14, wherein the personalized prior distribution is based on a convex hull defined by the given code for each image in the set of images.

16. The method according to claim 14, wherein the personalized prior distribution corresponds to a subset of elements of the convex hull defined by the given code for each image in the set of images.

17. The method according to any one of claims 14 to 16, further comprising generating a set of candidate images for a given image enhancement task using the personalized prior distribution.

18. Generating the aforementioned set of candidate images means This includes generating different sets of candidate coefficients, each of which has a corresponding code, and further includes generating a set of candidate images. The method according to claim 17, comprising generating the set of candidate images using the corresponding code.

19. To correspond to a specific stage of life associated with the subject, or The method according to any one of claims 14 to 18, further comprising aligning the set of images to correspond to a specific appearance associated with the subject.

20. The method according to claim 19, wherein the specific appearance corresponds to at least one of hairstyle, hair color, having a facial beard, or not having a facial beard.

21. Memory for storing the generative model, The system comprises one or more processors coupled to the memory, and the one or more processors are For each image in a set of subject images, the system is configured to identify a given code from a set of codes in a vector space, and identifying the given code involves using each code in the set of codes to compare each image in the set of images with its respective generated image, the given code having the smallest loss value, and each code in the set of codes corresponds to one of the subject images, and the one or more processors further, A processing system configured to use the generative model to generate a personalized prior distribution for the subject, wherein the personalized prior distribution corresponds to a given code for each image in the set of images.

22. The processing system according to claim 21, wherein the personalized prior distribution is based on a convex hull defined by the given code for each image in the set of images.

23. The processing system according to claim 21, wherein the personalized prior distribution corresponds to a subset of elements of the convex hull defined by the given code for each image in the set of images.

24. The processing system according to any one of claims 21 to 23, wherein one or more processors are further configured to generate a set of candidate images for a given image enhancement task using the personalized prior distribution.

25. The one or more processors described above are: To correspond to a specific stage of life associated with the subject, or The processing system according to any one of claims 21 to 24, further configured to match the set of images to correspond to a specific appearance associated with the subject.

Images