Facial Image Editing and Enhancement Using Personalized Priors

Abstract

Systems and methods for identifying personalized priors in a latent vector space of a generative model based on a set of images of a given object. In some examples, the techniques may further include using the personalized priors to restrict the input of the generative model to a latent vector space associated with the given object, such that when the model is tasked with editing the image of the object (e.g., to perform inpainting to fill in masked regions, to improve resolution, or to sharpen the image), the images it creates reflect the identifying features of the object.

Description

[Background technology]

[0001] Generative models can be used to perform tasks ranging from editing and enhancing images of a given object to generating photorealistic images (or portions of images) of a given object or synthetically generated objects. To adequately train such a model, a large set of images of many objects is generally required. As a result, using a generative model to edit, enhance, or fill in portions of an image of a known object may produce an image that appears photorealistic but resembles a different object. Summary of the Invention

[0002] The present technology relates to a system and method for identifying a personalized prior distribution in a latent vector space of a generative model based on a set of images of a given object. In some aspects, the present technology may further include using the personalized prior distribution (e.g., a convex hull defined by a set of codes generated based on a set of images of the object) to restrict the codes input to the generative model to reflect the identifying features of the object in the images that the model creates. For example, the generative model may be configured to highlight or fill in facial features in images of the object where cues related to the identification of the object are only partially present (e.g., due to motion blur, low light, low resolution, occlusion by other objects). In the absence of the present technology, the model may be able to successfully highlight or fill in such images, but may do so by creating images that appear to be of a different object. By using the present technology to focus the generative model, the images that the generative model creates may be more consistent with the appearance of the object.

[0003] In one aspect, the disclosure describes a computer-implemented method that includes: (1) for each given image of a set of images of a subject, using one or more processors of a processing system, testing a plurality of codes to identify an optimized code for the given image, the testing including: (a) for each code of the plurality of codes, generating a first image using a generative model and the code, and comparing, using the one or more processors, the first image to the given image to generate a first loss value for the code; and (b) using the one or more processors, comparing the generated first loss values ​​for each code of the plurality of codes to identify a code having a minimum first loss value as the optimized code for the given image; and (2) using the one or more processors, generating a personalized prior distribution for the subject based on a convex hull that includes each optimized code identified for each given image of the set of images of the subject. In some aspects, the method further includes: (1) for each given image of the set of images of the object, generating a second image using the generative model and the optimization code, and comparing, using one or more processors, the second image to the given image to generate a second loss value; and (2) using the one or more processors to modify one or more parameters of the generative model based at least in part on each generated second loss value to construct an adjusted generative model.In some aspects, the method further includes: (1) identifying a plurality of coefficient sets using one or more processors, where each coefficient set of the plurality of coefficient sets corresponds to a code in the convex hull; (2) for each given coefficient set of the plurality of coefficient sets, generating a third image using the one or more processors using the adjusted generative model and a given code corresponding to the given coefficient set; and comparing the third image to at least a portion of the input image of the subject to generate a third loss value for the third image, using the one or more processors; and (3) comparing the generated third loss values ​​for each third image, using the one or more processors, to identify the third image having the smallest third loss value as the personalized output image. In some aspects, the method further includes: (1) identifying a plurality of coefficient sets using one or more processors, where each coefficient set of the plurality of coefficient sets corresponds to a code in the convex hull; (2) identifying a plurality of code sets using one or more processors, where each code set of the plurality of code sets includes two or more individual codes, where each individual code corresponds to a coefficient set of the plurality of coefficient sets; (3) for each given code set of the plurality of codes, generating a third image using the adjusted generative model and the given code set using one or more processors, where each individual code of the given code set is provided to a different layer or set of layers of the adjusted generative model; and comparing the third image to at least a portion of the input image of the subject to generate a third loss value for the third image using the one or more processors; and (4) comparing the generated third loss values ​​for each third image using the one or more processors to identify the third image having the smallest third loss value as the personalized output image.In some aspects, the method further includes: (1) identifying a plurality of coefficient sets using one or more processors, where each coefficient set of the plurality of coefficient sets corresponds to a code in the convex hull; (2) for each given coefficient set of the plurality of coefficient sets, generating a third image using the generative model and a given code corresponding to the given coefficient set, using the one or more processors; comparing the third image to at least a portion of the input image of the subject to generate a third loss value for the third image, using the one or more processors; and (3) comparing the generated third loss values ​​for each third image, using the one or more processors, to identify the third image having the smallest third loss value as the personalized output image. In some aspects, the method further includes: (1) identifying a plurality of coefficient sets using one or more processors, where each coefficient set of the plurality of coefficient sets corresponds to a code in the convex hull; (2) identifying a plurality of code sets using one or more processors, where each code set of the plurality of code sets includes two or more individual codes, where each individual code corresponds to a coefficient set of the plurality of coefficient sets; (3) for each given code set of the plurality of codes, generating a third image using the generative model and the given code set using the one or more processors, where each individual code of the given code set is provided to a different layer or set of layers of the generative model; and comparing the third image to at least a portion of the input image of the subject to generate a third loss value for the third image using the one or more processors; and (4) comparing the generated third loss values ​​for each third image using the one or more processors to identify the third image having the smallest third loss value as the personalized output image. In some aspects, the multiple coefficient sets include a first coefficient set and a multiple series of coefficient sets selected using a gradient descent method based directly or indirectly on the first coefficient set.In some aspects, the input image of the subject includes a first portion of pixels preserved from the original image of the subject and a mask in place of a second portion of the pixels from the original image of the subject, and using one or more processors, comparing the third image to at least a portion of the input image of the subject to generate a third loss value for the third image includes comparing the third image to the first portion of the pixels to generate a third loss value for the third image. In some aspects, the input image has a first resolution, and the personalized output image has a second resolution that is higher than the first resolution. In some aspects, the plurality of codes includes a first code and a plurality of sets of codes selected using gradient descent based directly or indirectly on the first code. In some aspects, the first code represents a mean value of a latent vector space W, and the latent vector space W represents all possible codes that can be input to the generative model.

[0004] In another aspect, the present disclosure describes a processing system including a memory that stores a generative model and one or more processors coupled to the memory and configured to perform any of the aforementioned methods.

[0005] In another aspect, the disclosure describes a processing system that includes: (1) a memory that stores a generative model; and (2) one or more processors coupled to the memory and configured to generate a personalized prior distribution for a subject for use with the generative model, the one or more processors including: (a) for each given image of a set of images of the object, testing a plurality of codes to identify an optimized code for the given image, the testing including: (i) for each code of the plurality of codes, generating a first image using the generative model and the code and comparing the first image to the given image to generate a first loss value for the code; and (ii) comparing the generated first loss values ​​for each code of the plurality of codes to identify a code having a minimum first loss value as the optimized code for the given image; and (b) generating the personalized prior distribution for the subject based on a convex hull that includes each optimized code identified for each given image of the set of images of the object. In some aspects, the one or more processors are further configured to adjust the generative model, including: (1) for each given image of the set of images of the subject, generating a second image using the generative model and the optimization code, and comparing, using the one or more processors, the second image to the given image to generate a second loss value; and (2) using the one or more processors to modify one or more parameters of the generative model based at least in part on each generated second loss value to construct an adjusted generative model.In some aspects, the one or more processors are further configured to generate a personalized output image based on the input image of the subject, comprising: (1) identifying a plurality of coefficient sets, where each coefficient set of the plurality of coefficient sets corresponds to a code within a convex hull; (2) for each given coefficient set of the plurality of coefficient sets, generating a third image using the adjusted generative model and a given code corresponding to the given coefficient set, and comparing the third image to at least a portion of the input image of the subject to generate a third loss value for the third image; and (3) comparing the generated third loss values ​​for each third image to identify the third image having the smallest third loss value as the personalized output image. In some aspects, the one or more processors are further configured to generate a personalized output image based on the input image of the subject, comprising: (1) identifying a plurality of coefficient sets, where each coefficient set of the plurality of coefficient sets corresponds to a code in the convex hull; (2) identifying a plurality of code sets, where each code set of the plurality of code sets includes two or more individual codes, where each individual code corresponds to a coefficient set of the plurality of coefficient sets; (3) for each given code set of the plurality of code sets, generating a third image using the adjusted generative model and the given code set, where each individual code of the given code set is provided to a different layer or set of layers of the adjusted generative model; and comparing the third image to at least a portion of the input image of the subject to generate a third loss value for the third image; and (4) comparing the generated third loss values ​​for each third image to identify the third image having the smallest third loss value as the personalized output image.In some aspects, the one or more processors are further configured to generate a personalized output image based on the input image of the subject, including: (1) identifying a plurality of coefficient sets, where each coefficient set of the plurality of coefficient sets corresponds to a code within a convex hull; (2) for each given coefficient set of the plurality of coefficient sets, generating a third image using the generative model and a given code corresponding to the given coefficient set, and comparing the third image to at least a portion of the input image of the subject to generate a third loss value for the third image; and (3) comparing the generated third loss values ​​for each third image to identify the third image having the smallest third loss value as the personalized output image. In some aspects, the one or more processors are further configured to generate a personalized output image based on the input image of the subject, comprising: (1) identifying a plurality of coefficient sets, where each coefficient set of the plurality of coefficient sets corresponds to a code in the convex hull; (2) identifying a plurality of code sets, where each code set of the plurality of code sets includes two or more individual codes, where each individual code corresponds to a coefficient set of the plurality of coefficient sets; (3) for each given code set of the plurality of code sets, generating a third image using the generative model and the given code set, where each individual code of the given code set is provided to a different layer or set of layers of the generative model; and generating the third image; comparing the third image to at least a portion of the input image of the subject to generate a third loss value for the third image; and (4) comparing the generated third loss values ​​for each third image to identify the third image having the smallest third loss value as the personalized output image. In some aspects, the multiple coefficient sets include a first coefficient set and a multiple series of coefficient sets, and the one or more processors are further configured to select each coefficient set of the multiple series of coefficient sets using a gradient descent method based directly or indirectly on the first coefficient set.In some aspects, the input image of the subject includes a first portion of pixels preserved from the original image of the subject and a mask in place of a second portion of the pixels from the original image of the subject, and comparing the third image to at least a portion of the input image of the subject to generate a third loss value for the third image includes comparing the third image to the first portion of the pixels to generate a third loss value for the third image. In some aspects, the one or more processors are configured to generate a personalized output image based on the input image of the subject, the input image having a first resolution, and the personalized output image having a second resolution higher than the first resolution. In some aspects, the plurality of codes includes a first code and a plurality of series of codes, and the one or more processors are further configured to select each code of the plurality of series of codes using a gradient descent method based directly or indirectly on the first code. In some aspects, the one or more processors are further configured to select the first code representing a mean value of a latent vector space W, the latent vector space W representing all possible codes that can be input to the generative model. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a functional diagram of an exemplary system according to an aspect of the present disclosure. [Diagram 2] FIG. 1 is a functional diagram of an exemplary system according to an aspect of the present disclosure. [Diagram 3] 1A and 1B illustrate how different codes from within the latent vector space of a generative model can result in images of different objects, according to aspects of the present disclosure. [Figure 4] 1A and 1B show two different personalized prior distributions in the latent vector space of a generative model, according to an embodiment of the present disclosure, and how points in a given personalized prior distribution can result in different images of a given subject. [Diagram 5] 1 illustrates an example method for generating a personalized prior distribution based on a set of images of a subject, according to an aspect of the present disclosure. [Figure 6]FIG. 4B illustrates how a given set of images can be used to generate the personalized prior distribution of FIG. 4A according to an aspect of the disclosure. [Figure 7] 6 illustrates an example method for tuning a generative model following identification of an optimization code for each image in a set of images according to the method of FIG. 5 , in accordance with an aspect of the present disclosure. [Figure 8] 8 illustrates an example method for generating a personalized output image based on an input image and a personalized prior distribution generated by the method of FIG. 5 or FIG. 7 according to an embodiment of the present disclosure. [Figure 9] 8 illustrates an example method for generating a personalized output image based on an input image and a personalized prior distribution generated by the method of FIG. 5 or FIG. 7 according to an embodiment of the present disclosure. [Figure 10] 4B shows a comparative illustration of how a generative model, according to an embodiment of the present disclosure, can complete four example image enhancement tasks with and without the personalized prior of FIG. [Figure 11] 4B shows a comparative illustration of how a generative model, according to an embodiment of the present disclosure, can complete four example image inpainting tasks with and without the personalized prior of FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] The technology will now be described with reference to the following exemplary systems and methods: Reference numbers common between the figures shown and described below are intended to identify like features.

[0008] Exemplary System FIG. 1 illustrates a high-level system diagram 100 of an exemplary processing system 102 for performing the methods described herein. The processing system 102 may include one or more processors 104, as well as a memory 106 that stores instructions 108 and data 110. The instructions 108 and data 110 may include a generative model described herein (e.g., the generative model 306 of FIGS. 3A, 3B, 4A, 4B, 6, 10, and 11). In addition, the data 110 may store training examples used to train such a generative model, data used by the generative model when generating images, a set of images of a given object, personalized prior distributions based on a set of images of a given object (e.g., the personalized prior distributions 402 and 403 of FIGS. 4A, 4B, 6, 10, and 11), and / or images output by the generative model.

[0009] The processing system 102 may reside on a single computing device. For example, the processing system 102 may be a server, a personal computer, or a mobile device, and thus the generative model and its associated data may be local to that single computing device. Similarly, the processing system 102 may reside on a cloud computing system or other distributed system. In such cases, the generative model and / or its associated data may be distributed across two or more different physical computing devices. For example, in some aspects of the present technology, the processing system may include a first computing device that stores the generative model, as well as a second computing device that stores data used by the generative model in generating images, a set of images of a given subject, a personalized prior distribution based on the set of images of a given subject, and / or an image output by the generative model. Similarly, in some aspects of the present technology, the processing system may include a first computing device that stores layers 1 to n of a generative model having m layers, and a second computing device that stores layers n to m of the generative model.

[0010] Further in this regard, Figure 2 illustrates a high-level system diagram 200 in which the previously described exemplary processing system 102 is shown in communication with various websites and / or remote storage systems, including websites 210 and 218 and remote storage system 226, via one or more networks 208. In this example, each website 210 and 218 includes one or more servers 212a-212n and 220a-220n, respectively. Each of the servers 212a-212n and 220a-220n may have one or more processors (e.g., 214 and 222) and associated memory (e.g., 216 and 224) for storing instructions and data, including content for one or more web pages. Similarly, although not shown, the remote storage system 226 may also include one or more processors and memory for storing instructions and data. In some aspects of the technology, processing system 102 may be configured to obtain data, training examples, a set of images of a given subject, and / or input images of a given subject from one or more of website 210, website 218, and / or remote storage system 226 to provide to a generative model for training or tuning the generative model and / or to be used when generating images.

[0011] The processing systems described herein may be implemented on any type of computing device(s), such as any type of general computing device, server, or set thereof, and may further include other components typically present in a general purpose computing device or server. Similarly, the memory of such a processing system may be of any non-transitory type capable of storing information accessible by the processor(s) of the processing system. For example, the memory may include non-transitory media such as hard drives, memory cards, optical disks, solid state, tape memory, etc. Computing devices suitable for the roles described herein may include different combinations of the foregoing, whereby different portions of instructions and data are stored on different types of media.

[0012] In any case, the computing devices described herein may further include any other components typically used in connection with a computing device, such as a user interface subsystem. The user interface subsystem may include one or more user inputs (e.g., a mouse, keyboard, touch screen, and / or microphone), and one or more electronic displays (e.g., a monitor having a screen, or any other electrical device operable to display information). Output devices other than electronic displays, such as speakers, lights, and vibrating, pulsing, or tactile elements, may also be included in the computing devices described herein.

[0013] The one or more processors included in each computing device may be any conventional processor, such as a commercially available central processing unit ("CPU"), graphics processing unit ("GPU"), tensor processing unit ("TPU"). Alternatively, the one or more processors may be dedicated devices, such as an ASIC or other hardware-based processor. Each processor may have multiple cores capable of operating in parallel. The processor(s), memory, and other elements of a single computing device may be stored in a single physical housing or distributed among two or more housings. Similarly, the memory of a computing device may include a hard drive or other storage medium located in a housing different from the housing of the processor(s), such as in an external database or networked storage device. Thus, reference to a processor or computing device will be understood to include reference to a collection of processors or computing devices or memories that may or may not operate in parallel, and to one or more servers of a load-balanced server farm or cloud-based system.

[0014] A computing device described herein may store instructions (such as machine code) directly executable by a processor(s) or instructions (such as scripts) indirectly executable by a processor(s). A computing device may also store data, which may be retrieved, stored, or modified by one or more processors according to the instructions. The instructions may be stored as computing device code on a computing device readable medium. In that regard, the terms "instructions" and "programs" may be used interchangeably herein. The instructions may also be stored in object code format for direct processing by a processor(s) or in any other computing device language, including a collection of scripts or independent source code modules that are interpreted on demand or precompiled. By way of example, the programming language may be C#, C++, JAVA, or another computer programming language. Similarly, any component of the instructions or programs may be implemented in a computer scripting language, such as JavaScript, PHP, ASP, or any other computer scripting language. Furthermore, any one of these components may be implemented using a combination of a computer programming language and a computer scripting language.

[0015] Exemplary Methods 3A and 3B illustrate how different codes from within a latent vector space W (302) of a generative model 306 may result in different images (308, 309) of a subject, according to aspects of the present disclosure. In these examples, the generative model 306 may be any suitable model configured to generate an output image based on an input code, and the latent vector space W (302) represents the range of all possible input codes that may be provided to the generative model 306. In that regard, and solely for purposes of simplicity of explanation, the latent vector space W (302) is illustrated in FIGS. 3A and 3B (as well as FIGS. 4A, 4B, 6, 10, and 11) as being a two-dimensional space. However, the present techniques may be applied to any suitable generative model (e.g., GAN or bidirectional GAN ​​("BiGAN")) configured to generate an image based on an input vector of any suitable number of dimensions. For example, in some aspects of the present technology, the generative model 306 may be a generative adversarial network (e.g., StyleGAN, StarGAN) configured to use a latent vector space W having 512 dimensions.

[0016] As shown in FIG. 3A, the code at point 304 in the latent vector space W (302) is fed to a generative model 306, which generates an image 308. To explain the effect of the present technology, the examples shown in FIG. 3A, FIG. 3B, FIG. 4A, FIG. 4B, FIG. 10, and FIG. 11 each use an image of a subject whose appearance is recognizable to many people. Thus, in the example of FIG. 3A, it is assumed that the code at point 304 generates an image 308 of Barack Obama (the 44th President of the United States). Similarly, in the example of FIG. 3B, it is assumed that the code at point 305 generates an image 309 of Lady Gaga (an American singer-songwriter and actress, also known as Stefani Joanne Angelina Germanotta).

[0017] 4A and 4B show two different personalized prior distributions 402, 403 in the latent vector space W (302) of the generative model 306 according to aspects of the present disclosure, and further illustrate how points in a given personalized prior distribution (e.g., 404 vs. 304, 405 vs. 305) can result in different images of a given subject. Both examples of Figures 4A and 4B assume the use of the same latent vector space W (302) and the same generative model 306 used in Figures 3A and 3B, but further illustrate personalized prior distributions 402, 403 in the latent vector space W (302).

[0018] Each of the personalized priors 402 and 403 represents a vector space in the latent vector space W (302) that contains a subset of possible input codes that will create an image similar to a given subject. In this case, the personalized prior 402 is assumed to represent a range of codes that, when provided to the generative model 306, will create an image similar to Barack Obama. Thus, the personalized prior 402 includes a point 304 that represents the code that created image 308 of FIG. 3A, and another point 404 that represents the code that created a different image 406 of Barack Obama.

[0019] Similarly, personalized prior 403 is assumed to represent a range of codes that, when provided to generative model 306, will create an image that resembles Lady Gaga. Thus, personalized prior 403 includes point 304 that represents the code to create image 309 of FIG. 3B, and another point 405 that represents the code to create a different image 407 of Lady Gaga.

[0020] Again, for the sake of simplicity, the personalized priors 402 and 403 are shown in FIGS. 4A and 4B (as well as FIGS. 6, 10, and 11) as being two-dimensional spaces. However, because the present technology may be applied to any suitable generative model configured to generate images based on input vectors of any suitable number of dimensions, the personalized priors 402 and 403 may similarly be vector spaces of any number of dimensions equal to or less than the number of dimensions of the latent vector space W (302). Thus, for example, if the generative model 306 is a generative adversarial network configured to use a latent vector space W having 512 dimensions, then the personalized priors 402 and 403 may each be vector spaces of up to 512 dimensions. In addition, for the sake of clarity, the exemplary vector spaces 402 and 403 are shown completely separated. However, in practice, the personalized priors for different subjects may cross each other.

[0021] FIG. 5 illustrates an example method 500 for generating a personalized prior distribution based on a set of images of a subject, according to an aspect of the present disclosure.

[0022] In step 502, a processing system (e.g., processing system 102) selects a given image from a set of images of the subject. It will be appreciated that a larger number of images will generally produce a more representative personalized prior distribution than a smaller number of images. In that regard, it has been found that 100-200 images are generally sufficient for a generative model to produce images that appear realistic and consistent with the subject's identity. However, other aspects may also affect how well the personalized prior distribution represents the subject's appearance. For example, if the subject's appearance has changed significantly (e.g., due to a change in hairstyle, hair color, addition or removal of facial hair, or over a long period of time), it may be useful to tailor the set of images to a particular aspect so that the personalized prior distribution reflects a single "look" and the images produced by the generative model are consistent with that look. Similarly, for the same reasons, it may be useful to limit the set of images to a single stage of life (e.g., infancy, childhood, adolescence, adulthood, etc.). On the other hand, diversity in the set of images may be important. For example, a set of images that only show a subject's face from the front may not be as useful as a set of images that show the subject's face from a variety of different angles, a variety of different lighting conditions, etc. Generating a personalized prior from images that are too similar may overly restrict the generative model, causing it to create images that resemble the subject but do not necessarily look photorealistic.

[0023] As shown in step 504, after the processing system selects a given image, iteratively performs steps 506-514 to test multiple codes to identify an optimized code for the given image. In that regard, in step 506, the processing system identifies a code to be tested in the current pass. The code may be identified based on any suitable selection criteria. For example, in some aspects of the present technology, the processing system may be configured to blindly select a first code (e.g., using a random selection process or a preselected value such as the mean value of the latent vector space W) and then use a suitable optimization regime to select each successive code (in each successive pass through step 506) based directly or indirectly on the first code. Thus, in some aspects, the processing system may be configured to select each successive code using gradient descent based on the preceding code and an assessment of how well the image generated based on the preceding code matched the given image (e.g., the first loss value generated in the most recent pass through step 510).

[0024] At step 508, the processing system generates a first image using a generative model (e.g., generative model 306) and the code under test (identified in this pass through step 506). The generative model may be configured to create the first image in any suitable manner, including those described above with respect to Figures 3A, 3B, 4A, and 4B.

[0025] At step 510, the processing system compares the first image (generated in this pass through step 508) to the given image (selected at step 502) to generate a first loss value for the code under test. The first loss value may be generated in any suitable manner using any suitable function. For example, in some aspects of the present technology, the first loss value may be based on a comparison of the first image to the given image using a heuristic or learned similarity metric (e.g., Learned Perceptual Image Patch Similarity ("LPIPS"), Peak Signal-to-Noise Ratio ("PSNR"), Structural Similarity Index ("SSIM"), L1 or L2 loss, etc.).

[0026] At step 512, the processing system determines whether to test another code. Because it is assumed that there will be more than one code to be tested, the processing system automatically follows the "Yes" arrow back to step 506 when it first reaches step 512, whereby a second code is identified and tested. However, at any subsequent return to step 512, the processing system may determine whether to test another code based on any suitable criteria. Thus, as described above, in some aspects of the present technology, the processing system may determine when to stop testing another code based on a suitable optimization regime, such as gradient descent. In such a case, the determination at step 512 may be based on a comparison of a first loss value generated in a current pass through step 510 (or some other assessment of how well a first image matches a given image) to one or more of the first loss values ​​generated in a previous pass. For example, the processing system may be configured to stop testing a set of codes when a first loss value generated in a current pass through step 510 is equal to or greater than a first loss value generated in a previous pass through step 510.

[0027] Thus, the processing system repeats steps 506-512 with each next code until it is determined at step 512 that sufficient codes have been tested. At that point, the processing system follows the "No" arrow to step 514, where it compares the first loss values ​​generated (at step 510) for each code of the plurality of codes to identify the code having the smallest first loss value. The code with the smallest first loss value is selected as the optimized code for the given image.

[0028] The processing system then determines, in step 516, whether there are any more images in the set of images of the object. If there are, the processing system follows a "yes" arrow to step 518, where the processing system selects the next given image to test. The processing system then returns to step 504 to test the multiple codes to identify an optimization code for this new given image. In this manner, steps 504-518 are repeated as described above until an optimization code has been identified for every image in the set of images of the object. Once an optimization code has been selected (in step 514) for the last image in the set of images of the object, the processing system determines in step 516 that there are no more images in the set, and therefore follows a "no" arrow to step 520.

[0029] In step 520, the processing system generates, for each given image in the set of images of the object, a personalized prior distribution for the object based on a convex hull that includes each optimization code identified (in step 514). In that regard, in some aspects of the present technology, the personalized prior distribution may simply be a convex hull defined by each optimization code identified in step 514. In such a case, for n optimization codes {x1, x2, ..., x n}, the personalized prior distribution includes an arbitrary code c generated by linearly combining the optimized codes according to the following Equations 1 to 3. In the equations, the coefficients (alpha values ​​α1 to αn ) are greater than or equal to 0, and the sum of all the coefficients is 1.

[0030]

number

[0031] Similarly, in some aspects of the present technology, the personalized prior distribution may be a set of some subset of the convex hull defined by each optimization code identified in step 514, e.g., a set of some predetermined number (e.g., 100, 500, 1,000, 10,000, 100,000, 1,000,000) of codes or coefficient sets corresponding to a set of sampled points within the convex hull. Furthermore, in some aspects of the present technology, the personalized prior distribution may be a simpler hull (e.g., a hull with fewer vertices) that fits within or substantially overlaps with the actual convex hull defined by each optimization code identified in step 514. Additionally, in some aspects of the present technology, the personalized prior distribution may be based on the convex hull defined by Equations 1-3 above by encompassing a broader set of codes than those defined by Equations 1-3 above. For example, the personalized prior distribution may be a set of coefficients (α values ​​α1-α n ) is greater than or equal to some predetermined negative value (eg, −0.01, −0.05, −0.1).

[0032] FIG. 6 illustrates how a given set of n images 602a-612a may be used to generate the personalized prior distribution 402 of FIG. 4A in accordance with an aspect of the disclosure. In that regard, each image 602a-612a is assumed to be one of a set of n images of a subject, and the illustrated lines show how these six selected images correspond to different optimization codes 602b-612b in the latent vector space W (302) of a generative model (e.g., generative model 306). Each of these optimization codes 602b-612b may be found by steps 502-518 of the exemplary method of FIG. 5, as further described above. In addition, the exemplary diagram 600 of FIG. 6 illustrates how each of these optimization codes 602b-612b (along with optimization codes for the remainder of the set of n images, not shown) may be used to define a convex hull on which the personalized prior distribution 402 is based, as further described above with respect to step 520 of FIG. 5. Again, merely for purposes of simplicity of illustration, the personalized prior 403 in Figure 6 is shown as being a two-dimensional space, and only six sample images of a set of n images are shown, however, the personalized prior 403 may be based on any suitable number of points and may be a vector space of any suitable number of dimensions less than or equal to the dimensionality of the latent vector space W (302), as explained above.

[0033] Figure 7 illustrates an example method 700 for adjusting a generative model following identification of an optimization code for each image in a set of images according to the method of Figure 5, in accordance with aspects of the present disclosure. In that regard, the example method 700 represents a process that may be optionally performed after an optimization code has been identified for each image in a set of images (at step 514 of Figure 5).

[0034] Thus, at step 702, it is envisioned that a processing system (e.g., processing system 102) performs at least steps 502-518 of the exemplary method of FIG. 5 for each image in the set of images. Although the exemplary method of FIG. 7 shows steps 704-720 occurring after step 702, it will be understood that steps 704-720 may be performed in any suitable order with respect to steps 502-518 of the exemplary method of FIG. 5. For example, in some aspects of the present technology, the processing system may be configured to identify an optimization code for a given image of the set of images (at step 514 of FIG. 5) and then perform steps 704-712 for that given image prior to selecting a next given image (at step 518 of FIG. 5) or in parallel with testing multiple codes to identify an optimization code for the next given image (e.g., steps 504-514 of FIG. 5). In such a case, the processing system may be further configured to perform step 714 in parallel with steps 504-518 of FIG. 5 (after each batch of images has been processed by steps 704-712) such that the parameters of the generative model are periodically updated while the optimization code is continually identified (in successive passes through step 518).

[0035] Regardless of timing, in step 704, the processing system selects a given image from the set of images of the object. This set of images may be the entire set of images used in Figure 5, or any suitable subset thereof. For example, in some aspects of the present technology, the processing system may be configured to identify an optimization code for a set of 200 images of the object via steps 502-518 of Figure 5, but then to tune a generative model (e.g., generative model 306) based on only 100 of those images.

[0036] At step 706, the processing system generates a second image using the generative model and the optimization code identified for the given image (at step 514 of FIG. 5). Again, the generative model may be configured to create the second image in any suitable manner, including those described above with respect to FIGS. 3A, 3B, 4A, and 4B. Additionally, while step 706 refers to a "second image," it will be understood that this second image may in some cases be a copy of one of the "first images" generated at step 508 of FIG. 5. Thus, in some aspects of the present technology, for any "given image" in the first batch of images, the processing system may be configured to use the "first image" (or a copy thereof) associated with the optimization code identified at step 514 of FIG. 5 as the "second image" at step 706, rather than generating the image again at step 706. However, if the processing system modifies one or more parameters of the generative model in step 714 (described below), the generative model is changed such that each subsequent optimization code will produce a "second image" that is different from the "second image" that code would have previously generated. Thus, if method 700 is performed after method 500 (rather than in parallel with some or all of the steps of method 500), the processing system may be configured to generate a new "second image" in each successive pass through step 706 (rather than using a copy of the "first image" identified in step 508 of FIG. 5).

[0037] At step 708, the processing system compares the second image (generated at step 706) to the given image (selected at step 704) to generate a second loss value. Again, the second loss value may be generated in any suitable manner, using any suitable function. For example, in some aspects of the present technology, the second loss value may be based on a comparison of the second image to the given image using a heuristic or learned similarity metric (e.g., Learned Perceptual Image Patch Similarity ("LPIPS"), Peak Signal-to-Noise Ratio ("PSNR"), Structural Similarity Index ("SSIM"). Additionally, although step 708 refers to a "second loss value," it will be understood that this second loss value may, in some cases, be a copy of the "smallest first loss value" identified in step 514 of FIG. 5 for that image. Thus, in some aspects of the present technology, for any "given image" in the first batch of images, the processing system may generate a new loss value in step 708, rather than a new copy of the "smallest first loss value" identified in step 514 of FIG. 5 for that given image. 5 )。 As already mentioned, however, if the processing system modifies one or more parameters of the generative model in step 714 (described below), the generative model is changed such that each subsequent optimization code will produce a different "second image" than the "second image" that the code would have previously produced, and therefore a different "second loss value" when compared to the "given image". Thus, if method 700 is performed after method 500 (rather than in parallel with some or all of the steps of method 500), the processing system may be configured to generate a new "second loss value" in each successive pass through step 708 (rather than using a copy of the "smallest first loss value" identified in step 514 of FIG. 5 ).

[0038] At step 710, the processing system determines whether there are any more images in the batch. At that point, the set of images may be retained as a whole or may be divided into any suitable number of batches. If the set of images has not been divided, and thus there is a single "batch" that includes every image in the "set of images" of the object, the processing system follows a "yes" arrow to step 712 to select the next given image from the set of images of the object and repeats steps 706-710 for that newly selected image. This process is repeated until there are no more images in the set of images, at which point the processing system follows a "no" arrow to step 714. On the other hand, if the set of images has been divided into two or more batches (e.g., a set of 200 images may be divided into two 100-image batches, four 50-image batches, ten 20-image batches, 200 single-image "batches," etc.), steps 704-712 are repeated for each image until the end of the batch is reached.

[0039] As shown in step 714, after the "second loss values" are generated (at step 708) for every image in the batch, the processing system modifies one or more parameters of the generative model based at least in part on each generated second loss value. The processing system may be configured to modify one or more parameters in any suitable manner and even at any suitable interval based on these generated second loss values. Thus, in some aspects of the present technology, each "batch" may include a single image such that the processing system performs a backpropagation step to modify one or more parameters of the generative model each time a second loss value is generated. Similarly, when each "batch" includes two or more images, the processing system may be configured to add up each of the "second loss values" generated (at step 708) for each image in the batch into a total loss value (e.g., by summing or averaging multiple second loss values) and modify one or more parameters of the generative model based on the total loss value.

[0040] In step 716, the processing system determines whether there are any further batches in the set of images of the object. If the set of images has not been split and therefore there is a single "batch" that includes every image in the "set of images" of the object, the determination in step 716 is automatically "no" and then the method 700 ends as shown in step 720. However, if the set of images has been split into two or more batches, the processing system follows a "yes" arrow to step 718 to select a next given image from the set of images of the object. This then initiates another set of passes through steps 706-714 for each image in the next batch of images, and the process continues until there are no more batches. At that point, the processing system follows a "no" arrow to step 720.

[0041] Although method 700 is shown to end at step 720 once all images have been used to adjust the generative model, it will be appreciated that method 700 may be repeated any suitable number of times using the same set of images until the output of the generative model for each optimization code produces images that are sufficiently close to each given image. In that regard, in some aspects of the present technology, the processing system may be configured to aggregate all of the second loss values ​​generated during a given pass through method 700 and determine whether to repeat method 700 for the set of images based on the aggregated loss value. For example, in some aspects of the present technology, the processing system may be configured to repeat method 700 for the set of images if the total loss value for the most recent pass through method 700 was greater than some predefined threshold. Similarly, in some aspects, the processing system may be configured to make this determination using a gradient descent method, and thus repeat method 700 for the set of images until the total loss value in a given pass through method 700 is equal to or greater than the total loss value from the previous pass.

[0042] Figure 8 illustrates an example method 800 for generating a personalized output image based on an input image and a personalized prior distribution generated by the method of Figure 5 or Figure 7, according to aspects of the present disclosure. In that regard, the example method 800 represents a process that may be optionally performed after at least the personalized prior distribution has been generated (at step 520 of Figure 5), and may also be performed after the generative model has been further adjusted by the method of Figure 7. Thus, at step 802, it is envisioned that a processing system (e.g., processing system 102) performs at least method 500 of Figure 5, and optionally steps 704-720 of Figure 7, for each image in the set of images.

[0043] As described above, after the processing system generates a personalized prior (using method 500 of FIG. 5) and optionally trains a generative model (using method 700 of FIG. 7), it may use the personalized prior to generate different candidate coefficient sets, and then use codes corresponding to the candidate coefficient sets to generate candidate images for a given image enhancement task. Thus, as shown in step 804, for a particular input image of a subject, the processing system repeatedly performs steps 806-812 to test multiple coefficient sets, where each coefficient set of the multiple coefficient sets corresponds to a code within the convex hull (identified in step 520 of FIG. 5) to identify a personalized output image.

[0044] In each pass through step 806, the processing system identifies a coefficient set of the multiple coefficient sets and uses it to generate the given code. The coefficient set may be identified based on any suitable selection criteria. For example, in some aspects of the present technology, the processing system may be configured to blindly select a first coefficient set (e.g., using a random selection process or a preselected value such as the mean of a vector space represented by a personalized prior distribution) and then use a suitable optimization regime to select each successive coefficient set (in each successive pass through step 806) based directly or indirectly on the first coefficient set. Thus, in some aspects, the processing system may be configured to select each successive coefficient set using gradient descent based on the preceding coefficient set and an evaluation of how well an image generated based on the preceding coefficient set matched the input image (e.g., the third loss value generated in the most recent pass through step 810).

[0045] At step 808, the processing system generates a third image using the generative model (e.g., generative model 306, or a refined generative model resulting from one or more passes through steps 704-720 of FIG. 7) and the given code (generated in this pass through step 806). Again, the generative model may be configured to create the third image in any suitable manner, including those described above with respect to FIGS. 3A, 3B, 4A, and 4B.

[0046] At step 810, the processing system compares the third image (generated in this pass through step 808) with the input image of the subject to generate a third loss value for the third image. The third loss value may be generated in any suitable manner using any suitable function. For example, in some aspects of the present technology, the third loss value may be based on a comparison of the third image with the input image using a heuristic or learned similarity metric (e.g., Learned Perceptual Image Patch Similarity ("LPIPS"), Peak Signal-to-Noise Ratio ("PSNR"), Structural Similarity Index ("SSIM").

[0047] In step 812, the processing system determines whether to test another set of coefficients. Because it is assumed that there will be more than one set of coefficients to be tested, the processing system automatically follows the "Yes" arrow back to step 806 when it first reaches step 812, whereby a second set of coefficients is identified and tested. However, on any subsequent return to step 812, the processing system may determine whether to test another set of coefficients based on any suitable criteria. Thus, as described above, in some aspects of the present technology, the processing system may determine when to stop testing another set of coefficients based on a suitable optimization regime, such as gradient descent. In such a case, the determination in step 812 may be based on a comparison of the third loss value generated in the current pass through step 810 (or some other assessment of how well the third image matches the input image) to one or more of the third loss values ​​generated in the preceding pass. For example, the processing system may be configured to stop testing a series of coefficient sets when a third loss value generated in a current pass through step 810 is greater than or equal to a third loss value generated in a previous pass through step 810.

[0048] Thus, the processing system repeats steps 806-812 with each of the next coefficient sets until it is determined at step 812 that sufficient coefficient sets have been tested. At that point, the processing system follows the "no" arrow to step 814. The processing system then compares the third loss values ​​generated (at step 810) for each third image to identify the third image with the smallest third loss value. The third image with the smallest third loss value is used as the personalized output image. In this manner, method 800 may be used to create a personalized output image. This personalized output image is generated using the code in the personalized prior (so that the output image is more likely to resemble the subject than the output image would have been if the code had not been so limited) and is optimized to closely match the input image (so that when performing an image editing or enhancement task, it is ensured that the image created by the model is still consistent with the image that could have been obtained from the input image).

[0049] FIG. 9 illustrates another exemplary method 900 for generating a personalized output image based on an input image and a personalized prior distribution generated by the method of FIG. 5 or FIG. 7, according to an aspect of the present disclosure. In that respect, the exemplary method 900 also represents a process that may be optionally performed after at least a personalized prior distribution is generated (at step 520 of FIG. 5), and may also be performed after the generative model has been further tuned by the method of FIG. 7. The only difference between the method 800 of FIG. 8 and the method 900 of FIG. 9 is that a set of given codes is tested in each pass through steps 906-912, where each individual code in the set of given codes is within the convex hull identified in step 520 of FIG. 5.

[0050] Thus, as noted above, in step 902, it is assumed that a processing system (e.g., processing system 102) performs at least method 500 of Figure 5, and optionally steps 704-720 of Figure 7, for each image in the set of images. Similarly, as shown in step 904, for a particular input image of a subject, the processing system repeatedly performs steps 906-912 to test multiple code sets, where each code in the code set lies within the convex hull (identified in step 520 of Figure 5), to identify a personalized output image.

[0051] In each pass through step 906, the processing system identifies a given code set that includes two or more individual codes. Again, this given code set may be identified based on any suitable selection criteria. For example, in some aspects of the present technology, the processing system may be configured to blindly select a first given code set (e.g., using a random selection process or by assigning a preselected value, such as the mean of a vector space represented by a personalized prior distribution, to each individual code), and then use a suitable optimization regime to select each successive code set (in each successive pass through step 906) based directly or indirectly on the first given code set. Thus, in some aspects, the processing system may be configured to select each successive code set using gradient descent based on the preceding code set and an evaluation of how well an image generated based on the preceding code set matched the input image (e.g., the third loss value generated in the most recent pass through step 910).

[0052] At step 908, the processing system generates the third image using the generative model (e.g., generative model 306, or an adjusted generative model resulting from one or more passes through steps 704-720 of FIG. 7) and the given code set (generated in this pass through step 906). In that regard, at step 908, the processing system provides each individual code of the given code set to a different layer or set of layers of the generative model. Again, the generative model may be configured to create the third image in any suitable manner, including those described above with respect to FIGS. 3A, 3B, 4A, and 4B.

[0053] At step 910, the processing system compares the third image (generated in this pass through step 908) with the input image of the subject to generate a third loss value for the third image. Again, the third loss value may be generated in any suitable manner using any suitable function. For example, in some aspects of the present technology, the third loss value may be based on a comparison of the third image with the input image using a heuristic or learned similarity metric (e.g., Learned Perceptual Image Patch Similarity ("LPIPS"), Peak Signal-to-Noise Ratio ("PSNR"), Structural Similarity Index ("SSIM").

[0054] In step 912, the processing system determines whether to test another code set. Because it is assumed that there will be more than one code set to be tested, the processing system automatically follows the "Yes" arrow back to step 906 when it first reaches step 912, whereby a second code set is identified and tested. However, on any subsequent return to step 912, the processing system may determine whether to test another code set based on any suitable criteria. Thus, as described above, in some aspects of the present technology, the processing system may determine when to stop testing another code set based on a suitable optimization regime, such as gradient descent. In such a case, the determination in step 912 may be based on a comparison of the third loss value generated in the current pass through step 910 (or some other assessment of how well the third image matches the input image) to one or more of the third loss values ​​generated in the preceding pass. For example, the processing system may be configured to stop testing a series of code sets when a third loss value generated in a current pass through step 910 is greater than or equal to a third loss value generated in a previous pass through step 910.

[0055] Thus, similar to method 800, the processing system repeats steps 906-912 with each successive code set until it is determined at step 912 that sufficient code sets have been tested. At that point, the processing system follows the "no" arrow to step 914. There, the processing system compares the third loss values ​​generated (at step 910) for each third image to identify the third image having the smallest third loss value. Again, the third image identified as having the smallest third loss value is used as the personalized output image.

[0056] FIG. 10 is a diagram 1000 showing a comparative illustration of how a generative model (e.g., generative model 306) according to an aspect of the present disclosure may complete four example image enhancement tasks with and without using the personalized prior distribution 402 of FIG. 4A.

[0057] In Figure 10, a row of four input images 1002a-1002d are shown in the center of diagram 1000. Each of the input images 1002a-1002d is blurred, as it is assumed that the generative model is tasked with creating output images that resemble the input images but with enhanced resolution.

[0058] The left column of images 1004a-d show potential outputs of the generative model when it is not constrained to select a code in any particular portion of the latent vector space W (302). In that regard, dashed lines connect each input image to a point in the latent vector space W (302) that represents the code that the generative model ultimately selected (e.g., after a selection process such as that described above in method 800 of FIG. 8 or method 900 of FIG. 9), and arrows connect that point to the corresponding output image that the generative model created based on that code. As can be seen, although output images 1004a-d are visually consistent with each of input images 1002a-d, respectively, these output images do not all appear to be of the same subject.

[0059] In contrast, the right column of images 1006a-d show potential outputs of the generative model when constrained to select a code in the personalized prior 402 of FIG. 4A. Again, dashed lines connect each input image to a point in the personalized prior 402 representing the code that the generative model ultimately selects (e.g., after a selection process such as that described above with respect to method 800 of FIG. 8 or method 900 of FIG. 9), and arrows connect that point to the corresponding output image that the generative model creates based on that code. As can be seen, this ultimately produces output images 1006a-d, which are visually consistent with each of the input images 1002a-d and all appear to be of the same subject. Specifically, personalized prior distribution 402 in FIG. 4A represents a range of code that, when provided to generative model 306 (as described above), creates an image that resembles Barack Obama, such that each of output images 1006a-1006d appears to show an image of Barack Obama that is visually consistent with input images 1002a-1002d.

[0060] FIG. 11 is a diagram 1100 showing a comparative illustration of how a generative model (e.g., generative model 306) according to an aspect of the present disclosure may complete four example image inpainting tasks with and without using the personalized prior distribution 403 of FIG. 4B.

[0061] Again, a row of four input images 1102a-d is shown in the center of diagram 1000. It is assumed that the generative model is tasked with creating an output image that resembles the input images but with masked parts filled in, and so each of the input images 1102a-d includes a black mask 1103a-d that represents the pixels to be replaced.

[0062] In FIG. 1100, the left column of images 1104a-d show potential outputs of the generative model when it is not constrained to select a code in any particular portion of the latent vector space W (302). Again, dashed lines connect each input image to a point in the latent vector space W (302) representing the code that the generative model ultimately selects (e.g., after a selection process such as that described above with respect to method 800 of FIG. 8 or method 900 of FIG. 9), and arrows connect that point to the corresponding output image that the generative model creates based on that code. As can be seen, although the output images 1104a-d are visually consistent with the unmasked portions of each of the input images 1102a-d, respectively, these output images do not all appear to be of the same subject.

[0063] In contrast, the right column of images 1106a-d show the potential outputs of the generative model when constrained to select a code in the personalized prior 403 of FIG. 4B. Again, dashed lines connect each input image to a point in the personalized prior 403 representing the code that the generative model ultimately selects (e.g., after a selection process such as that described above with respect to method 800 of FIG. 8 or method 900 of FIG. 9), and arrows connect that point to the corresponding output image that the generative model creates based on that code. As can be seen, this ultimately produces output images 1106a-d, which are visually consistent with each of the input images 1102a-d and all appear to be of the same subject. Specifically, because personalized prior distribution 403 in FIG. 4B represents a range of code that, when provided to generative model 306 (as described above), creates images that resemble Lady Gaga, each of output images 1106a-1106d appears to show an image of Lady Gaga that is visually consistent with the unmasked portions of input images 1102a-1102d.

[0064] Thus, both diagrams 1000 and 1100 focus on the code used by a generative model to provide an example of how personalized priors can be used to enable the generative model to create output images that visually match both the input image and the identity of a particular subject. Thus, when the subject of an input image is already known, personalized priors may be selected and used to bias the generative model toward creating a more representative, and therefore more appropriate, output image.

[0065] Unless otherwise stated, the foregoing alternatives are not mutually exclusive and may be implemented in various combinations to achieve inherent advantages. These and other variations and combinations of the features described above may be utilized without departing from the subject matter defined by the claims, and therefore the foregoing description of the exemplary systems and methods should be taken as illustrative, and not limiting, of the subject matter defined by the claims. In addition, the provision of examples described herein, and phrases such as "such as," "including," and "comprising," should not be construed as limiting the subject matter of the claims to any particular examples, but rather the examples are intended to illustrate only some of many possible embodiments. Additionally, the same reference numbers in different drawings may identify the same or similar elements.

Claims

1. 1. A computer-implemented method comprising: for each given image of the set of images of the object, using one or more processors of a processing system, testing a plurality of codes to identify an optimized code for the given image, said testing including: For each code of the plurality of codes, generating a first image using a generative model and the code; using the one or more processors to compare the first image with the given image to generate a first loss value for the code; using the one or more processors, comparing the generated first loss value for each code of the plurality of codes and identifying the code having a minimum first loss value as the optimized code for the given image; The method comprises: The method further includes using the one or more processors to generate a personalized prior distribution for the object based on a convex hull that includes each of the optimization codes identified for each given image of a set of images of the object.

2. for the optimization code identified for each given image of the set of images of the object, generating a second image using the generative model and the optimization code; and comparing, using the one or more processors, the second image to the given image to generate a second loss value; 2. The method of claim 1 , further comprising: using the one or more processors to modify one or more parameters of the generative model based at least in part on each generated second loss value to construct an adjusted generative model.

3. and further comprising: identifying a plurality of coefficient sets using the one or more processors, each coefficient set of the plurality of coefficient sets corresponding to a code within the convex hull, the method comprising: For each given coefficient set of the plurality of coefficient sets, generating, using the one or more processors, a third image using the adjusted generative model and a given code corresponding to the given set of coefficients; using the one or more processors to compare the third image with at least a portion of the input image of the object to generate a third loss value for the third image; 3. The method of claim 2, further comprising: using the one or more processors to compare the third loss values ​​generated for each of the third images and identify the third image having a minimum third loss value as a personalized output image.

4. and further comprising: identifying a plurality of coefficient sets using the one or more processors, each coefficient set of the plurality of coefficient sets corresponding to a code within the convex hull, the method comprising: and further comprising: identifying, using the one or more processors, a plurality of code sets, each code set of the plurality of code sets including two or more individual codes, each of the individual codes corresponding to a coefficient set of the plurality of coefficient sets, the method comprising: For each given code set of the plurality of code sets, and generating, using the one or more processors, a third image using the adjusted generative model and the given code set, wherein each individual code of the given code set is provided to a different layer or set of layers of the adjusted generative model, the method comprising: For each given code set of the plurality of code sets, using the one or more processors to compare the third image with at least a portion of the input image of the object to generate a third loss value for the third image; 3. The method of claim 2, further comprising: using the one or more processors to compare the third loss values ​​generated for each of the third images and identify the third image having a minimum third loss value as a personalized output image.

5. and further comprising: identifying a plurality of coefficient sets using the one or more processors, each coefficient set of the plurality of coefficient sets corresponding to a code within the convex hull, the method comprising: For each given coefficient set of the plurality of coefficient sets, generating, using the one or more processors, a third image using the generative model and a given code corresponding to the given set of coefficients; using the one or more processors to compare the third image with at least a portion of the input image of the object to generate a third loss value for the third image; 2. The method of claim 1, further comprising: using the one or more processors to compare the third loss values ​​generated for each of the third images and identify the third image having a minimum third loss value as a personalized output image.

6. and further comprising: identifying a plurality of coefficient sets using the one or more processors, each coefficient set of the plurality of coefficient sets corresponding to a code within the convex hull, the method comprising: and further comprising: identifying, using the one or more processors, a plurality of code sets, each code set of the plurality of code sets including two or more individual codes, each of the individual codes corresponding to a coefficient set of the plurality of coefficient sets, the method comprising: For each given code set of the plurality of code sets, and generating, using the one or more processors, a third image using the generative model and the given code set, wherein each individual code of the given code set is provided to a different layer or set of layers of the generative model, the method further comprising: For each given code set of the plurality of code sets, using the one or more processors to compare the third image with at least a portion of the input image of the object to generate a third loss value for the third image; 2. The method of claim 1, further comprising: using the one or more processors to compare the third loss values ​​generated for each of the third images and identify the third image having a minimum third loss value as a personalized output image.

7. 6. The method of claim 3 or claim 5, wherein the plurality of coefficient sets comprises a first coefficient set and a plurality of successive coefficient sets selected using a gradient descent method based directly or indirectly on the first coefficient set.

8. the input image of the object includes a first portion of pixels preserved from the original image of the object and a mask in place of a second portion of pixels from the original image of the object; 8. The method of claim 3, wherein using the one or more processors to compare the third image with at least a portion of the input image of the subject to generate the third loss value for the third image comprises comparing the third image with a first portion of the pixels to generate the third loss value for the third image.

9. The method of any one of claims 3 to 7, wherein the input image has a first resolution and the personalized output image has a second resolution that is higher than the first resolution.

10. 10. The method of claim 1, wherein the plurality of codes comprises a first code and a plurality of successive codes selected using a gradient descent method based directly or indirectly on the first code.

11. 11. The method of claim 10, wherein the first code represents a mean value of a latent vector space W, the latent vector space W representing all possible codes that can be input to the generative model.

12. A memory for storing a generative model; and one or more processors coupled to the memory and configured to generate a personalized prior distribution for the subject for use with the generative model, the one or more processors comprising: For each given image of the set of images of the object, testing a plurality of codes to identify an optimized code for the given image, said testing including: For each code of the plurality of codes, generating a first image using the generative model and the code; comparing the first image with the given image to generate a first loss value for the code; comparing the generated first loss value for each code of the plurality of codes to identify the code having a smallest first loss value as the optimized code for the given image, wherein the one or more processors further include: A processing system that generates the personalized prior distribution for the object based on a convex hull that includes each of the optimization codes identified for each given image in a set of images of the object.

13. The one or more processors are further configured to adjust the generative model, the one or more processors: for the optimization code identified for each given image of the set of images of the object, generating a second image using the generative model and the optimization code; using the one or more processors to compare the second image with the given image to generate a second loss value; The processing system of claim 12 , further comprising: modifying one or more parameters of the generative model based at least in part on each generated second loss value to construct an adjusted generative model, using the one or more processors.

14. The one or more processors are further configured to generate a personalized output image based on the input image of the subject, the one or more processors further configured to: identifying a plurality of coefficient sets, each coefficient set of the plurality of coefficient sets corresponding to a code within the convex hull, the one or more processors further comprising: For each given coefficient set of the plurality of coefficient sets, generating a third image using the adjusted generative model and a given code corresponding to the given set of coefficients; comparing the third image with at least a portion of the input image of the object to generate a third loss value for the third image; 14. The processing system of claim 13, further comprising: comparing the third loss values ​​generated for each of the third images; and identifying the third image having the smallest third loss value as a personalized output image.

15. The one or more processors are further configured to generate a personalized output image based on the input image of the subject, the one or more processors further configured to: identifying a plurality of coefficient sets, each coefficient set of the plurality of coefficient sets corresponding to a code within the convex hull, the one or more processors further comprising: identifying a plurality of code sets, each code set of the plurality of code sets including two or more individual codes, each of the individual codes corresponding to a coefficient set of the plurality of coefficient sets, the one or more processors further comprising: For each given code set of the plurality of code sets, generating a third image using the adjusted generative model and the given code set, wherein each individual code of the given code set is provided to a different layer or set of layers of the adjusted generative model, and the one or more processors further comprise: For each given code set of the plurality of code sets, comparing the third image with at least a portion of the input image of the object to generate a third loss value for the third image; 14. The processing system of claim 13, further comprising: comparing the third loss values ​​generated for each of the third images; and identifying the third image having the smallest third loss value as a personalized output image.

16. The one or more processors are further configured to generate a personalized output image based on the input image of the subject, the one or more processors further configured to: identifying a plurality of coefficient sets, each coefficient set of the plurality of coefficient sets corresponding to a code within the convex hull, the one or more processors further comprising: For each given coefficient set of the plurality of coefficient sets, generating a third image using the generative model and a given code corresponding to the given set of coefficients; comparing the third image with at least a portion of the input image of the object to generate a third loss value for the third image; 13. The processing system of claim 12, further comprising: comparing the third loss values ​​generated for each of the third images; and identifying the third image having the smallest third loss value as a personalized output image.

17. The one or more processors are further configured to generate a personalized output image based on the input image of the subject, the one or more processors further configured to: identifying a plurality of coefficient sets, each coefficient set of the plurality of coefficient sets corresponding to a code within the convex hull, the one or more processors further comprising: identifying a plurality of code sets, each code set of the plurality of code sets including two or more individual codes, each of the individual codes corresponding to a coefficient set of the plurality of coefficient sets, the one or more processors further comprising: For each given code set of the plurality of code sets, generating a third image using the generative model and the given code set, each individual code of the given code set being provided to a different layer or set of layers of the generative model, the one or more processors further comprising: For each given code set of the plurality of code sets, comparing the third image with at least a portion of the input image of the object to generate a third loss value for the third image; 13. The processing system of claim 12, further comprising: comparing the third loss values ​​generated for each of the third images; and identifying the third image having the smallest third loss value as a personalized output image.

18. the plurality of coefficient sets includes a first coefficient set and a plurality of successive coefficient sets; 17. The processing system of claim 14 or claim 16, wherein the one or more processors are further configured to select each coefficient set of the plurality of series of coefficient sets using a gradient descent method based directly or indirectly on the first coefficient set.

19. the input image of the object includes a first portion of pixels preserved from the original image of the object and a mask in place of a second portion of pixels from the original image of the object; 19. The processing system of claim 14, wherein comparing the third image with at least a portion of the input image of the subject to generate the third loss value for the third image comprises comparing the third image with a first portion of the pixels to generate the third loss value for the third image.

20. 19. The processing system of claim 14, wherein the one or more processors are configured to generate the personalized output image based on an input image of the subject, the input image having a first resolution, and the personalized output image having a second resolution higher than the first resolution.

21. the plurality of codes includes a first code and a plurality of series of codes; 21. The processing system of claim 12, wherein the one or more processors are further configured to select each code of the plurality of sequences of codes using a gradient descent method based directly or indirectly on the first code.

22. 22. The processing system of claim 21 , wherein the one or more processors are further configured to select a first code that represents a mean value of a latent vector space W, where the latent vector space W represents all possible codes that can be input to the generative model.

23. A memory for storing a generative model; A processing system comprising: one or more processors coupled to the memory and configured to perform the method of any one of claims 1 to 11.

24. A program for causing one or more processors to execute the method according to any one of claims 1 to 11.

Images