Facial image editing and enhancement using personalized priors

Abstract

Systems and methods for identifying a personalized prior within a latent vector space of a generative model based on a set of images of a given subject. In some examples, the technology can also include using the personalized prior to constrain the input to the generative model to the latent vector space associated with the given subject, such that when the model is tasked with editing an image of the subject (e.g., performing inpainting to fill in masked regions, increasing resolution, or deblurring the image), the identifying features of the subject will be reflected in the image produced by the model.

Description

BACKGROUND

[0001] Generative models can be used to perform tasks ranging from editing and augmenting images of a given object to generating realistic images (or image portions) of a given object or a synthetically generated object. To adequately train such models, a large set of images, typically of a large set of objects, is required. Thus, when generative models are used to edit, augment, or fill in a portion of an image of a known object, they can produce images that look realistic but resemble a different object. SUMMARY

[0002] The present technology relates to systems and methods for identifying a personalized prior within a latent vector space of a generative model based on a set of images of a given object. In some aspects, the present technology can also include using the personalized prior (e.g., a convex hull defined by a set of codes generated based on the set of images of the object) to restrict codes input to the generative model such that identified features of the object will be reflected in images produced by the model. For example, a generative model can be configured to augment or fill in facial features in an image of an object where only partial cues related to the identity of the object are present (e.g., due to motion blur, low light, low resolution, being occluded by other objects). Without the present technology, the model can successfully augment or fill in such an image but can do so by producing an image that looks like it belongs to a different object. The present technology can be used to focus the generative model such that images it produces are more consistent with the appearance of the object.

[0003] In one aspect, the present disclosure describes a computer-implemented method comprising: (1) for each given image in a set of images of an object, using one or more processors of a processing system, testing a plurality of codes to identify an optimized code for the given image, including: (a) for each code in the plurality of codes: using a generative model and the code to generate a first image; and using the one or more processors, comparing the first image to the given image to generate a first loss value for the code; and (b) using the one or more processors, comparing the first loss value generated for each code in the plurality of codes to identify the code with the lowest first loss value as the optimized code for the given image; and (2) using the one or more processors, generating a personalized prior for the object based on a convex hull comprising each optimized code identified for each given image in the set of images of the object. In some aspects, the method further comprises: (1) for each optimized code identified for each given image in the set of images of the object: using the generative model and the optimized code to generate a second image; and using the one or more processors, comparing the second image to the given image to generate a second loss value; and (2) using the one or more processors, modifying one or more parameters of the generative model based at least in part on each generated second loss value to create an adjusted generative model. In some aspects, the method further comprises: (1) using the one or more processors, identifying a plurality of coefficient sets, each coefficient set in the plurality of coefficient sets corresponding to a code within the convex hull; (2) for each given coefficient set in the plurality of coefficient sets: using the one or more processors, using the adjusted generative model and a given code corresponding to the given coefficient set to generate a third image; and using the one or more processors, comparing the third image to at least a portion of an input image of the object to generate a third loss value for the third image; and (3) using the one or more processors, comparing the third loss value generated for each third image to identify the third image with the lowest third loss value as a personalized output image.In some aspects, the method further comprises: (1) identifying, using the one or more processors, a plurality of coefficient sets, each coefficient set of the plurality of coefficient sets corresponding to a code within the convex hull; (2) identifying, using the one or more processors, a plurality of code sets, each code set of the plurality of code sets comprising two or more individual codes, each individual code corresponding to a coefficient set of the plurality of coefficient sets; (3) for each given code set of the plurality of code sets: generating, using the one or more processors, a third image using the adjusted generative model and the given code set, each individual code of the given code set being provided to a different layer or set of layers of the adjusted generative model; and comparing, using the one or more processors, the third image to at least a portion of the input image of the subject to generate a third loss value for the third image; and (4) comparing, using the one or more processors, the third loss value generated for each third image to identify the third image having the lowest third loss value as the personalized output image. In some aspects, the method further comprises: (1) identifying, using the one or more processors, a plurality of coefficient sets, each coefficient set of the plurality of coefficient sets corresponding to a code within the convex hull; (2) for each given coefficient set of the plurality of coefficient sets: generating, using the one or more processors, a third image using the generative model and a given code corresponding to the given coefficient set; and comparing, using the one or more processors, the third image to at least a portion of the input image of the subject to generate a third loss value for the third image; and (3) comparing, using the one or more processors, the third loss value generated for each third image to identify the third image having the lowest third loss value as the personalized output image. In some aspects, the method further comprises: (1) identifying, using the one or more processors, a plurality of coefficient sets, each coefficient set of the plurality of coefficient sets corresponding to a code within the convex hull; (2) identifying, using the one or more processors, a plurality of code sets, each code set of the plurality of code sets comprising two or more individual codes, each individual code corresponding to a coefficient set of the plurality of coefficient sets; (3) for each given code set of the plurality of code sets: generating, using the one or more processors, a third image using the generative model and the given code set, each individual code of the given code set being provided to a different layer or set of layers of the generative model; and comparing, using the one or more processors, the third image to at least a portion of the input image of the subject to generate a third loss value for the third image; and (4) comparing, using the one or more processors, the third loss value generated for each third image to identify the third image having the lowest third loss value as the personalized output image.In some aspects, the plurality of coefficient sets includes a first coefficient set, and a plurality of successive coefficient sets selected directly or indirectly based on the first coefficient set using gradient descent. In some aspects, the input image of the object includes a first pixel portion preserved from an original image of the object, and a mask that replaces a second pixel portion from the original image of the object, and comparing, using the one or more processors, the third image to at least a portion of the input image of the object to generate the third loss value for the third image includes comparing the third image to the first pixel portion to generate the third loss value for the third image. In some aspects, the input image has a first resolution, and the personalized output image has a second resolution that is higher than the first resolution. In some aspects, the plurality of codes includes a first code, and a plurality of successive codes selected directly or indirectly based on the first code using gradient descent. In some aspects, the first code represents a mean value of a latent vector space W that represents all possible codes that can be input into the generative model.

[0004] In another aspect, the disclosure describes a processing system comprising: a memory storing a generative model; and one or more processors coupled to the memory and configured to perform any of the methods just described.

[0005] In another aspect, the present disclosure describes a processing system comprising: (1) a memory storing a generative model; and (2) one or more processors coupled to the memory and configured to generate a personalized prior for a subject for use with the generative model, comprising: (a) for each given image in a set of images of the subject, testing a plurality of codes to identify an optimal code for the given image, comprising: (i) for each code in the plurality of codes: generating a first image using the generative model and the code; and comparing the first image to the given image to generate a first loss value for the code; and (ii) comparing the first loss value generated for each code in the plurality of codes to identify the code with the lowest first loss value as the optimal code for the given image; and (b) generating the personalized prior for the subject based on a convex hull comprising each optimal code identified for each given image in the set of images of the subject. In some aspects, the one or more processors are further configured to tune the generative model, comprising: (1) for each optimal code identified for each given image in the set of images of the subject: generating a second image using the generative model and the optimal code; and comparing the second image to the given image using the one or more processors to generate a second loss value; and (2) modifying one or more parameters of the generative model based at least in part on each generated second loss value using the one or more processors to create a tuned generative model. In some aspects, the one or more processors are further configured to generate a personalized output image based on an input image of the subject, comprising: (1) identifying a plurality of coefficient sets, each coefficient set in the plurality of coefficient sets corresponding to a code within the convex hull; (2) for each given coefficient set in the plurality of coefficient sets: generating a third image using the tuned generative model and a given code corresponding to the given coefficient set; and comparing the third image to at least a portion of the input image of the subject to generate a third loss value for the third image; and (3) comparing the third loss value generated for each third image to identify the third image with the lowest third loss value as the personalized output image.In some aspects, the one or more processors are further configured to generate the personalized output image based on the input image of the subject, which includes: (1) identifying a plurality of coefficient sets, each coefficient set of the plurality of coefficient sets corresponding to a code within the convex hull; (2) identifying a plurality of code sets, each code set of the plurality of code sets including two or more individual codes, each individual code corresponding to a coefficient set of the plurality of coefficient sets; (3) for each given code set of the plurality of code sets: generating a third image using the adjusted generative model and the given code set, each individual code of the given code set being provided to a different layer or set of layers of the adjusted generative model; and comparing the third image to at least a portion of the input image of the subject to generate a third loss value for the third image; and (4) comparing the third loss values generated for each third image to identify the third image having the lowest third loss value as the personalized output image. In some aspects, the one or more processors are further configured to generate the personalized output image based on the input image of the subject, which includes: (1) identifying a plurality of coefficient sets, each coefficient set of the plurality of coefficient sets corresponding to a code within the convex hull; (2) for each given coefficient set of the plurality of coefficient sets: generating a third image using the generative model and a given code corresponding to the given coefficient set; and comparing the third image to at least a portion of the input image of the subject to generate a third loss value for the third image; and (3) comparing the third loss values generated for each third image to identify the third image having the lowest third loss value as the personalized output image. In some aspects, the one or more processors are further configured to generate the personalized output image based on the input image of the subject, which includes: (1) identifying a plurality of coefficient sets, each coefficient set of the plurality of coefficient sets corresponding to a code within the convex hull; (2) identifying a plurality of code sets, each code set of the plurality of code sets including two or more individual codes, each individual code corresponding to a coefficient set of the plurality of coefficient sets; (3) for each given code set of the plurality of code sets: generating a third image using the generative model and the given code set, each individual code of the given code set being provided to a different layer or set of layers of the generative model; and comparing the third image to at least a portion of the input image of the subject to generate a third loss value for the third image; and (4) comparing the third loss values generated for each third image to identify the third image having the lowest third loss value as the personalized output image. In some aspects, the plurality of coefficient sets includes a first coefficient set and a plurality of consecutive coefficient sets, and the one or more processors are further configured to select each coefficient set of the plurality of consecutive coefficient sets based on the first coefficient set directly or indirectly using gradient descent.In some aspects, the input image of the object includes a first portion of pixels preserved from an original image of the object, and a mask that replaces a second portion of pixels from the original image of the object, and comparing the third image to at least a portion of the input image of the object to generate the third loss value for the third image includes comparing the third image to the first portion of pixels to generate the third loss value for the third image. In some aspects, the one or more processors are configured to generate the personalized output image based on the input image of the object, wherein the input image has a first resolution, and the personalized output image has a second resolution higher than the first resolution. In some aspects, the plurality of codes includes a first code and a plurality of consecutive codes, and the one or more processors are further configured to select each code of the plurality of consecutive codes based on the first code directly or indirectly using gradient descent. In some aspects, the one or more processors are further configured to select the first code that represents a mean value of a latent vector space W, the latent vector space W representing all possible codes that can be input into the generative model. BRIEF DESCRIPTION OF DRAWINGS

[0006] FIG. 1 is a functional diagram of an example system in accordance with aspects of the present disclosure.

[0007] FIG. 2 is a functional diagram of an example system in accordance with aspects of the present disclosure.

[0008] FIG. 3A and FIG. 3B shows how different codes within a latent vector space from a generative model can produce images of different objects in accordance with aspects of the present disclosure.

[0009] FIG. 4A and FIG. 4B shows two different personalized priors within a latent vector space of a generative model, and how points within a given personalized prior can produce different images of a given object in accordance with aspects of the present disclosure.

[0010] FIG. 5 sets forth an example method for generating a personalized prior based on a set of images of an object in accordance with aspects of the present disclosure.

[0011] FIG. 6 shows how a personalized prior of FIG. 4A can be generated using a given set of images in accordance with aspects of the present disclosure.

[0012] FIG. 7 sets forth an example method for adjusting a generative model after a method in accordance with FIG. 5 identifies an optimal code for each image in a set of images.

[0013] FIG. 8 Exemplary methods for generating personalized output images based on an input image and a personalized prior generated according to FIG. 5 or FIG. 7 are set forth according to aspects of the present disclosure.

[0014] FIG. 9 Exemplary methods for generating personalized output images based on an input image and a personalized prior generated according to FIG. 5 or FIG. 7 are set forth according to aspects of the present disclosure.

[0015] FIG. 10 A comparative illustration of how a generative model according to aspects of the present disclosure can complete four exemplary image enhancement tasks with and without personal priors of FIG. 4A is shown.

[0016] FIG. 11 A comparative illustration of how a generative model according to aspects of the present disclosure can complete four exemplary image inpainting tasks with and without personal priors of FIG. 4B is shown. DETAILED DESCRIPTION

[0017] The present technology will now be described with respect to the following exemplary systems and methods. Common reference numbers are intended to identify the same features between the drawings depicted and described hereafter.

[0018] Example System

[0019] FIG. 1 A high-level system diagram 100 of an exemplary processing system 102 for performing the methods described herein is shown. The processing system 102 can include one or more processors 104 and a memory 106 storing instructions 108 and data 110. The instructions 108 and data 110 can include a generative model (e.g., a generative model 306 of FIG. 3A , FIG. 3B , FIG. 4A , FIG. 4B , FIG. 6 , FIG. 10 and FIG. 11 as described herein. Further, the data 110 can store training examples used to train such a generative model, data used by the generative model in generating images, a set of images of a given subject, a personalized prior based on the set of images of the given subject (e.g., a personalized prior 308 of FIG. 4A , FIG. 4B , FIG. 6 , FIG. 10 and FIG. 11personalized priors 402 and 403) and / or images output by the generative model.

[0020] The processing system 102 can reside on a single computing device. For example, the processing system 102 can be a server, a personal computer, or a mobile device, and thus, the generative models and data associated therewith can be local to that single computing device. Similarly, the processing system 102 can reside on a cloud computing system or other distributed system. In such cases, the generative models and / or data associated therewith can be distributed across two or more different physical computing devices. For example, in some aspects of the technology, the processing system can include a first computing device that stores a generative model, and a second computing device that stores data to be used by the generative model when generating images, a set of images of a given object, personalized priors based on the set of images of the given object, and / or images output by the generative model. Likewise, in some aspects of the technology, the processing system can include a first computing device that stores 1-n layers of a generative model having m layers, and a second computing device that stores n-m layers of the generative model.

[0021] Further in this aspect, FIG. 2 An advanced system diagram 200 is shown in which the example processing system 102 just described is shown in communication with various websites and / or remote storage systems, including websites 210 and 218 and a remote storage system 226, over one or more networks 208. In this example, the websites 210 and 218 each include one or more servers 212a-212n and 220a-220n, respectively. Each of the servers 212a-212n and 220a-220n can have one or more processors (e.g., 214 and 222) and associated memory (e.g., 216 and 224) that stores instructions and data, including content of one or more webpages. Likewise, although not shown, the remote storage system 226 can also include one or more processors and memory that stores instructions and data. In some aspects of the technology, the processing system 102 can be configured to retrieve data, training examples, a set of images of a given object, and / or an input image of the given object from one or more of the websites 210, the websites 218, and / or the remote storage system 226 to provide to a generative model to train or condition the generative model, and / or use when generating images.

[0022] The processing systems described herein can be implemented on any type of computing device, such as any type of general purpose computing device, server, or collection thereof, and can also include other components typically found in general purpose computing devices or servers. Likewise, the memory of such processing systems can be of any non-transitory type capable of storing information accessible by the processors of the processing systems. For example, the memory can include non-transitory media, such as a hard drive, memory card, optical disk, solid state, tape memory, etc. Computing devices suitable for the roles described herein can include different combinations of the foregoing, whereby different portions of instructions and data are stored on different types of media.

[0023] In all cases, the computing devices described herein can also include any other components typically used in computing devices, such as user interface subsystems. User interface subsystems can include one or more user inputs (e.g., mouse, keyboard, touch screen, and / or microphone) and one or more electronic displays (e.g., a monitor with a screen or any other electrical device operable to display information). Output devices other than electronic displays, such as speakers, lights, and vibrating, pulsing, or haptic elements, can also be included in the computing devices described herein.

[0024] The one or more processors included in each computing device can be any conventional processor, such as commercially available central processing units ("CPUs"), graphics processing units ("GPUs"), tensor processing units ("TPUs"), etc. Alternatively, the one or more processors can be a special-purpose device, such as an ASIC or other hardware-based processor. Each processor can have multiple cores capable of operating in parallel. The processors, memory, and other elements of a single computing device can be stored within a single physical housing, or can be distributed among two or more housings. Similarly, the memory of a computing device can include a hard drive or other storage media located in a different housing from that of the processors, such as located in an external database or networked storage device. Thus, references to a processor or computing device will be understood to include references to a collection of processors or computing devices or memory that can or can not operate in parallel, as well as references to a load-balanced server farm or one or more servers of a cloud-based system.

[0025] The computing devices described herein can store instructions (such as machine code) that can be executed directly by a processor or instructions (such as scripts) that are executed indirectly by a processor. The computing devices can also store data that can be retrieved, stored, or modified by one or more processors in accordance with the instructions. The instructions can be stored as computing device code on a computing device-readable medium of the computing device. In this respect, the terms “instructions” and “programs” can be used interchangeably herein. The instructions can also be stored in object code format for direct processing by a processor, or in any other computing device language that can be read directly by a processor, including scripts or collections of independent source code modules that are interpreted on demand or compiled just in time. By way of example, a programming language can be C#, C++, JAVA, or another computer programming language. Similarly, any assembly of the instructions or programs can be implemented in a computer script language, such as JavaScript, PHP, ASP, or any other computer script language. In addition, any of these components can be implemented using a combination of computer programming languages and computer script languages.

[0026] Example Method

[0027] FIG. 3A and FIG. 3B It is shown how different codes within a latent vector space W (302) from a generative model 306 can produce images (308, 309) of different objects, in accordance with aspects of the present disclosure. In these examples, the generative model 306 can be any suitable model configured to generate an output image based on an input code, and the latent vector space W (302) represents the range of all possible input codes that can be provided to the generative model 306. In this respect, the latent vector space W (302) is shown as a two-dimensional space for simplicity of illustration only. FIG. 3A and FIG. 3B (And FIG. 4A , FIG. 4B , FIG. 6 , FIG. 10 and FIG. 11 ) are shown as two-dimensional spaces. However, the present technology can be applied to any suitable generative model (e.g., a GAN or a bidirectional GAN (“BiGAN”)) configured to generate images based on input vectors of any suitable number of dimensions. For example, in some aspects of the technology, the generative model 306 can be a generative adversarial network configured to use a latent vector space W having 512 dimensions (e.g., StyleGAN, StarGAN).

[0028] As shown in FIG. 3A , a code at a point 304 of the latent vector space W (302) is fed to the generative model 306, resulting in an image 308. To illustrate the effect of the present technology, FIG. 3A ,FIG. 3B , FIG. 4A , FIG. 4B , FIG. 10 and FIG. 11 The examples described each use images of objects whose portraits many people would be able to recognize. Therefore, in FIG. 3A In the example, it has been assumed that the code at point 304 will produce image 308 of a well-known American figure. Similarly, in FIG. 3B In the example, it is assumed that the code at point 305 will produce image 309 of a well-known American singer, songwriter, and actor.

[0029] FIG. 4A and FIG. 4B Two distinct personalized priors 402, 403 within the latent vector space W(302) of the generative model 306 are shown, and it is also shown how points within a given personalized prior (e.g., 304 and 404, 305 and 405) can generate different images of a given object according to various aspects of this disclosure. FIG. 4A and FIG. 4B The examples all assume the use of FIG. 3A and FIG. 3B The same latent vector space W(302) and the same generative model 306 are used, but the individual priors 402 and 403 within the latent vector space W(302) are described in addition.

[0030] Personal priors 402 and 403 each represent a subset of the latent vector space W(302), comprising a subset of possible input codes that produce an image resembling a given object. In this case, assume that personal prior 402 represents a range of codes that, when provided to generative model 306, produce an image resembling a well-known American figure. Therefore, personal prior 402 includes point 304 (representing the production...). FIG. 3A Image 308 (code) and another point 404 (code 406 representing the generation of a different image of a famous American figure).

[0031] Similarly, suppose personal prior 403 represents a range of codes that, when provided to generative model 306, produce images resembling a famous American singer, songwriter, and actor. Therefore, personal prior 403 includes point 304 (representing the generation of...). FIG. 3B Image 309 (code) and another point 405 (code that produces different images 407 of a famous American singer, songwriter and actor).

[0032] Similarly, here, for the sake of simplicity, personal priors 402 and 403 are... FIG. 4A and FIG. 4B In China (and in FIG. 6 , FIG. 10 andFIG. 11 The personal priors 402 and 403 are shown as being two-dimensional spaces. However, since the present technology can be applied to any suitable generative model configured to generate images based on input vectors of any suitable number of dimensions, the personal priors 402 and 403 can likewise be vector spaces of any number of dimensions less than or equal to the number of dimensions in the latent vector space W (302). Thus, for example, where the generative model 306 is a generative adversarial network configured to use a latent vector space W having 512 dimensions, the personal priors 402 and 403 can each be vector spaces of up to 512 dimensions. Moreover, for clarity of illustration, the example vector spaces 402 and 403 are shown as being completely separate. However, in practice, the personal priors for different subjects can cross over one another.

[0033] FIG. 5 An example method 500 for generating a personalized prior based on a set of images of a subject in accordance with aspects of the present disclosure is set forth.

[0034] In step 502, a processing system (e.g., the processing system 102) selects a given image from a set of images of a subject. It will be appreciated that a larger number of images will generally yield a more representative personal prior than a smaller number of images. In this regard, it has been found that a number of images between 100 and 200 will generally enable a generative model to produce images that look realistic and consistent with the subject’s identity. However, other factors can also influence how well a personal prior represents the appearance of a subject. For example, if the appearance of a subject changes significantly (e.g., due to a change in hairstyle, hair color, addition or removal of facial hair, or passage of a significant amount of time), it can be helpful to tailor the set of images to a particular stage, such that the personal prior will reflect a single “look” and the images produced by the generative model will be consistent with that look. Likewise, it can be helpful to constrain the set of images to a single stage of life (e.g., infancy, childhood, adolescence, adulthood, etc.) for the same reason. On the other hand, variability in the set of images can also be important. For example, a set of images showing the subject’s face only from the front can not be as useful as a set of images showing the subject’s face from a variety of different angles, under different lighting conditions, etc. A personal prior generated from too similar images can overly constrain the generative model, resulting in images that resemble the subject but do not always look realistic.

[0035] As shown in step 504, after the processing system selects a given image, it will repeat the execution of steps 506-514 in order to test multiple codes to identify an optimal code for the given image. In this regard, in step 506, the processing system will identify a code to be tested in the current pass. This code can be identified based on any suitable selection criteria. For example, in some aspects of the technology, the processing system can be configured to blindly select a first code (e.g., using a random selection process, or a pre-selected value such as the mean value of the latent vector space W), and then select each successive code (in each successive pass through step 506) directly or indirectly based on the first code using a suitable optimization scheme. Thus, in some aspects, the processing system can be configured to use gradient descent to select each successive code based on the previous code and an assessment of how well the image generated based on the previous code matches the given image (e.g., the first loss value generated in the most recent pass through step 510).

[0036] In step 508, the processing system generates a first image using the generative model (e.g., generative model 306) and the code (identified here in step 506). The generative model can be configured to produce the first image in any suitable manner, including in the manner as described above with respect to FIG. 3A 、 FIG. 3B 、 FIG. 4A and FIG. 4B .

[0037] In step 510, the processing system compares the first image (generated here in step 508) to the given image (selected in step 502) to generate a first loss value for the code. The first loss value can be generated in any suitable manner using any suitable function. For example, in some aspects of the technology, the first loss value can be based on a comparison of the first image to the given image using a heuristic or learned similarity metric (e.g., learned perceptual image patch similarity (“LPIPS”), peak signal-to-noise ratio (“PSNR”), structural similarity index measurement (“SSIM”), LI or L2 loss, etc.).

[0038] At step 512, the processing system determines whether another code should be tested. Since multiple codes are assumed to be tested, the first time the processing system reaches step 512, it will automatically follow the "yes" arrow and return to step 506, such that a second code will be identified and tested. However, on all subsequent returns to step 512, the processing system can determine whether to test another code based on any suitable criteria. Thus, as mentioned above, in some aspects of the technology, the processing system can determine when to stop testing another code based on a suitable optimization scheme, such as gradient descent. In this case, the determination in step 512 can be based on a comparison of the first loss value generated in the current pass through step 510 (or some other assessment of how well the first image matches the given image) to one or more of the first loss values generated in previous passes. For example, the processing system can be configured to stop testing consecutive codes when the first loss value generated in the current pass through step 510 is equal to or greater than the first loss value generated in the previous pass through step 510.

[0039] Thus, the processing system will loop through steps 506-512 for each successive code until it is determined at step 512 that enough codes have been tested. At this point, the processing system will follow the "no" arrow to step 514, where it will compare the first loss values generated (in step 510) for each of the plurality of codes to identify the code with the lowest first loss value. This code with the lowest first loss value is selected as the optimized code for the given image.

[0040] Then, in step 516, the processing system will determine whether there are any additional images in the set of images of the object. If yes, the processing system will follow the "yes" arrow to step 518, where the processing system will select the next given image to be tested. The processing system will then return to step 504 to test the plurality of codes to identify an optimized code for this new given image. In this way, steps 504-518 will be repeated as described above until an optimized code is identified for each image in the set of images of the object. Once an optimized code is selected for the last image in the set of images of the object (at step 514), the processing system will determine at step 516 that there are no additional images in the set, and thus follow the "no" arrow to step 520.

[0041] In step 520, the processing system will generate a personalized prior for the object based on the convex hull comprising each optimized code identified (in step 514) for each given image in the set of images of the object. In this regard, in some aspects of the technology, the personalized prior can simply be the convex hull defined by each optimized code identified in step 514. In this case, assuming there is a set of n optimized codes {x1, x2,..., xn}, the personalized prior can be defined as: x = arg min x∈{x1, x2,..., xn} f(x)n} the personalized prior will include any code c generated by optimizing a linear combination of the codes according to Equations 1-3 below, where each of the coefficients (a values a1through a n ) is greater than or equal to 0, and the sum of all coefficients is 1.

[0042] c = a1x1+ a2x2+... + a n x n (1)

[0043]

[0044]

[0045] Likewise, in some aspects of the technology, the personalized prior can be some subset of the convex hull defined by each of the optimization codes identified in step 514, such as a set of some predetermined number (e.g., 100, 500, 1,000, 10,000, 100,000, 1,000,000) of the codes or a set of coefficients corresponding to a sampled set of points within the convex hull. Further, in some aspects of the technology, the personalized prior can be a simpler hull (e.g., a hull with fewer vertices) that coincides with or substantially overlaps the actual convex hull defined by each of the optimization codes identified in step 514. Further, in some aspects of the technology, the personalized prior can be implemented by encompassing a broader set of codes than the set of codes defined by Equations 1-3 above based on the convex hull defined by Equations 1-3 above. For example, the personalized prior can include any code c generated according to Equations 1 and 3 above, where the coefficients (a values a1through a n ) are greater than or equal to some predetermined negative value (e.g., -0.01, -0.05, -0.1).

[0046] FIG. 6 It is shown how a given set of n images 602a-612a can be used to generate a personalized prior 402 of FIG. 4A in accordance with aspects of the present disclosure. In this aspect, each image 602a-612a is assumed to be one image of a set of n images of some object, and the depicted lines show how these six selected images correspond to different optimization codes 602b-612b within the latent vector space W (302) of a generative model (e.g., generative model 306). These optimization codes 602b-612b can each be found according to steps 502-518 of the example method of FIG. 5 Further, FIG. 6Exemplary illustration 600 shows how each of these optimized codes 602b-612b (along with the optimized codes for the rest of the set of n images, not shown) can be used to define the convex hull on which the personalized prior 402 is based, as described above relative to... FIG. 5 Step 520 is further described. Again, for the sake of simplicity only, FIG. 6 The personal prior 403 is shown as a two-dimensional space, and only six sample images from a set of n images are shown. However, the personal prior 403 can be based on any suitable number of points and can be a vector space of any suitable number of dimensions less than or equal to the number of dimensions in the latent vector space W(302), as explained above.

[0047] FIG. 7 The various aspects of this disclosure are described for use in accordance with... FIG. 5 An exemplary method 700 is described, which adjusts the generative model after identifying optimized code for each image in an image set. In this respect, exemplary method 700 indicates that optimized code can be identified for each image in the image set (in... FIG. 5 The process that can be optionally executed after optimizing the code (step 514).

[0048] Therefore, in step 702, it is assumed that the processing system (e.g., processing system 102) will perform at least [the following steps] for each image in the image set. FIG. 5 Steps 502-518 of the exemplary method. Although FIG. 7 The exemplary method shows that steps 704-720 occur after step 702, but it should be understood that steps 704-720 can be relative to... FIG. 5 Steps 502-518 of the exemplary method are performed in any suitable order. For example, in some aspects of this technology, the processing system can be configured to optimize code for recognition of a given image in a set of images (in... FIG. 5 In step 514), and then in selecting the next given image (in FIG. 5 In step 518) before or with optimized code that tests multiple codes to identify the next given image (e.g., FIG. 5 Steps 704-712 are performed on the given image in parallel with steps 504-514). In this case, the processing system can also be configured to work with... FIG. 5 Steps 504-518 are performed in parallel with step 714 (after processing each batch of images according to steps 704-712), so that the parameters of the generated model are periodically updated as the optimization code continues to be identified (in successive steps 518).

[0049] Regardless of the timing, in step 704, the processing system selects a given image from the image set of the object. This image set can be... FIG. 5 The entire set of images used, or any suitable subset thereof. For example, in some aspects of this technique, the processing system can be configured to... FIG. 5 Steps 502-518 are code for identifying a set of 200 images of the object, but can then be configured to tune the generative model based on only 100 of these images (e.g., generative model 306).

[0050] In step 706, the processing system uses a generative model and recognition for a given image (in... FIG. 3A The optimized code in step 514 is used to generate the second image. Also here, the generative model can be configured in any suitable manner (including as described above relative to...). FIG. 3B , FIG. 4A , FIG. 4B and FIG. 5 The second image is generated in the manner described. Furthermore, although step 706 refers to a "second image," it should be understood that this second image can, in some cases, be... FIG. 5 A copy of one of the "first images" generated in step 508. Therefore, in some aspects of this technique, for any "given image" in the first batch of images, the processing system can be configured to use... FIG. 5 The "first image" (or a copy thereof) associated with the optimization code identified in step 514 is used as the "second image" in step 706, instead of generating the image a second time in step 706. However, once the processing system modifies one or more parameters of the generative model in step 714 (discussed below), the generative model changes such that each subsequent optimization code will produce a "second image" different from the second image previously produced by that code. Therefore, in the case where method 700 is executed after method 500 (instead of being executed in parallel with some or all of the steps of method 500), the processing system can be configured to generate a new "second image" in each successive pass through step 706 (instead of using...). FIG. 5 (The copy of the "first image" identified in step 508).

[0051] In step 708, the processing system compares the second image (generated in step 706) to the given image (selected in step 704) to generate a second loss value. Again, here, the second loss value can be generated in any suitable manner using any suitable function. For example, in some aspects of the technology, the second loss value can be based on a comparison of the second image to the given image using a heuristic or learned similarity metric (e.g., learned perceptual image patch similarity (“LPIPS”), peak signal-to-noise ratio (“PSNR”), structural similarity index measurement (“SSIM”)). Moreover, although step 708 references a “second loss value,” it will be appreciated that this second loss value can in some cases be a copy of the “lowest first loss value” identified for the image in step 514 of the method 500. Thus, in some aspects of the technology, for any “given image” in the first batch of images, the processing system can be configured to use the “lowest first loss value” identified for the given image in step 514 of the method 500 as the “second loss value” of step 708, rather than regenerating the loss value in step 708. However, as already mentioned, once the processing system modifies one or more parameters of the generative model in step 714 (discussed below), the generative model changes, such that each successive optimization code will produce a different “second image” than the second image previously produced by that code, and thus a different “second loss value” when compared to the “given image.” Thus, in cases where the method 700 is performed after the method 500 (rather than in parallel with some or all steps of the method 500), the processing system can be configured to generate a new “second loss value” in each successive pass through step 708 (rather than using a copy of the “lowest first loss value” identified in step 514 of the method 500). FIG. 5 FIG. 5 FIG. 8

[0052] ​​​In step 710, the processing system determines whether there are additional images in the batch. In this regard, the image collection can remain intact, or be divided into any suitable number of batches. In the case where the image collection has not been split, and thus there is a single “batch” containing every image in the “image collection” of the object, the processing system will follow the “yes” arrow to step 712 to select the next given image from the image collection of the object, and repeat steps 706-710 for the newly selected image. This process will be repeated until there are no additional images remaining in the image collection, at which point the processing system will follow the “no” arrow to step 714. On the other hand, in the case where the image collection is divided into two or more batches (e.g., a collection of 200 images can be divided into two batches of 100 images each, four batches of 50 images each, ten batches of 20 images each, 200 “batches” of a single image, etc.), steps 704-712 will be repeated for each image until the end of the batch is reached.

[0053] As shown in step 714, after generating a “second loss value” (in step 708) for each image in the batch, the processing system modifies one or more parameters of the generative model based at least in part on each generated second loss value. The processing system can be configured to modify the one or more parameters in any suitable manner and at any suitable interval based on the generated second loss values. Thus, in some aspects of the technology, each “batch” can include a single image, such that the processing system will perform a backpropagation step in which the processing system modifies the one or more parameters of the generative model each time a second loss value is generated. Likewise, in the case where each “batch” includes two or more images, the processing system can be configured to combine each of the “second loss values” generated (in step 708) for each image of the batch into an aggregate loss value (e.g., by summing or averaging the multiple second loss values), and modify the one or more parameters of the generative model based on the aggregate loss value.

[0054] In step 716, the processing system determines whether there are additional batches in the image collection of the object. In the case where the image collection has not been split, and thus there is a single “batch” containing every image in the “image collection” of the object, the determination in step 716 will automatically be “no,” and the method 700 will end thereafter, as shown in step 720. However, in the case where the image collection is divided into two or more batches, the processing system will follow the “yes” arrow to step 718 to select the next given image from the image collection of the object. This will then begin another set of steps 706-714 for each image in the next batch of images, and the process will continue until there are no additional batches remaining, at which point the processing system will follow the “no” arrow to step 720.

[0055] Although the method 700 is shown as ending in step 720 once all images have been used to adjust the generative model, it should be understood that the method 700 can be repeated any suitable number of times using the same set of images until its output for each optimization code produces an image that is sufficiently close to each given image. In this regard, in some aspects of the technology, the processing system can be configured to aggregate all second loss values generated during a given pass through the method 700 and determine whether to repeat the method 700 for the set of images based on the aggregated loss value. For example, in some aspects of the technology, the processing system can be configured to repeat the method 700 for the set of images if the aggregated loss value for the most recent pass through the method 700 is greater than some predetermined threshold. Also, in some aspects, the processing system can be configured to use gradient descent to make this determination and thus repeat the method 700 for the set of images until the aggregated loss value for the given pass through the method 700 is equal to or greater than the aggregated loss value in the previous iteration.

[0056] FIG. 5 An example method 800 for generating a personalized output image based on an input image and a personalized prior generated according to the method of FIG. 7 or FIG. 5 is set forth. In this regard, the example method 800 represents a process that can be optionally performed after generating at least one personalized prior (at step 520 of the method of FIG. 7 and can also be performed after further adjusting the generative model according to the method of FIG. 5 Thus, in step 802, assume that the processing system (e.g., the processing system 102) will perform at least the method 500 of FIG. 7 for each image in the set of images and optionally perform steps 704-720 of the method of FIG. 5

[0057] As described above, after the processing system generates a personalized prior (using the method 500 of FIG. 7 and optionally adjusts the generative model (using the method 700 of FIG. 5 the personalized prior can be used to generate different sets of candidate coefficients and then a candidate image can be generated for a given image enhancement task using a code corresponding to the set of candidate coefficients. Thus, as shown in step 804, for a particular input image of an object, the processing system will repeatedly perform steps 806-812 in order to test multiple sets of coefficients to identify a personalized output image, where each of the multiple sets of coefficients corresponds to a code within the convex hull (identified in step 520 of the method of FIG. 7

[0058] ​​At each pass step 806, the processing system identifies a coefficient set of the plurality of coefficient sets, and uses the coefficient set to generate a given code. This coefficient set can be identified based on any suitable selection criteria. For example, in some aspects of the technology, the processing system can be configured to blindly select a first coefficient set (e.g., using a random selection process, or a pre-selected value such as the mean value of the vector space expressed in a personalized prior), and then select each successive coefficient set (in each successive pass step 806) based directly or indirectly on the first coefficient set using a suitable optimization scheme. Thus, in some aspects, the processing system can be configured to use gradient descent to select each successive coefficient set based on the previous coefficient set and an assessment of how well the image generated based on the previous coefficient set matches the input image (e.g., the third loss value generated in the most recent pass step 810).

[0059] At step 808, the processing system generates a third image using the generative model (e.g., the generative model 306, or an adjusted generative model produced by one or more of the pass steps 704-720) and the given code (generated here in pass step 806). Again here, the generative model can be configured to produce the third image in any suitable manner, including in the manner as described above with respect to the generation of the second image in pass step 806. FIG. 3A FIG. 3B FIG. 4A FIG. 4B FIG. 9

[0060] At step 810, the processing system compares the third image (generated here in pass step 808) to the input image of the object to generate a third loss value for the third image. The third loss value can be generated in any suitable manner using any suitable function. For example, in some aspects of the technology, the third loss value can be based on a comparison of the third image to the input image using a heuristic or learned similarity metric (e.g., learned perceptual image patch similarity (“LPIPS”), peak signal-to-noise ratio (“PSNR”), structural similarity index measurement (“SSIM”)).

[0061] ​​​​​In step 812, the processing system determines whether another set of coefficients should be tested. Since it is assumed that multiple sets of coefficients will be tested, when the processing system first reaches step 812, it will automatically follow the "yes" arrow and return to step 806, allowing a second set of coefficients to be identified and tested. However, on all subsequent returns to step 812, the processing system can determine whether to test another set of coefficients based on any suitable criterion. Therefore, as mentioned above, in some aspects of this technique, the processing system can determine when to stop testing another set of coefficients based on a suitable optimization scheme (such as gradient descent). In this case, the determination in step 812 can be based on a comparison of the third loss value currently generated through step 810 (or some other evaluation of how well the third image matches the input image) with one or more of the third loss values ​​generated in previous iterations. For example, the processing system can be configured to stop testing a continuous set of coefficients when the third loss value currently generated through step 810 is equal to or greater than the third loss value generated in the previous iteration through step 810.

[0062] Therefore, the processing system iterates through steps 806-812 for each subsequent set of coefficients until a sufficiently tested set of coefficients is determined at step 812. At this point, the processing system follows the "No" arrow to step 814, where it compares the third loss value generated for each third image (in step 810) to identify the third image with the lowest third loss value. This third image with the lowest third loss value is used as the personalized output image. In this way, method 800 can be used to generate a personalized output image that is both generated using code within a personalized prior (making it more likely to resemble the object than a personalized output image generated without code constraints) and optimized to closely match the input image (thus ensuring that the image generated by the model is consistent with the image that can be collected from the input image when performing image editing or enhancement tasks).

[0063] FIG. 5 The present disclosure describes various aspects of the method for using an input image and based on... FIG. 7 or FIG. 5 Another exemplary method 900 is to generate a personalized output image using a personalized prior generated by the method. In this respect, exemplary method 900 also indicates that at least one personalized prior can be generated (in...) FIG. 7 Optional execution is performed after step 520, and can also be performed according to... FIG. 8 The method further adjusts the process executed after generating the model. FIG. 9 Method 800 and FIG. 5The only difference between the methods 900 is that, in each pass through step 906-912, a given set of codes is tested, where each individual code in the given set of codes is within the convex hull identified in step 520 of the method 500. FIG. 5

[0064] Thus, as described above, in step 902, the processing system (e.g., the processing system 102) will perform at least the method 500 on each image in the set of images, and optionally perform steps 704-720 of the method 600. Likewise, as shown in step 904, for a particular input image of the object, the processing system will repeat steps 906-912 to test multiple sets of codes to identify a personalized output image, where each code in the set of codes is within the convex hull (identified in step 520 of the method 500) of the object. FIG. 7 FIG. 5 FIG. 7

[0065] In each pass through step 906, the processing system identifies a given set of codes that includes two or more individual codes. Again here, this given set of codes can be identified based on any suitable selection criteria. For example, in some aspects of the technology, the processing system can be configured to blindly select a first given set of codes (e.g., using a random selection process, or by assigning a pre-selected value to each individual code, such as the mean value of the vector space represented by the personalization priors), and then use a suitable optimization scheme to select each successive set of codes (in each successive pass through step 906) based directly or indirectly on that first given set of codes. Thus, in some aspects, the processing system can be configured to use gradient descent to select each successive set of codes based on the previous set of codes and an evaluation of how well the image generated based on the previous set of codes matches the input image (e.g., the third loss value generated in the most recent pass through step 910).

[0066] In step 908, the processing system generates a third image using the generative model (e.g., the generative model 306, or an adjusted generative model produced by one or more passes through steps 704-720 of the method 600) and the given set of codes (generated in this pass through step 906). In this aspect, in step 908, the processing system provides each individual code in the given set of codes to a different layer or set of layers of the generative model. Again here, the generative model can be configured to produce the third image in any suitable manner, including in the manner described above with respect to the method 600. FIG. 3A FIG. 3B FIG. 4A FIG. 4B FIG. 10

[0067] ​​​​​​​​​In step 910, the processing system compares the third image (here generated by step 908) to the input image of the subject to generate a third loss value for the third image. Again here, the third loss value can be generated in any suitable manner using any suitable function. For example, in some aspects of the technology, the third loss value can be based on a comparison of the third image to the input image using a heuristic or learned similarity metric (e.g., learned perceptual image patch similarity (“LPIPS”), peak signal-to-noise ratio (“PSNR”), structural similarity index measurement (“SSIM”)).

[0068] In step 912, the processing system determines whether another code set should be tested. Since multiple code sets are assumed to be tested, the first time the processing system reaches step 912, it will automatically follow the “yes” arrow and return to step 906 so that a second code set will be identified and tested. However, on all subsequent returns to step 912, the processing system can determine whether to test another code set based on any suitable criteria. Thus, as mentioned above, in some aspects of the technology, the processing system can determine when to stop testing another code set based on a suitable optimization scheme, such as gradient descent. In this case, the determination in step 912 can be based on a comparison of the third loss value (or some other assessment of how well the third image matches the input image) generated by the current pass through step 910 to one or more of the third loss values generated in previous passes. For example, the processing system can be configured to stop testing successive code sets when the third loss value generated by the current pass through step 910 is equal to or greater than the third loss value generated by the previous pass through step 910.

[0069] Similar to aspect 800, the processing system will thus loop through steps 906-912 for each next code set until it is determined at step 912 that enough code sets have been tested. At this point, the processing system will follow the “no” arrow to step 914 where it compares the third loss values generated (in step 910) for each third image to identify the third image having the lowest third loss value. Again here, the third image identified as having the lowest third loss value is used as the personalized output image.

[0070] FIG. 4A is a comparative illustration of how a generative model (e.g., generative model 306) according to aspects of the present disclosure can complete four exemplary image enhancement tasks with or without the use of a personal prior 402 of FIG. 10 is a comparative illustration of how a generative model (e.g., generative model 306) according to aspects of the present disclosure can complete four exemplary image enhancement tasks with or without the use of a personal prior 402 of

[0071] In FIG. 8In the diagram, a column of four input images 1002a-1002d is shown at the center of schema 1000. Since it is assumed that the generative model is responsible for producing output images that are similar to the input images but with improved resolution, each of the input images 1002a-1002d is blurred.

[0072] Images 1004a-1004d in the left column show the latent output of the generative model, where the model is not constrained to select codes within any particular part of its latent vector space W(302). In this respect, dashed lines connect each input image to a point within the latent vector space W(302), representing the point ultimately chosen by the generative model (e.g., in relation to the above). FIG. 9 Method 800 or FIG. 4A The code following the selection process described in method 900 is used, and the arrow connects the point to the corresponding output image generated by the generative model based on that code. As can be seen, although the output images 1004a-1004d are visually consistent with their corresponding input images 1002a-1002d, the output images do not all appear to belong to the same object.

[0073] In contrast, the right-hand columns of images 1006a-1006d show the latent output of the generative model, where the generative model is constrained to select... FIG. 8 The code within the personal prior 402. Also here, the dashed lines connect each input image to a point within the personal prior 402, representing the point the generative model ultimately chooses (e.g., in relation to the points mentioned above). FIG. 9 Method 800 or FIG. 4A The code following the selection process described in method 900, and the arrow connects that point to the corresponding output image generated by the generative model based on that code. As can be seen, this ultimately produces output images 1006a-1006d, which are visually consistent with their corresponding input images 1002a-1002d and all appear to belong to the same object. Specifically, because FIG. 11 The personalized prior 402 indicates that when provided to the generative model 306, it produces an image similar to a certain American celebrity within a certain code range (as discussed above), so that each of the output images 1006a-1006d appears to display an image of a certain American celebrity that is visually consistent with the input images 1002a-1002d.

[0074] FIG. 4B This illustrates how generative models (e.g., generative model 306) according to various aspects of this disclosure can be used with or without using FIG. 8 Diagram 1100 compares the completion of four exemplary image restoration tasks under the individual prior 403.

[0075] Also here, at the center of Figure 1000, a column of four input images 1102a-1102d is shown. Since it is assumed that the generative model is responsible for producing an output image that is similar to the input images but fills in the masked parts, each of the input images 1102a-1102d includes a black mask 1103a-1103d representing the pixels to be replaced.

[0076] In Figure 1100, the left column of images 1104a-1104d shows the latent output of the generative model, where the generative model is not constrained to select codes within any particular part of its latent vector space W(302). Here again, the dashed lines connect each input image to a point within the latent vector space W(302), representing the point that the generative model ultimately chooses (e.g., in relation to the above). FIG. 9 Method 800 or FIG. 4B The code following the selection process described in method 900 is used, and the arrow connects the point to the corresponding output image generated by the generative model based on the code. As can be seen, although the output images 1104a-1104d are visually consistent with the unmasked portions of their respective input images 1102a-1102d, the output images do not all appear to belong to the same object.

[0077] In contrast, the right-hand columns of images 1106a-1106d show the latent output of the generative model, where the generative model is constrained to select... FIG. 8 The code within the personal prior 403. Also here, the dashed lines connect each input image to a point within the personal prior 403, representing the point the generative model ultimately chooses (e.g., in relation to the points mentioned above). FIG. 9 Method 800 or FIG. 4B The code following the selection process described in method 900, and the arrow connects that point to the corresponding output image generated by the generative model based on that code. As can be seen, this ultimately produces output images 1106a-1106d, which are visually consistent with their corresponding input images 1102a-1102d and all appear to belong to the same object. Specifically, due to ​ The personalized prior 403 represents a certain range of codes (as discussed above) that, when provided to the generative model 306, will produce images similar to a famous American singer, songwriter, and actor. Therefore, each of the output images 1106a-1106d appears to display an image of a famous American singer, songwriter, and actor that is visually consistent with the unmasked portion of the input images 1102a-1102d.

[0078] Thus, both illustrations 1000 and 1100 illustrate how a personalized prior can be used to focus the code used by the generative model, such that the generative model can produce examples of output images that are both visually consistent with the input image and consistent with the identity of the particular object. Thus, where the object of the input image is already known, a personalized prior can be selected and used such that the generative model will be biased to produce more representative, and thus better, output images.

[0079] The foregoing alternative examples are not mutually exclusive of one another unless otherwise noted, but can be implemented in various combinations to achieve unique advantages. As these and other variations and combinations of features discussed above can be susceptible to variation and modification, the foregoing description should be considered as illustrative only and not restrictive in character, as the scope of the exemplary systems and methods defined by the claims should be construed in accordance with the entire written specification if the claims are incorporated by reference, and, wherever aspects are described in the above description in terms of a process, operation, or technique, it is understood that the skilled artisan can employ alternative implementations of the described process, operation, or technique and that one variation or implementation described does not preclude the use of another variation or implementation. Also, conclusions expressed by the claims are not necessarily limited to those specifically stated. The description as set forth is not intended to be exhaustive or to be limited to the precise forms disclosed. As such, many modifications and other embodiments are possible in light of the above teachings. Further, the description is not intended to be limited to the forms shown and described. Rather, it is intended to cover any and all modifications and equivalents within the scope of the claims.

Claims

1. A computer-implemented method, comprising: For each given image in the image set of the object, multiple codes are tested using one or more processors of the processing system to identify optimized code for the given image, including: For each of the multiple codes: The generative model and the code are used to generate the first image; and The first image is compared with the given image using the one or more processors to generate a first loss value for the code; and The first loss values ​​generated for each of the plurality of codes are compared using the one or more processors to identify the code with the lowest first loss value as the optimized code for the given image; and Using the one or more processors, a personalized prior for the object is generated based on the convex hull within the latent vector space W of the generative model, the convex hull including each optimized code for recognition of each given image in the image set of the object, the latent vector space W representing all possible codes that can be input into the generative model.

2. The method of claim 1, further comprising: The optimized code for recognizing each given image in the image set for the object: The second image is generated using the generative model and the optimized code; as well as The second image is compared with the given image using the one or more processors to generate a second loss value; as well as Using the one or more processors, at least in part based on each generated second loss value, one or more parameters of the generative model are modified to create an adjusted generative model.

3. The method of claim 2, further comprising: The one or more processors are used to identify multiple sets of coefficients, each of which corresponds to a code within the convex hull. For each given set of coefficients in the plurality of coefficient sets: Using the one or more processors, the adjusted generative model, and a given code corresponding to the given set of coefficients, a third image is generated; as well as The third image is compared with at least a portion of the input image of the object using the one or more processors to generate a third loss value for the third image; as well as The third loss value generated for each third image is compared using the one or more processors to identify the third image with the lowest third loss value as the personalized output image.

4. The method of claim 3, wherein the plurality of coefficient sets includes a first coefficient set and a plurality of consecutive coefficient sets selected directly or indirectly based on the first coefficient set using gradient descent.

5. The method of claim 3, wherein the input image of the object comprises a first pixel portion retained from an original image of the object, and a mask replacing a second pixel portion from the original image of the object, and The third loss value of the third image is generated by comparing the third image with at least a portion of the input image of the object using the one or more processors. This includes comparing the third image with the first pixel portion to generate the third loss value of the third image.

6. The method of claim 3, wherein the input image has a first resolution, and the personalized output image has a second resolution higher than the first resolution.

7. The method of claim 2, further comprising: The one or more processors are used to identify multiple sets of coefficients, each of which corresponds to a code within the convex hull. The one or more processors are used to identify multiple sets of codes, each of the multiple sets of codes comprising two or more individual codes, each individual code corresponding to a set of coefficients in the multiple sets of coefficients; For each given code set among the plurality of code sets: Using the one or more processors, the adjusted generative model and the given set of codes are used to generate a third image, each individual code in the given set of codes being provided to a different layer or set of layers of the adjusted generative model; as well as The third image is compared with at least a portion of the input image of the object using the one or more processors to generate a third loss value for the third image; as well as The third loss value generated for each third image is compared using the one or more processors to identify the third image with the lowest third loss value as the personalized output image.

8. The method of claim 1, further comprising: The one or more processors are used to identify multiple sets of coefficients, each of which corresponds to a code within the convex hull. For each given set of coefficients in the plurality of coefficient sets: Using the one or more processors, the generative model, and given code corresponding to the given set of coefficients, a third image is generated; as well as The third image is compared with at least a portion of the input image of the object using the one or more processors to generate a third loss value for the third image; as well as The third loss value generated for each third image is compared using the one or more processors to identify the third image with the lowest third loss value as the personalized output image.

9. The method of claim 1, further comprising: The one or more processors are used to identify multiple sets of coefficients, each of which corresponds to a code within the convex hull. The one or more processors are used to identify multiple sets of codes, each of the multiple sets of codes comprising two or more individual codes, each individual code corresponding to a set of coefficients in the multiple sets of coefficients; For each given code set among the plurality of code sets: Using the one or more processors, the generative model and the given set of codes are used to generate a third image, each individual code in the given set of codes being provided to a different layer or set of layers of the generative model; as well as The third image is compared with at least a portion of the input image of the object using the one or more processors to generate a third loss value for the third image; as well as The third loss value generated for each third image is compared using the one or more processors to identify the third image with the lowest third loss value as the personalized output image.

10. The method of claim 1, wherein the plurality of codes includes a first code and a plurality of successive codes selected directly or indirectly based on the first code using gradient descent.

11. The method of claim 10, wherein the first code represents the average value of the potential vector space W.

12. A processing system, comprising: The memory stores the generative model; as well as One or more processors, coupled to the memory and configured to generate personalized priors for use with the generative model, including: For each given image in the image set of the object, multiple codes are tested to identify the optimized code for the given image, including: For each of the multiple codes: The first image is generated using the generative model and the code; and The first image is compared with the given image to generate a first loss value for the code; and The first loss values ​​generated for each of the plurality of codes are compared to identify the code with the lowest first loss value as the optimized code for the given image; and The personalized prior of the object is generated based on the convex hull within the latent vector space W of the generative model, the convex hull including each optimized code for recognition of each given image in the image set of the object, the latent vector space W representing all possible codes that can be input into the generative model.

13. The processing system of claim 12, wherein the one or more processors are further configured to regulate the generative model, including: The optimized code for recognizing each given image in the image set for the object: The second image is generated using the generative model and the optimized code; as well as The second image is compared with the given image using the one or more processors to generate a second loss value; as well as Using the one or more processors, at least in part based on each generated second loss value, one or more parameters of the generative model are modified to create an adjusted generative model.

14. The processing system of claim 13, wherein the one or more processors are further configured to generate a personalized output image based on an input image of the object, comprising: Identify multiple sets of coefficients, each of which corresponds to a code within the convex hull; For each given set of coefficients in the plurality of coefficient sets: A third image is generated using the adjusted generative model and a given code corresponding to the given set of coefficients; as well as The third image is compared with at least a portion of the input image of the object to generate a third loss value for the third image; as well as The third loss value generated for each third image is compared to identify the third image with the lowest third loss value as the personalized output image.

15. The processing system of claim 14, wherein the plurality of coefficient sets includes a first coefficient set and a plurality of continuous coefficient sets, and The one or more processors are further configured to use gradient descent to select each of the plurality of consecutive coefficient sets directly or indirectly based on the first coefficient set.

16. The processing system of claim 14, wherein the input image of the object includes a first pixel portion retained from an original image of the object, and a mask replacing a second pixel portion from the original image of the object, and The third loss value of the third image is generated by comparing the third image with at least a portion of the input image of the object. This includes comparing the third image with the first pixel portion to generate the third loss value of the third image.

17. The processing system of claim 14, wherein the one or more processors are configured to generate the personalized output image based on the input image of the object, wherein the input image has a first resolution and the personalized output image has a second resolution higher than the first resolution.

18. The processing system of claim 13, wherein the one or more processors are further configured to generate a personalized output image based on an input image of the object, comprising: Identify multiple sets of coefficients, each of which corresponds to a code within the convex hull; Identify multiple sets of codes, each of which includes two or more individual codes, each individual code corresponding to a set of coefficients in the multiple sets of coefficients; For each given code set among the plurality of code sets: The adjusted generative model and the given set of codes are used to generate a third image, wherein each individual code in the given set of codes is provided to a different layer or set of layers of the adjusted generative model; as well as The third image is compared with at least a portion of the input image of the object to generate a third loss value for the third image; as well as The third loss value generated for each third image is compared to identify the third image with the lowest third loss value as the personalized output image.

19. The processing system of claim 12, wherein the one or more processors are further configured to generate a personalized output image based on an input image of the object, comprising: Identify multiple sets of coefficients, each of which corresponds to a code within the convex hull; For each given set of coefficients in the plurality of coefficient sets: A third image is generated using the generative model and a given code corresponding to the given set of coefficients; as well as The third image is compared with at least a portion of the input image of the object to generate a third loss value for the third image; as well as The third loss value generated for each third image is compared to identify the third image with the lowest third loss value as the personalized output image.

20. The processing system of claim 12, wherein the one or more processors are further configured to generate a personalized output image based on an input image of the object, comprising: Identify multiple sets of coefficients, each of which corresponds to a code within the convex hull; Identify multiple sets of codes, each of which includes two or more individual codes, each individual code corresponding to a set of coefficients in the multiple sets of coefficients; For each given code set among the plurality of code sets: The generative model and the given set of codes are used to generate a third image, wherein each individual code in the given set of codes is provided to a different layer or set of layers of the generative model; as well as The third image is compared with at least a portion of the input image of the object to generate a third loss value for the third image; as well as The third loss value generated for each third image is compared to identify the third image with the lowest third loss value as the personalized output image.

21. The processing system of claim 12, wherein the plurality of codes comprises a first code and a plurality of consecutive codes, and The one or more processors are further configured to use gradient descent to select each of the plurality of consecutive codes directly or indirectly based on the first code.

22. The processing system of claim 21, wherein the one or more processors are further configured to select a first code representing the average value of the potential vector space W.

Images