Facial image editing and enhancement using personalized priori

Abstract

Systems and methods for identifying a personalized priori within a potential vector space of a generative model based on a set of images of a given object. In some examples, the techniques may also include using the personalized priori to constrain the input to generate a model to a potential vector space associated with the given object such that when the model is responsible for editing an image of the object (e.g., performing a repair to fill a masked area, improve resolution, or deblurring the image), the model is not responsible for generating the model. Recognition features of the object will be reflected in an image generated by the model.

Description

Background Art

[0001] Generative models can be used to perform tasks ranging from editing and enhancing images of a given object to generating realistic images (or portions of images) of a given object or synthetically generating objects. In order to adequately train such models, large collections of images are required, typically of large collections of objects. Thus, when generative models are used to edit, enhance, or fill in a portion of an image of a known object, they can produce images that look real but resemble a different object. Summary of the invention

[0002] The present technology relates to systems and methods for identifying personalized priors within a latent vector space of a generative model based on a set of images of a given object. In some aspects, the present technology may also include using the personalized prior (e.g., a convex hull defined by a set of codes generated based on a set of images of the object) to constrain the codes input to the generative model so that the identifying features of the object will be reflected in the images produced by the model. For example, the generative model may be configured to enhance or fill in facial features in an image of an object where only partial clues related to the identity of the object are present (e.g., due to motion blur, low brightness, low resolution, occlusion by other objects). In the absence of the present technology, the model may successfully enhance or fill in such an image, but may achieve this by producing an image that appears to belong to a different object. The present technology can be used to focus the generative model so that the images it produces are more consistent with the appearance of the object.

[0003] In one aspect, the present disclosure describes a computer-implemented method comprising: (1) for each given image in a set of images of an object, testing a plurality of codes using one or more processors of a processing system to identify an optimized code for the given image, comprising: (a) for each code in the plurality of codes: generating a first image using a generative model and the code; and comparing the first image with the given image using the one or more processors to generate a first loss value for the code; and (b) comparing the first loss values ​​generated for each code in the plurality of codes using the one or more processors to identify the code with the lowest first loss value as the optimized code for the given image; and (2) generating a personalized prior for the object based on a convex hull using the one or more processors, the convex hull including each optimized code identified for each given image in the set of images of the object. In some aspects, the method further comprises: (1) for each given image in the set of images of the object, the optimization code is identified: generating a second image using the generative model and the optimization code; and comparing the second image with the given image using the one or more processors to generate a second loss value; and (2) modifying one or more parameters of the generative model based at least in part on each generated second loss value to create an adjusted generative model using the one or more processors. In some aspects, the method further comprises: (1) identifying a plurality of coefficient sets using the one or more processors, each of the plurality of coefficient sets corresponding to a code within the convex hull; (2) for each given coefficient set in the plurality of coefficient sets: generating a third image using the one or more processors using the adjusted generative model and a given code corresponding to the given coefficient set; and comparing the third image with at least a portion of the input image of the object using the one or more processors to generate a third loss value for the third image; and (3) comparing the third loss values ​​generated for each third image using the one or more processors to identify the third image with the lowest third loss value as the personalized output image.In some aspects, the method further includes: (1) using the one or more processors to identify multiple coefficient sets, each coefficient set in the multiple coefficient sets corresponding to a code within the convex hull; (2) using the one or more processors to identify multiple code sets, each code set in the multiple code sets includes two or more individual codes, each individual code corresponding to a coefficient set in the multiple coefficient sets; (3) for each given code set in the multiple code sets: using the one or more processors to generate a third image using the adjusted generative model and the given code set, each individual code in the given code set being provided to a different layer or set of layers of the adjusted generative model; and using the one or more processors to compare the third image with at least a portion of the input image of the object to generate a third loss value for the third image; and (4) using the one or more processors to compare the third loss values ​​generated for each third image to identify the third image with the lowest third loss value as the personalized output image. In some aspects, the method further includes: (1) identifying, using the one or more processors, a plurality of sets of coefficients, each set of coefficients in the plurality of sets of coefficients corresponding to a code within the convex hull; (2) for each given set of coefficients in the plurality of sets of coefficients: generating, using the one or more processors, a third image using the generative model and a given code corresponding to the given set of coefficients; and comparing, using the one or more processors, the third image with at least a portion of the input image of the object to generate a third loss value for the third image; and (3) comparing, using the one or more processors, the third loss values ​​generated for each third image to identify the third image having the lowest third loss value as the personalized output image. In some aspects, the method further includes: (1) using the one or more processors to identify multiple coefficient sets, each coefficient set in the multiple coefficient sets corresponding to a code within the convex hull; (2) using the one or more processors to identify multiple code sets, each code set in the multiple code sets includes two or more individual codes, each individual code corresponding to a coefficient set in the multiple coefficient sets; (3) for each given code set in the multiple code sets: using the one or more processors, using the generative model and the given code set to generate a third image, each individual code in the given code set being provided to a different layer or set of layers of the generative model; and using the one or more processors to compare the third image with at least a portion of the input image of the object to generate a third loss value for the third image; and (4) using the one or more processors to compare the third loss values ​​generated for each third image to identify the third image with the lowest third loss value as the personalized output image.In some aspects, the plurality of coefficient sets include a first coefficient set, and a plurality of continuous coefficient sets selected directly or indirectly based on the first coefficient set using gradient descent. In some aspects, the input image of the object includes a first pixel portion retained from the original image of the object, and a mask replacing a second pixel portion of the original image from the object, and using the one or more processors to compare the third image with at least a portion of the input image of the object to generate the third loss value of the third image includes comparing the third image with the first pixel portion to generate the third loss value of the third image. In some aspects, the input image has a first resolution, and the personalized output image has a second resolution higher than the first resolution. In some aspects, the plurality of codes include a first code, and a plurality of continuous codes selected directly or indirectly based on the first code using gradient descent. In some aspects, the first code represents an average value of a potential vector space W, and the potential vector space W represents all possible codes that can be input into the generative model.

[0004] In another aspect, the present disclosure describes a processing system comprising: a memory storing a generative model; and one or more processors coupled to the memory and configured to perform any of the methods just described.

[0005] In another aspect, the present disclosure describes a processing system comprising: (1) a memory storing a generative model; and (2) one or more processors coupled to the memory and configured to generate a personalized prior for an object for use with the generative model, comprising: (a) for each given image in a set of images of the object, testing a plurality of codes to identify an optimized code for the given image, comprising: (i) for each of the plurality of codes: generating a first image using the generative model and the code; and comparing the first image with the given image to generate a first loss value for the code; and (ii) comparing the first loss values ​​generated for each of the plurality of codes to identify the code with the lowest first loss value as the optimized code for the given image; and (b) generating the personalized prior for the object based on a convex hull comprising each optimized code identified for each given image in the set of images of the object. In some aspects, the one or more processors are further configured to adjust the generative model, which includes: (1) for each given image in the set of images of the object, the optimization code is identified: using the generative model and the optimization code to generate a second image; and using the one or more processors to compare the second image with the given image to generate a second loss value; and (2) using the one or more processors, at least in part based on each generated second loss value, modifying one or more parameters of the generative model to create an adjusted generative model. In some aspects, the one or more processors are further configured to generate a personalized output image based on the input image of the object, which includes: (1) identifying a plurality of coefficient sets, each of the plurality of coefficient sets corresponding to a code within the convex hull; (2) for each given coefficient set in the plurality of coefficient sets: generating a third image using the adjusted generative model and a given code corresponding to the given coefficient set; and comparing the third image with at least a portion of the input image of the object to generate a third loss value for the third image; and (3) comparing the third loss values ​​generated for each third image to identify the third image with the lowest third loss value as the personalized output image.In some aspects, the one or more processors are further configured to generate a personalized output image based on the input image of the object, comprising: (1) identifying a plurality of coefficient sets, each of the plurality of coefficient sets corresponding to a code within the convex hull; (2) identifying a plurality of code sets, each of the plurality of code sets comprising two or more individual codes, each individual code corresponding to a coefficient set in the plurality of coefficient sets; (3) for each given code set in the plurality of code sets: generating a third image using the adjusted generative model and the given code set, each individual code in the given code set being provided to a different layer or set of layers of the adjusted generative model; and comparing the third image with at least a portion of the input image of the object to generate a third loss value for the third image; and (4) comparing the third loss values ​​generated for each third image to identify the third image having the lowest third loss value as the personalized output image. In some aspects, the one or more processors are further configured to generate a personalized output image based on the input image of the object, comprising: (1) identifying a plurality of coefficient sets, each of the plurality of coefficient sets corresponding to a code within the convex hull; (2) for each given coefficient set in the plurality of coefficient sets: generating a third image using the generative model and a given code corresponding to the given coefficient set; and comparing the third image with at least a portion of the input image of the object to generate a third loss value for the third image; and (3) comparing the third loss values ​​generated for each third image to identify the third image having the lowest third loss value as the personalized output image. In some aspects, the one or more processors are further configured to generate a personalized output image based on an input image of the object, comprising: (1) identifying a plurality of coefficient sets, each of the plurality of coefficient sets corresponding to a code within the convex hull; (2) identifying a plurality of code sets, each of the plurality of code sets comprising two or more individual codes, each individual code corresponding to a coefficient set in the plurality of coefficient sets; (3) for each given code set in the plurality of code sets: generating a third image using the generative model and the given code set, each individual code in the given code set being provided to a different layer or set of layers of the generative model; and comparing the third image with at least a portion of the input image of the object to generate a third loss value for the third image; and (4) comparing the third loss values ​​generated for each third image to identify the third image having the lowest third loss value as the personalized output image. In some aspects, the plurality of coefficient sets comprises a first coefficient set and a plurality of consecutive coefficient sets, and the one or more processors are further configured to select each of the plurality of consecutive coefficient sets based on the first coefficient set directly or indirectly using gradient descent.In some aspects, the input image of the object includes a first pixel portion retained from the original image of the object, and a mask that replaces a second pixel portion from the original image of the object, and comparing the third image with at least a portion of the input image of the object to generate the third loss value of the third image includes comparing the third image with the first pixel portion to generate the third loss value of the third image. In some aspects, the one or more processors are configured to generate the personalized output image based on the input image of the object, wherein the input image has a first resolution and the personalized output image has a second resolution higher than the first resolution. In some aspects, the plurality of codes include a first code and a plurality of continuous codes, and the one or more processors are also configured to select each of the plurality of continuous codes based on the first code directly or indirectly using gradient descent. In some aspects, the one or more processors are also configured to select a first code representing an average value of a latent vector space W, the latent vector space W representing all possible codes that can be input into the generative model. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a functional diagram of an example system according to aspects of the present disclosure.

[0007] Figure 2 is a functional diagram of an example system according to aspects of the present disclosure.

[0008] Figure 3A and Figure 3B It is shown how different codes within the latent vector space from a generative model can produce images of different objects in accordance with aspects of the present disclosure.

[0009] Figure 4A and Figure 4B Two different personalized priors within the latent vector space of a generative model are shown, and how points within a given personalized prior can produce different images of a given object in accordance with aspects of the present disclosure.

[0010] Figure 5 Exemplary methods for generating personalized priors based on a collection of images of an object in accordance with aspects of the present disclosure are set forth.

[0011] Figure 6 shows how a given set of images can be used to generate Figure 4A Personalized priors.

[0012] Figure 7 The present invention describes various aspects of the present disclosure for use in Figure 5 An exemplary method of adjusting a generative model after identifying an optimized code for each image in a collection of images.

[0013] Figure 8 According to various aspects of the present disclosure, a method for Figure 5 or Figure 7 An exemplary method for generating a personalized prior to generate a personalized output image.

[0014] Fig. 9 According to various aspects of the present disclosure, a method for Figure 5 or Figure 7 An exemplary method for generating a personalized prior to generate a personalized output image.

[0015] Fig.10 shows how a generative model according to aspects of the present disclosure can be used with and without Figure 4A Comparison of four exemplary image enhancement tasks with individual priors.

[0016] Fig.11 shows how a generative model according to aspects of the present disclosure can be used with or without Figure 4B Comparison of four exemplary image inpainting tasks with individual priors. DETAILED DESCRIPTION

[0017] The present technology will now be described with respect to the following exemplary systems and methods.Common reference numerals between the figures depicted and described below are intended to identify like features.

[0018] Example System

[0019] Figure 1 A high-level system diagram 100 of an exemplary processing system 102 for performing the methods described herein is shown. The processing system 102 may include one or more processors 104 and a memory 106 storing instructions 108 and data 110. The instructions 108 and data 110 may include generating models as described herein (e.g., Figure 3A , Figure 3B , Figure 4A , Figure 4B , Figure 6 , Fig.10 and Fig.11 In addition, data 110 may store training examples for training such a generative model, data used by the generative model when generating images, a collection of images of a given object, personalized priors based on the collection of images of a given object (e.g., Figure 4A , Figure 4B , Figure 6 , Fig.10 and Fig.11personalized priors 402 and 403) and / or images output by a generative model.

[0020] The processing system 102 may reside on a single computing device. For example, the processing system 102 may be a server, a personal computer, or a mobile device, and therefore, the generative model and data associated with it may be local to the single computing device. Similarly, the processing system 102 may reside on a cloud computing system or other distributed system. In this case, the generative model and / or data associated with it may be distributed on two or more different physical computing devices. For example, in some aspects of the technology, the processing system may include a first computing device storing a generative model, and a second computing device storing data to be used by the generative model when generating an image, a set of images of a given object, a personalized prior based on a set of images of a given object, and / or an image output by the generative model. Similarly, in some aspects of the technology, the processing system may include a first computing device storing 1-n layers of a generative model having m layers, and a second computing device storing nm layers of a generative model.

[0021] Further in this regard, Figure 2 A high-level system diagram 200 is shown in which the exemplary processing system 102 just described is shown in communication with various websites and / or remote storage systems (including websites 210 and 218 and remote storage system 226) via one or more networks 208. In this example, websites 210 and 218 each include one or more servers 212a-212n and 220a-220n, respectively. Each of servers 212a-212n and 220a-220n can have one or more processors (e.g., 214 and 222) and associated memory (e.g., 216 and 224) that stores instructions and data (including the content of one or more web pages). Likewise, although not shown, remote storage system 226 can also include one or more processors and memory that stores instructions and data. In some aspects of the technology, processing system 102 can be configured to retrieve data, training examples, image collections of given objects, and / or input images of given objects from one or more of website 210, website 218, and / or remote storage system 226 to provide to the generative model to train or adjust the generative model, and / or for use in generating images.

[0022] The processing systems described herein may be implemented on any type of computing device, such as any type of general purpose computing device, server, or collection thereof, and may also include other components typically found in a general purpose computing device or server. Likewise, the memory of such a processing system may be of any non-transitory type capable of storing information accessible by a processor of the processing system. For example, the memory may include non-transitory media, such as hard drives, memory cards, optical disks, solid-state, tape storage, and the like. Computing devices suitable for the roles described herein may include different combinations of the foregoing, whereby different portions of instructions and data are stored on different types of media.

[0023] In all cases, the computing devices described herein may also include any other components commonly used in conjunction with computing devices, such as a user interface subsystem. The user interface subsystem may include one or more user inputs (e.g., a mouse, keyboard, touch screen, and / or microphone) and one or more electronic displays (e.g., a monitor with a screen or any other electrical device operable to display information). Output devices other than electronic displays (such as speakers, lights, and vibration, pulse, or tactile elements) may also be included in the computing devices described herein.

[0024] The one or more processors included in each computing device may be any conventional processor, such as a commercially available central processing unit ("CPU"), a graphics processing unit ("GPU"), a tensor processing unit ("TPU"), and the like. Alternatively, one or more processors may be dedicated devices, such as ASICs or other hardware-based processors. Each processor may have multiple cores capable of operating in parallel. The processor, memory, and other elements of a single computing device may be stored within a single physical housing, or may be distributed between two or more housings. Similarly, the memory of a computing device may include a hard drive or other storage medium located in a housing different from the housing of the processor (such as located in an external database or networked storage device). Therefore, references to processors or computing devices will be understood to include a collection of processors or computing devices or memories that may or may not operate in parallel, as well as references to one or more servers of a load-balanced server farm or a cloud-based system.

[0025] The computing device described herein can store instructions (such as machine code) that can be directly executed by a processor or instructions (such as scripts) that are indirectly executed. The computing device can also store data, which can be retrieved, stored or modified by one or more processors according to the instructions. The instructions can be stored on a computing device readable medium as a computing device code. In this regard, the terms "instructions" and "programs" can be used interchangeably herein. The instructions can also be stored in an object code format to be directly processed by the processor, or stored in any other computing device language, which includes a collection of scripts or independent source code modules that are interpreted or compiled in advance on demand. For example, the programming language can be C#, C++, JAVA or another computer programming language. Similarly, any component of an instruction or program can be implemented with a computer scripting language (such as JavaScript, PHP, ASP or any other computer scripting language). In addition, any of these components can be implemented using a combination of a computer programming language and a computer scripting language.

[0026] Example Method

[0027] Figure 3A and Figure 3B 308, 309, shows how different codes within a latent vector space W (302) from a generative model 306 can produce images (308, 309) of different objects, in accordance with aspects of the present disclosure. In these examples, the generative model 306 can be any suitable model configured to generate an output image based on an input code, and the latent vector space W (302) represents the range of all possible input codes that can be provided to the generative model 306. In this regard, for purposes of simplicity of illustration only, the latent vector space W (302) is shown in FIG. Figure 3A and Figure 3B (as well as Figure 4A , Figure 4B , Figure 6 , Fig.10 and Fig.11 ) is shown as a two-dimensional space. However, the present technology can be applied to any suitable generative model (e.g., GAN or bidirectional GAN ​​("BiGAN")) configured to generate images based on input vectors of any suitable number of dimensions. For example, in some aspects of the technology, the generative model 306 can be a generative adversarial network (e.g., StyleGAN, StarGAN) configured to use a latent vector space W having 512 dimensions.

[0028] like Figure 3A As shown, the code at point 304 of the latent vector space W (302) is fed to a generative model 306, thereby generating an image 308. To illustrate the effect of the present technique, Figure 3A , Figure 3B , Figure 4A , Figure 4B , Fig.10 and Fig.11 The examples described in each use images of subjects whose likenesses many people would be able to recognize. Figure 3A In the example of , it has been assumed that the code at point 304 will produce an image 308 of a certain President of the United States. Figure 3B In the example of , it has been assumed that the code at point 305 will produce image 309 of a well-known American singer, songwriter and actor.

[0029] Figure 4A and Figure 4B Two different personalized priors 402, 403 within the latent vector space W(302) of the generative model 306 are shown, and also shown how points within a given personalized prior (e.g., 304 and 404, 305 and 405) can produce different images of a given object according to aspects of the present disclosure. Figure 4A and Figure 4B The examples all assume the use of Figure 3A and Figure 3B The same latent vector space W (302) and the same generative model 306 are used in , but personal priors 402, 403 within the latent vector space W (302) are additionally depicted.

[0030] Personal priors 402 and 403 each represent a vector space within the latent vector space W (302) that includes a subset of possible input codes that produce images similar to a given object. In this case, assume that personal prior 402 represents a certain range of codes that, when provided to generative model 306, produce images similar to a certain U.S. president. Therefore, personal prior 402 includes point 304 (representing the code that produces Figure 3A 308) and another dot 404 (representing a code that produces a different image 406 of a certain President of the United States).

[0031] Likewise, assume that personal prior 403 represents a range of codes that, when provided to generative model 306, produce images similar to a well-known American singer, songwriter, and actor. Thus, personal prior 403 includes point 304 (representing the range of codes that produce Figure 3B 309) and another dot 405 (representing a code for generating a different image 407 of a well-known American singer, songwriter and actor).

[0032] Here again, for the purpose of simplifying the explanation only, individual priors 402 and 403 are Figure 4A and Figure 4B in (and in Figure 6 , Fig.10 and Fig.11) is shown as a two-dimensional space. However, since the present technology can be applied to any suitable generative model configured to generate an image based on an input vector of any suitable number of dimensions, personal priors 402 and 403 can also be vector spaces of any number of dimensions less than or equal to the number of dimensions in the latent vector space W (302). Therefore, for example, in the case where the generative model 306 is a generative adversarial network configured to use a latent vector space W with 512 dimensions, personal priors 402 and 403 can each be a vector space of up to 512 dimensions. In addition, for clarity of illustration, the exemplary vector spaces 402 and 403 are shown as completely separated. However, in practice, personal priors of different objects can intersect with each other.

[0033] Figure 5 An exemplary method 500 for generating a personalized prior based on a collection of images of an object in accordance with aspects of the present disclosure is set forth.

[0034] In step 502, a processing system (e.g., processing system 102) selects a given image from a collection of images of the subject. It will be appreciated that a larger number of images will generally produce a more representative personal prior than a smaller number of images. In this regard, it has been found that a number of images between 100 and 200 images generally enables a generative model to produce images that appear realistic and consistent with the subject's identity. However, other aspects may also affect how well the personal prior represents the subject's appearance. For example, if the subject's appearance changes significantly (e.g., due to a change in hairstyle, hair color, the addition or removal of facial hair, or the passage of a significant amount of time), it may be helpful to tailor the image collection to a particular stage so that the personal prior will reflect a single "look" and the images produced by the generative model will be consistent with that look. Likewise, for the same reason, it may be helpful to constrain the image collection to a single life stage (e.g., infancy, childhood, adolescence, adulthood, etc.). On the other hand, the variability of the image collection may also be important. For example, a collection of images that only shows the subject's face from the front may not be as useful as a collection of images that shows the subject's face from a variety of different angles, different lighting conditions, etc. Personal priors generated from images that are too similar may constrain the generative model too much, causing it to produce images that resemble the object but do not always look real.

[0035] As shown in step 504, after the processing system selects a given image, it will repeat steps 506-514 to test multiple codes to identify the optimized code for a given image. In this regard, in step 506, the processing system will identify the code to be tested in the current pass. This code can be identified based on any suitable selection criteria. For example, in some aspects of the technology, the processing system can be configured to blindly select a first code (e.g., using a random selection process, or a pre-selected value, such as the average value of the potential vector space W), and then use a suitable optimization scheme to directly or indirectly select each continuous code based on the first code (in each continuous pass step 506). Therefore, in some aspects, the processing system can be configured to use gradient descent to select each continuous code based on the previous code and the evaluation of the degree of matching of the image generated by the previous code with the given image (e.g., the first loss value generated in the most recent pass step 510).

[0036] In step 508, the processing system generates a first image using a generative model (e.g., generative model 306) and code (herein identified in step 506). The generative model may be configured in any suitable manner (including as described above with respect to Figure 3A , Figure 3B , Figure 4A and Figure 4B The first image is generated in the manner described above.

[0037] In step 510, the processing system compares the first image (here generated by step 508) with the given image (selected in step 502) to generate a first loss value for the code. The first loss value can be generated in any suitable manner using any suitable function. For example, in some aspects of the technology, the first loss value can be based on a comparison of the first image with the given image using a heuristic or learned similarity metric (e.g., learned perceptual image patch similarity ("LPIPS"), peak signal-to-noise ratio ("PSNR"), structural similarity index measure ("SSIM"), L1 or L2 loss, etc.).

[0038] In step 512, the processing system determines whether another code should be tested. Since it is assumed that multiple codes will be tested, the first time the processing system reaches step 512, it will automatically follow the "yes" arrow and return to step 506 so that the second code will be identified and tested. However, on all subsequent returns to step 512, the processing system can determine whether to test another code based on any suitable criteria. Therefore, as mentioned above, in some aspects of the technology, the processing system can determine when to stop testing another code based on a suitable optimization scheme (such as gradient descent). In this case, the determination in step 512 can be based on a comparison of the first loss value generated in the current pass step 510 (or some other evaluation of the degree of match of the first image to the given image) with one or more of the first loss values ​​generated in the previous pass. For example, the processing system can be configured to stop testing consecutive codes when the first loss value generated in the current pass step 510 is equal to or greater than the first loss value generated in the previous pass step 510.

[0039] Thus, the processing system will loop through steps 506-512 for each subsequent code until it is determined at step 512 that enough codes have been tested. At this point, the processing system will follow the "no" arrow to step 514 where it will compare the first loss values ​​generated (in step 510) for each of the plurality of codes to identify the code with the lowest first loss value. The code with the lowest first loss value is selected as the optimized code for the given image.

[0040] Then, in step 516, the processing system will determine whether there are any additional images in the object's image set. If so, the processing system will follow the "yes" arrow to step 518, at which the processing system will select the next given image to be tested. The processing system will then return to step 504 to test multiple codes to identify the optimization code for this new given image. In this way, steps 504-518 will be repeated as described above until an optimization code is identified for each image in the object's image set. Once an optimization code is selected for the last image in the object's image set (at step 514), the processing system will determine at step 516 that there are no additional images in the set, and therefore follow the "no" arrow to step 520.

[0041] In step 520, the processing system generates a personalized prior for the object based on the convex hull that includes each optimization code identified (in step 514) for each given image in the image set of the object. In this regard, in some aspects of the technique, the personalized prior can simply be the convex hull defined by each optimization code identified in step 514. In this case, assume that there is a set of n optimization codes {x1, x2, ..., xn}, the personalized prior will include any code c generated by optimizing the linear combination of codes according to the following formulas 1-3, where the coefficients (α values ​​α1 to α n ) is greater than or equal to 0, and the sum of all coefficients is 1.

[0042] c=α1x1+α2x2+...+α n x n (1)

[0043]

[0044] Likewise, in some aspects of the technique, the personalized prior may be some subset of the convex hull defined by each optimization code identified in step 514, such as a set of some predetermined number (e.g., 100, 500, 1,000, 10,000, 100,000, 1,000,000) of codes or a set of coefficients corresponding to a sampled set of points within the convex hull. Further, in some aspects of the technique, the personalized prior may be a simpler hull (e.g., a hull with fewer vertices) that conforms to or substantially overlaps the actual convex hull defined by each optimization code identified in step 514. Further, in some aspects of the technique, the personalized prior may be implemented based on the convex hull defined by formulas 1-3 above by encompassing a wider set of codes than the set of codes defined by formulas 1-3 above. For example, the personalized prior may include any code c generated according to formulas 1 and 3 above, where the coefficients (α values ​​α1 to α n ) is greater than or equal to a predetermined negative value (e.g., -0.01, -0.05, -0.1).

[0045] Figure 6 shows how a given set of n images 602a-612a may be used to generate a Figure 4A 402 of personalized priors. In this regard, each image 602a-612a is assumed to be one of a set of n images of a certain object, and the lines depicted show how these six selected images correspond to different optimization codes 602b-612b within the latent vector space W(302) of the generative model (e.g., generative model 306). Each of these optimization codes 602b-612b can be further described above according to Figure 5 The exemplary method steps 502-518 can be found. In addition, Figure 6 600 shows how each of these optimization codes 602b-612b (along with optimization codes for the rest of the set of n images, not shown) can be used to define a convex hull on which the personalized prior 402 is based, as described above with respect to Figure 5Step 520 is further described. Again, for the purpose of simplifying the description only, Figure 6 The personal prior 403 is shown as a two-dimensional space, and only six sample images are shown in a set of n images. However, the personal prior 403 can be based on any suitable number of points and can be a vector space with any suitable number of dimensions less than or equal to the number of dimensions in the latent vector space W (302), as explained above.

[0046] Figure 7 The present invention describes various aspects of the present disclosure for use in Figure 5 The exemplary method 700 of adjusting the generative model after the method identifies the optimized code for each image in the image collection. In this regard, the exemplary method 700 represents that the optimized code can be identified for each image in the image collection (in Figure 5 A process optionally performed after optimizing the code at step 514).

[0047] Therefore, in step 702, it is assumed that the processing system (e.g., processing system 102) will perform at least Figure 5 Although Figure 7 The exemplary method of FIG. 7 shows steps 704-720 occurring after step 702, but it should be understood that steps 704-720 may occur relative to step 702. Figure 5 Steps 502-518 of the exemplary method of are performed in any suitable order. For example, in some aspects of the technology, the processing system can be configured to identify the optimized code for a given image in the image collection (in Figure 5 ), and then in selecting the next given image (in step 514 of Figure 5 518) before or in conjunction with testing multiple codes to identify the next optimized code for a given image (e.g., Figure 5 In this case, the processing system may also be configured to perform steps 704-712 in parallel with steps 504-514 of the image. Figure 5 Steps 504-518 are performed in parallel with step 714 (after each batch of images is processed according to steps 704-712), so that the parameters of the generative model are periodically updated as optimization codes continue to be identified (in successive passes through step 518).

[0048] Regardless of the timing, in step 704, the processing system selects a given image from a set of images of the object. This set of images may be Figure 5 For example, in some aspects of the technology, the processing system can be configured to process the entire image set used in the image processing, or any suitable subset thereof. Figure 5Steps 502-518 of identify optimized code for a set of 200 images of an object, but may then be configured to adjust a generative model (eg, generative model 306) based on only 100 of those images.

[0049] In step 706, the processing system uses the generative model and the recognition of the given image (in Figure 5 Here again, the generation model can be configured in any suitable manner (including as described above with respect to Figure 3A , Figure 3B , Figure 4A and Figure 4B In addition, although step 706 refers to a "second image", it should be understood that this second image may be, in some cases, Figure 5 Thus, in some aspects of the technique, for any "given image" in the first batch of images, the processing system may be configured to use the same Figure 5 The processing system may be configured to generate a new “second image” (or a copy thereof) associated with the optimization code identified in step 514 of step 706 as the “second image” of step 706, rather than generating the image a second time in step 706. However, once the processing system modifies one or more parameters of the generative model (discussed below) in step 714, the generative model changes so that each subsequent optimization code will produce a “second image” that is different from the second image previously produced by that code. Thus, in the event that method 700 is executed after method 500 (rather than in parallel with some or all of the steps of method 500), the processing system may be configured to generate a new “second image” (rather than using a new “second image”) in each successive pass through step 706. Figure 5 a copy of the “first image” identified in step 508 ).

[0050] In step 708, the processing system compares the second image (generated in step 706) to the given image (selected in step 704) to generate a second loss value. Also here, the second loss value can be generated in any suitable manner using any suitable function. For example, in some aspects of the technology, the second loss value can be based on a comparison of the second image to the given image using a heuristic or learned similarity metric (e.g., learned perceptual image patch similarity ("LPIPS"), peak signal-to-noise ratio ("PSNR"), structural similarity index measure ("SSIM")). In addition, although step 708 references a "second loss value", it should be understood that this second loss value may be, in some cases, a second loss value. Figure 5Thus, in some aspects of the technique, for any "given image" in the first batch of images, the processing system may be configured to use Figure 5 The “lowest first loss value” identified for the given image in step 514 of the processing system is used as the “second loss value” of step 708, rather than regenerating a loss value in step 708. However, as already mentioned, once the processing system modifies one or more parameters of the generative model in step 714 (discussed below), the generative model is changed so that each subsequent optimization code will produce a “second image” that is different from the second image previously produced by the code, and therefore produces a different “second loss value” when compared to the “given image”. Therefore, in the case where method 700 is executed after method 500 (rather than in parallel with some or all of the steps of method 500), the processing system can be configured to generate a new “second loss value” in each consecutive pass through step 708 (rather than using Figure 5 a copy of the “lowest first loss value” identified in step 514 of ).

[0051] In step 710, the processing system determines whether there are additional images in the batch. In this regard, the image collection can remain intact, or be divided into any suitable number of batches. In the event that the image collection has not been split, and therefore there is a single "batch" containing every image in the "image collection" of the object, the processing system will follow the "yes" arrow to step 712 to select the next given image from the image collection of the object, and repeat steps 706-710 for the newly selected image. This process will be repeated until there are no additional images remaining in the image collection, at which point the processing system will follow the "no" arrow to step 714. On the other hand, in the event that the image collection is divided into two or more batches (e.g., a collection of 200 images can be divided into two batches of 100 images, four batches of 50 images, ten batches of 20 images, 200 "batches" of a single image, etc.), steps 704-712 will be repeated for each image until the end of the batch is reached.

[0052] As shown in step 714, after generating the "second loss value" for each image in the batch (in step 708), the processing system modifies one or more parameters of the generative model based at least in part on each generated second loss value. The processing system can be configured to modify the one or more parameters in any suitable manner and at any suitable interval based on these generated second loss values. Thus, in some aspects of the technology, each "batch" can include a single image, so that the processing system will perform a back-propagation step, wherein each time a second loss value is generated, the processing system modifies one or more parameters of the generative model. Similarly, where each "batch" includes two or more images, the processing system can be configured to combine each of the "second loss values" generated (in step 708) for each image of the batch into an aggregate loss value (e.g., by summing or averaging multiple second loss values), and modify one or more parameters of the generative model based on the aggregate loss value.

[0053] In step 716, the processing system determines whether there are additional batches in the image collection of the object. In the event that the image collection has not been split, and therefore there is a single "batch" containing every image in the "image collection" of the object, the determination in step 716 will automatically become "no", and the method 700 will then end, as shown in step 720. However, in the event that the image collection is divided into two or more batches, the processing system will follow the "yes" arrow to step 718 to select the next given image from the image collection of the object. This will then begin another set of steps 706-714 for each image in the next batch of images, and the process will continue until there are no additional batches remaining, at which point the processing system will follow the "no" arrow to step 720.

[0054] Although method 700 is shown as ending in step 720 once all images have been used to adjust the generative model, it should be understood that method 700 may be repeated any suitable number of times using the same set of images until it produces images that are sufficiently close to the output of each optimization code for each given image. In this regard, in some aspects of the technology, the processing system may be configured to aggregate all second loss values ​​generated during a given pass through method 700, and determine whether to repeat method 700 for the set of images based on the aggregated loss value. For example, in some aspects of the technology, if the aggregated loss value of the most recent pass through method 700 is greater than some predetermined threshold, the processing system may be configured to repeat method 700 for the set of images. Likewise, in some aspects, the processing system may be configured to use gradient descent to make this determination, and therefore repeat method 700 for the set of images until the aggregated loss value for a given pass through method 700 is equal to or greater than the aggregated loss value in the pass before it.

[0055] Figure 8 According to various aspects of the present disclosure, a method for Figure 5 or Figure 7 In this regard, the exemplary method 800 represents an exemplary method that can generate at least one personalized prior (in Figure 5 ) is optionally performed after step 520 of Figure 7 The method further adjusts the process performed after the model is generated. Therefore, in step 802, it is assumed that the processing system (e.g., processing system 102) will perform at least Figure 5 The method 500 and optionally performing Figure 7 Steps 704-720.

[0056] As described above, the processing system generates a personalized prior (using Figure 5 Method 500) and optionally adjust the generative model (using Figure 7 After the method 700 of FIG. 8 , the personalized prior can be used to generate different candidate coefficient sets, and the code corresponding to the candidate coefficient sets can then be used to generate candidate images for a given image enhancement task. Therefore, as shown in step 804, for a particular input image of the object, the processing system will repeatedly perform steps 806-812 to test multiple coefficient sets to identify a personalized output image, where each coefficient set in the multiple coefficient sets is consistent with the convex hull (in FIG. Figure 5 Corresponding to the code in step 520 of identification).

[0057] In each pass through step 806, the processing system identifies a set of coefficients from among the plurality of sets of coefficients and uses the set of coefficients to generate a given code. This set of coefficients may be identified based on any suitable selection criteria. For example, in some aspects of the technology, the processing system may be configured to blindly select a first set of coefficients (e.g., using a random selection process, or a pre-selected value, such as an average value of a vector space represented by a personalized prior), and then select each successive set of coefficients (in each successive pass through step 806) directly or indirectly based on the first set of coefficients using a suitable optimization scheme. Thus, in some aspects, the processing system may be configured to use gradient descent to select each successive set of coefficients based on a previous set of coefficients and an evaluation of how well an image generated based on the previous set of coefficients matches the input image (e.g., a third loss value generated in the most recent pass through step 810).

[0058] In step 808, the processing system uses a generative model (e.g., generative model 306, or Figure 7The generative model can be configured to generate a third image in any suitable manner (including as described above with respect to Figure 3A , Figure 3B , Figure 4A and Figure 4B The third image is generated in the manner described above.

[0059] In step 810, the processing system compares the third image (here generated by step 808) with the input image of the object to generate a third loss value for the third image. The third loss value can be generated in any suitable manner using any suitable function. For example, in some aspects of the technology, the third loss value can be based on a comparison of the third image with the input image using a heuristic or learned similarity metric (e.g., learned perceptual image patch similarity ("LPIPS"), peak signal-to-noise ratio ("PSNR"), structural similarity index measure ("SSIM")).

[0060] In step 812, the processing system determines whether another set of coefficients should be tested. Since it is assumed that multiple sets of coefficients will be tested, the first time the processing system reaches step 812, it will automatically follow the "yes" arrow and return to step 806 so that the second set of coefficients will be identified and tested. However, on all subsequent returns to step 812, the processing system can determine whether to test another set of coefficients based on any suitable criteria. Therefore, as mentioned above, in some aspects of the technology, the processing system can determine when to stop testing another set of coefficients based on a suitable optimization scheme (such as gradient descent). In this case, the determination in step 812 can be based on a comparison of the third loss value generated in the current pass step 810 (or some other assessment of the degree of match of the third image to the input image) with one or more of the third loss values ​​generated in the previous pass. For example, the processing system can be configured to stop testing the consecutive set of coefficients when the third loss value generated in the current pass step 810 is equal to or greater than the third loss value generated in the previous pass step 810.

[0061] Thus, the processing system will loop through steps 806-812 for each subsequent set of coefficients until, at step 812, it is determined that enough sets of coefficients have been tested. At this point, the processing system will follow the "NO" arrow to step 814, where the processing system will compare the third loss values ​​generated (in step 810) for each third image to identify the third image with the lowest third loss value. The third image with the lowest third loss value is used as the personalized output image. In this way, method 800 can be used to produce a personalized output image that is both generated using a code within a personalized prior (so that it is more likely to resemble an object than a personalized output image that would be generated if the code were unconstrained) and optimized to closely match the input image (thereby ensuring that the image generated by the model is consistent with the image that can be gleaned from the input image when performing an image editing or enhancement task).

[0062] Fig. 9 According to various aspects of the present disclosure, a method for Figure 5 or Figure 7 Another exemplary method 900 for generating a personalized output image using a personalized prior generated by a method. In this regard, the exemplary method 900 also represents a method that can generate at least one personalized prior (in Figure 5 ) is optionally performed after step 520 of Figure 7 The method further regulates the process performed after the generated model. Figure 8 Method 800 with Fig. 9 The only difference between the method 900 and the method 900 is that in each pass through steps 906-912, a given code set is tested, wherein each individual code in the given code set is tested in Figure 5 within the convex hull identified in step 520 .

[0063] Thus, as described above, in step 902, it is assumed that the processing system (e.g., processing system 102) will perform at least Figure 5 The method 500 and optionally performing Figure 7 Similarly, as shown in step 904, for a particular input image of the object, the processing system will repeatedly perform steps 906-912 to test multiple code sets to identify the personalized output image, where each code in the code set is in the convex hull (in Figure 5 identi?ed in step 520).

[0064] In each pass through step 906, the processing system identifies a given code set that includes two or more individual codes. Here again, this given code set can be identified based on any suitable selection criteria. For example, in some aspects of the technology, the processing system can be configured to blindly select a first given code set (e.g., using a random selection process, or by assigning a pre-selected value to each individual code, such as an average value of a vector space represented by a personalized prior), and then select each consecutive code set (in each consecutive pass through step 906) directly or indirectly based on the first given code set using a suitable optimization scheme. Therefore, in some aspects, the processing system can be configured to use gradient descent to select each consecutive code set based on the previous code set and an evaluation of the degree of match between the image generated by the previous code set and the input image (e.g., the third loss value generated in the most recent pass through step 910).

[0065] In step 908, the processing system uses a generative model (e.g., generative model 306, or Figure 7 The processing system generates a third image by using one or more of the adjusted generative models generated by steps 704-720) and a given set of codes (here generated by step 906). In this regard, in step 908, the processing system provides each individual code in the given set of codes to a different layer or set of layers of the generative model. Also here, the generative model can be configured in any suitable manner (including as described above with respect to Figure 3A , Figure 3B , Figure 4A and Figure 4B The third image is generated in the manner described above.

[0066] In step 910, the processing system compares the third image (here generated by step 908) with the input image of the object to generate a third loss value for the third image. Also here, the third loss value can be generated in any suitable manner using any suitable function. For example, in some aspects of the technology, the third loss value can be based on a comparison of the third image with the input image using a heuristic or learned similarity metric (e.g., learned perceptual image patch similarity ("LPIPS"), peak signal-to-noise ratio ("PSNR"), structural similarity index measure ("SSIM")).

[0067] In step 912, the processing system determines whether another code set should be tested. Since it is assumed that multiple code sets will be tested, the first time the processing system reaches step 912, it will automatically follow the "yes" arrow and return to step 906 so that the second code set will be identified and tested. However, on all subsequent returns to step 912, the processing system can determine whether to test another code set based on any suitable criteria. Therefore, as mentioned above, in some aspects of the technology, the processing system can determine when to stop testing another code set based on a suitable optimization scheme (such as gradient descent). In this case, the determination in step 912 can be based on a comparison of the third loss value generated in the current pass step 910 (or some other evaluation of the degree of match of the third image to the input image) with one or more of the third loss values ​​generated in the previous pass. For example, the processing system can be configured to stop testing the continuous code set when the third loss value generated in the current pass step 910 is equal to or greater than the third loss value generated in the previous pass step 910.

[0068] Similar to aspect 800, the processing system will therefore loop through steps 906-912 for each subsequent code set until it is determined at step 912 that enough code sets have been tested. At this point, the processing system will follow the "no" arrow to step 914, where it will compare the third loss values ​​generated (in step 910) for each third image to identify the third image with the lowest third loss value. Here again, the third image identified as having the lowest third loss value is used as the personalized output image.

[0069] Fig.10 is an illustration of how a generative model (e.g., generative model 306) according to aspects of the present disclosure may be used or not used. Figure 4A Graph 1000 illustrates a comparison of four exemplary image enhancement tasks performed with personal priors 402 .

[0070] exist Fig.10 1000, a column of four input images 1002a-1002d is shown in the center of diagram 1000. Since the generative model is assumed to be responsible for producing an output image that is similar to the input image but with increased resolution, each of the input images 1002a-1002d is blurred.

[0071] The left column of images 1004a-1004d shows the latent output of a generative model where the generative model is not constrained to select a code within any particular portion of its latent vector space W(302). In this regard, a dashed line connects each input image to a point within the latent vector space W(302) that the generative model ultimately selects (e.g., in a region such as the region described above with respect to FIG. Figure 8Method 800 or Fig. 9 900) and an arrow connects the point to the corresponding output image generated by the generative model based on the code. As can be seen, although the output images 1004a-1004d are each visually consistent with their corresponding input images 1002a-1002d, the output images do not all appear to belong to the same object.

[0072] In contrast, the right column of images 1006a-1006d shows the latent outputs of a generative model that is constrained to select Figure 4A Here again, a dashed line connects each input image to a point in personal prior 402, which represents the point that the generative model ultimately chooses (e.g., in the example above relative to Figure 8 Method 800 or Fig. 9 900 ), and an arrow connects the point to the corresponding output image generated by the generative model based on the code. As can be seen, this ultimately produces output images 1006a-1006d that are visually consistent with their corresponding input images 1002a-1002d and appear to belong to the same object. Specifically, due to Figure 4A The personalized prior 402 represents a certain code range that, when provided to the generative model 306, produces an image similar to that of the President of the United States (as discussed above), so that each of the output images 1006a-1006d appears to show an image of the President of the United States that is visually consistent with the input images 1002a-1002d.

[0073] Fig.11 is an illustration of how a generative model (e.g., generative model 306) according to aspects of the present disclosure may be used or not used. Figure 4B FIG1100 is a diagram illustrating a comparison of four exemplary image restoration tasks performed with personal priors 403.

[0074] Here again, a column of four input images 1102a-1102d is shown in the center of illustration 1000. Since the generative model is assumed to produce output images that are similar to the input images but with the masked portions filled in, each of the input images 1102a-1102d includes a black mask 1103a-1103d representing the pixels to be replaced.

[0075] In diagram 1100, the left column of images 1104a-1104d show the latent outputs of a generative model where the generative model is not constrained to select codes within any particular portion of its latent vector space W(302). Here again, a dashed line connects each input image to a point within the latent vector space W(302) that the generative model ultimately selects (e.g., in a region such as the region described above with respect to FIG. Figure 8 Method 800 or Fig. 9 900) and an arrow connects the point to the corresponding output image generated by the generative model based on the code. As can be seen, although the output images 1104a-1104d are each visually consistent with the unmasked portions of their corresponding input images 1102a-1102d, the output images do not all appear to belong to the same object.

[0076] In contrast, the right column of images 1106a-1106d shows the latent outputs of a generative model that is constrained to select Figure 4B Here again, a dashed line connects each input image to a point in personal prior 403, which represents the point that the generative model ultimately chooses (e.g., in the example above relative to Figure 8 Method 800 or Fig. 9 900 ), and an arrow connects the point to the corresponding output image generated by the generative model based on the code. As can be seen, this ultimately produces output images 1106a-1106d that are visually consistent with their corresponding input images 1102a-1102d and appear to belong to the same object. Specifically, due to Figure 4B The personalized prior 403 represents a certain range of codes that, when provided to the generative model 306, produces an image similar to the famous American singer, songwriter, and actor (as discussed above), so that each of the output images 1106a-1106d appears to show an image of the famous American singer, songwriter, and actor that is visually consistent with the unmasked portions of the input images 1102a-1102d.

[0077] Thus, both diagrams 1000 and 1100 illustrate how personalized priors can be used to focus the code used by a generative model so that the generative model can produce examples of output images that are both visually consistent with the input image and consistent with the identity of a particular object. Thus, given that the object of the input image is already known, a personalized prior can be selected and used so that the generative model will be biased toward producing a more representative and therefore better output image.

[0078] Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the foregoing description of the exemplary systems and methods should be made by way of illustration rather than by way of limitation of the subject matter defined by the claims. In addition, the examples described herein and the provision of clauses expressed as "such as," "including," "comprising," etc. should not be interpreted as limiting the subject matter of the claims to specific examples; rather, these examples are intended to illustrate only some of the many possible embodiments. In addition, the same reference numerals in different figures may identify the same or similar elements.

Claims

1. A computer-implemented method comprising: For each given image in the collection of images of the object, testing a plurality of codes using one or more processors of the processing system to identify an optimized code for the given image includes: For each of the multiple codes: generating a first image using the generative model and the code; and comparing the first image to the given image using the one or more processors to generate a first loss value for the code; and comparing, using the one or more processors, the first loss values ​​generated for each of the plurality of codes to identify the code having the lowest first loss value as the optimized code for the given image; and Using the one or more processors, a personalized prior for the object is generated based on a convex hull that includes each optimized code identified for each given image in the collection of images of the object.

2. The method of claim 1, further comprising: The optimized code identified for each given image in the set of images of the object: generating a second image using the generative model and the optimized code; as well as comparing the second image to the given image using the one or more processors to generate a second loss value; as well as Using the one or more processors, one or more parameters of the generative model are modified based at least in part on each generated second loss value to create an adjusted generative model.

3. The method of claim 2, further comprising: identifying, using the one or more processors, a plurality of sets of coefficients, each set of coefficients in the plurality of sets of coefficients corresponding to a code within the convex hull; For each given set of coefficients in the plurality of sets of coefficients: generating, using the one or more processors, a third image using the adjusted generative model and a given code corresponding to the given set of coefficients; as well as comparing, using the one or more processors, the third image to at least a portion of an input image of the object to generate a third loss value for the third image; as well as The third loss values ​​generated for each third image are compared using the one or more processors to identify the third image having the lowest third loss value as the personalized output image.

4. The method of claim 2, further comprising: identifying, using the one or more processors, a plurality of sets of coefficients, each set of coefficients in the plurality of sets of coefficients corresponding to a code within the convex hull; identifying, using the one or more processors, a plurality of code sets, each code set in the plurality of code sets comprising two or more individual codes, each individual code corresponding to a set of coefficients in the plurality of sets of coefficients; For each given code set in the plurality of code sets: generating, using the one or more processors, a third image using the adjusted generative model and the given set of codes, each individual code in the given set of codes being provided to a different layer or set of layers of the adjusted generative model; as well as comparing, using the one or more processors, the third image to at least a portion of an input image of the object to generate a third loss value for the third image; as well as The third loss values ​​generated for each third image are compared using the one or more processors to identify the third image having the lowest third loss value as the personalized output image.

5. The method of claim 1, further comprising: identifying, using the one or more processors, a plurality of sets of coefficients, each set of coefficients in the plurality of sets of coefficients corresponding to a code within the convex hull; For each given set of coefficients in the plurality of sets of coefficients: generating, using the one or more processors, a third image using the generative model and a given code corresponding to the given set of coefficients; as well as comparing, using the one or more processors, the third image to at least a portion of an input image of the object to generate a third loss value for the third image; as well as The third loss values ​​generated for each third image are compared using the one or more processors to identify the third image having the lowest third loss value as the personalized output image.

6. The method of claim 1, further comprising: identifying, using the one or more processors, a plurality of sets of coefficients, each set of coefficients in the plurality of sets of coefficients corresponding to a code within the convex hull; identifying, using the one or more processors, a plurality of code sets, each code set in the plurality of code sets comprising two or more individual codes, each individual code corresponding to a set of coefficients in the plurality of sets of coefficients; For each given code set in the plurality of code sets: generating, using the one or more processors, a third image using the generative model and the given set of codes, each individual code in the given set of codes being provided to a different layer or set of layers of the generative model; as well as comparing, using the one or more processors, the third image to at least a portion of an input image of the object to generate a third loss value for the third image; as well as The third loss values ​​generated for each third image are compared using the one or more processors to identify the third image having the lowest third loss value as the personalized output image.

7. The method of claim 3 or claim 5, wherein the plurality of coefficient sets comprises a first coefficient set, and a plurality of consecutive coefficient sets selected directly or indirectly based on the first coefficient set using gradient descent.

8. The method of any one of claims 3 to 7, wherein the input image of the object comprises a first pixel portion retained from an original image of the object, and a mask replacing a second pixel portion from the original image of the object, and Wherein comparing the third image with at least a portion of the input image of the object using the one or more processors to generate the third loss value of the third image includes comparing the third image with the first pixel portion to generate the third loss value of the third image.

9. The method of any one of claims 3 to 7, wherein the input image has a first resolution and the personalized output image has a second resolution higher than the first resolution.

10. The method of any one of claims 1 to 9, wherein the plurality of codes comprises a first code, and a plurality of consecutive codes selected directly or indirectly based on the first code using gradient descent.

11. The method of claim 10, wherein the first code represents an average value of a latent vector space W, the latent vector space W representing all possible codes that can be input into the generative model.

12. A processing system comprising: A memory storing a generation model; as well as One or more processors coupled to the memory and configured to generate a personalized prior for an object for use with the generative model, comprising: For each given image in the set of images of the object, a plurality of codes are tested to identify an optimized code for the given image, including: For each of the multiple codes: generating a first image using the generative model and the code; and comparing the first image to the given image to generate a first loss value for the code; and comparing the first loss values ​​generated for each of the plurality of codes to identify the code having the lowest first loss value as the optimized code for the given image; and The personalized prior for the object is generated based on a convex hull that includes each optimized code identified for each given image in the set of images of the object.

13. The processing system of claim 12, wherein the one or more processors are further configured to adjust the generative model, comprising: The optimized code identified for each given image in the set of images of the object: generating a second image using the generative model and the optimized code; as well as comparing the second image to the given image using the one or more processors to generate a second loss value; as well as Using the one or more processors, one or more parameters of the generative model are modified based at least in part on each generated second loss value to create an adjusted generative model.

14. The processing system of claim 13, wherein the one or more processors are further configured to generate a personalized output image based on the input image of the object, comprising: identifying a plurality of sets of coefficients, each set of coefficients in the plurality of sets of coefficients corresponding to a code within the convex hull; For each given set of coefficients in the plurality of sets of coefficients: generating a third image using the adjusted generative model and a given code corresponding to the given set of coefficients; as well as comparing the third image to at least a portion of the input image of the object to generate a third loss value for the third image; as well as The third loss values ​​generated for each third image are compared to identify the third image having the lowest third loss value as the personalized output image.

15. The processing system of claim 13, wherein the one or more processors are further configured to generate a personalized output image based on the input image of the object, comprising: identifying a plurality of sets of coefficients, each set of coefficients in the plurality of sets of coefficients corresponding to a code within the convex hull; identifying a plurality of code sets, each code set in the plurality of code sets comprising two or more individual codes, each individual code corresponding to a coefficient set in the plurality of coefficient sets; For each given code set in the plurality of code sets: generating a third image using the adjusted generative model and the given set of codes, each individual code in the given set of codes being provided to a different layer or set of layers of the adjusted generative model; as well as comparing the third image to at least a portion of the input image of the object to generate a third loss value for the third image; as well as The third loss values ​​generated for each third image are compared to identify the third image having the lowest third loss value as the personalized output image.

16. The processing system of claim 12, wherein the one or more processors are further configured to generate a personalized output image based on the input image of the object, comprising: identifying a plurality of sets of coefficients, each set of coefficients in the plurality of sets of coefficients corresponding to a code within the convex hull; For each given set of coefficients in the plurality of sets of coefficients: generating a third image using the generative model and a given code corresponding to the given set of coefficients; as well as comparing the third image to at least a portion of the input image of the object to generate a third loss value for the third image; as well as The third loss values ​​generated for each third image are compared to identify the third image having the lowest third loss value as the personalized output image.

17. The processing system of claim 12, wherein the one or more processors are further configured to generate a personalized output image based on the input image of the object, comprising: identifying a plurality of sets of coefficients, each set of coefficients in the plurality of sets of coefficients corresponding to a code within the convex hull; identifying a plurality of code sets, each code set in the plurality of code sets comprising two or more individual codes, each individual code corresponding to a coefficient set in the plurality of coefficient sets; For each given code set in the plurality of code sets: generating a third image using the generative model and the given set of codes, each individual code in the given set of codes being provided to a different layer or set of layers of the generative model; as well as comparing the third image to at least a portion of the input image of the object to generate a third loss value for the third image; as well as The third loss values ​​generated for each third image are compared to identify the third image having the lowest third loss value as the personalized output image.

18. The processing system of claim 14 or claim 16, wherein the plurality of coefficient sets comprises a first coefficient set and a plurality of consecutive coefficient sets, and The one or more processors are further configured to select each set of coefficients in the plurality of consecutive sets of coefficients based directly or indirectly on the first set of coefficients using gradient descent.

19. The processing system of any one of claims 14 to 18, wherein the input image of the object comprises a first pixel portion retained from an original image of the object, and a mask replacing a second pixel portion from the original image of the object, and Wherein comparing the third image with at least a portion of the input image of the object to generate the third loss value of the third image includes comparing the third image with the first pixel portion to generate the third loss value of the third image.

20. The processing system of any one of claims 14 to 18, wherein the one or more processors are configured to generate the personalized output image based on the input image of the object, wherein the input image has a first resolution and the personalized output image has a second resolution higher than the first resolution.

21. The processing system of any one of claims 12 to 20, wherein the plurality of codes comprises a first code and a plurality of consecutive codes, and Wherein the one or more processors are further configured to select each of the plurality of consecutive codes based directly or indirectly on the first code using gradient descent.

22. The processing system of claim 21, wherein the one or more processors are further configured to select a first code representing an average value of a latent vector space W representing all possible codes that can be input into the generative model.

23. A processing system comprising: A memory storing a generation model; as well as One or more processors coupled to the memory and configured to perform the method of any one of claims 1 to 11.

Images