Improving facial recognition tasks using enhanced facial images

Abstract

Methods and systems for generating a composite image based on an input image are described. A method may include receiving an input image; generating, using an encoder, a first potential code vector representation based on the input image; receiving a potential code corresponding to a feature to be added to the input image; modifying the first potential code vector representation based on a potential code corresponding to a feature to be added; generating, by an image decoder, a composite image based on the modified first potential code vector representation; identifying one or more landmarks in the base image using a landmark detector; identifying, using a flag detector, one or more flags in the composite image; determining a similarity measure between the indicia identified on the base image and the indicia identified in the composite image; and discarding the composite image based on the comparison.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application No. 17 / 751,393, filed on May 23, 2022, entitled USING AUGMENTED FACE IMAGES TO IMPROVE FACIAL RECOGNITION TASKS, the entire contents of which are incorporated herein by reference in their entirety. Background Art

[0003] The number of samples required to train a machine learning model is often proportional to the network capacity, the number of output dimensions, and the complexity of the task. For tasks that need to operate in arbitrary wild environments, the amount of data involved can be prohibitively large. For example, the amount of effort required to simply capture and label enough datasets to cover all possible conditions for face-related recognition tasks such as predicting age, ethnicity, hairstyle, facial hair, accessories (e.g., goggles, earrings), lighting, etc. can only grow exponentially.

[0004] As the visualization and video game industries push for realism in computer-generated imagery (CGI), there is a trend toward using traditional CGI techniques to generate synthetic datasets for ML model training. Some of the benefits of having a synthetic dataset generation system include nearly perfect labels that are available for free since they are simply control variables or derived properties of the above parametric models, and the number of samples is virtually unlimited since the dataset generation system can generate data when needed. Nonetheless, creating such a generation system, which can include controllable face and body generators with a comprehensive library of 3D assets for parts including, for example, hair and accessories, can still be a daunting task.

[0005] Therefore, there is a need to identify and utilize low-cost but effective ways to obtain data samples for robust model training. Embodiments have been described with respect to these and other general considerations. Although relatively specific problems have been discussed, the examples described herein should not be limited to solving the specific problems identified in the above background technology. Summary of the invention

[0006] According to examples of the present disclosure, a low-cost but effective way to enhance existing data samples for robust model training is described. In the example, a framework incorporating StyleGAN manipulation to generate enhanced images for machine learning training improves the inference quality of face recognition tasks, especially for underrepresented samples encountered during field testing.

[0007] Therefore, according to an example of the present disclosure, a method for generating a synthetic image based on an input image. The method may include receiving an input image, and generating a first latent code vector representation based on the input image using an encoder. The method may also include receiving a latent code corresponding to a feature to be added to the input image, and modifying the first latent code vector representation based on the latent code corresponding to the feature to be added. In addition, the method may also include generating a synthetic image based on the modified first latent code vector representation by an image decoder. In an example, the method may also include identifying one or more landmarks in a base image using a landmark detector, identifying one or more landmarks in a synthetic image using a landmark detector, determining a similarity measure between landmarks identified on the base image and landmarks identified in the synthetic image, and discarding the synthetic image based on the comparison.

[0008] According to an example of the present disclosure, a system for generating a synthetic image based on an input image is described. The system may include one or more hardware processors configured by machine-readable instructions to: receive an input image; generate a first latent code vector representation based on the input image using an encoder; receive a latent code corresponding to a feature to be added to the input image; modify the first latent code vector representation based on the latent code corresponding to the feature to be added; generate a synthetic image based on the modified first latent code vector representation by an image decoder; identify one or more landmarks in a base image using a landmark detector; identify one or more landmarks in a synthetic image using a landmark detector; determine a similarity measure between landmarks identified on the base image and landmarks identified in the synthetic image; and discard the synthetic image based on the comparison.

[0009] According to an example of the present disclosure, a non-transitory computer-readable storage medium is described, including instructions for generating a synthetic image based on an input image. In the example, the instructions can be executed by one or more processors to cause the one or more processors to: receive an input image; generate a first potential code vector representation based on the input image using an encoder; receive a potential code corresponding to a feature to be added to the input image; modify the first potential code vector representation based on the potential code corresponding to the feature to be added; generate a synthetic image based on the modified first potential code vector representation by an image decoder; identify one or more landmarks in a base image using a landmark detector; identify one or more landmarks in a synthetic image using a landmark detector; determine a similarity measure between landmarks identified on the base image and landmarks identified in the synthetic image; and discard the synthetic image based on the comparison.

[0010] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Non-limiting and non-exhaustive examples are described with reference to the following figures.

[0012] Figure 1 Depicts an image enhancement and generation framework according to examples of the present disclosure.

[0013] Figure 2 Depicts examples of input and output images according to examples of the present disclosure.

[0014] Figure 3 Depicted are examples of directly evaluating input and output images based on one or more landmarks associated with facial expressions according to examples of the present disclosure.

[0015] Figure 4 Depicts additional examples of output images that include augmented data and are generated from corresponding input images.

[0016] Figure 5 Depicted are aspects of a synthetic data generator for generating an augmented image according to examples of the present disclosure.

[0017] Figure 6 Depicted are block diagrams illustrating the physical components (eg, hardware) of a computing system with which aspects of the present disclosure may be practiced.

[0018] Figure 7 One aspect of a system architecture for processing data received at a computing system from a remote source according to examples of the present disclosure is described.

[0019] Figure 8 Details of a first method according to an example of the present disclosure are described.

[0020] Fig. 9 Describes details of a second method according to examples of the present disclosure. DETAILED DESCRIPTION

[0021] In the following detailed description, reference is made to the accompanying drawings forming a part thereof, and specific embodiments or examples are shown by way of illustration in the accompanying drawings. These aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from the present disclosure. The embodiments may be practiced as methods, systems, or devices. Therefore, the embodiments may take the form of hardware implementations, complete software implementations, or implementations combining software and hardware aspects. Therefore, the following detailed description should not be regarded as restrictive, and the scope of the present disclosure is limited by the appended claims and their equivalents.

[0022] The process of training a machine learning (ML) model is primarily about tuning its parameters so that it can map any input to a specific output. The number of samples required to train a model is typically proportional to the network capacity, the number of output dimensions, and most importantly, the complexity of the task. For tasks that need to operate in arbitrary wild environments, the amount of data can be prohibitively large. For example, the effort of simply capturing and labeling enough datasets to cover all possible conditions for face-related recognition tasks such as predicting age, ethnicity, hairstyle, facial hair, accessories (goggles, earrings), lighting, etc. can only grow exponentially.

[0023] As the visualization and video game industries push for realism in computer-generated imagery (CGI), there is a trend toward using traditional CGI techniques to generate synthetic datasets for ML model training. Some of the benefits of having a synthetic dataset generation system include nearly perfect labels that are available for free since they are simply control variables or derived properties of the above parametric models, and the number of samples is virtually unlimited since the dataset generation system can generate data when needed. Nonetheless, creating such a generation system, which can include controllable face and body generators with a comprehensive library of 3D assets for parts including, for example, hair and accessories, can still be a daunting task.

[0024] Therefore, systems between real capture methods and pure synthesis methods are so-called enhancement methods. Among typical enhancement operations on image data, geometric or color enhancement, random erasing and feature space enhancement operations can be performed. Recently, StyleGAN and its variants are able to generate high-quality and very natural-looking synthetic facial images. These techniques can also be applied to image enhancement techniques, mainly in the fields of image transfer and stylization. InterFaceGAN also provides a way to identify semantic latent vectors in StyleGAN for additional manipulation and use. As mentioned above, a framework for manipulating images to generate enhanced image data for machine learning is presented. Therefore, such a framework can improve the inference quality of facial recognition tasks. As an example, the task of predicting facial blend shape activations from facial images is used as a task trained using such enhanced images.

[0025] The task of predicting facial blend shape activations (e.g., jaw opening, blinking, etc.) from facial images is commonly used in facial animation applications. Blend shape models are linear deformable models based on a set of predefined shapes that represent surface movements of various facial regions, and "activations" are represented as blend weights of the linear model. In order to train a neural network to perform such a task, a paired dataset is prepared, where each sample consists of a facial image and its corresponding blend shape activation; thus, the neural network can be trained in a supervised manner. In order to diversify the training samples, features like goggles or beards can be added to real capture or original images. However, such additional features generally do not change the facial expression in the image because it is related to paired activations. As described herein, two main stages are involved in the enhanced image generation framework described herein. First, the input image or the original image can be encoded into a latent space so that the latent code can be edited and a pre-trained model such as, but not limited to, StyleGAN2 is used to generate the final enhanced output image from the modified latent code. Second, after performing the data augmentation generation described in the first step, a landmark detector can be used to extract facial landmarks from the original input image and the enhanced output image; such facial landmarks can be compared to identify or filter good enhanced images from poor enhanced images. Finally, the enhanced images can be used for training a neural network to improve its performance, as explained below.

[0026] For data augmentation, the enhanced image generation framework can utilize InterFaceGAN to identify those semantic latent vectors present in StyleGAN. For example, a pre-trained StyleGAN encoder can be used to map an image into a latent space with some attribute labels. Therefore, an independent linear support vector machine (SVM) can be applied to each attribute. The hyperplane of each SVM and its corresponding normal vector can be associated with such semantic meaning. In order to enhance one or more images in the dataset, each image is encoded into one or more latent codes, and then randomly scaled semantic vectors can be added to the latent code, where randomly scaled semantic vectors can be added to promote enhanced diversity. Such enhanced codes (e.g., the resulting semantic vectors) can then be forwarded to the pre-trained StyleGAN2, where the final enhanced output image can be synthesized.

[0027] In the example task, the goal of the augmented image generation framework is to create training images that can reuse the original output as much as possible (e.g., blend shape activations). However, if the augmented image contains undesired facial expression changes, which will violate the underlying blend shape activations, this can jeopardize the inference quality. Specifically, for each input image x'∈R H×W×3 , the generated image should be x^∈R H×W×3, to maintain the task-relevant attributes except the modified attributes (i.e., beard, glasses). To measure the effectiveness of the generated samples, a facial keypoint detector can be used to extract facial landmarks: Φ: R H×W×3 →R 240 ×2 . A normalized quality score can then be generated as the root mean square error between the landmarks from the original image and its generated enhanced counterpart. Generated images with quality scores above a given threshold can then be used to train a machine learning model in a dataset, where the dataset includes images with expected visual attributes. Therefore, the trained model is more robust to those attributes that were previously underrepresented. In the example, a convolutional neural network can be used as a feature extractor, and a small number of fully connected layers can be used to regress task-related values. A smooth L1 loss can be used, and the machine learning model including the neural network is trained using the Adam optimizer.

[0028] Thus, some of the generated images can be visualized using the selected attributes, as depicted in the accompanying figures, to demonstrate that the image augmentation framework can generate realistic images with modified attributes, but maintain facial expressions intact. As depicted in the accompanying figures described herein, the proposed landmark inconsistency metric service (such as, correlation metric) can be used as a hint to filter out low-quality samples that are not suitable for training machine learning models. Let the machine learning model learn to improve performance using augmented data, especially when the low-quality data is filtered such as using some heuristic rules (i.e., landmark error) to further improve model performance.

[0029] Figure 1 An image enhancement framework 100 is depicted according to an example of the present disclosure. The image enhancement framework 100 can generate a training image such as an enhanced data image 102 from an input image or original image 106, wherein the input image or original image 106 is selected from a plurality of input images or original images 104. Thus, for example, the image enhancement framework 100 can utilize a data generation pipeline 122, wherein the input image or original image 106 is provided to an encoder, such as an image encoder 108 configured to map the input image 106 to a latent space or latent vector code.

[0030] The latent space 110 may include latent vectors, where the latent space code corresponds to the input image 106 as encoded by the image encoder 108. In an example, the latent code editor 112 may receive a plurality of attribute selections, for example from the attribute selector 114, and emphasize or otherwise enhance the input image 106 or the latent vector code associated with the input image 106 to include one or more desired attributes. According to an example of the present disclosure, one or more inputs 116, such as selections corresponding to features of beards, facial hair, accessories, earrings, glasses, etc., may be provided to the attribute selector 114. In an example, the attribute selector 114 may identify a semantic latent vector corresponding to the input 116. Thus, for example, the attribute selector 114 may correspond to the semantic vector identified from the attribute selector 114, and may be used to scale or otherwise adjust or modify one or more latent codes from the original input image 106, thereby enhancing or emphasizing the desired features or attributes provided as input 116 in the latent space. The resulting latent code may then be provided to the image decoder 118. For example, a StyleGAN decoder or a pre-trained StyleGAN decoder may be utilized, wherein the output image 120 may then be synthesized. According to examples of the present disclosure, synthesizing or otherwise generating the enhanced image 104 from the input image 102 may generate a plurality of enhanced images 124 .

[0031] For example, the enhanced image 120 of the plurality of enhanced images 124 may be provided to a quality assessor 126, where the generated image 120 may be evaluated based on a quality metric or other heuristic evaluation process. For example, a facial landmark detector may be utilized to identify facial landmarks, such that the facial landmarks associated with the enhanced image 120 may be compared to the facial landmarks identified from the original input image 106, and in the case where the quality of the enhanced output image generated based on one or more evaluation features or heuristics is low, the correlation between the facial landmarks of the original input image 106 and the facial landmarks of the generated output image 120 as determined by the one or more landmark detectors may not meet a specified threshold. Therefore, the generated image may be discarded, for example, by being deleted or erased at 128. Therefore, good or otherwise positive samples may include the enhanced generated output image corresponding to the image set 130. The image set 130 may then be provided to, for example, a machine learning model 132, where the machine learning model 132 may be trained using the generated enhanced image.

[0032] The machine learning model may include 132, which may be any other model that may be trained using or otherwise on the input image, as described herein. Thus, for example, the blend shape activation model 132 may include a convolutional neural network 134 having fully connected layers 136, where such training may be evaluated based on some task loss 138 associated with the blend shape activations.

[0033] Figure 2 Depicts examples of input and output images according to examples of the present disclosure. The first input image 204 may correspond to Figure 1 The input image 106 and the output image 206 may correspond to Figure 1 The output image 206 includes a plurality of attributes or additional features, such as a beard 210 and a mustache 208. As another example, the first input image 212 may correspond to Figure 1 The input image 106 and the output image 214 may correspond to Figure 1 The output image 214 includes a plurality of attributes or additional features, such as a mustache 218 and glasses 216. As another example, the first input image 220 may correspond to Figure 1 The input image 106 and the output image 222 may correspond to Figure 1 The output image 222 includes a plurality of attributes or additional features, such as glasses 224. As another example, the first input image 226 may correspond to Figure 1 The input image 106 and the output image 228 may correspond to Figure 1 The output image 228 includes a plurality of attributes or additional features such as glasses 232 , and other accessories such as earrings 230 .

[0034] Figure 3 An example of directly evaluating an input image and an output image based on one or more landmarks associated with a facial expression according to an example of the present disclosure is depicted. That is, one or more features of the input image 302 may be compared to one or more features of the output image 304 to determine the extent to which such features correspond to each other. Figure 3 , features 308A to 308E may correspond well to features 310A to 310E, respectively, where output image 304 has been enhanced to include glasses 306. In an example, a landmark detector may be used to identify features for comparison. In some examples, features may correspond to expected facial expressions, although such features may not be so limited.

[0035] like Figure 3 As depicted in , output image 312 may include mouth 312B that is not very correlated with mouth 308B, thereby causing too much variation in the facial expression of input image 302. Therefore, output image 312 may be discarded.

[0036] Figure 4Additional examples of output images 404, 414, and 430 including augmented data and generated from respective input images 402, 412, and 428 are depicted. In the example, facial features 408 may be similar to facial features 410, where the output image includes glasses 406. However, in some examples, eye 424 corresponding to eye 426 may be different, such that output image 414 with mustache 418 and glasses 416 is not close enough or does not have the same facial features as the input image, where such features may be included in the comparison set of 422 and 420. Output image 430 generally illustrates that, in some examples, features may be removed from the input image. That is, for example, augmented output image 430 may not have features that input image 428 had. In such an example, output image 430 does not have mustache 436, but has glasses 432 and hat 434 added.

[0037] Figure 5 Depicts aspects of a synthetic data generator 502 for generating an enhanced image according to an example of the present disclosure. In an example, the synthetic data generator 502 can be a hardware device, such as a server. In an example, the synthetic data generator 502 can include a communication interface 504, a processor 506, and a computer-readable storage device 508 including one or more applications 510. In an example, the communication interface 504 can be coupled to a network and receive a plurality of input images, such as an image set 532, as input 512. An image acquisition manager 516 can manage the acquisition of the images 532, and in some cases, can perform pre-processing of the images to prepare them for enhancement. The image set 532 can include, for example, the input images 102 ( Figure 1 In an example, the image encoder 520 may be configured with Figure 1 The image encoder 108 is the same or similar. In an example, the potential code editor 522 can be the same as Figure 1 The potential code editor 112 is the same or similar to the image decoder 524. In an example, the image decoder 524 can be the same as the potential code editor 112. Figure 1 The attribute selector 526 may be the same as or similar to the image decoder 118 of FIG. Figure 1 The composite image analyzer 530 may be the same or similar to the attribute selector 114 of Figure 1The quality assessor 126 of the image set analyzer 520 may be the same or similar. In some examples, the image set analyzer 520 may be used to analyze the input image set and determine one or more features or attributes that are not adequately represented by such input image set. Thus, the image set analyzer 530 may provide or otherwise select one or more attributes to be expressed or otherwise included in the enhancement process. Thus, the synthetic data generator 502 may generate an image set 534 containing synthetic data and provide this image set as output. In an example, the image set 534 containing synthetic data may also include the original input images from the image set 532.

[0038] Figure 6 6 is a block diagram illustrating the physical components (e.g., hardware) of a computing system 600 with which aspects of the present disclosure may be practiced. The computing device components described below may be suitable for the computing and / or processing devices described above. In a basic configuration, the computing system 600 may include at least one processing unit 602 and a system memory 604. Depending on the configuration and type of the computing device, the system memory 604 may include, but is not limited to, volatile storage devices (e.g., random access memory (RAM)), non-volatile storage devices (e.g., read-only memory (ROM)), flash memory, or any combination of such memories.

[0039] The system memory 604 may include an operating system 605 and one or more program modules 606 suitable for running software applications 620, such as one or more components supported by the systems described herein. As an example, the system memory 604 may include an image encoder 621, a potential code editor 622, an image decoder 623, a composite image analyzer 625, an image set analyzer 626, and an attribute selector 624, where such components are supported by the systems described herein. For example, the operating system 605 may be suitable for controlling the operation of the computing system 600.

[0040] Furthermore, the examples of the present disclosure may be practiced in conjunction with a graphics library, other operating systems, or any other application, and are not limited to any particular application or system. Figure 6 608. The computing system 600 may have additional features or functionality. For example, the computing system 600 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage devices may be Figure 6 609 and non-removable storage device 610.

[0041] As described above, a plurality of program modules and data files may be stored in system memory 604. When executed on processing unit 602, program module 606 (e.g., software application 620) may perform processes including, but not limited to, the various aspects described herein. Other program modules may be used in accordance with various aspects of the present disclosure. In addition, examples of the present disclosure may be practiced in circuit discrete electronic components, packaged or integrated electronic chips containing logic gates, circuits utilizing microprocessors, or on a single chip containing electronic components or microprocessors. For example, embodiments of the present disclosure may be implemented via a processor or a processor. Figure 6 Each or many of the components described in the embodiment can be integrated into a system on a chip (SOC) on a single integrated circuit for practice. Such SOC devices may include one or more processing units, graphics units, communication units, system virtualization units, and various application functions, all of which are integrated (or "burned") onto a chip substrate as a single integrated circuit. When operated via the SOC, the functions described herein regarding the ability to switch protocols for the client can be operated via dedicated logic integrated with other components of the computing system 600 on a single integrated circuit (chip). Embodiments of the present disclosure may also be practiced using other technologies capable of performing logical operations (such as AND, OR, and NOT), including but not limited to mechanical, optical, fluid, and quantum technologies. In addition, embodiments of the present disclosure may be practiced within a general-purpose computer or within any other circuit or system.

[0042] The computing system 600 may also have one or more input devices 612, such as a keyboard, a mouse, a pen, a sound or voice input device, a touch or sliding input device, etc. The one or more input devices 612 may include an image sensor. An output device 614 may also be included, such as a display, a speaker, a printer, etc. The above devices are examples and other devices may also be used. Figure 6 As shown, computing system 600 may include one or more communication connections 616 that allow communication with other computing devices / systems 650. Examples of suitable communication connections 616 include, but are not limited to, radio frequency (RF) transmitters, receivers, and / or transceiver circuitry; universal serial bus (USB), parallel and / or serial ports. Computing system 600 may represent one or more of synthetic data generators 502.

[0043] The term computer-readable media as used herein may include computer storage media. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, or program modules. System memory 604, removable storage device 609, and non-removable storage device 610 are all examples of computer storage media (e.g., memory storage devices). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or any other article that can be used to store information and can be accessed by the computing system 600. Any such computer storage media may be part of the computing system 600. Computer storage media do not include carrier waves or other propagated or modulated data signals.

[0044] Communication media may be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may describe a signal that has one or more characteristics set or changed in such a manner as to encode the information in the signal. By way of example and not limitation, communication media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0045] Figure 7 One aspect of a system architecture for processing data received at a computing system from a remote source such as a personal computer 704, a tablet computing device 706, or a mobile computing device 708 is illustrated. The personal computer 704, the tablet computing device 706, or the mobile computing device 708 may include one or more applications 720; such applications may include, but are not limited to, an image encoder 520, a potential code editor 522, an image decoder 524, an attribute selector 526, a composite image analyzer 528, an image set analyzer 530, and an application 510. The content at the server device 702 may be stored in different communication channels or other storage types. For example, a directory service, a web portal, a stylized image service, an instant messaging store, or a social networking service may be used to store various documents.

[0046] As described above, one or more of the aforementioned program modules 604 or software applications 606 may be employed by server device 702 and / or personal computer 704, tablet computing device 706, or mobile computing device 708. For example, server device 702 may include synthetic data generation framework 718 as described above.

[0047] The server device 702 can provide data to and from client computing devices such as a personal computer 704, a tablet computing device 706, and / or a mobile computing device 708 (e.g., a smart phone) over a network 717. By way of example, the computer system described above can be embodied in a personal computer 704, a tablet computing device 706, and / or a mobile computing device 708 (e.g., a smart phone). In addition to receiving graphics data that can be used for pre-processing at a graphics originating system or post-processing at a receiving computing system, any of these examples of computing devices can obtain content from a repository 716.

[0048] In addition, the aspects and functions described herein can be operated on a distributed system (e.g., a cloud-based computing system), where application functions, memory, data storage and retrieval, and various processing functions can be remotely operated from each other on a distributed computing network (such as the Internet or an intranet). The user interface and various types of information can be displayed via an onboard computing device display or via a remote display unit associated with one or more computing devices. For example, the user interface and various types of information can be displayed and interacted on a wall onto which the user interface and various types of information are projected. Interaction with multiple computing systems through which embodiments of the present invention can be practiced includes key input, touch screen input, voice or other audio input, gesture input in which the associated computing device is equipped with a detection (e.g., camera) function for capturing and interpreting user gestures to control the functions of the computing device, etc.

[0049] Figure 8 An exemplary method 800 for generating enhanced output data based on an original image is depicted according to examples of the present disclosure. Figure 8 800. Generally, method 800 begins at 802 and ends at 820. Method 800 may include more or fewer steps, or may be performed in a different order than described above. Figure 8 The steps shown are arranged in the order of the steps. Method 800 can be performed as a set of computer executable instructions executed by a computer system and encoded or stored on a computer readable medium. In an example, aspects of method 800 are performed by one or more processing devices such as a computer or server. In addition, method 800 can be performed by gates or circuits associated with a processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a system on a chip (SOC), a neural processing unit, or other hardware device. Hereinafter, reference will be made to the method 800 in conjunction with the method 800. Figures 1 to 7 The described systems, components, modules, software, data structures, user interfaces, etc. are used to explain method 800.

[0050] Method 800 begins at 802, where the process can proceed to 804. At 804, a base image set including a plurality of input images can be received. At 808, the base image set can be analyzed to determine or otherwise identify attributes that are not adequately represented in the base image set. For example, the attributes can be a beard, an earring, a hat, an accessory, glasses, etc. At 810, each of the input images can be provided to the data generation pipeline 122, as previously described, so that a plurality of enhanced or synthesized output images can be generated. At 812, the generated synthetic images can be evaluated relative to the corresponding or corresponding base images, so that at 816, high-quality synthetic images are provided as training data. At 818, the synthetic images can be used to train a machine learning model. Method 800 can end at 820.

[0051] Fig. 9 Depicted is an exemplary method 900 for generating enhanced output data based on an original image according to examples of the present disclosure. Fig. 9 900. Generally, method 900 begins at 902 and ends at 918. Method 900 may include more or fewer steps, or may be performed in a different order than described above. Fig. 9 The steps shown are arranged in the order of the steps. Method 900 can be performed as a set of computer executable instructions executed by a computer system and encoded or stored on a computer readable medium. In an example, aspects of method 900 are performed by one or more processing devices such as a computer or server. In addition, method 800 can be performed by gates or circuits associated with a processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a system on a chip (SOC), a neural processing unit, or other hardware device. Hereinafter, reference will be made to the method 900 in conjunction with the computer system. Figures 1 to 8 The described systems, components, modules, software, data structures, user interfaces, etc. are used to explain method 900.

[0052] Method 900 begins at 902, where the flow may proceed to 904. At 904, a base image is received. At 906, the base image may be encoded to generate a latent code representation of the base image. At 908, one or more attributes to be included or otherwise "generated" and included in the image are identified, and at 910, the one or more attributes are added to or removed from the base image by editing the latent space code representation of the image. At 912, the modified latent space code may be provided to an image generator or decoder, and a synthetic image may be generated. At 914, one or more markers of the base image may be identified and compared to one or more markers of the synthetically generated image. At 916, based on such comparison, it may be determined to retain only those synthetically generated images having correlations between one or more markers that satisfy a particular threshold. At 918, method 900 may end.

[0053] In addition, the aspects and functions described herein can be operated on a distributed system (e.g., a cloud-based computing system), where application functions, memory, data storage and retrieval, and various processing functions can be remotely operated from each other on a distributed computing network (such as the Internet or an intranet). The user interface and various types of information can be displayed via an onboard computing device display or via a remote display unit associated with one or more computing devices. For example, the user interface and various types of information can be displayed and interacted on a wall onto which the user interface and various types of information are projected. Interaction with multiple computing systems through which embodiments of the present invention can be practiced includes key input, touch screen input, voice or other audio input, gesture input in which the associated computing device is equipped with a detection (e.g., camera) function for capturing and interpreting user gestures to control the functions of the computing device, etc.

[0054] The present disclosure relates to systems and methods for generating textured images according to at least the examples provided in the following sections:

[0055] (A) According to at least one aspect of the present disclosure, a method for generating a synthetic image based on an input image is described. The method may include receiving an input image; generating a first latent code vector representation based on the input image using an encoder; receiving a latent code corresponding to a feature to be added to the input image; modifying the first latent code vector representation based on the latent code corresponding to the feature to be added; generating a synthetic image based on the modified first latent code vector representation by an image decoder; identifying one or more landmarks in a base image using a landmark detector; identifying one or more landmarks in a synthetic image using a landmark detector; determining a similarity measure between landmarks identified on the base image and landmarks identified in the synthetic image; and discarding the synthetic image based on the comparison.

[0056] (A2) According to at least one aspect of A1 above, the latent code corresponding to the feature to be added is a randomly scaled semantic vector.

[0057] (A3) According to at least one aspect of A1-A2 above, the method includes analyzing multiple input images to determine multiple under-represented features; and identifying features to be added to the input images based on the analysis.

[0058] (A4) According to at least one aspect of A1-A3 above, the feature to be added corresponds to at least one of a beard, glasses, earrings or a mustache.

[0059] (A5) According to at least one aspect of A1-A4 above, the image decoder is a StyleGAN image decoder.

[0060] (A6) According to at least one aspect of A1-A5 above, the method includes using multiple synthetic images to train a machine learning neural network model, wherein the generated synthetic image is included in the multiple synthetic images.

[0061] (A7) According to at least one aspect of A1-A3 above, the input image includes a facial image of a person.

[0062] On yet another hand, some examples include a system comprising one or more processors and a memory coupled to the one or more processors, the memory storing one or more instructions that, when executed by the one or more processors, cause the one or more processors to perform any of the methods described herein (e.g., A1-A7 above).

[0063] On yet another hand, some examples include a non-transitory computer-readable storage medium storing one or more programs for execution by one or more processors of the storage device, the one or more programs including instructions for performing any of the methods described herein (e.g., A1-A7 above).

[0064] For example, the above text describes various aspects of the present disclosure with reference to the block diagrams and / or operational descriptions of the methods, systems, and computer program products according to various aspects of the present disclosure. The functions / actions marked in the frame may not occur in the order shown in any flow chart. For example, two frames shown in succession may actually be performed substantially simultaneously, or these frames may sometimes be performed in the opposite order, depending on the functions / actions involved.

[0065] The description and illustration of one or more aspects provided in this application are not intended to limit or restrict the scope of the present disclosure in any way. The aspects, examples and details provided in this application are considered to be sufficient to convey the best mode of possession and enable others to make and use the disclosure claimed. The disclosure claimed should not be interpreted as being limited to any aspect, example or detail provided in this application. Whether combined or individually shown and described, it is intended to selectively include or omit various features (of structure and method) to produce an embodiment with a specific feature set. The description and illustration of this application have been provided, and those skilled in the art can envision the changes, modifications and alternative aspects that fall within the spirit of the broader aspects of the overall inventive concept embodied in this application, which do not depart from the broader scope of the disclosure claimed.

Claims

1. A method for generating a composite image based on an input image, the method comprising: receiving an input image; generating, using an encoder, a first latent code vector representation based on the input image; receiving a latent code corresponding to a feature to be added to the input image; modifying the first latent code vector representation based on the latent code corresponding to the feature to be added; generating, by an image decoder, a synthetic image based on the modified first latent code vector representation; identifying one or more landmarks in the base image using a landmark detector; identifying one or more landmarks in the composite image using a landmark detector; determining a similarity measure between the landmark identified on the base image and the landmark identified in the composite image; as well as The composite image is discarded based on the similarity measure.

2. The method according to claim 1, wherein the latent code corresponding to the feature to be added is a randomly scaled semantic vector.

3. The method according to claim 2, further comprising: analyzing a plurality of input images to determine a plurality of under-represented features; as well as The features to be added to the input image are identified based on the analysis.

4. The method of claim 1, wherein the feature to be added corresponds to at least one of a beard, glasses, an earring, or a moustache.

5. The method of claim 1, wherein the image decoder is a StyleGAN image decoder.

6. The method according to claim 1, further comprising: A machine learning neural network model is trained using a plurality of synthetic images, wherein the generated synthetic image is included in the plurality of synthetic images. The method of claim 1 , wherein the input image comprises a facial image of a person.

8. A system comprising: One or more hardware processors configured by machine-readable instructions to: receiving an input image; generating, using an encoder, a first latent code vector representation based on the input image; receiving a latent code corresponding to a feature to be added to the input image; modifying the first latent code vector representation based on the latent code corresponding to the feature to be added; generating, by an image decoder, a synthetic image based on the modified first latent code vector representation; identifying one or more landmarks in the base image using a landmark detector; identifying one or more landmarks in the composite image using a landmark detector; determining a similarity measure between the landmark identified on the base image and the landmark identified in the composite image; as well as The composite image is discarded based on the similarity measure.

9. The system of claim 8, wherein the latent code corresponding to the feature to be added is a randomly scaled semantic vector.

10. The system according to claim 9, further comprising: analyzing a plurality of input images to determine a plurality of under-represented features; as well as The features to be added to the input image are identified based on the analysis.

11. The system of claim 8, wherein the feature to be added corresponds to at least one of a beard, glasses, earrings, or a moustache.

12. The system of claim 8, further comprising: A machine learning neural network model is trained using a plurality of synthetic images, wherein the generated synthetic image is included in the plurality of synthetic images.

13. The system of claim 8, wherein the input image comprises a facial image of a person.

14. The system of claim 8, wherein the image decoder is a StyleGAN image decoder.

15. A non-transitory computer-readable storage medium comprising instructions executable by one or more processors to cause the one or more processors to: receiving an input image; generating, using an encoder, a first latent code vector representation based on the input image; receiving a latent code corresponding to a feature to be added to the input image; modifying the first latent code vector representation based on the latent code corresponding to the feature to be added; generating, by an image decoder, a synthetic image based on the modified first latent code vector representation; identifying one or more landmarks in the base image using a landmark detector; identifying one or more landmarks in the composite image using a landmark detector; determining a similarity measure between the landmark identified on the base image and the landmark identified in the composite image; as well as The composite image is discarded based on the similarity measure.

16. The computer-readable storage medium of claim 15, wherein the latent code corresponding to the feature to be added is a randomly scaled semantic vector.

17. The computer-readable storage medium of claim 16, further comprising: A machine learning neural network model is trained using a plurality of synthetic images, wherein the generated synthetic image is included in the plurality of synthetic images.

18. The computer-readable storage medium of claim 15, wherein the input image comprises a facial image of a person.

19. The computer-readable storage medium of claim 15, wherein the image decoder is a StyleGAN image decoder.

20. The computer-readable storage medium of claim 15, wherein the feature to be added corresponds to at least one of a beard, glasses, an earring, or a mustache.

Images