Systems and methods for improved skin tone rendering in digital images

Abstract

Embodiments of the present disclosure may include an improved skin tone rendering system for a user's skin tone in a digital image, the system including a first computing device. Embodiments may also include a computing device camera. Embodiments may also include a user skin tone analysis device. In some embodiments, the first computing device may be configured to receive a set of user skin tone images of the user. In some embodiments, the set of user skin tone images may include at least an unprocessed user skin tone image of the user acquired from the computing device camera. In some embodiments, the first computing device may be configured to acquire a skin tone analysis user skin tone image for the user. In some embodiments, the skin tone analysis user skin tone image may be captured using a user skin tone analysis device.

Description

[Technical Field]

[0001] The present invention relates to improved skin tone rendering in digital images using a skin tone analyzer attached to a computing device. [Background technology]

[0002] Computing devices (such as smartphones, tablets, and digital cameras) are often capable of taking photographs, in which accurately capturing and displaying accurate and true skin tones in images across a variety of skin colors and shades, varying lighting, and other factors is a known challenge.

[0003] Although many approaches exist for rendering more accurate skin tones, the problem remains largely unsolved, primarily due to limitations in the computing devices used for image capture and the resulting lack of adequate processing capabilities for the images.

[0004] Therefore, there is a need for an improved method and system for improved skin tone rendering in digital images. Summary of the Invention

[0005] 1. An improved skin tone rendering system for a user's skin tone in a digital image, the system including a first computing device configured to: receive a user skin tone image set for the user, the first computing device including at least an unprocessed user skin tone image of the user acquired from a computing device camera; obtain a skin tone assembly user skin tone image for the user, the skin tone assembly user skin tone image for the user, the skin tone assembly user skin tone image obtained using a user skin tone analysis device; extract user skin tone color values ​​for the user from each image in the user skin tone image set and the skin tone assembly user skin tone image for the user; calculate a set of user skin tone rendering adjustment coefficients from the skin tone color values; apply one or more user skin tone rendering coefficients from the set of user skin tone rendering adjustment coefficients to the unprocessed user skin tone image to obtain an adjusted user skin tone image; and output the adjusted user skin tone image.

[0006] The first computing device may further include a first computing device camera and a user skin tone analysis device attached to the first computing device in front of the first computing device camera, wherein the acquisition utilizes the first computing device camera together with the user skin tone analysis device in front of the computing device camera, and the skin tone assembly user skin tone image for the user is an image of the user.

[0007] The skin tone assembly user skin tone image is at 10x or greater magnification.

[0008] The system may further include a database of skin tone assembly skin tone images from a second computing device camera having a second user skin tone analysis device in front of it, wherein the acquisition is from the database of skin tone assembly skin tone images, and the skin tone assembly user skin tone image for the user is selected based on a comparison of the unprocessed user skin tone image of the user to a set of skin tone assembly user skin tone images in the database of skin tone assembly skin tone images, rather than an image of the user.

[0009] The user skin tone color value is L * Channel, a * Channel and b * It may include a channel.

[0010] The set of user skin tone images may include unprocessed skin tone images and processed human skin tone images.

[0011] The extraction includes identifying, for each image in the set of user skin tone images, a set of image pixels that includes the user's skin surface, and for each pixel in the set of image pixels, * Channel, said a * Channel and b * Sum the channels and L * Channel, said a * Channel and b * Divide the sum of the channels by the number of pixels in the image pixel set and calculate the average L * Channel, average a * Channel and average b * This may include calculating the channel.

[0012] The skin tone rendering adjustment factor set is a set of adjustment factors for the a between the skin tone assembly skin tone image and the raw skin tone image. *a first skin tone rendering adjustment factor including a first difference between channels, and a first difference between the skin tone assembly skin tone image and the raw skin tone image; * and a second skin tone rendering adjustment factor comprising a second difference between the channels.

[0013] The set of skin tone rendering adjustment factors may be a function of the L between the human processed skin tone image and the unprocessed skin tone image. * The application may further include a third skin tone rendering adjustment factor including a third difference between channels, and the application may include the first skin tone rendering adjustment factor, the second skin tone rendering adjustment factor, and the third skin tone rendering adjustment factor.

[0014] The human-processed skin tone image further includes generating the human-processed skin tone image from the unprocessed skin tone image by a human adjusting the unprocessed skin tone image using image processing software to make the user's skin tone in the human-processed skin tone image appear empirically more similar to the user's skin tone as seen by the human in the real world.

[0015] The extraction includes identifying a first set of image pixels that include the user's skin surface in the raw skin tone image and a second set of image pixels that include the user's skin surface in the skin tone assembly user skin tone image for the user, and calculating a first average L for the first set of image pixels. * channel and a second average L for the second set of image pixels. * and estimating a channel and a pixel value from the first set of image pixels and the second set of image pixels based on the estimation. * establishing a mapping of channel values ​​and using the mapping, for each pixel in the first set of image pixels, * Channel adjustment coefficient and b * Generate pixel skin tone rendering adjustment factors including the channel adjustment factors, and for each pixel,* The channel adjustment coefficient and the b * This may further include using a channel adjustment factor.

[0016] Each user skin tone image in the set of user skin tone images may include an extracted skin tone snippet of the user, and the application may be to the extracted skin tone snippet in the raw user skin tone image.

[0017] The output may include one or more of displaying the adjusted user skin tone image on a screen of the computing device or storing the adjusted user skin tone image on a memory of a computing device.

[0018] Also provided is an improved skin tone rendering method for a user's user skin tone in a digital image, the method including: receiving, by a computing device, a user skin tone image set for the user including at least an unprocessed user skin tone image of the user acquired from a computing device camera; obtaining a skin tone assembly user skin tone image for the user obtained using a user skin tone analysis device; extracting user skin tone color values ​​for the user from each image in the user skin tone image set and the skin tone assembly user skin tone image for the user; calculating a user skin tone rendering adjustment coefficient set from the skin tone color values; applying one or more user skin tone rendering coefficients from the user skin tone rendering adjustment coefficient set to the unprocessed user skin tone image to obtain an adjusted user skin tone image; and outputting the adjusted user skin tone image.

[0019] The acquisition may be via the computing device, the computing device further including a computing device camera and a user skin tone analyzer, the user skin tone analyzer being in front of the computing device camera, and the skin tone assembly user skin tone image for the user being an image of the user.

[0020] The skin tone assembly user skin tone image may be at a magnification of 10x or greater.

[0021] The acquisition may be from a database of skin tone assembly skin tone images, and the skin tone assembly user skin tone image for the user is selected based on a comparison of the unprocessed user skin tone image of the user to a set of skin tone assembly user skin tone images in the database of skin tone assembly skin tone images, rather than an image of the user.

[0022] The user skin tone color value is L * Channel, a * Channel and b * It may include a channel.

[0023] The set of user skin tone images may include unprocessed skin tone images and processed human skin tone images.

[0024] The extraction may include identifying, for each image in the set of user skin tone images, a set of image pixels that include the user's skin surface; and, for each pixel in the set of image pixels, calculating the L * Channel, said a * Channel and b * Summing the channels and L * Channel, said a * Channel and b * Divide the sum of the channels by the number of pixels in the image pixel set and calculate the average L * Channel, average a* Channel and average b * This may include calculating the channel.

[0025] The skin tone rendering adjustment factor set is a set of adjustment factors for the a between the skin tone assembly skin tone image and the raw skin tone image. * a first skin tone rendering adjustment factor including a first difference between the channels, and the b between the skin tone assembly skin tone image and the raw skin tone image; * and a second skin tone rendering adjustment factor comprising a second difference between the channels.

[0026] The set of skin tone rendering adjustment factors may be a function of the L between the human processed skin tone image and the unprocessed skin tone image. * The application may further include a third skin tone rendering adjustment factor including a third difference between channels, and the application may include the first skin tone rendering adjustment factor, the second skin tone rendering adjustment factor, and the third skin tone rendering adjustment factor.

[0027] The method further includes generating an processed skin tone image by a human using image processing software to adjust the unprocessed skin tone image to make the user's skin tone in the processed skin tone image appear empirically more similar to the user's skin tone as seen by the human in the real world.

[0028] The extraction includes identifying a first set of image pixels that include the user's skin surface in the raw skin tone image and a second set of image pixels that include the user's skin surface in the skin tone assembly user skin tone image for the user; * channel and a second average L for the second set of image pixels. * and estimating a channel from the first set of image pixels and the second set of image pixels based on the estimation.* establishing a channel value mapping; and using the mapping, for each pixel of the first set of image pixels, * Channel adjustment coefficient and b * Generate pixel skin tone rendering adjustment factors including the channel adjustment factors, and for each pixel, * The channel adjustment coefficient and the b * This may further include using a channel adjustment factor.

[0029] Each user skin tone image in the set of user skin tone images may include an extracted skin tone snippet of the user, and the application may be to the extracted skin tone snippet in the raw user skin tone image.

[0030] The output may include one or more of displaying the adjusted user skin tone image on a screen of the computing device or storing the adjusted user skin tone image on a memory of the computing device. [Brief explanation of the drawings]

[0031] [Figure 1] FIG. 1 is a block diagram illustrating a system according to some embodiments of the present disclosure. [Figure 2] FIG. 2 is a block diagram further illustrating the system of FIG. 1 in accordance with some embodiments of the present disclosure. [Figure 3] FIG. 3 is a flowchart illustrating a method according to some embodiments of the present disclosure. [Figure 4] FIG. 4 is a flowchart further illustrating the method from FIG. 3 according to some embodiments of the present disclosure. [Figure 5] FIG. 5 is a flowchart further illustrating the method from FIG. 3 according to some embodiments of the present disclosure. [Figure 6] FIG. 6 is an example of a user skin tone image following various stages of the methods described herein, according to some embodiments of the present disclosure. [Figure 7] FIG. 7 is an example of identifying skin tone image snippets according to some embodiments of the present disclosure. Detailed Description of the Invention

[0032] 1 is a block diagram illustrating a system 110 according to some embodiments of the present disclosure. In some embodiments, the system 110 may include a first computing device 112, a computing device camera 114, a user skin tone analyzer 116, and one or more images of a user 130 a stored in volatile or non-volatile memory (not shown) on the computing device 112, such as an unprocessed user skin tone image 122, a processed user skin tone image 124, a skin tone analysis user skin tone image 126, and an adjusted user skin tone image 128.

[0033] Generally, system 110, as shown in FIG. 1, illustrates an embodiment in which user 130a may take a photograph of themselves (unprocessed user skin tone image 122—an example can be seen at 602) and may also take a skin tone assembly user skin tone image 126 (taken with the skin tone assembly, which may be at ten times or more (10X+) magnification, using cross-polarized light, and under controlled lighting conditions—the skin tone assembly comprises computing device 112 and computing device 114, along with user skin tone analyzer 116—an example can be seen at 604), and then process unprocessed skin tone image 122 (e.g., using a photo app on their computing device) to make their photograph appear more accurate, generate processed skin tone image 124 (or “human-processed user skin tone image 124”), and perform the functions described herein, and arrive at adjusted user skin tone image 128 (an example can be seen at 606). An embodiment of the system 110 may use only the image of the user 130a.

[0034] The first computing device 112 may be configured to receive a set of user skin tone images 120 of the user from either the computing device camera 114 and storage on the computing device 112 (e.g., after human processing in an app on the computing device 112) or from an external source. The set of user skin tone images 120 may include at least one unprocessed user skin tone image 122 of the user obtained from the computing device camera 114 or an external camera.

[0035] The first computing device 112 may be configured to obtain a skin tone analysis user skin tone image for the user. The skin tone analysis user skin tone image may be captured using the user skin tone analyzer 116.

[0036] The user skin tone analysis device (USTAD) 116 / 216 may be hardware described in PCT / CA2020 / 050216 or PCT / CA2017 / 050503, or may include another skin tone analysis system capable of capturing images of a user's skin, where the images have sufficient characteristics for the analysis described herein. The USTAD may have an SDK running on it such that an application on a computing device can enable, control, or verify the methods described herein. In particular, as noted above, system 110 requires the ability to acquire user skin tone images that enable the processing described herein. In one embodiment, the user skin tone image, particularly the skin tone analyzer user skin tone image 126, may be captured with a 10 megapixel camera, for example, at a magnification of 10x to 30x, using cross-polarized light (e.g., to remove glare or light source reflections from the image of the skin). Image magnification and cross-polarization can be used for comparison and analysis purposes and can help overcome some of the limitations of computing device hardware that make accurate skin tone assessment and rendering difficult.

[0037] The user skin tone image is calculated by LAB (L for each pixel)* Channel, a * Channel and b * The user skin image may be in one of several color formats, such as RGB (with multiple channels) or RGB. The user skin image may be of virtually any quality, type, format, or size / file size, so long as the methods herein are applicable. For example, the image may be compressed or uncompressed, raw or processed, and in a variety of file formats.

[0038] In some embodiments, first computing device 112 may be configured to extract user skin tone color values ​​for the user from each image in user skin tone image set 120 and the user's skin tone analysis user skin tone image. First computing device 112 may be configured to calculate a set of user skin tone rendering adjustment factors from the skin tone color values. First computing device 112 may be configured to apply one or more user skin tone rendering adjustment factors from the set of user skin tone rendering adjustment factors to raw user skin tone image 122 to obtain an adjusted user skin tone image.

[0039] In some embodiments, the user skin tone analyzer 116 / 216 may be attached to a computing device (such as 112 or 212) in front of the computing device camera 114. The acquisition may be via the computing device camera 114, with the user skin tone analyzer 116 in front of the computing device camera 114. The skin tone analysis user skin tone image for the user may be an image of the user.

[0040] FIG. 2 is a block diagram further illustrating the system 110 from FIG. 1, according to some embodiments of the present disclosure.

[0041] 2, system 110 illustrates an embodiment in which user 130a may take a photo including themselves (unprocessed user skin tone image 122) and use a skin tone assembly user skin tone image 126 (taken on a computing device 212, along with a skin tone assembly-skin tone analyzer 216, and a computing device or database 214, which may or may not include them) that may include themselves in the image, and then process the unprocessed skin tone image 122 (e.g., using a photo app on their computing device) to make their photo appear more accurate (generating an processed user skin tone image, or “human-processed user skin tone image”), or proceed without human processing and use the functionality described herein that does not require such human intervention to perform the functions described herein. Embodiments of system 110 may use the image of user 130a along with an image for (but not “of”) the user to provide the functionality described herein.

[0042] In some embodiments, system 110 may include a database 214 of skin tone analysis skin tone images, a second computing device camera 215, a second user skin tone analysis device 216 in front of second computing device camera 215, and a network 220 (e.g., the Internet, one or more local or wide area networks, including wired and wireless configurations known in the art and which may include a variety of hardware and software configurations). Database 214 of skin tone analysis skin tone images may be acquired via or from second computing device camera 215 and may be for many different users (130b and others). Acquisition may be from database 214 of skin tone analysis skin tone images, and a skin tone analysis user skin tone image for a user need not be an image of the user and may be selected based on a comparison of the user's unprocessed user skin tone image 122 with database 214 of skin tone analysis skin tone images.

[0043] Database 214 may be a server that stores and processes skin tone images, such as skin tone assembly user skin tone image 126 (from one or more of users 130a, 130b, and others), as described herein. Database 214 may be any combination of web servers, application servers, and database servers known to those skilled in the art. Each such server may include a typical server configuration, including a processor, volatile and non-volatile storage devices, and software instructions executable thereon. Database 214 may communicate via an app to perform the functions described herein, including the exchange of skin images, product recommendations, and e-commerce functions. Of course, the app may perform these functions alone or in combination with database 214.

[0044] The database 214 may include a database server that receives and stores all skin tone images from all users in user profiles for each registered and guest user. These may be received from one or more USTADs 116 / 216, although the app may be configured to store skin images only locally (though this may exclude some of the outcome information based on population and attribute comparisons). The database 214 (e.g., via a database server not shown) may provide various analytical functions as described herein and provide various display functions as described herein.

[0045] 3 is a flowchart illustrating a method according to some embodiments of the present disclosure. In some embodiments, at 310, the method may include receiving, by a computing device, a set of user skin tone images of a user, where the set of user skin tone images includes at least an unprocessed user skin tone image of the user captured from a computing device camera. At 320, the method may include acquiring a skin tone analysis user skin tone image for the user, where the skin tone analysis user skin tone image is captured using the user skin tone analysis device.

[0046] In some embodiments, at 330, the method may include extracting user skin tone color values ​​for the user from each image of the set of user skin tone images and the skin tone analysis user skin tone image for the user. At 340, the method may include calculating a set of user skin tone rendering adjustment factors from the skin tone color values. At 350, the method may include applying one or more user skin tone rendering adjustment factors from the set of user skin tone rendering adjustment factors to the raw user skin tone image to obtain an adjusted user skin tone image.

[0047] In some embodiments, the acquisition may be via a computing device camera with the user skin tone analysis device in front of the computing device camera. The skin tone analysis user skin tone image for the user may be an image of the user. In some embodiments, the skin tone analysis user skin tone image may be at a magnification of 10x or more. In some embodiments, the application may be to an extracted skin tone snippet in an unprocessed user skin tone image.

[0048] In some embodiments, the acquisition may be from a database of skin tone analysis skin tone images, and the skin tone analysis user skin tone image for the user may not be an image of the user, but may be selected based on a comparison of the unprocessed user skin tone image of the user with the database of skin tone analysis skin tone images.

[0049] In some embodiments, the user skin tone color value is L * Channel, a * Channel, and b * In some embodiments, the set of user skin tone images may include an unprocessed skin tone image and a human-processed skin tone image. In some embodiments, the method includes the human using image processing software to adjust or process the unprocessed skin tone image to generate the human-processed skin tone image, such that the user skin tone in the human-processed skin tone image appears empirically more similar to the user's skin tone as seen by a human in the real world. This includes L * It may also involve a person adjusting various aspects of the raw skin tone image, including adjusting channels (such as adjusting "light" in a camera app).

[0050] In some embodiments, each user skin tone image in the set of user skin tone images may include an extracted skin tone snippet of the user. In some embodiments, the method may include outputting the results of the color processing. Outputting may include one or more of displaying the adjusted user skin tone image on a screen of a computing device or storing the adjusted user skin tone image in memory of the computing device. Of course, this may ultimately include transmitting various images and data to database 214.

[0051] 4 is a flowchart further illustrating the method, particularly the extraction, from FIG. 3, according to some embodiments of the present disclosure. In some embodiments, extracting skin tone color values ​​may include, for each image in the user skin tone image set, steps 410 through 430. At 410, extracting skin tone color values ​​may include identifying a set of image pixels that constitute the user's skin surface. At 420, the extraction may include, for each pixel in the image pixel set, calculating L * Channel, a * Channel and b * At 430, the extraction may include summing the L * Channel, a * channel, and b * Divide the sum of the channels by the number of pixels in the image pixel set to get the average L * Channel, average a * channel, and average b * This may include calculating the channel.

[0052] In some embodiments, the skin tone rendering adjustment factor set is a function of the skin tone assembly / analysis skin tone image and the raw skin tone image. * a first skin tone rendering adjustment factor that includes the first difference between the channels and between the skin tone assembly / analysis skin tone image and the raw skin tone image; * and a second skin tone rendering adjustment factor comprising a second difference between the channels.

[0053] In some embodiments, the set of skin tone rendering adjustment factors may be a linear regression coefficient between the processed and unprocessed human skin tone images. * The method may further include a third skin tone rendering adjustment factor that includes a third difference between the channels, and the application includes the first skin tone rendering adjustment factor, the second skin tone rendering adjustment factor, and the third skin tone rendering adjustment factor.

[0054] As an example, with respect to the method of FIGS. 3-4, in one embodiment there may be an unprocessed user skin tone image 122, a processed user skin tone image 124, and a skin tone analyzer user skin tone image 126. From each of these images, the method identifies the user's face or skin (assuming only one face is present; embodiments of the invention can process multiple users in each image, treating each user as an independent user and rendering them more accurately while attempting to maintain any adjustment factors so that each user matches both each other and the rest of the pixels that make up the subject and skin or face in the adjusted image). For each pixel in a given image, the channel values ​​of the LAB values ​​are summed. The sum of a particular channel is then divided by the number of pixels to get the average channel value for the skin pixels in a particular image. This becomes the channel value for that image. After extraction, the method has the average LAB values ​​for each channel for the skin pixels in each of the three images. A typical example of a set of user skin tone rendering adjustment factors is as follows: 1)a * Channel average (from image 126) - a * Channel average (from image 122) = delta(a * ), for example, 122-118=4. 2)b * Channel average (from image 126) - b * Channel average (from image 122) = delta(b *), for example, 115-111=4. At this point, the two user skin tone rendering adjustment factors are delta(a * )=4 and delta(b * )=4. 3) L * Channel Average (from image 124) - L * Channel Average (from image 122) = Delta(L * ), for example, 87-73=14. In this case, delta (L * )=14.

[0055] Here, delta (a * )=4, Delta(b * )=4, Delta(L * )=14, each pixel in the raw portrait photo is considered, and the delta (L * ), Delta (a * ), Delta (b * ) values ​​are applied to each pixel. So if the raw pixel has LAB values ​​(73,122,122), the application is (73,122,122)-delta(L * =14), Delta(a * )=4 and delta(b * )=4, and the new LAB value for the pixel in the (adjusted) image 128 is (87,126,126).

[0056] 5 is a flowchart further illustrating the method, particularly the calculation, from FIG. 3, according to some embodiments of the present disclosure. In some embodiments, the extraction can include steps 510 through 550. At 510, the extraction can include identifying a first set of image pixels that include the user's skin surface in the raw skin tone image (e.g., as shown in FIG. 7, where 704 is raw user skin tone image 122 and 702 is a set of image pixels—white—that includes the skin of user 130a) and a second set of image pixels that include the user's skin surface in a skin tone analysis user skin tone image for the user (e.g., which can be substantially every pixel based on USTAD 116). At 520, the extraction can include identifying a first average L for the first set of image pixels.* channel and the second average L for the second set of image pixels * At 530, the extraction may include estimating L from the first set of image pixels and the second set of image pixels based on the estimation (e.g., using deltas from the mean, weighting of the mean, etc.). * At 540, the extraction may include establishing a mapping of channel values. For each pixel in the first set of image pixels, the mapping is used to determine a * Channel adjustment coefficient and b * At 550, the extraction may include generating pixel skin tone rendering adjustment factors that include the channel adjustment factors. * Channel adjustment factor and b * This may include using a channel adjustment factor.

[0057] As an example, in the method of Figure 5, all pixels of the skin region are used from the raw user skin tone image. They are stored in an array, duplicates (where all LAB channels match) are removed, and then the resulting image is added to the image, e.g., L * can be sorted by L * When plotted as a histogram (vertical axis is pixel count) of L, it can result in a bell curve. The same can be done for a user skin analysis user skin tone image (which can show skin texture more clearly due to increased detail), resulting in two pixel arrays - one from the portrait photo (unprocessed user skin tone image) and one from the skin texture. This can result in similar histograms, but with different L * Only the average of the pixels from both arrays can be in different positions. From there, a formula is determined to map pixels from the portrait pixel array to the skin texture array. As explained, this is just a matter of finding the L *This may be an average delta of 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 60, 61, 62, 63, 64, 65, 66, 70, 71, * 48.5 is the L from the skin texture pixel array (image 126) * 54.5, and for every pixel in the image, a delta a * and b * This can be used to calculate the adjusted user skin tone image 128, which can provide a more accurate representation of skin texture details, details such as pores, moles, wrinkles, etc., as well as a better representation of highlights and shadows.

[0058] It is desirable to eliminate the need for either or both (I) the user's 130a's own skin tone assembly user skin tone image 126 (relying on the database 214 and the skin tone assembly user skin tone images 126 therein - selecting the one that best suits the user 130a to ensure accuracy of the methods herein) and (II) a human processed image 124. 1) Omit the user's 130a own skin tone assembly user skin tone image 126. This can be achieved, for example, by training an ML model that can generate matching skin color LAB values ​​(from the raw image 122) to results obtained by scanning the user's skin with a scanner (i.e., using the user skin tone analyzer 116 to obtain the user's actual image 126). The closest match in database 214 is then used to calculate a set of user skin tone rendering adjustment factors, one of the sets described herein. In particular, information from the computing device camera (beyond the raw user skin tone image 122), such as information about magnification and lighting, may be used to derive the best match from database 214. 2) Omitting the human processed image 124. This is done, for example, by subtracting delta (L *) can be achieved by using ML to train a model that outputs the delta (L ) between the raw user skin tone image 122 and the processed user skin tone image 124. Similarly, information from the computing device camera (used to capture the raw image 122) could be used (estimated ambient light parameters, average LAB of background and foreground pixels, skin texture and location, etc.). For example, the method could take many photos in different lighting conditions and estimate the delta (L ) between the raw user skin tone image 122 and the processed user skin tone image 124. * ), the processing engine of the computing device can be utilized (e.g., via an SDK and API).

[0059] In practice, embodiments of the present invention for correcting skin tone rendering in raw user skin tone images can be implemented either before and / or after the AI / ML solution is trained. Prior to training, at least one raw user skin tone image can be combined with at least one skin tone assembly user skin tone image for a user (where "for" means either "of" or selected for the user, e.g., from database 214), and optionally with at least one human-processed skin tone image (generally "user's"). At that point, the system has a skin tone assembly user skin tone image for the user and knows how that skin looks under a known illuminant (e.g., D65, as might be used in a skin tone analyzer). Various skin tone rendering adjustment factors—e.g., to adjust for light present in the raw user skin tone image—can be applied to the skin tone assembly user skin tone image. The skin tone rendering adjustment factors are determined as described herein and applied as described herein to adjust the skin tone of a given unprocessed user skin tone image. Of course, as described herein, when it is desired to achieve more accurate skin tone rendering of a given unprocessed user skin tone image, it is preferable to be able to determine and apply appropriate skin tone rendering adjustment factors to the unprocessed user skin tone image without relying on a human to generate the processed skin tone image or having a skin analyzer present with the user / human. In such cases, the database 214 and trained AI / ML model can be used to avoid such a need, while still providing the skin tone rendering adjustment factors necessary to arrive at the adjusted user skin tone image, as described herein.

[0060] The above-described embodiments of the present disclosure can be implemented in any of numerous ways. For example, the embodiments can be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or collection of processors, whether provided on a single computer or distributed across multiple computers.

[0061] Additionally, the various methods or processes outlined herein may be coded as software executable on one or more processors employing any one of a variety of operating systems or platforms, and such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and may be compiled as executable machine language code or intermediate code that runs on a framework or virtual machine.

[0062] In this regard, the concepts disclosed herein may be embodied as a non-transitory computer-readable medium (or media) (e.g., computer memory, one or more floppy disks, compact disks, optical disks, magnetic tapes, flash memory, circuitry in a field programmable gate array or other semiconductor device, or other non-transitory tangible computer storage medium) encoded with one or more programs that, when executed on one or more computers or other processors, perform methods that implement the various embodiments of the present disclosure described above. The computer-readable medium or media may be portable, and the program or programs stored thereon may be loaded onto one or more different computers or other processors to implement various aspects of the present disclosure, as described above.

[0063] As used herein, the terms "program," "app," "application," or "software" are used to refer to any type of computer code or collection of computer-executable instructions that may be employed to program a computer or other processor to implement various aspects of the present disclosure, as described above. Furthermore, it should be noted that, in accordance with one aspect of the present embodiment, one or more computer programs that perform the methods of the present disclosure need not reside on a single computer or processor, but may be distributed in the form of modules among multiple different computers or processors to implement various aspects of the present disclosure.

[0064] Computer-executable instructions may exist in many forms, such as program modules executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0065] Additionally, data structures may be stored on computer-readable media in any suitable form. For ease of illustration, data structures may be shown as having fields that are related via location within the data structure. Such relationships may similarly be achieved by allocating storage to the fields with locations within the computer-readable media that convey the relationship between the fields. However, any suitable mechanism may be used to establish relationships between information within fields of a data structure, including the use of pointers, tags, or other mechanisms that establish relationships between data elements.

[0066] Various features and aspects of the present disclosure may be used alone, in combination of two or more, or in various arrangements not specifically described in the previously described embodiments, and therefore, its application is not limited to the details and arrangements of construction set forth in the foregoing description or drawings. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.

[0067] Additionally, the concepts disclosed herein may be implemented as a method, an example of which is shown. The acts performed as part of the method may be reordered in any suitable manner. Thus, embodiments may comprise operations performed in an order different from that shown, and may include performing some operations simultaneously, even if shown as sequential in the illustrated embodiments.

[0068] The use of ordinal terms such as "first," "second," "third," etc. in the claims to modify claim elements does not, by itself, imply any priority, precedence, or ordering of one claim element relative to another or chronological order of method acts performed, but is merely used as a label to distinguish a claim element having a certain name from other elements having the same name (except for the use of ordinal terms).

[0069] Also, the phrases and terms used herein are for purposes of description and should not be regarded as limiting. As used herein, the use of "including," "comprising," "having," "containing," "involving," and variations thereof, is meant to include the items listed thereafter and equivalents of additional items.

[0070] Some (or different) elements below and / or in the claims may be described as being "coupled," "in communication," or "configured to communicate." This term is intended to be non-limiting and is to be interpreted to include, without limitation, wired and wireless communications, where appropriate, using any one or more suitable protocols, as well as communication methods that are constantly maintained, periodically performed, and / or performed or initiated on an as-needed basis.

[0071] Embodiments may also be implemented in a cloud computing environment. For purposes of this specification and the following claims, "cloud computing" may be defined as a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications and services) that can be rapidly provisioned through virtualization, released with minimal management effort or service provider interaction, and then scaled accordingly. The cloud model may be comprised of a variety of characteristics (e.g., on-demand self-service, pervasive network access, resource pooling, rapid elasticity, metered service), service models (e.g., Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS)), and deployment models (e.g., private cloud, community cloud, public cloud, hybrid cloud).

[0072] This description uses examples to disclose the invention and will enable any person skilled in the art to make and use the invention. The patentable scope of the invention is defined by the claims, and may include other examples that occur to those skilled in the art. Such other examples are within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements that do not differ substantially from the literal language of the claims.

[0073] It should be understood that the above-described assemblies and modules can be interconnected as needed to perform the desired functions and tasks, to the extent that such combinations and permutations are possible by one skilled in the art without explicitly describing each one. No particular assembly or component is superior to any of the equivalents available to those skilled in the art. No particular mode of performing the disclosed subject matter is superior to other modes, so long as the function can be performed. All important aspects of the disclosed subject matter are believed to have been provided herein. The scope of the present invention is understood to be limited to the scope provided by the independent claims, and is not limited to: (i) the dependent claims, (ii) the detailed description of non-limiting embodiments, (iii) the summary, (iv) the abstract, and / or (v) any description provided outside this specification (i.e., outside this application as filed, prosecuted, and / or patented). In this specification, the term "includes" is understood to be equivalent to the term "comprising." The above outlines non-limiting embodiments (examples). The non-limiting embodiments should be understood to be merely illustrative by way of example.

Claims

1. 1. An improved skin tone rendering system for a user's skin tone in a digital image, comprising: The system includes a first computing device; The first computing device receiving a set of user skin tone images of the user, the set including at least an unprocessed user skin tone image of the user obtained from a computing device camera; obtaining a skin tone assembly user skin tone image for said user, said user skin tone image obtained using a user skin tone analysis device; extracting a user skin tone color value for the user from each image in the user skin tone image set and the skin tone assembly user skin tone image for the user; calculating a set of user skin tone rendering adjustment factors from the skin tone color values; applying one or more user skin tone rendering factors from the set of user skin tone rendering adjustment factors to the raw user skin tone image to obtain an adjusted user skin tone image; and and configured to output the adjusted user skin tone image.

2. 2. The system of claim 1, wherein the first computing device further includes a first computing device camera and a user skin tone analysis device attached to the first computing device in front of the first computing device camera, wherein the acquisition utilizes the first computing device camera together with the user skin tone analysis device in front of the computing device camera, and wherein the skin tone assembly user skin tone image for the user is an image of the user.

3. 3. The system of claim 2, wherein the skin tone assembly user skin tone image is at a magnification of 10x or greater.

4. 10. The system of claim 1, further comprising a database of skin tone assembly skin tone images from a second computing device camera having a second user skin tone analysis device in front of it, wherein said obtaining is from said database of skin tone assembly skin tone images, and wherein said skin tone assembly user skin tone image for said user is selected based on a comparison of said unprocessed user skin tone image of said user with a set of skin tone assembly user skin tone images in said database of skin tone assembly skin tone images, rather than an image of said user.

5. 2. The system of claim 1, wherein the user skin tone color value is L ＊ Channel, a ＊ Channel and b ＊ Includes channels.

6. 6. The system of claim 5, wherein the set of user skin tone images includes unprocessed skin tone images and human processed skin tone images.

7. 7. The system of claim 6, wherein the extraction comprises, for each image in the set of user skin tone images: identifying a set of image pixels that includes a skin surface of the user; For each pixel in the set of image pixels, ＊ channel, said a ＊ channel and said b ＊ Sum the channels, and L ＊ channel, said a ＊ channel and said b ＊ Divide the sum of the channels by the number of pixels in the image pixel set to find the average L ＊ Channel, average a ＊ Channel and average b ＊ This includes calculating the channel.

8. 8. The system of claim 7, wherein the set of skin tone rendering adjustment factors is a function of the a between the skin tone assembly skin tone image and the raw skin tone image. ＊ a first skin tone rendering adjustment factor including a first difference between channels, and a first difference between the skin tone assembly skin tone image and the raw skin tone image; ＊ and a second skin tone rendering adjustment factor comprising a second difference between the channels.

9. 9. The system of claim 8, wherein the set of skin tone rendering adjustment factors is a function of the L between the human processed skin tone image and the unprocessed skin tone image. ＊ Further comprising a third skin tone rendering adjustment factor comprising a third difference between channels, wherein the application comprises the first skin tone rendering adjustment factor, the second skin tone rendering adjustment factor, and the third skin tone rendering adjustment factor.

10. 7. The system of claim 6, wherein the human-processed skin tone image is generated from the unprocessed skin tone image by a human adjusting the unprocessed skin tone image using image processing software, thereby making the user's skin tone in the human-processed skin tone image appear empirically more similar to the user's skin tone as seen by the human in the real world.

11. 6. The system of claim 5, wherein the extraction comprises: identifying a first set of image pixels comprising the user's skin surface in the raw skin tone image and a second set of image pixels comprising the user's skin surface in the skin tone assembly user skin tone image for the user; a first average L for the first set of image pixels; ＊ a second average L for the second set of image pixels; ＊ Estimate the channel and Based on the estimation, L ＊ Set the mapping of channel values, Using the mapping, for each pixel in the first set of image pixels, ＊ Channel adjustment coefficient and b ＊ generating pixel skin tone rendering adjustment factors including the channel adjustment factors; For each pixel, ＊ The channel adjustment coefficient and the b ＊ Further comprising using a channel adjustment factor.

12. The system of claim 1 , wherein each user skin tone image in the set of user skin tone images comprises an extracted skin tone snippet of the user.

13. 13. The system of claim 12, wherein the applying is to the extracted skin tone snippet in the raw user skin tone image.

14. 10. The system of claim 1, Outputting includes one or more of displaying the adjusted user skin tone image on a screen of the computing device or storing the adjusted user skin tone image on a memory of the computing device.

15. 1. A method for improved skin tone rendering in a digital image, for a user's skin tone, comprising: The method comprises: receiving, by a computing device, a set of user skin tone images of the user, the set including at least an unprocessed user skin tone image of the user obtained from a computing device camera; obtaining a skin tone assembly user skin tone image for said user, said skin tone image obtained using a user skin tone analysis device; extracting user skin tone color values ​​for the user from each image in the user skin tone image set and the skin tone assembly user skin tone image of the user; calculating a set of user skin tone rendering adjustment factors from the skin tone color values; applying one or more user skin tone rendering factors from the set of user skin tone rendering adjustment factors to the raw user skin tone image to obtain an adjusted user skin tone image; and outputting the adjusted user skin tone image.

16. 16. The method of claim 15, wherein the acquiring is via the computing device, the computing device further comprising a computing device camera and a user skin tone analyzer, the user skin tone analyzer being in front of the computing device camera, and the skin tone assembly user skin tone image for the user being an image of the user.

17. 17. The method of claim 16, wherein the skin tone assembly user skin tone image is at a magnification of 10x or greater.

18. 16. The method of claim 15, wherein the obtaining is from a database of skin tone assembly skin tone images, and the skin tone assembly user skin tone image for the user is selected based on a comparison of the unprocessed user skin tone image of the user to a set of skin tone assembly user skin tone images in the database of skin tone assembly skin tone images, rather than an image of the user.

19. 16. The method of claim 15, wherein the user skin tone color value is L ＊ Channel, a ＊ Channel and b ＊ Includes channels.

20. 20. The method of claim 19, wherein the set of user skin tone images includes unprocessed skin tone images and human processed skin tone images.

21. 21. The method of claim 20, wherein the extraction comprises, for each image in the set of user skin tone images: identifying a set of image pixels comprising a skin surface of the user; For each pixel in the set of image pixels, ＊ channel, said a ＊ channel and said b ＊ summing channels, and L ＊ channel, said a ＊ channel and said b ＊ Divide the sum of the channels by the number of pixels in the image pixel set to find the average L ＊ Channel, average a ＊ Channel and average b ＊ This includes calculating the channel.

22. 22. The method of claim 21, wherein the set of skin tone rendering adjustment factors is a function of the a between the skin tone assembly skin tone image and the raw skin tone image. ＊ a first skin tone rendering adjustment factor comprising a first difference between the channels; and a first difference between the b skin tone assembly skin tone image and the raw skin tone image. ＊ and a second skin tone rendering adjustment factor comprising a second difference between the channels.

23. 23. The method of claim 22, wherein the set of skin tone rendering adjustment factors is a function of the L between the processed and unprocessed human skin tone images. ＊ Further comprising a third skin tone rendering adjustment factor comprising a third difference between channels, wherein the application comprises the first skin tone rendering adjustment factor, the second skin tone rendering adjustment factor, and the third skin tone rendering adjustment factor.

24. 21. The method of claim 20, further comprising generating an processed skin tone image by a human using image processing software to adjust the unprocessed skin tone image to make the user's skin tone in the processed skin tone image appear empirically more similar to the user's skin tone as seen by the human in the real world.

25. 20. The method of claim 19, wherein the extraction comprises: identifying a first set of image pixels comprising a skin surface of the user in the raw skin tone image and a second set of image pixels comprising a skin surface of the user in the skin tone assembly user skin tone image for the user; a first average L for the first set of image pixels; ＊ a second average L for the second set of image pixels; ＊ estimating the channel; Based on the estimation, L ＊ setting a channel value mapping; Using the mapping, for each pixel in the first set of image pixels, ＊ Channel adjustment coefficient and b ＊ generating pixel skin tone rendering adjustment factors including the channel adjustment factors; For each pixel, ＊ The channel adjustment coefficient and the b ＊ Further comprising using a channel adjustment factor.

26. The method of claim 15 , wherein each user skin tone image in the set of user skin tone images comprises an extracted skin tone snippet of the user.

27. 17. The method of claim 16, wherein the applying is to the extracted skin tone snippet in the raw user skin tone image.

28. 16. The method of claim 15, wherein the outputting comprises one or more of displaying the adjusted user skin tone image on a screen of the computing device or storing the adjusted user skin tone image on a memory of the computing device.

Images