System and method for improved skin ton rendering in digital images

Abstract

Embodiments of the present disclosure may include a system for improved skin tone rendering of a user's skin tone of a user in a digital image, the system including a first computing device. Embodiments may also include a computing device camera. Embodiments may also include a user skin hue analysis device. In some embodiments, a first computing device may be configured to receive a set of user skin tone images of a user. In some embodiments, the set of user skin tone images may include at least one unprocessed user skin tone image of the user obtained from a computing device camera. In some embodiments, a first computing device may be configured to obtain a skin tone analysis user skin tone image for a user. In some embodiments, a skin tone analysis user skin tone image may be captured using a user skin tone analysis device.

Description

Technical Field

[0001] The present invention relates to improved skin tone rendering in digital images using a skin tone analysis device attached to a computing device. Background Art

[0002] Computing devices (smartphones, tablets, digital cameras, etc.) can generally take pictures. For these pictures, accurately capturing and displaying accurate and realistic skin tones in the image (across various colors and shades of skin and various lighting and other factors) is a known challenge.

[0003] Despite many methods for rendering more accurate skin tones, the challenge remains largely unresolved - mainly due to the limitations of the computing devices used to capture the images and the resulting lack of ability to correctly process the images.

[0004] Therefore, there is a need for an improved method and system for improved skin tone rendering in digital images. Summary of the Invention

[0005] There is a system for improved skin tone rendering of a user's skin tone in a digital image, the system comprising: a first computing device configured to: receive a set of user skin tone images of a user, the set at least including an unprocessed user skin tone image of the user obtained from a computing device camera; obtain a user skin tone assembly user skin tone image for the user, the user skin tone assembly user skin tone image being taken using a user skin tone analysis device; extract user skin tone color values of the user from each image in the set of user skin tone images and the user skin tone assembly user skin tone image of the user; compute a set of user skin tone rendering adjustment factors based on the skin tone color values; apply one or more user skin tone rendering adjustment factors from the set of user skin tone rendering adjustment factors to the unprocessed user skin tone image to obtain an adjusted user skin tone image; and output the adjusted user skin tone image.

[0006] The first computing device may further include a first computing device camera and a user skin tone analysis device attached to the computing device in front of the first computing device camera, and wherein the obtaining is performed via the first computing device camera using the user skin tone analysis device in front of the computing device camera, and wherein the user skin tone analysis user skin tone image for the user is an image of the user.

[0007] The magnification of the user skin tone assembly user skin tone image is not less than 10 times.

[0008] The system may also include a database of skin tone component skin tone images from a camera of a second computing device, having a second user skin tone analysis device in front of the camera of the second computing device, and wherein the obtaining is performed from the database of skin tone component skin tone images, and the skin tone component user skin tone image for the user is not an image of the user and is selected based on comparing an unprocessed user skin tone image of the user with a set of skin tone component user skin tone images in the database of skin tone component skin tone images.

[0009] User skin tone color values may include an L* channel, an a* channel, and a b* channel.

[0010] The set of user skin tone images may include unprocessed skin tone images and human-processed skin tone images.

[0011] For each image in the set of user skin tone images, the extraction may further include: identifying a set of image pixels including the user's skin surface; for each pixel in the set of image pixels, summing the L* channel, the a* channel, and the b* channel; and for the L* channel, the a* channel, and the b* channel, dividing the sum by the number of pixels in the set of image pixels to obtain an average L* channel, an average a* channel, and an average b* channel.

[0012] The set of skin tone rendering adjustment factors may include a first skin tone rendering adjustment factor and a second skin tone rendering adjustment factor, the first skin tone rendering adjustment factor including a first difference between the a* channels between the skin tone component skin tone image and the unprocessed skin tone image, and the second skin tone rendering adjustment factor including a second difference between the b* channels between the skin tone component skin tone image and the unprocessed skin tone image.

[0013] The set of skin tone rendering adjustment factors may further include a third skin tone rendering adjustment factor, the third skin tone rendering adjustment factor including a third difference between the L* channels between the human-processed skin tone image and the unprocessed skin tone image, and the application includes the first skin tone rendering adjustment factor, the second skin tone rendering adjustment factor, and the third skin tone rendering adjustment factor.

[0014] The human-processed skin tone image may be created from the unprocessed skin tone image by a human using image processing software to adjust the unprocessed skin tone image such that the user's skin tone in the human-processed skin tone image empirically appears more similar to how a human would see the user's skin tone in real life.

[0015] The extraction may further include: identifying a first set of image pixels including the user's skin surface in the unprocessed skin tone image, and identifying a second set of image pixels including the user's skin surface in the user skin tone assembly user skin tone image for the user; deriving a first mean L* channel of the first set of image pixels and a second mean L* channel of the second set of image pixels; based on the derivation, setting a mapping of L* channel values from the first set of image pixels and the second set of image pixels; using the mapping to create a pixel skin tone rendering adjustment factor for each pixel in the first set of image pixels, the pixel skin tone rendering adjustment factor including an a* channel adjustment factor and a b* channel adjustment factor; and applying the a* channel adjustment factor and the b* channel adjustment factor for each pixel.

[0016] Each user skin tone image in the set of user skin tone images may include an extracted skin tone snippet of the user, and the application may be for the extracted skin tone snippet in the unprocessed user skin tone image.

[0017] The output may include one or more of the following: displaying the adjusted user skin tone image on the screen of the computing device or storing the adjusted user skin tone image in the memory of the computing device.

[0018] There is also a method for improved skin tone rendering of a user's skin tone in a digital image, the method including: receiving, by a computing device, a set of user skin tone images of the user, the set including at least an unprocessed user skin tone image of the user obtained from a computing device camera; obtaining a user skin tone assembly user skin tone image for the user, the user skin tone assembly user skin tone image being captured using a user skin tone analysis device; extracting user skin tone color values of the user from each image in the set of user skin tone images and the user skin tone assembly user skin tone image of the user; calculating a set of user skin tone rendering adjustment factors based on the skin tone color values; applying one or more user skin tone rendering adjustment factors from the set of user skin tone rendering adjustment factors to the unprocessed user skin tone image to obtain an adjusted user skin tone image; outputting the adjusted user skin tone image.

[0019] The obtaining may be performed via the computing device, the computing device further including a computing device camera and a user skin tone analysis device, wherein the user skin tone analysis device is in front of the computing device camera, and wherein the user skin tone assembly user skin tone image for the user is an image of the user.

[0020] The magnification of the user skin tone assembly user skin tone image may be not less than 10 times.

[0021] This acquisition can be performed from a database of skin tone assembly skin tone images, and the user skin tone image for the skin tone assembly of the user is not an image of the user and is selected based on comparing the unprocessed user skin tone image of the user with a set of skin tone assembly user skin tone images in the database of skin tone assembly skin tone images.

[0022] User skin tone color values can include an L* channel, an a* channel, and a b* channel.

[0023] The set of user skin tone images can include unprocessed skin tone images and human-processed skin tone images.

[0024] This extraction can also include, for each image in the set of user skin tone images: identifying a set of image pixels including the user's skin surface; for each pixel in the set of image pixels, summing the L* channel, the a* channel, and the b* channel; and for the L* channel, the a* channel, and the b* channel, dividing the sum by the number of pixels in the set of image pixels to obtain an average L* channel, an average a* channel, and an average b* channel.

[0025] The set of skin tone rendering adjustment factors can include a first skin tone rendering adjustment factor and a second skin tone rendering adjustment factor. The first skin tone rendering adjustment factor includes a first difference between the a* channels between the skin tone assembly skin tone image and the unprocessed skin tone image, and the second skin tone rendering adjustment factor includes a second difference between the b* channels between the skin tone assembly skin tone image and the unprocessed skin tone image.

[0026] The set of skin tone rendering adjustment factors can also include a third skin tone rendering adjustment factor. The third skin tone rendering adjustment factor includes a third difference between the L* channels between the human-processed skin tone image and the unprocessed skin tone image, and the application includes the first skin tone rendering adjustment factor, the second skin tone rendering adjustment factor, and the third skin tone rendering adjustment factor.

[0027] The method can also include creating a human-processed skin tone image by a human using image processing software to adjust the unprocessed skin tone image so that the user's skin tone in the human-processed skin tone image empirically appears more similar to how a human sees the user's skin tone in real life.

[0028] The extraction may further include: identifying a first set of image pixels including a user's skin surface in an unprocessed skin tone image, and identifying a second set of image pixels including the user's skin surface in a user skin tone image for the user's skin tone assembly; deriving a first mean L* channel of the first set of image pixels and a second mean L* channel of the second set of image pixels; based on the derivation, setting a mapping of L* channel values from the first set of image pixels and the second set of image pixels; using the mapping to create a pixel skin tone rendering adjustment factor for each pixel in the first set of image pixels, the pixel skin tone rendering adjustment factor including an a* channel adjustment factor and a b* channel adjustment factor; and adopting the a* channel adjustment factor and the b* channel adjustment factor for each pixel.

[0029] Each user skin tone image in the set of user skin tone images may include an extracted skin tone segment of the user, and the application is for the extracted skin tone segment in the unprocessed user skin tone image.

[0030] The output may include one or more of the following: displaying an adjusted user skin tone image on a screen of a computing device or storing the adjusted user skin tone image in a memory of the computing device. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a block diagram showing a system according to some embodiments of the present disclosure.

[0032] Figure 2 is a further block diagram showing a system from Figure 1 of the present disclosure.

[0033] Figure 3 is a flowchart showing a method according to some embodiments of the present disclosure.

[0034] Figure 4 is a further flowchart showing a method from Figure 3 of the present disclosure.

[0035] Figure 5 is a further flowchart showing a method from Figure 3 of the present disclosure.

[0036] Figure 6 is an example of a user skin tone image through various stages of the method according to some embodiments of the present disclosure.

[0037] Figure 7 is an example of identifying a skin tone image segment according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0038] Figure 1 It is a block diagram depicting system 110 according to some embodiments of the present disclosure. In some embodiments, system 110 may include a first computing device 112, a computing device camera 114, a user skin tone analysis device 116, and one or more images of user 130a (such as (one or more) unprocessed user skin tone images 122, (one or more) processed user skin tone images 124, skin tone analysis user skin tone images 126, and (one or more) adjusted user skin tone images 128) stored in volatile or non-volatile memory (not shown) on computing device 112.

[0039] Broadly speaking, as Figure 1 shown, system 110 illustrates an embodiment in which user 130a can take a photo including themselves (unprocessed user skin tone image 122 - an example of which can be seen at 602), and can also take a skin tone assembly user skin tone image 126 (taken using the skin tone assembly, and having a magnification of 10x+ and using cross-polarized light and under controlled lighting conditions - computing device 112 and computing device 114 together with user skin tone analysis device 116 - an example of which can be seen at 604), and then process the unprocessed skin tone image 122 (e.g., using the photo app on their computing device) to make their photo look more accurate, and create a processed skin tone image 124 (or "human-processed user skin tone image 124") to allow the functions described herein to be performed, and obtain an adjusted user skin tone image 128 ( - an example of which can be seen at 606). Embodiments of system 110 may use only the images of user 130a.

[0040] The first computing device 112 may be configured to receive a set of user skin tone images 120 of the user from the computing device camera 114 and storage on computing device 112 (such as after human processing via an app on computing device 112) or from an external source. The set of user skin tone images 120 may include at least one unprocessed user skin tone image 122 of the user obtained from the computing device camera 114 or an external camera.

[0041] The first computing device 112 may be configured to obtain a skin tone analysis user skin tone image for the user. The skin tone analysis user skin tone image may be taken using the user skin tone analysis device 116.

[0042] The User Skin Tone Analysis Device (USTAD) 116 / 216 can be hardware as described in PCT / CA2020 / 050216 or PCT / CA2017 / 050503, or can include another skin tone analysis system capable of taking images of the user's skin, which images have characteristics sufficient for the analysis described herein. The USTAD can have an SDK running thereon, thereby allowing applications on a computing device to enable, control, or view the methods described herein. Notably, and as mentioned, the system 110 requires the ability to obtain user skin tone images that permit the processing described herein. In one embodiment, the user skin tone image and in particular the user skin tone image 126 of the skin tone analysis device of the user can be taken using cross-polarized light (e.g., to eliminate glare or reflections of light sources from the skin image) (e.g., using a 10 megapixel camera with a magnification of not less than 10 times and up to 30 times). The magnification and cross-polarized light in the images that can be used for comparison and analysis purposes can help overcome some of the hardware limitations of the computing device that make accurate skin tone assessment and rendering difficult.

[0043] The user skin image can be one of several color formats, such as LAB (having an L* channel, an a* channel, and a b* channel for each pixel) or RGB. The user skin image can be essentially of any quality, type, format, or size / file size, as long as the methods herein can be applied. For example, the image can be compressed or uncompressed, raw or processed, and in various file formats.

[0044] In some embodiments, the first computing device 112 can be configured to extract the user's user skin tone color values from each image in the set of user skin tone images 120 and the user's skin tone analysis user skin tone image of the user. The first computing device 112 can be configured to compute a set of user skin tone rendering adjustment factors based on the skin tone color values. The first computing device 112 can be configured to apply one or more user skin tone rendering adjustment factors from the set of user skin tone rendering adjustment factors to the unprocessed user skin tone image 122 to obtain an adjusted user skin tone image.

[0045] In some embodiments, the user skin tone analysis device 116 / 216 can be attached in front of a computing device camera 114 to a computing device (such as 112 or 212). The obtaining can be via the computing device camera 114 using the user skin tone analysis device 116 in front of the computing device camera 114. The user skin tone analysis user skin tone image for the user can be an image of the user.

[0046] Figure 2 is further described from some embodiments according to the present disclosure Figure 1Block diagram of system 110.

[0047] Broadly speaking, as Figure 2 shown, system 110 illustrates an embodiment in which user 130a can take a photo that includes themselves (unprocessed user skin tone image 122), and can also use a skin tone assembly to generate a user skin tone image 126, which may or may not include themselves in the image (captured using a skin tone assembly - computing device 212, and computing device or database 214 together with user skin tone analysis device 216, where their computing device 112 may not require or have USTAD 116), and then process the unprocessed skin tone image 122 (e.g., using a photo app on their computing device) to make their photo look more accurate (generating a processed user skin tone image, or "human-processed user skin tone image"), or may continue without any human processing and use the functions described herein that do not require such human intervention to allow the execution of the functions described herein. Embodiments of system 110 can use an image of user 130a together with images for the user (but not "of the user") to provide the functions described herein.

[0048] In some embodiments, system 110 can include a database 214 for skin tone analysis of skin tone images, a second computing device camera 215, and a second user skin tone analysis device 216 in front of the second computing device camera 215 and a network 220 (such as the Internet, one or more local area networks or wide area networks, and which can have wired and wireless components and can include various hardware and software components known in the art). The database 214 for skin tone analysis of skin tone images can be obtained via or from the second computing device camera 215 and can be of many different users (130b and other users). This obtaining can be performed from the database 214 for skin tone analysis of skin tone images, and the skin tone analysis user skin tone image for the user may not be an image of the user and can be selected based on comparing the unprocessed user skin tone image 122 of the user with the database 214 for skin tone analysis of skin tone images.

[0049] The database 214 can be a server that stores and processes skin tone images, such as the user skin tone images 126 of the skin tone assembly (from one or more users 130a, 130b, and other users), as described herein. The database 214 can be any combination of a web server, an application server, and a database server, as known to those skilled in the art. Each such server can include typical server components, which include a processor, volatile and non-volatile memory storage devices, and software instructions executable thereon. The database 214 can communicate via the app to perform the functions described herein (including exchanging skin images, product recommendations, e-commerce capabilities, etc.). Of course, the app can also perform these functions alone or in combination with the database 214.

[0050] The database 214 can include a database server that receives all skin tone images from all users and stores them in user profiles for each registered user and guest user. These can be received from one or more USTADs 116 / 216, although the app can be configured to store skin images only locally (although this may exclude some result information based on population and demographic comparisons). The database 214 (e.g., via a database server, not shown) can provide various analysis functions as described herein and can provide various display functions as described herein.

[0051] Figure 3 is a flowchart depicting a method according to some embodiments of the present disclosure. In some embodiments, at 310, the method can include receiving, by a computing device, a set of user skin tone images of a user (including at least an unprocessed user skin tone image of the user obtained from a computing device camera). At 320, the method can include obtaining a skin tone analysis user skin tone image for the user, the skin tone analysis user skin tone image being captured using a user skin tone analysis device.

[0052] In some embodiments, at 330, the method can include extracting user skin tone color values of the user from each image in the set of user skin tone images and the skin tone analysis user skin tone image of the user. At 340, the method can include computing a set of user skin tone rendering adjustment factors based on the skin tone color values. At 350, the method can include applying one or more user skin tone rendering adjustment factors from the set of user skin tone rendering adjustment factors to the unprocessed user skin tone image to obtain an adjusted user skin tone image.

[0053] In some embodiments, the obtaining can be performed via a computing device camera using a user skin tone analysis device in front of the computing device camera. The user skin tone image for user skin tone analysis can be an image of the user. In some embodiments, the magnification of the user skin tone image for skin tone analysis can be not less than 10 times. In some embodiments, the application can be directed to the extracted skin tone segments in the unprocessed user skin tone image.

[0054] In some embodiments, the obtaining can be performed from a database of skin tone analysis skin tone images, and the user skin tone image for user skin tone analysis can not be an image of the user and can be selected based on comparing the unprocessed user skin tone image of the user with the database of skin tone analysis skin tone images.

[0055] In some embodiments, the user skin tone color values can include an L* channel, an a* channel, and a b* channel. In some embodiments, the set of user skin tone images can include unprocessed skin tone images and human-processed skin tone images. In some embodiments, the method can include creating a human-processed skin tone image by a human using image processing software to adjust or process the unprocessed skin tone image such that the user skin tone in the human-processed skin tone image empirically appears more similar to how a human sees the user skin tone in real life. This can involve a human adjusting various aspects of the unprocessed skin tone image (including adjusting the L* channel (e.g., by adjusting "Light" in the camera app)).

[0056] In some embodiments, each user skin tone image in the set of user skin tone images can include the extracted skin tone segments of the user. In some embodiments, the method can include outputting the result of color processing. The output can include one or more of the following: displaying the adjusted user skin tone image on the screen of the computing device or storing the adjusted user skin tone image in the memory of the computing device. Of course, this can ultimately also include sending various images and data to database 214.

[0057] Figure 4 is further described from some embodiments according to the present disclosure Figure 3Flowchart of a method (and in particular extraction). In some embodiments, for each image in the set of user skin tone images, extracting the skin tone color value may include 410 to 430. At 410, extracting the skin tone color value may include identifying a set of image pixels that includes the user's skin surface. At 420, the extraction may include, for each pixel in the set of image pixels, summing the L* channel, a* channel, and b* channel. At 430, the extraction may include, for the L* channel, a* channel, and b* channel, dividing the sum by the number of pixels in the set of image pixels to obtain an average L* channel, average a* channel, and average b* channel.

[0058] In some embodiments, the set of skin tone rendering adjustment factors may include a first skin tone rendering adjustment factor and a second skin tone rendering adjustment factor. The first skin tone rendering adjustment factor includes a first difference between the a* channels between the skin tone assembly / analyzed skin tone image and the unprocessed skin tone image. The second skin tone rendering adjustment factor includes a second difference between the b* channels between the skin tone assembly / analyzed skin tone image and the unprocessed skin tone image.

[0059] In some embodiments, the set of skin tone rendering adjustment factors may further include a third skin tone rendering adjustment factor. The third skin tone rendering adjustment factor includes a third difference between the L* channels between the human-processed skin tone image and the unprocessed skin tone image, and the application includes the first skin tone rendering adjustment factor, the second skin tone rendering adjustment factor, and the third skin tone rendering adjustment factor.

[0060] For example, for Figure 3 - Figure 4 the method in, in one embodiment, there may be an unprocessed user skin tone image 122, a processed user skin tone image 124, and a skin tone analysis device user skin tone image 126. These methods identify the user's face or skin from each of these images (assuming only one face exists. Note that embodiments of the present invention can handle multiple users in each photo, treating each user as a separate user for more accurate rendering while attempting to maintain any adjustment factors such that each user looks matched to each other and to the rest of the subject of the adjusted image and the pixels that make up the skin or face. For each of these pixels in a given image, the LAB value channel values are added. Then the total for a particular channel is divided by the number of pixels to obtain the average channel value for the skin pixels in a particular image. This becomes the channel value for that image. After extraction, then, for the skin pixels in each of the three images, these methods have an average LAB value in each channel. Then, an exemplary set of user skin tone rendering adjustment factors may be:

[0061] 1) a * ChannelAvg(from Image 126) - a * ChannelAvg(from Image 122) = Delta(a*), for example, it can be 122 – 118 = 4.

[0062] 2) b * ChannelAvg(from Image 126) - b * ChannelAvg(from Image 122) = Delta(b*), for example, it can be 115 – 111 = 4. At this time, the two user skin tone rendering adjustment factors will be Delta(a*) = 4 and Delta(b*) = 4.

[0063] 3) L * ChannelAvg(from Image 124) - L * ChannelAvg(from Image 122) = Delta(L*), for example, it can be 87 – 73 = 14. This will provide Delta(L*) = 14.

[0064] Now, with Delta(a*) = 4, and Delta(b*) = 4, and Delta(L*) = 14, consider each pixel in the unprocessed portrait photo and apply each of the Delta(L*), Delta(a*), and Delta(b*) values to each pixel. Thus, if the LAB value of an unprocessed pixel is (73, 122, 122), the application will be (73, 122, 122) - Delta(L* = 14), Delta(a*) = 4, and Delta(b*) = 4, such that the new LAB value of the pixel in (now adjusted) Image 128 will be (87, 126, 126).

[0065] Figure 5 is a flowchart further depicting the method (and in particular the operations) from Figure 3 In some embodiments, the extraction can include 510 to 550. At 510, the extraction can include identifying a first set of image pixels in the unprocessed skin tone image that includes the user's skin surface (e.g., as Figure 7As shown, where 704 is the unprocessed user skin tone image 122, and 702 is the set of image pixels including the white of the user 130a's skin) and identify a second set of image pixels including the skin surface of the user in the user skin tone image for user skin tone analysis (which can be, for example, substantially each pixel based on USTAD 116). At 520, the extraction can include deriving a first mean L* channel of the first set of image pixels and a second mean L* channel of the second set of image pixels. At 530, the extraction can include setting a mapping of L* channel values from the first set of image pixels and the second set of image pixels based on inference (such as using a delta from the mean, weighted mean, etc.). At 540, the extraction can include creating a pixel skin tone rendering adjustment factor for each pixel in the first set of image pixels using the mapping, the pixel skin tone rendering adjustment factor including an a* channel adjustment factor and a b* channel adjustment factor. At 550, the extraction can include adopting the a* channel adjustment factor and the b* channel adjustment factor for each pixel.

[0066] For example, for Figure 5 the method in, all pixels from the skin area can be used from the unprocessed user skin tone image. These can be placed in an array, and duplicates (where all LAB channels match) can be removed, and then sorted by L*, for example. If plotted as a histogram of L* (pixel count on the vertical axis), this may result in a bell curve. The same can be done for the user skin analysis user skin tone image (which can more clearly show skin texture via increased detail), resulting in two pixel arrays - one from the portrait photo (unprocessed user skin tone image), and one from the skin texture. This may result in histograms that look similar, only the L* means may be in different locations. From there, a formula will be determined to map pixels from the portrait photo pixel array to the skin texture array. As described above, this may just be the delta of the L* means from the two arrays, or may be one or more different methods. However, the result is a formula where you can say for each pixel: 48.5 of L* from the portrait photo (image 122) pixel array maps to 54.5 of L* from the skin texture pixel array (image 126), and then you can use it to operate on the delta a*'s and b*'s for each pixel in the image. This can provide more skin texture details and a more accurate detailed representation (such as pores, moles, lines, etc.) in the adjusted user skin tone image 128, and can also better show highlights and shadows.

[0067] It may be desirable to omit the need for one or both of the following: (I) the user 130a's own skin tone assembly user skin tone image 126 (which relies on the database 214 and the skin tone assembly user skin tone image 126 therein - to select the best match for the user 130a to render the methods herein accurately) and (II) the human-processed image 124.

[0068] 1) Omit the user 130a's own skin tone assembly user skin tone image 126. For example, this can be done by training an ML model that can produce the LAB values of the skin color (from the unprocessed image 122), which match the results that would be obtained by scanning the user's skin with a scanner (i.e., obtaining the actual image 126 of the user using the user skin tone analysis device 116). With that, the closest match in the database 214 will be used to compute a set of user skin tone rendering adjustment factors (such as one of the sets described herein). Notably, information from the computing device camera (in addition to the unprocessed user skin tone image 122) can be used, such as information about magnification and lighting, to obtain the best match from the database 214.

[0069] 2) Omit the human-processed image 124. For example, this can also be done using ML to train a model that will output Delta(L*) based on the unprocessed image 122. Similarly, information from the computing device camera (used to take the unprocessed image 122) will be used (such as estimated ambient light parameters, average LAB of background and foreground pixels, skin texture, and location, etc.). For example, the method can use the processing engine of the computing device (such as via their SDK and API) (by taking many photos under different lighting conditions and measuring the Delta(L*) between the unprocessed user skin tone image 122 and the processed user skin tone image 124).

[0070] In practice, embodiments of the present invention can be implemented before and / or after training an AI / ML solution to correct skin tone rendering in an unprocessed user skin tone image. Before training, at least one unprocessed user skin tone image can be paired with at least one skin tone assembly user skin tone image for the user ("for (used for)" the user means "of" the user or selected for the user (e.g., from database 214), and optionally paired with at least one human-processed skin tone image (typically "of" the user). At this time, the system will have the skin tone assembly user skin tone image for the user and will know what the skin looks like under a known light source (e.g., D65 as can be used in a skin tone analysis device). Various skin tone rendering adjustment factors can be determined as described herein and applied as shown herein - for example to adjust the illumination present in the unprocessed user skin tone image. Of course, and as described herein, it may be preferably possible to determine the appropriate skin tone rendering adjustment factor and apply it to the unprocessed user skin tone image without relying on a human to create a human-processed skin tone image, or having the skin analysis device present with the user / human when the user / human wants a more accurate skin tone rendering of a given unprocessed user skin tone image. In this case, database 214 and the trained AI / ML model can be used to eliminate this need while providing the required skin tone rendering adjustment factor to obtain an adjusted user skin tone image as described herein.

[0071] The above-described embodiments of the present disclosure can be implemented in any of a variety of ways. For example, an embodiment can be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or set of processors, whether provided in a single computer or distributed among multiple computers.

[0072] Furthermore, the various methods or processes outlined herein can be encoded as software executable on one or more processors that employ any of a variety of operating systems or platforms. Additionally, such software can be written using any of a variety of suitable programming languages and / or programming or scripting tools, and can also be compiled into executable machine language code or intermediate code that executes on a framework or virtual machine.

[0073] In this regard, the concepts disclosed herein can be embodied as a non-transitory computer-readable medium (or media) (e.g., computer memory, one or more floppy disks, compact discs, optical discs, magnetic tapes, flash memories, circuit configurations in a field-programmable gate array or other semiconductor device, or other non-transitory, tangible computer storage media) encoded with one or more programs that, when executed on one or more computers or other processors, perform the methods of implementing the various embodiments of the present disclosure above. One or more computer-readable media can be transportable such that one or more programs stored thereon can be loaded onto one or more different computers or other processors to implement various aspects of the present disclosure as described above.

[0074] The terms "program," "app," or "application" or "software" are used herein to refer to any type of computer code or set of computer-executable instructions that can be used to program a computer or other processor to implement various aspects of the present disclosure as described above. Additionally, it should be understood that, according to one aspect of this embodiment, one or more computer programs that perform the methods of the present disclosure when executed need not reside on a single computer or processor, but can be distributed in a modular fashion among multiple different computers or processors to implement various aspects of the present disclosure.

[0075] Computer-executable instructions can be in many forms, such as program modules executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Generally, in various embodiments, the functions of program modules can be combined or distributed as needed.

[0076] Furthermore, data structures can be stored in a computer-readable medium in any suitable form. For simplicity of illustration, a data structure can be shown as having fields related by their positions in the data structure. This relationship can equally be achieved by allocating storage in the computer-readable medium for fields having positions that convey the relationship between the fields. However, any suitable mechanism can be used to establish the relationship between the information in the fields of a data structure (including by using pointers, tags, or other mechanisms for establishing relationships between data elements).

[0077] The various features and aspects of the present disclosure can be used alone, in any combination of two or more, or in various arrangements not specifically discussed in the embodiments described above, and thus their application is not limited to the details and arrangements of the components set forth in the above description or shown in the drawings. For example, aspects described in one embodiment can be combined with aspects described in other embodiments in any manner.

[0078] In addition, the concepts disclosed herein may be embodied as methods, for which examples have been provided. The acts performed as part of the method may be ordered in any suitable way. Accordingly, embodiments may be constructed in which the acts are performed in an order different from that shown, which may include performing some acts simultaneously, even if shown as sequential acts in illustrative embodiments.

[0079] The use of ordinal terms (such as “first,” “second,” and “third”) in the claims to modify a claim element itself does not mean any precedence, priority, or order of one claim element with respect to another, or the temporal order of acts of a method, but is merely used as a label to distinguish one claim element having a particular name from another claim element having the same name (if the ordinal term were not used) to distinguish the claim elements.

[0080] Furthermore, the words and terms used herein are for descriptive purposes only and should not be regarded as limiting. As used herein, “including,” “comprising,” “having,” “containing,” “involving,” and variations thereof are intended to include the items listed hereinafter and their equivalents as well as additional items.

[0081] Several (or different) elements discussed and / or claimed hereinafter are described as “coupled,” “in communication with,” or “configured to communicate with.” The term is intended to be non - limiting and, where appropriate, should be construed to include, but not be limited to, wired and wireless communication using any one or more suitable protocols, and communication methods that are continuously maintained, performed periodically, and / or initiated as needed.

[0082] Embodiments may also be implemented in a cloud computing environment. In this description and the appended claims, “cloud computing” may be defined as a model for enabling ubiquitous, convenient, on - demand network access to a shared pool of configurable computing resources (such as networks, servers, storage, applications, and services) that can be rapidly provisioned via virtualization and released with minimal management effort or service provider interaction, and then scaled accordingly. The cloud model may include various characteristics (such as on - demand self - service, broad network access, resource pooling, rapid elasticity, measured service, etc.), service models (such as software as a service (“SaaS”), platform as a service (“PaaS”), infrastructure as a service (“IaaS”)), and deployment models (such as private cloud, community cloud, public cloud, hybrid cloud, etc.).

[0083] This written description uses examples to disclose the invention and also enables those skilled in the art to make and use the invention. The patentable scope of the invention is defined by the claims and may include other examples that occur to those skilled in the art. If these other examples have structural elements that do not differ from the literal language of the claims or if they include equivalent structural elements that do not differ substantially from the literal language of the claims, then these other examples are within the scope of the claims.

[0084] It will be understood that the above assemblies and modules can be connected to each other as needed to perform the desired functions and tasks within the scope of those skilled in the art, such that such combinations and arrangements can be made without having to describe each and every one in explicit terms. No particular assembly or component can be superior to any equivalents available to those skilled in the art. There is no particular mode of practicing the disclosed subject matter that is superior to any other mode so long as these functions can be performed. It is believed that all key aspects of the disclosed subject matter have been provided in this document. It should be understood that the scope of the invention is limited to the scope provided by the independent claim(s) and it should also be understood that the scope of the invention is not limited to: (i) the dependent claims, (ii) the detailed description of the non-limiting embodiments, (iii) the summary of the invention, (iv) the abstract, and / or (v) the description provided outside of this document (i.e., outside of the present application as filed, prosecuted, and / or granted). For this document, it should be understood that the phrase “includes” is equivalent to the word “comprising”. Non-limiting embodiments (examples) have been outlined above. Specific non-limiting embodiments (examples) have been described. It should be understood that the non-limiting embodiments are provided only as examples.

Claims

1. A system for improved skin tone rendering of a user's skin tone in a digital image, the system comprising: A first computing device configured to: Receive a set of user skin tone images of the user, including at least the unprocessed user skin tone image of the user obtained from a computing device camera; Obtain a skin tone assembly user skin tone image for the user, the skin tone assembly user skin tone image being taken using a user skin tone analysis device; Extract user skin tone color values of the user from each image in the set of user skin tone images and the skin tone assembly user skin tone image of the user; Calculate a set of user skin tone rendering adjustment factors based on the skin tone color values; Apply one or more user skin tone rendering adjustment factors from the set of user skin tone rendering adjustment factors to the unprocessed user skin tone image to obtain an adjusted user skin tone image; And Output the adjusted user skin tone image.

2. The system according to claim 1, wherein the first computing device further comprises a first computing device camera and a user skin tone analysis device, the user skin tone analysis device being attached to the first computing device in front of the first computing device camera, and wherein the obtaining is performed via the first computing device camera using the user skin tone analysis device in front of the computing device camera, and wherein the skin tone assembly user skin tone image for the user is an image of the user.

3. The system according to claim 2, wherein the magnification of the skin tone assembly user skin tone image is not less than 10 times.

4. The system according to claim 1, further comprising a database of skin tone assembly skin tone images from a second computing device camera, having a second user skin tone analysis device in front of the second computing device camera, and wherein the obtaining is performed from the database of skin tone assembly skin tone images, and the skin tone assembly user skin tone image for the user is not an image of the user and is selected based on comparing the unprocessed user skin tone image of the user with a set of skin tone assembly user skin tone images in the database of skin tone assembly skin tone images.

5. The system according to claim 1, wherein the user skin tone color values include an L* channel, an a* channel, and a b* channel.

6. The system according to claim 5, wherein the set of user skin tone images includes an unprocessed skin tone image and a human-processed skin tone image.

7. The system according to claim 6, wherein the extraction further comprises, for each image in the set of user skin tone images: Identifying a set of image pixels including the user's skin surface; For each pixel in the set of image pixels, summing the L* channel, the a* channel, and the b* channel; And For the L* channel, the a* channel, and the b* channel, divide the sum by the number of pixels in the set of image pixels to obtain an average L* channel, an average a* channel, and an average b* channel.

8. The system according to claim 7, wherein the set of skin tone rendering adjustment factors includes a first skin tone rendering adjustment factor and a second skin tone rendering adjustment factor, the first skin tone rendering adjustment factor includes a first difference between the a* channels between the skin tone assembly skin tone image and the unprocessed skin tone image, and the second skin tone rendering adjustment factor includes a second difference between the b* channels between the skin tone assembly skin tone image and the unprocessed skin tone image.

9. The system according to claim 8, wherein the set of skin tone rendering adjustment factors further includes a third skin tone rendering adjustment factor, the third skin tone rendering adjustment factor includes a third difference between the L* channels between the human-processed skin tone image and the unprocessed skin tone image, and the application includes the first skin tone rendering adjustment factor, the second skin tone rendering adjustment factor, and the third skin tone rendering adjustment factor.

10. The system according to claim 6, wherein the human-processed skin tone image is created from the unprocessed skin tone image by a human using image processing software to adjust the unprocessed skin tone image such that the user's skin tone in the human-processed skin tone image empirically appears more similar to how the human sees the user's skin tone in real life.

11. The system according to claim 5, wherein the extraction further includes: Identifying a first set of image pixels including the user's skin surface in the unprocessed skin tone image and identifying a second set of image pixels including the user's skin surface in the skin tone assembly user skin tone image for the user; Deriving a first mean L* channel of the first set of image pixels and a second mean L* channel of the second set of image pixels; Based on the derivation, setting a mapping of L* channel values from the first set of image pixels and the second set of image pixels; Using the mapping to create a pixel skin tone rendering adjustment factor for each pixel in the first set of image pixels, the pixel skin tone rendering adjustment factor including an a* channel adjustment factor and a b* channel adjustment factor; For each pixel, adopting the a* channel adjustment factor and the b* channel adjustment factor.

12. The system according to claim 1, wherein each user skin tone image in the set of user skin tone images includes the extracted skin tone segment of the user.

13. The system according to claim 12, wherein the application is for the extracted skin tone segment in the unprocessed user skin tone image.

14. The system according to claim 1, wherein the output includes one or more of the following: displaying the adjusted user skin tone image on the screen of the computing device or storing the adjusted user skin tone image in the memory of the computing device.

15. A method for improved skin tone rendering of a user's skin tone in a digital image, the method comprising: receiving, by a computing device, a set of user skin tone images of the user, the set including at least the unprocessed user skin tone image of the user obtained from a computing device camera; obtaining a skin tone assembly user skin tone image for the user, the skin tone assembly user skin tone image being taken using a user skin tone analysis device; extracting user skin tone color values of the user from each image in the set of user skin tone images and the skin tone assembly user skin tone image of the user; computing a set of user skin tone rendering adjustment factors based on the skin tone color values; applying one or more user skin tone rendering adjustment factors from the set of user skin tone rendering adjustment factors to the unprocessed user skin tone image to obtain an adjusted user skin tone image; and outputting the adjusted user skin tone image.

16. The method according to claim 15, wherein the obtaining is performed by the computing device, the computing device further including a computing device camera and a user skin tone analysis device, wherein the user skin tone analysis device is in front of the computing device camera, and wherein the skin tone assembly user skin tone image for the user is an image of the user.

17. The method according to claim 16, wherein the magnification of the skin tone assembly user skin tone image is not less than 10 times.

18. The method according to claim 15, wherein the obtaining is performed from a database of skin tone assembly skin tone images, and the skin tone assembly user skin tone image for the user is not an image of the user and is selected based on comparing the unprocessed user skin tone image of the user with a set of skin tone assembly user skin tone images in the database of skin tone assembly skin tone images.

19. The method according to claim 15, wherein the user skin tone color values include an L* channel, an a* channel, and a b* channel.

20. The method according to claim 19, wherein the set of user skin tone images includes an unprocessed skin tone image and a human-processed skin tone image.

21. The method according to claim 20, wherein the extracting further includes, for each image in the set of user skin tone images: identifying a set of image pixels including the skin surface of the user; for each pixel in the set of image pixels, summing the L* channel, the a* channel, and the b* channel; and For the L* channel, the a* channel, and the b* channel, divide the sum by the number of pixels in the set of image pixels to obtain an average L* channel, an average a* channel, and an average b* channel.

22. The method according to claim 21, wherein the set of skin tone rendering adjustment factors includes a first skin tone rendering adjustment factor and a second skin tone rendering adjustment factor, the first skin tone rendering adjustment factor includes a first difference between the a* channels between the skin tone assembly skin tone image and the unprocessed skin tone image, and the second skin tone rendering adjustment factor includes a second difference between the b* channels between the skin tone assembly skin tone image and the unprocessed skin tone image.

23. The method according to claim 22, wherein the set of skin tone rendering adjustment factors further includes a third skin tone rendering adjustment factor, the third skin tone rendering adjustment factor includes a third difference between the L* channels between the human-processed skin tone image and the unprocessed skin tone image, and the application includes the first skin tone rendering adjustment factor, the second skin tone rendering adjustment factor, and the third skin tone rendering adjustment factor.

24. The method according to claim 20, further comprising creating the human-processed skin tone image by a human using image processing software to adjust the unprocessed skin tone image such that the user's skin tone in the human-processed skin tone image empirically appears more similar to how the human sees the user's skin tone in real life.

25. The method according to claim 19, wherein the extraction further includes: identifying a first set of image pixels including the user's skin surface in the unprocessed skin tone image and identifying a second set of image pixels including the user's skin surface in the skin tone assembly user skin tone image for the user; deriving a first mean L* channel of the first set of image pixels and a second mean L* channel of the second set of image pixels; based on the derivation, setting a mapping of L* channel values from the first set of image pixels and the second set of image pixels; using the mapping to create a pixel skin tone rendering adjustment factor for each pixel in the first set of image pixels, the pixel skin tone rendering adjustment factor including an a* channel adjustment factor and a b* channel adjustment factor; for each pixel, adopting the a* channel adjustment factor and the b* channel adjustment factor.

26. The method according to claim 15, wherein each user skin tone image in the set of user skin tone images includes the extracted skin tone segment of the user.

27. The method according to claim 16, wherein the application is for the extracted skin tone segment in the unprocessed user skin tone image.

28. The method according to claim 15, wherein the output comprises one or more of the following: displaying the adjusted user skin tone image on a screen of the computing device or storing the adjusted user skin tone image in a memory of the computing device.

Images