Creating a color cloud

Abstract

A bracket can be used to capture video of an object such as hair. The video can then be used to extract and visualize color information associated with the material using a color cloud. The color cloud contains all colors present in one or more frames of the video. The bracket includes a video recording device capable of viewing the material through a window on the bracket. A smartphone can be used with the bracket to act as the video recording device and to provide lighting.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. Application No. 17 / 107,168, filed November 30, 2020, and French Application No. FR2101816, filed February 25, 2021, the entire contents of each of which are incorporated herein by reference. Background Technology

[0003] Spectrophotometers are used to read the color of hair samples, but they only provide a single digital representation of the material in the form of a single RGB, L*a*b*, LCh, or HSV triplet, and typically only read areas smaller than 1 cm x 1 cm. Furthermore, when comparing two materials, analysis is limited to Euclidean distances in color spaces between the two (e.g., ...). Furthermore, the threshold for detection in hair color has not yet been determined. Additionally, a single color (e.g., RGB or L*a*b*) is the average color of the measured area, and the average color is not concrete but a mathematical concept. Theoretically, for example, one might take the average color of a hair region, and that color might not even exist in that hair region. Summary of the Invention

[0004] In one aspect, among other things, this disclosure relates particularly to systems, apparatus, and methods for creating three-dimensional virtual representations, color clouds, etc., of objects, materials, etc., from one or more digital images, videos, optical sensor information, etc.

[0005] This disclosure describes generating a color cloud for an object, wherein the color cloud contains every color present in one or more frames across a plurality of frames; and using the color cloud to identify features of hair.

[0006] In one embodiment, the color cloud comprises the frequency of each color present in one or more frames of a plurality of frames.

[0007] One embodiment also includes comparing a first color cloud with a second color cloud; and, based on the comparison, visualizing the similarities and differences between the first color cloud and the second color cloud.

[0008] One embodiment also includes displaying a color cloud in a color space.

[0009] In one embodiment, the hair features include identifying the level of shine.

[0010] A system is also disclosed, comprising: a color cloud unit including computational circuitry operatively coupled to an image sensor and configured to generate a three-dimensional virtual representation of color information associated with multiple digital images of an object; and a salient feature unit including computational circuitry for identifying one or more salient features of the object and generating one or more virtual instances of identifiers indicating salient features of the object on a graphical user interface display based on one or more inputs associated with the three-dimensional virtual representation of the object's color information.

[0011] In one aspect, a color cloud unit comprising a computational circuit configured to generate a three-dimensional virtual representation of color information associated with multiple digital images of an object includes a computational circuit configured to generate a three-dimensional virtual representation of color information, the color information including regions indicating the frequency of color occurrence, color variation, or color intensity distribution associated with the object.

[0012] In one aspect, the multiple digital images of the object include multiple digital images of hair; and the color cloud unit of the computational circuit configured to generate a three-dimensional virtual representation of color information associated with the multiple digital images of the object includes the computational circuit configured to generate a three-dimensional virtual representation of hair color information associated with the multiple digital images of hair.

[0013] In one aspect, the multiple digital images of the object include multiple digital images of hair; and the color cloud unit of the computational circuit configured to generate a three-dimensional virtual representation of color information associated with the multiple digital images of the object includes the computational circuit configured to generate a three-dimensional virtual representation of hair color information and a three-dimensional virtual representation of hair features associated with the multiple digital images of hair.

[0014] In one aspect, a color cloud unit comprising a computational circuit configured to generate a three-dimensional virtual representation of color information associated with multiple digital images of an object includes a computational circuit configured to generate a three-dimensional virtual representation of color information, which includes voxels with varying intensities to indicate the frequency of color occurrence of specific features associated with the imaged object.

[0015] On the other hand, the multiple digital images of the object include multiple digital images of hair; and the computational circuitry configured to generate a three-dimensional virtual representation of color information, the color information including voxels with varying intensities to indicate the frequency of color occurrence of specific features associated with the imaged object, further includes computational circuitry configured to generate a three-dimensional virtual representation of hair feature information, the hair feature information including voxels indicating gray hair coverage, gloss, brightness, and uniformity associated with the multiple digital images of hair. Attached Figure Description

[0016] Figure 1 The video shows a user dragging a bracket over their hair to capture video and create a color cloud for it.

[0017] Figure 2 This demonstrates one method for creating and utilizing color clouds.

[0018] Figure 3 A first embodiment of the bracket is shown.

[0019] Figure 4 A second embodiment of the bracket is shown, wherein the bracket can be attached to a smartphone.

[0020] Figure 5A The image shows a user's hair used to capture video, where the video is captured across the length of the user's hair (starting from the roots and ending at the ends).

[0021] Figure 5B An example of a video with multiple frames arranged in chronological order is shown, in which the user captures a video of their hair, starting at the roots and ending at the tips.

[0022] Figure 6 An example of a single color cloud is shown, displaying all the colors from each frame of the video.

[0023] Figure 7 An example of a color cloud with color and frequency information is shown.

[0024] Figure 8 Examples of four different color clouds are shown. These four different color clouds were created from samples and then superimposed in the L*a*b* color space (projected on the L* and b* dimensions) to visualize their similarities and differences.

[0025] Figure 9 The video directly shows the shared and unique colors among different hair types. Detailed Implementation

[0026] This disclosure describes a physical device and a unique algorithm that can quantify the optical reflectance of any material (such as hair samples and real hair on a human head) in a highly repeatable, portable manner and workable in any lighting environment. The device (which will be referred to as a bracket) can be used on the material (such as hair) and a color cloud algorithm is used to analyze the material to obtain a holistic digital representation of the material (defined herein as a color cloud). This color cloud can be compared with other reflective materials or other color clouds of the same reflective material at different points in time (e.g., colored hair before and after washing). Embodiments disclosed herein extract existing colors from video captures of the material (and do not manipulate them in any arithmetic way to produce a net result). This technique can use a robust color set as its basis for detecting minute color changes.

[0027] Figure 1 An example use case is illustrated. User 101 can pick up bracket 102 and drag it over their hair 103. As bracket 102 is dragged over the user's hair 103, the bracket can acquire a video of the hair 103. A color cloud algorithm can be used to analyze the video to generate a color cloud.

[0028] Figure 2 This is a flowchart illustrating one embodiment. In S202, a video of material (such as a user's hair) is obtained using a bracket. For example, to create a color cloud for a user's hair, the user can move the bracket along the entire length of their hair (i.e., from root to tip). In S204, a color cloud is created from one or more frames of the video. Essentially, all colors in these one or more frames are extracted and displayed in the color cloud. In S206, the color cloud is compared with other color clouds. This comparison can be used for color matching and / or color differentiation.

[0029] A bracket is a device that allows people to capture video of materials, thereby creating a color cloud of the material. Figure 3An example of a bracket is shown (solid lines indicate the exterior of the bracket, while dashed lines indicate the interior). The bracket has: a first end 301; a second end 302; a window 303 on the second end 302; and a wall 304 connecting the first end 301 and the second end 302. In one embodiment, the first end 301 and the second end 302 are at opposite ends of the bracket. An image sensor (such as a video recording device 305) can be attached to the first end 301 inside the bracket to capture video of any material 306 (e.g., hair) visible through the window 303. Additionally, the bracket may have processing circuitry 308 connected to the video recording device 305 and configured to: create a color cloud from the video of the material 306, compare the created color cloud with a previous color cloud, etc. The second end 302 of the bracket may be colored with a predetermined set of colors 309 adjacent to the window 303 for color separation when captured by the video recording device 305. The video can capture both material 306 passing through window 303 and a predetermined set of colors 309 adjacent to window 303; then, when determining the color of material 306, the predetermined (i.e., known) colors can be used as a reference. A lamp 307 can also be attached to the bracket to provide an adequately lit field of view from the video recording device 305 to window 303. The bracket can be used to maintain a consistent distance between the video recording device 305 and the material 306 when recording video. The bracket may also have a handle 310.

[0030] Processing circuitry 308 may include a color cloud unit and a salient feature unit. The color cloud unit may include circuitry coupled to an image sensor and configured to generate a 3D virtual representation of color information associated with multiple digital images of an object, such as regions indicating color frequency, color variation, or color intensity distribution associated with the object. The salient feature unit may include circuitry configured to identify one or more salient features of the object based on one or more inputs associated with the 3D virtual representation of the object's color information, and to generate one or more virtual instances of identifiers indicating the salient features of the object on a graphical user interface display. In one embodiment, the object may be hair, and the 3D virtual representation may be hair color information and / or hair feature information associated with the multiple digital images of the hair. Additionally, the 3D virtual representation may include voxels with varying intensities to indicate the color frequency of a specific feature associated with the imaged object. Examples of feature information include: object identification data, object feature data, color frequency data, color variation data, color intensity data, presence or absence of color data, frame-by-frame or pixel-by-pixel color analysis information, etc. Non-limiting color information may include: hue, tint, tone, shade, etc. In one embodiment, the hue of the base color is a lighter form of that color, and the chroma is a darker form. In another embodiment, hue refers to the brightness (color) or darkness (chroma) of the base color. If the image object is hair, the 3D virtual representation may include hair feature information, which includes voxels indicating the following: gray hair coverage, gloss, radiance, and uniformity associated with the plurality of digital images of the hair. Examples of hair features may include: hair damage state, concentration of artificial colorants, color distribution, gray hair coverage, texture, gloss, distribution density, scalp features, etc.

[0031] Figure 3 The main body of the bracket is essentially a hollow rectangular prism with a window 303 at one end. In one embodiment, when the window 303 is covered, the only light inside the bracket body can come from the lamp 307. This allows for consistent lighting conditions within the bracket. Furthermore, in other embodiments, the bracket can have different shapes, such as cylinders, cubes, trapezoidal prisms, etc. The bracket can also have a screen that can be used to visualize color clouds.

[0032] In one embodiment, the first end of the bracket can be configured to use a camera from a mobile device (e.g., a smartphone) as a video recording device. Figure 4An example is shown where the first end 401 of the holder can be an attached smartphone 402. A camera 403 on the smartphone 402 can capture video of the material 404 through a window 405 on the second end 406. Additionally, processing circuitry already in the smartphone 402 can be used to create a color cloud. The color cloud and other relevant information can be visualized directly on the smartphone's screen. Furthermore, the smartphone's flash 407 can be used to provide a well-lit field of view from the smartphone's camera 403 to the window 405. The holder may also have a retainer 408 that allows the smartphone 402 to be properly attached to the holder when in use and detached from the holder when not in use.

[0033] The captured video can be material from which color clouds need to be generated. The bracket should be positioned such that the video recording device can capture video of the material through the opening in the bracket provided by the window. The window can be positioned directly against the material to prevent external light from entering the interior of the bracket. An example of material could be hair on a head. In this case, the bracket can start by capturing video from the roots of the hair and then be dragged across the hair to the ends.

[0034] In one embodiment, the video (which comprises multiple frames) can be at least six seconds long and at least 60 frames per second. If the video captures human hair or a hair sample, the hair can be straight or curly.

[0035] Videos used to obtain material can include videos capturing someone's hair. Figure 5A and Figure 5B An example is shown in the image. First, refer to... Figure 5A The video captures the entire length of the model's hair, starting from the root 51a and moving downwards towards the tip 52a. Figure 5B This shows each frame of the captured video laid out in chronological order (from...). Figure 5A ), where the top left frame 51b is the hair root 51a and the bottom right frame 52b is the hair tip 52a. In another embodiment, the video used to acquire the material may include video capturing a length less than the entire length of a person's hair (e.g., capturing only half of a person's hair). Note that video can be captured in other directions besides downwards, such as upwards or at an angle.

[0036] A color cloud algorithm can be used to create a color cloud from one or more frames of a video. The color cloud algorithm can extract color information from the one or more frames. The color cloud contains every color present in one or more of the frames. In another embodiment, the color cloud contains: every color present in one or more of the frames, and the frequency of every color present in one or more of the frames. In yet another embodiment, the color cloud contains: every color present in one or more of the frames, and the frequency of every color present in one or more of the frames, wherein the frequency exceeds a minimum predetermined threshold.

[0037] In one embodiment, a color cloud algorithm can generate a color cloud by extracting all colors from a single frame of a video. For example, in Figure 5B In this approach, a separate color cloud can be created for each frame. The color cloud created for each frame can include all the colors present in that single frame, and in another embodiment, it can include the frequency of each color present in that single frame.

[0038] In another embodiment, a color cloud algorithm can generate a color cloud by extracting cumulative colors from a set of frames in a video (rather than from a single frame). For example, refer again... Figure 5B A color cloud can be generated for each group of 10 frames, wherein the color cloud created for each group of 10 frames includes all colors present in the 10 frames of that particular group, and in another embodiment includes the frequency of each color present in the 10 frames of that group.

[0039] Figure 6 The image shows an example of a color cloud, where the color cloud uses a color space to display every color across all frames of the video. Because the video is time-based, selecting a specific frame allows for the creation of a color cloud at a specific point in time and / or a time range.

[0040] Once the color cloud algorithm has extracted all the colors from one or more frames of a video, the color cloud can be displayed in a variety of ways. For example, colors can be displayed in a color space; color spaces can use coordinates (such as L*a*b*, RGB, LCh, HSV, etc.) to convey color information.

[0041] In another embodiment, the color cloud may include the frequency (i.e., number of times) that each color is present in the one or more frames of the video. Figure 7An example is shown where a brighter pixel represents a higher frequency of color, and vice versa. An outer color cloud 71 (i.e., the complete color cloud) captures every color present in the one or more frames, while an inner color cloud 72 (i.e., the optimized color cloud) captures colors present at higher frequencies. The threshold used to determine whether a color exists at a higher frequency can be adjusted depending on the application. In one case, only the inner color cloud 72 is used, and low-frequency (i.e., low-information) colors are removed as noise.

[0042] In another embodiment, the color cloud can be used to identify hair features such as gray coverage, shine, brightness, evenness, glow, and so on. For example, color and frequency information from the color cloud can be used to filter “low-information” colors and enhance color detection to extract the amount of gray hair a person has. As another example, the shine level of hair can be calculated by looking at the mean, variance, and spatially connected cluster of L* values ​​in the hair's color cloud; high variability indicates a wide range of L* values ​​from dark to light, and high spatially connected L* values ​​indicate bands of shine, both of which, when combined, relate to a person's perception of shine.

[0043] Color clouds can be compared with other color clouds to identify similarities and differences. This enables color change detection and color matching. Figure 8 The example shown is a comparison with other color clouds. Color clouds are generated for two unwashed samples and the same two samples after cleaning. Overlapping color cloud portions indicate color similarity (i.e., colors present in both samples), while non-overlapping portions indicate color differences (i.e., colors not shared between samples). This technique quantifies the overlap and non-overlap of groups of colors in a color space (e.g., the number of groups, group similarity, average color of the groups).

[0044] In addition, the similarities and differences between hair colors can be visualized back on the video. Figure 9 An example is shown where color clouds were created for samples before and after washing. Darker areas on the hair indicate shared colors, while lighter areas indicate unique colors. Similarities and differences between these color clouds can be identified and visualized directly on the samples. In other words, a user can replay one of the videos used to capture the hair and display similarities / differences directly on the hair in the video. This is done by simply matching the color of a video pixel against a color contained in either of the two different color clouds or the overlap of the two different color clouds. For example, a video pixel color that matches either a color in the overlap or intersection of the color clouds will be colored darker in the visualized video.

[0045] In another embodiment, the atlas of hair samples may have a catalog of color clouds created for each hair sample. Any subsequent capture of hair can then generate a color cloud, which is compared to samples in the atlas to determine the closest match (i.e., the most similar in color). For good differentiation, a sample should be more similar to itself than to other samples.

[0046] The technologies described in this article can capture a wide range of reflections (e.g., from hair samples and hair heads) to accurately and robustly measure heterogeneous materials (such as color) and to accurately and robustly measure hair color changes. Applications of these technologies can include: measuring color changes more accurately and reproducibly than existing instruments (e.g., spectrophotometers); measuring the color of human hair (as a hair colorist would); measuring gray hair coverage (e.g., measuring the amount of gray hair in a person's hair roots to indicate how much hair coloring product to use); measuring hair health (e.g., based on shine levels, where too little shine can indicate dullness and hair damage); dynamic visualization of hair before and after coloring and / or washing, and more. The technologies described in this article can also allow for color assessment at home, in-store display and visualization, and product recommendations at home. For example, consumers can download an app to their phones to capture video and create color clouds. The app can then further recommend hair products based on certain characteristics of the hair (e.g., specific hair color products can be recommended based on the user's hair color, gray hair level, and a survey performed on the app). In addition, the trays can be portable (e.g., foldable cardboard trays, 3D printed trays, etc.).

Claims

1. A method of creating a color cloud for a material, comprising: acquiring a video of the material, wherein the video comprises a plurality of frames, and the material is a user's hair and wherein the video is based on an image sensor moving over the user's hair; creating a color cloud for the material based on one or more of the plurality of frames, wherein the color cloud contains each color present in one or more of the plurality of frames; and identifying a feature of the hair using the color cloud; wherein the color cloud contains a frequency of each color present in one or more of the plurality of frames, the frequency being a number of times each color is present in the plurality of frames, and wherein the color cloud comprises an outer color cloud and an inner color cloud, wherein the outer color cloud captures each color present in the plurality of frames, and the inner color cloud captures colors present at a frequency above a threshold frequency, indicating that a number of times the inner color cloud color is present in the plurality of frames is above a threshold number, and wherein the outer color cloud is shown in a two-dimensional image as surrounding the inner color cloud.

2. The method of claim 1, further comprising: displaying the color cloud in a color space.

3. The method of claim 1, wherein the feature of the hair comprises identifying a level of shine.

4. A system for creating a color cloud for a material, comprising: processing circuitry configured to, acquire a video of the material, wherein the video comprises a plurality of frames, and the material is a user's hair and wherein the video is based on an image sensor moving over the user's hair; create a color cloud for the material based on one or more of the plurality of frames, wherein the color cloud contains each color present in one or more of the plurality of frames; and identify a feature of the hair using the color cloud; wherein the color cloud contains a frequency of each color present in one or more of the plurality of frames, the frequency being a number of times each color is present in the plurality of frames, and wherein the color cloud comprises an outer color cloud and an inner color cloud, wherein the outer color cloud captures each color present in the plurality of frames, and the inner color cloud captures colors present at a frequency above a threshold frequency, indicating that a number of times the inner color cloud color is present in the plurality of frames is above a threshold number, and wherein the outer color cloud is shown in a two-dimensional image as surrounding the inner color cloud.

5. The system of claim 4, wherein the processing circuitry is further configured to display the color cloud in a color space.

6. The system of claim 4, wherein the feature of the hair comprises identifying a level of shine.

7. A system, comprising: a color cloud unit comprising computing circuitry, operably coupled to an image sensor, and configured to generate a three-dimensional virtual representation of color information associated with a plurality of digital images of hair, wherein the three-dimensional virtual representation contains a frequency of each color present in the plurality of digital images, the frequency being a number of times each color is present in the plurality of digital images, and wherein the three-dimensional virtual representation includes an outer color cloud and an inner color cloud, wherein the outer color cloud captures every color present in the plurality of digital images, and the inner color cloud captures colors present at a frequency above a threshold frequency, indicating that the inner color cloud colors are present in the plurality of digital images a number of times above a threshold number, and wherein the outer color cloud is shown in the two-dimensional image as surrounding the inner color cloud; and a feature unit including computing circuitry to identify one or more features of the hair and generate one or more virtual instances of an identification of the features of the hair on the graphical user interface display based on one or more inputs associated with the three-dimensional virtual representation of color information of the hair.

8. The system of claim 7, wherein the color cloud unit including computing circuitry configured to generate a three-dimensional virtual representation of color information associated with the plurality of digital images of the hair includes computing circuitry configured to generate a three-dimensional virtual representation of color information including regions indicative of color frequency of occurrence, color variation, or color intensity distribution associated with the hair.

9. The system of claim 7, wherein the color cloud unit including computing circuitry configured to generate a three-dimensional virtual representation of color information associated with the plurality of digital images of the hair includes computing circuitry configured to generate a three-dimensional virtual representation of hair color information and a three-dimensional virtual representation of hair features associated with the plurality of digital images of the hair.

10. The system of claim 7, wherein, The color cloud unit including computing circuitry configured to generate a three-dimensional virtual representation of color information associated with the plurality of digital images of the hair includes computing circuitry configured to generate a three-dimensional virtual representation of color information including voxels having varying intensities to indicate color frequency of occurrence of a particular feature associated with the imaged hair.

11. The system of claim 10, wherein the computing circuitry configured to generate a three-dimensional virtual representation of color information including voxels having varying intensities to indicate color frequency of occurrence of a particular feature associated with the imaged hair further includes computing circuitry configured to generate a three-dimensional virtual representation of hair feature information including voxels indicative of white hair coverage, shine, luster, or uniformity associated with the plurality of digital images of the hair.

Images