Illumination compensation in imaging

Abstract

In one embodiment, a method (100) is described. The method includes obtaining (102) a three-dimensional representation of a body surface. The method also includes obtaining illumination information indicative of an orientation of the three-dimensional representation of the body surface relative to a reference axis. The method also includes determining illumination compensation information (104). The illumination compensation information is used (106) to compensate for apparent illumination changes in the illumination information from an image of the body surface.

Description

Technical Field

[0001] The present invention relates to methods, apparatuses, and tangible machine-readable media for imaging, for example, in certain lighting settings. Background Technology

[0002] A topic of interest within the less prominent fields of measurement and monitoring involves skin sensing for personal care and health applications. Skin sensing systems are being developed that aim to quantify and monitor features in the skin that are too small to be detected, too weak to be noticed, or too slow to be followed. To provide user-acceptable results, such skin sensing systems may need to offer sensitivity and specificity when performing skin sensing. Measurements performed by such skin sensing systems have proven robust and reliable, allowing users to build trust in them.

[0003] For example, it is known from US patent application US2011 / 182520A1 that the position of the light source illuminating the object can be derived from an image of the object. Based on this, the orientation of the object can be determined.

[0004] Imaging-based skin sensing systems may need to determine information that can be affected by difficult-to-control parameters such as ambient lighting. For example, certain uncontrolled environments, such as those in a user's home, may have undefined and / or variable ambient lighting. Such uncontrolled environments may lead to inaccurate measurements of the user's skin, which in turn may result in unacceptable or untrustworthy results for the user. Consequently, any personal care or health treatment plans based on such results may be compromised.

[0005] The performance of other imaging-based sensing systems used to acquire information from surfaces other than skin can also be adversely affected by certain uncontrolled environments. For example, for a particular application in an uncontrolled environment, a user looking to acquire an image of an object may find that the image may have unacceptable variations in lighting, which could affect how the image is perceived or subsequently processed.

[0006] Therefore, the goal is to improve imaging in certain lighting settings. Summary of the Invention

[0007] The aspects or embodiments described herein relate to improving imaging in certain lighting settings. The aspects or embodiments described herein can eliminate one or more problems associated with imaging in uncontrolled environments.

[0008] In a first aspect, a method is described. This method is a computer-implemented approach. The method includes obtaining a three-dimensional representation of a body surface. The method also includes obtaining illumination information of the three-dimensional representation. The illumination information indicates the orientation of the body surface relative to a reference axis. The method further includes determining illumination compensation information configured to normalize apparent illumination variations in the illumination information. The method also includes determining the orientation of the body surface based on analysis of the three-dimensional reconstruction of the body surface. The method further includes using the illumination compensation information to compensate for illumination variations in an image of the body surface. The method also includes determining information for characterizing the skin of the body surface based on the image compensated for the illumination variations.

[0009] Three-dimensional reconstruction can be derived from imaging data of the body surface.

[0010] In some embodiments, the information used to characterize the skin includes measurements of the skin to determine the process of actions performed as part of a personal care application.

[0011] In some embodiments, determining the lighting compensation information includes determining a normalized map representing multiple spatial locations on a body surface. Each spatial location may be associated with a lighting compensation factor to be applied to the corresponding spatial location in the image to correct for lighting variations in the image.

[0012] In some embodiments, the lighting information includes a three-dimensional lighting map representing multiple spatial locations on a body surface. Each spatial location in the lighting map may be associated with a lighting parameter indicating at least one of the following: properties of a lighting source; and the relative position of the lighting source with respect to the spatial location.

[0013] In some embodiments, the method includes determining illumination information by acquiring imaging data of a body surface. The method may further include estimating illumination parameters of the spatial location of the body surface based on analysis of the acquired imaging data. This analysis may take into account the orientation of the body surface.

[0014] In some embodiments, the properties of the lighting source include at least one of the following: the brightness of the lighting source; and the divergence of the lighting source.

[0015] In some embodiments, determining illumination compensation information includes obtaining imaging data of a body surface. The method may also include determining illumination information based on the obtained imaging data. The method may further include identifying any apparent changes in illumination within the illumination information to determine illumination compensation information.

[0016] In some embodiments, determining illumination information from imaging data includes comparing the imaging data with previously acquired imaging data to determine whether imaging data was acquired in a previous time frame. If imaging data was not previously acquired, the method may include generating illumination information from the imaging data. The method may also include storing the illumination information associated with the imaging data in memory. If imaging data was previously acquired, the method may include retrieving the illumination information associated with the imaging data from memory.

[0017] In some embodiments, obtaining a three-dimensional representation of a body surface includes obtaining imaging data of the body surface. The method may also include determining a three-dimensional representation based on the obtained imaging data.

[0018] In some embodiments, determining a three-dimensional representation from imaging data includes: examining a memory that stores a database of body surfaces with any previously identified features to determine whether a body surface can be identified. If no body surface is identified, the method may include determining a three-dimensional representation from the imaging data. The method may also include storing a three-dimensional representation associated with the body surface in memory. If a body surface is identified, the method may include obtaining a three-dimensional representation associated with the identified body surface from memory.

[0019] In some embodiments, the body surface includes the subject's face. Three-dimensional representation may include a three-dimensional reconstruction of the face.

[0020] In some embodiments, using illumination compensation information to compensate for illumination variations in an image of a body surface includes obtaining an image of the body surface. The method may also include using illumination compensation information to compensate for illumination variations in the image.

[0021] In a second aspect, an apparatus is described. The apparatus includes processing circuitry. The processing circuitry includes an acquisition module. The acquisition module is configured to acquire a three-dimensional representation of a body surface. The acquisition module is also configured to acquire illumination information of the three-dimensional representation. The illumination information indicates the orientation of the body surface relative to a reference axis. The processing circuitry also includes a determination module. The determination module is configured to determine illumination compensation information, which is configured to normalize apparent illumination variations in the illumination information. The determination module is also configured to determine the orientation of the body surface based on an analysis of a three-dimensional reconstruction of the body surface derived from imaging data of the body surface. The processing circuitry also includes a correction module. The correction module is configured to use the illumination compensation information to compensate for illumination variations in an image of the body surface. The determination module is also configured to determine information for characterizing the skin of the body surface based on the image compensated for the illumination variations.

[0022] In some embodiments, the device further includes an imaging module. The imaging module may be configured to enable the imaging device to acquire images of the body surface.

[0023] In a third aspect, a tangible machine-readable medium is described. The tangible machine-readable medium stores instructions that, when executed by at least one processor, cause the at least one processor to obtain a three-dimensional representation of a body surface. The instructions further cause the at least one processor to obtain illumination information of the three-dimensional representation. The illumination information indicates the orientation of the body surface relative to a reference axis. The instructions further cause the at least one processor to determine illumination compensation information configured to normalize apparent illumination variations in the illumination information. The instructions also cause the at least one processor to determine the orientation of the body surface based on an analysis of a three-dimensional reconstruction of the body surface derived from imaging data of the body surface. The instructions further cause the at least one processor to use the illumination compensation information to compensate for illumination variations in an image of the body surface. The instructions further cause the at least one processor to determine information for characterizing the skin of the body surface based on the image compensated for the illumination variations.

[0024] These and other aspects of the invention will become apparent and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0025] Exemplary embodiments of the invention will now be described by way of example only with reference to the following figures, wherein:

[0026] Figure 1 The invention relates to a method for improving imaging in certain lighting settings according to one embodiment;

[0027] Figure 2 This is a schematic diagram of a system for improving imaging in certain lighting settings, according to one embodiment;

[0028] Figure 3 The invention relates to a method for improving imaging in certain lighting settings according to one embodiment;

[0029] Figure 4 This is a schematic diagram of an apparatus for improving imaging in certain lighting settings, according to one embodiment;

[0030] Figure 5 This is a schematic diagram of an apparatus for improving imaging in certain lighting settings, according to one embodiment; and

[0031] Figure 6 This is a schematic diagram of a machine-readable medium for improving imaging in certain lighting settings, according to one embodiment. Detailed Implementation

[0032] Figure 1A method 100 (e.g., a computer-implemented method) for improving imaging in certain lighting settings is illustrated. For example, imaging may be affected by certain lighting settings, such as in an uncontrolled environment where the lighting is undefined and / or may vary. As will be described in more detail below, method 100 can correct for any undefined and / or potentially varying lighting in an uncontrolled environment such as in an object's home.

[0033] Method 100 includes obtaining a three-dimensional representation of the body surface at box 102.

[0034] A three-dimensional representation of a body surface can refer to a three-dimensional map or surface profile of the body surface. For example, a three-dimensional representation can be generated for the body surface of a part of the human body (e.g., face, torso, or limb) or any other body (e.g., the body of an inanimate object such as a product or other object). The three-dimensional representation can include a map of vertices (e.g., location coordinates) corresponding to locations on the body surface. In some examples, the three-dimensional representation can represent a polygonal mesh that can be defined by at least one of the following: a set of vertices, edges connecting adjacent vertices, and faces (e.g., triangles, quadrilaterals, or other polygonal shapes) defined by a closed set of edges.

[0035] A 3D representation can include a dataset involving certain features of a body surface. For example, certain dataset elements may include indications of the vertices defining the body surface and / or any information that can be used to define the 3D representation. Thus, a 3D representation can refer to any information that can be used to depict or generate a 3D replica of a body surface.

[0036] Box 102 of method 100 also includes obtaining lighting information in a three-dimensional representation. The lighting information can indicate the orientation of the body surface relative to a reference axis.

[0037] Three-dimensional lighting information can refer to the illumination distribution on a body surface. This illumination distribution can indicate the orientation of the body surface relative to a reference axis. In some examples, the body surface may be situated within a lighting setup where the illumination distribution on the body surface can provide information about the relationship between the light source and the body surface. For example, if the apparent illumination intensity at one part of the body surface is greater than the apparent illumination intensity at another part of the body surface, this can indicate that one part of the body surface is facing the light source and / or is closer to the light source than another part of the body surface. In other words, by analyzing the illumination distribution on the body surface, the orientation of the body surface relative to a reference axis can be determined or estimated.

[0038] A reference axis can refer to an axis around which a body surface can rotate, thus defining the orientation of the body surface. More than one reference axis can be defined, such that rotation about at least one axis can change the orientation of the body surface. A reference axis can be defined relative to a three-dimensional representation. For example, a three-dimensional representation of the body surface can be determined, and one or more reference axes can be defined relative to that three-dimensional representation.

[0039] Method 100 further includes, in block 104, determining lighting compensation information to compensate for apparent lighting changes in the lighting information.

[0040] Apparent illumination variations in illumination information derived from a 3D representation can indicate that the illumination of a body surface is undefined and / or potentially variable (e.g., spatial variations and / or variations as a function of time). For example, a body surface may have a non-uniform illumination distribution due to undefined and / or potentially varying illumination. Illumination compensation information can provide compensation for the illumination distribution determined from imaging data acquired for a specific lighting setup.

[0041] Illumination compensation information may include illumination compensation factors associated with spatial locations on the body surface. Each spatial location on the body surface may be associated with a specific illumination parameter, such as intensity and / or spectral content (e.g., red, green, and blue intensity levels), which can be determined based on imaging data of the body surface. In some cases, uniformly distributed illumination may produce the same illumination parameter values ​​at every spatial location. However, non-uniformly distributed illumination may produce different illumination parameter values ​​at different spatial locations on the body surface.

[0042] An illumination compensation factor can be determined based on a specific illumination distribution defined for a body surface within a given lighting setup. Therefore, an illumination compensation factor for a specific spatial location can be applied to illumination parameter values ​​determined at that location, thereby compensating for any variations in illumination.

[0043] In some examples, lighting compensation information can normalize lighting variations on a body surface. For instance, if a specific spatial location on a body surface is determined to be illuminated at an intensity 10% higher than another spatial location, a lighting compensator factor can be calculated for the specific spatial location and one or both of the other spatial locations to compensate for the lighting variation. In this case, the lighting compensation factor for the specific spatial location could indicate that the intensity detected at that location would be reduced by a factor of 1.1 (e.g., to compensate for the 10% higher intensity at that specific location) – although other methods could be used to normalize the lighting variation.

[0044] Method 100 also includes, in box 106, using illumination compensation information to compensate for illumination variations in an image of the body surface.

[0045] Once lighting compensation information is determined for a specific lighting setup, lighting variations in an image of a body surface can be compensated for, ensuring that the lighting distribution on the body surface meets specific criteria. Such criteria may include, for example, the uniformity of the lighting distribution, so that every location on the body surface appears to be uniformly illuminated.

[0046] This standard can define a range of lighting parameter values ​​(e.g., intensity level, spectral content, etc.) that are considered acceptable for the applications involved. For example, a particular application may determine lighting parameter values ​​from an image, and the application may make decisions based on these lighting parameter values, assuming no changes in lighting.

[0047] That is, an application may incorrectly assume that the lighting settings are suitable for its intended purpose. If the lighting parameter values ​​of the image are not within acceptable ranges, any decision made may be compromised because it may not take into account any changes in lighting.

[0048] Therefore, method 100 can improve imaging in certain lighting settings because it can compensate for lighting variations that might otherwise lead to incorrect decisions by applications using the image. Thus, the image provided by method 100 can be used to improve decision-making in any application that may be affected by lighting variations such as undefined and / or potential changes found in certain lighting settings. Examples of such applications are described in more detail below.

[0049] Furthermore, as mentioned earlier, lighting information can indicate the orientation of a body surface. In some applications, the orientation of a body surface can shift over time, causing the lighting on the body surface to change as it moves.

[0050] Method 100 can adapt to body surface movement, allowing illumination compensation information to be updated as the body surface moves. Therefore, in some examples, imaging data can be repeatedly acquired (the frequency of which can depend on the speed of the movement). If the imaging data indicates that the body surface has moved, the illumination information can change, causing the illumination compensation information to be updated accordingly. On the other hand, if there is no substantial movement of the body surface, the illumination information may not change significantly. In this case, updating the illumination compensation information may not be necessary.

[0051] Therefore, at least one block of method 100 can be implemented on a repetitive basis, enabling the detection of any changes in orientation and / or illumination, and the corresponding updating of illumination compensation information. This ability to adapt to body surface movement can be useful in scenarios where imaging data is acquired while the body surface can move relative to a reference axis. Thus, method 100 can facilitate timely tracking of the body surface while also compensating for any changes in illumination. Further examples of how this capability can be used are provided in more detail below.

[0052] As described above, a three-dimensional representation of the body surface can be obtained. This three-dimensional representation can be used to help determine the lighting information of the body surface.

[0053] Depending on the lighting setup, the contours and other features of a body surface can be illuminated in a certain way. For example, one side of a contour or feature may be illuminated with a greater intensity than the other side. This illumination distribution can be observed from imaging data (which can define a two-dimensional illumination map), which can be used to determine the lighting information.

[0054] Information from the 3D representation can be combined with the imaging data to determine lighting information (e.g., defining a 3D lighting map). For example, information about contours or features can be used to help determine the relative positioning between a body surface and a lighting source.

[0055] Therefore, three-dimensional representation can be used to improve the reliability and / or accuracy of determining lighting information. In other words, additional information about lighting setups can be determined by utilizing the knowledge provided by the three-dimensional representation, which might otherwise not be apparent from two-dimensional lighting diagrams alone.

[0056] Figure 2 This is a schematic diagram of a system 200 for implementing some of the methods described herein. For example, system 200 may implement method 100 described above. Therefore, reference is made to method 100 in the following description of system 200.

[0057] The system 200 of this embodiment is configured to acquire an image of an object's face 202, which is an example of a body surface. According to this embodiment, the body surface involved in method 100 includes the object's face, and the three-dimensional representation includes a three-dimensional reconstruction of the face 202.

[0058] Light source 204 (e.g., light bulb, light-emitting diode (LED), natural light source such as sunlight and / or any other light source) provides illumination to face 202.

[0059] The relative positioning of the light source 204 and the face 202 affects the distribution of illumination on the face 202. For example, the light source 204 is depicted as being above the face 202. Therefore, the forehead is closer to the light source 204 than the chin, which can cause the forehead to appear to be illuminated with greater intensity than the chin. Similarly, due to the contours or features of the face 202, certain parts of the face 202 may appear to be illuminated with greater intensity. For example, the top and front of the nose on the face 202 face the light source 204, while the base of the nose faces away from the light source 204. This could mean that the top and front of the nose can be illuminated with greater intensity than the base of the nose.

[0060] Therefore, the illumination distribution on face 202 may be non-uniform due to various factors. In some examples, the relationship between face 202 and light source 204 can affect the illumination distribution on face 202. This relationship can refer to, for example, the distance and / or positioning of face 202 relative to light source 204 (or the orientation of face 202).

[0061] As previously mentioned, movement of face 202 may affect the illumination distribution on face 202. In some examples, the characteristics of illumination source 204 may additionally or alternatively affect the illumination distribution on face 202. The properties of illumination source 204 may refer to, for example, the spatial illumination distribution provided by the illumination source (e.g., divergence and / or uniformity), its luminance, spectral content, and / or any other property of the illumination source that may affect the illumination distribution on face 202.

[0062] System 200 includes an imaging device 206 (e.g., a camera) for acquiring images (e.g., "imaging data") of a face 202 and a processing device 208 for processing the images. The imaging device 206 and the processing device 208 are communicatively coupled to each other for transmitting imaging data and / or image acquisition instructions therebetween. The imaging device 206 and the processing device 208 may be provided as part of the same device (e.g., a smart device, such as a telephone, tablet, mirror, or other device with image acquisition and processing capabilities), or they may be provided separately (e.g., the imaging device 206 may be communicatively coupled to a separate processing entity, such as a server, via a communication network). The processing device 208 may implement some of the methods described herein. At this point, the functionality of the processing device 208 will now be described with respect to method 100.

[0063] The processing device 208 is configured to obtain a three-dimensional representation of the face 202. The three-dimensional representation can be obtained or determined from imaging data of the face 202 (e.g., obtained via one or more images acquired by the imaging device 206 and / or from data provided in a memory accessible to the processing device 208).

[0064] Imaging data can be processed to determine a three-dimensional representation. For example, the three-dimensional representation can be based on a model used to reconstruct three-dimensional features revealed from two-dimensional imaging data. More than one image can be used to reconstruct three-dimensional features. For example, images taken at different angles relative to face 202 can collectively provide information for allowing the reconstruction of the three-dimensional contours and features of face 202.

[0065] The processing device 208 is also configured to obtain lighting information indicating a three-dimensional representation of the orientation of the face 202 relative to a reference axis. In this embodiment, the orientation of the face 202 is defined by a first orthogonal reference axis 210a and a second orthogonal reference axis 210b. Rotation about the first reference axis 210a can correspond to movement of the face 202 when the object rotates its head to the left and right. Rotation about the second reference axis 210b can correspond to movement of the face 202 when the object tilts its head up and down.

[0066] Illumination information for a three-dimensional representation can be obtained from imaging data of face 202 (e.g., obtained via one or more images acquired by imaging device 206 and / or data provided from memory accessible by processing device 208).

[0067] Imaging data may include indications of illumination parameters observed by imaging device 206 at a specific spatial location on face 202. Illumination parameters may refer to, for example, intensity values ​​and / or spectral content (e.g., red, green, and blue intensity levels) observed by imaging device 206 at a specific location. Illumination parameters observed by imaging device 206 may depend on factors such as the luminance and / or spectral content of illumination source 204, the distance between illumination source 204 and the specific spatial location on face 202, reflectance and / or scattering at the specific spatial location, and other factors.

[0068] The illumination parameters can be indicated by pixel intensity values ​​within the imaging data corresponding to a specific spatial location on face 202. For example, face 202 can be illuminated such that a spatial location on face 202 is observed to have a higher intensity than other spatial locations on the surface. As previously stated, for Figure 2 The specific lighting setup shown may have higher intensity at the top or front of the forehead and / or nose compared to the base of the chin and / or nose. Pixels within imaging data corresponding to spatial locations such as the top or front of the forehead and / or nose may have higher intensity values ​​than other pixels corresponding to other spatial locations such as the base of the chin and / or nose. Therefore, information derived from the imaging data can be used to determine or estimate illumination parameters for spatial locations on the face 202.

[0069] The illumination parameters can be time-dependent, meaning they can vary as a function of time (e.g., due to changes in illumination level and / or due to movement of face 202 relative to illumination source 204). Image sequences of face 202 can be used to identify such changes in the illumination parameters by comparing the image sequences.

[0070] The processing device 208 is configured to determine lighting compensation information to compensate for apparent lighting changes in the lighting information.

[0071] based on Figure 2 The lighting setup shown allows the lighting compensation information to take into account the lighting distribution at different spatial locations on the face 202. This means that when an image of the face 202 is acquired, the lighting compensation factor is applied to the lighting parameters, resulting in the normalization of any apparent non-uniformity in the lighting distribution in the image. In other words, the lighting compensation information is configured to normalize lighting variations in the image.

[0072] In one embodiment, determining the lighting compensation information includes determining a normalized map representing multiple spatial locations of a body surface (e.g., face 202). Each spatial location may be associated with a lighting compensation factor to be applied to the corresponding spatial location in the image to correct for lighting variations in the image.

[0073] The processing device 208 is configured to use illumination compensation information to compensate for illumination changes in the image of the face 202.

[0074] Therefore, the processing device 208 can use information that can be analyzed (e.g., from imaging data and / or from any other information stored in a database) to determine whether any lighting changes exist for a given lighting setting. In the image of face 202, any such lighting changes can be compensated for (e.g., normalized) using lighting compensation information.

[0075] Various applications of the methods and apparatus described herein are envisioned. For example, based on Figure 2 The system 200 shown can determine information about a face 202 from an image. Such information can be used, for example, for characterizing facial skin. As previously mentioned, topics of interest in the less conspicuous fields of measurement and monitoring relate to skin sensing for personal care and health applications. The methods and apparatus described herein enable the determination of certain information about facial skin that takes into account any changes in illumination under certain lighting conditions.

[0076] Therefore, objects using system 200 can have a certain level of confidence, meaning that the images captured by imaging device 206 and used for a specific personal care or health application are robust and reliable enough to allow objects to establish trust in system 200. Examples of personal care or health applications include: characterizing skin papules, determining the health status of skin or tissue beneath the skin surface, characterizing blood perfusion, etc. Any algorithm configured to characterize the health status of skin or tissue beneath the skin surface can make decisions based on some erroneous assumptions about the illumination distribution. For example, erroneous assumptions about the illumination distribution may lead to incorrect classification or characterization of the health status of the skin or tissue beneath the skin. An example could be characterizing skin papules. Incorrect characterization of skin papule types (e.g., caused by erroneous assumptions about the illumination distribution) may lead to recommending incorrect treatment plans for skin papule types.

[0077] Therefore, the embodiments described herein can facilitate non-obtrusive measurements of the skin to determine the processes of action as part of a personal care or health application. Home users may find the embodiments described herein more reliable and / or robust for imaging, where imaging data is used to make certain decisions for specific applications (e.g., related to particular personal care or health applications), compared to systems that do not take into account the uncontrolled environment of the home.

[0078] The apparatus and methods described herein can be used in uncontrolled environments while avoiding erroneous measurements that would otherwise lead to unacceptable or unreliable results for the object. Therefore, home users may have a degree of confidence that any personal care or health treatment based on such results is unlikely or improbable to be impaired by any erroneous measurements caused by lighting settings.

[0079] Furthermore, by considering the orientation of face 202 when determining lighting compensation information, objects can have greater confidence that any movement of their face 202 will not adversely affect the results. Similarly, objects can find the system 200 easier to use because they do not need to worry about the orientation of their face 202 when acquiring an image of it.

[0080] although Figure 2 A face 202 is depicted, but system 200 is capable of compensating for lighting variations on any type of body surface. Therefore, the methods and apparatus described herein can be used in a variety of applications beyond facial skin characterization. For example, the skin of other parts of the body can be characterized, for instance, for certain personal care or health treatment programs.

[0081] Further unclaimed applications can be extended beyond skin features, for example, to ensure that the image of any particular body surface (including inanimate bodies or objects) takes into account lighting settings and / or the orientation of the body surface, so that any subsequent use or processing of the image will not be adversely affected by the lighting settings.

[0082] Figure 3 A method 300 for improving imaging in certain lighting settings is illustrated (e.g., a computer-implemented method). Method 300 may include or reference information about... Figure 1 Certain boxes are described. Method 300 can be implemented by certain devices or systems described herein, such as those concerning... Figure 2 The system described in System 200 is implemented as follows, for ease of reference. Figure 2 System 200. Although Figure 2 This relates to systems for characterizing facial skin, but the method 300 described below can be used to improve imaging of any body surface or object surface, whether or not it is alive.

[0083] In one embodiment, method 300 includes obtaining imaging data of a body surface at block 302 (e.g., by causing imaging device 206 to capture an image of the body surface). This imaging data can be used to obtain or determine a three-dimensional representation of the body surface. Therefore, method 300 also includes determining a three-dimensional representation based on the obtained imaging data. Method 300 includes determining the orientation of the body surface based on analysis of a three-dimensional reconstruction of the body surface derived from the imaging data of the body surface. The determination of the three-dimensional representation can be performed as described below.

[0084] In one embodiment, determining a three-dimensional representation based on imaging data includes, at block 304, determining whether a body surface can be identified. For example, method 300 includes checking 306 a memory storing a database of any previously identified body surfaces.

[0085] If no body surface is identified (i.e., "No" in box 306), method 300 includes determining a three-dimensional representation based on the imaging data at box 308. This three-dimensional representation may be referred to as a three-dimensional (3D) map or a 3D contour An, where "n" refers to the nth image or the most recent image obtained from box 302. In the example where the body surface is a face, the three-dimensional representation may be referred to as a 3D face map or a 3D face contour.

[0086] Method 300 also includes, at block 310, causing a three-dimensional representation (i.e., An) associated with the body surface to be stored in a memory (which may be local memory on the smart device and / or other user device, or in a server or online / cloud storage communicatively coupled to the smart device, user device, and / or imaging device 206).

[0087] If a body surface is identified (i.e., “Yes” in box 306), method 300 includes: at box 312 obtaining a three-dimensional representation associated with the identified body surface from memory (e.g., the same memory used to store the three-dimensional representations obtained via boxes 308 and 310). Since this three-dimensional representation is previously obtained, the 3D image or 3D contour An-1 obtained from memory is associated with the previously obtained image or “n-1” image.

[0088] Determining the 3D representation using boxes 308 and 310 utilizes computational resources such as processing power and time. By using body surface recognition and retrieving a previously determined 3D map or 3D contour An-1 from memory, fewer computational resources (e.g., processing power and / or time) can be used when implementing method 300. From the user's perspective, if a 3D representation of the body surface has already been determined (and retrieved from memory), they can find the representation of the body surface faster.

[0089] Then, the three-dimensional representation obtained through boxes 308 and 310 or 312 is used in other boxes of method 300, which will be described in more detail below.

[0090] Once the three-dimensional representation of the body surface is determined, method 300 also includes obtaining lighting information of the three-dimensional representation at box 316. The lighting information can indicate the orientation of the body surface relative to a reference axis.

[0091] In one embodiment, imaging device 206 may acquire at least one additional image of the body surface at frame 314 to obtain updated information about lighting settings and / or orientation. If the body surface has moved due to the acquisition of the image at frame 302, the additional image may provide information indicating the orientation of the body surface relative to a reference axis.

[0092] Therefore, the real-time orientation (or at least timestamp orientation) of the body surface can be determined based on the imaging data obtained at box 308 (e.g., for tracking purposes). If the body surface has not moved or is unlikely to have moved, illumination information can be obtained at box 316 using the same image previously obtained (e.g., at box 302). The illumination information can be referred to as illumination map Fn, where 'n' refers to the nth image obtained at box 314.

[0093] The orientation of the body surface can be determined based on information derived from imaging data obtained at one or both of boxes 302 and 314. For example, analysis of the imaging data when a three-dimensional representation is obtained can provide information about the orientation of the body surface. Additionally or alternatively, analysis of the imaging data when illumination information is obtained can provide information about the orientation of the body surface. For example, a specific distribution of illumination can indicate the orientation of the body surface.

[0094] In one embodiment, an imaging data sequence can be obtained, which can be used, for example, to determine whether the orientation of the body surface has changed and / or whether the lighting settings have changed since previous imaging data were obtained. To determine whether a change has occurred, method 300 includes, at block 318, comparing the lighting information Fn of the nth (i.e., the latest) image with the lighting obtained for the (n-1)th (i.e., the previous) image.

[0095] In one embodiment, determining illumination information from imaging data includes: at block 318, comparing the imaging data with previously acquired imaging data to determine whether imaging data has been acquired in a previous time frame (in other words, whether similar or indistinguishable imaging data has been acquired previously). If imaging data has not been acquired previously (i.e., "No" at block 318), then method 300 includes generating illumination information Bn from the imaging data (i.e., for the nth image) at block 320, and storing the illumination information associated with the imaging data in a memory (which may be the same as or different from the memory described above) at block 322.

[0096] If imaging data has been previously obtained (i.e., “Yes” in box 318), then method 300 includes obtaining illumination information Bn-1 associated with the imaging data (i.e., for the previously obtained (n-1)th image) from memory at box 324.

[0097] Determining lighting information according to boxes 320 and 322 uses computational resources such as processing power and time. By retrieving previously determined lighting information Bn-1 from memory (e.g., according to box 324), fewer computational resources (e.g., processing power and / or time) can be used when implementing method 300. From the user's perspective, if the lighting information has been previously determined (and retrieved from memory), they may find that determining the lighting information is faster.

[0098] In one embodiment, the illumination information defined in blocks 320 and 322 or 324 includes a three-dimensional illumination map. The three-dimensional illumination map can be generated using imaging data obtained at block 314 to map illumination parameters of spatial locations on a body surface to corresponding spatial locations in a three-dimensional representation. Therefore, the three-dimensional illumination map can include information about the contours and / or features of the body surface, as well as corresponding illumination information for these contours and / or features.

[0099] In one embodiment, a three-dimensional illumination map represents multiple spatial locations on a body surface. Each spatial location of the illumination map may be associated with an illumination parameter indicating at least one of the following: the properties of the illumination source, and the relative position of the illumination source with respect to the spatial location.

[0100] In some embodiments, the illumination parameters for the spatial location of the body surface can be based on analysis of the acquired imaging data. This analysis may take into account the orientation of the body surface.

[0101] In some embodiments, the properties of a lighting source may include at least one of the following: the luminance of the lighting source; and the divergence of the lighting source. In some embodiments, the characteristics of a lighting source may refer to luminance, spatial illumination distribution, divergence, spectral content, and / or any other characteristics that may affect the illumination distribution provided by the lighting source.

[0102] Based on knowledge of illumination parameters at a specific spatial location in the illumination map and the properties of the illumination source, it is possible to determine the relative position of the illumination source with respect to the spatial location. In other words, the analysis of the illumination map can be used to estimate the relationship between a body surface and the illumination source. For example, method 300 may include estimating certain properties of the illumination source and / or the relationship between the body surface and the illumination source. This estimation may refer to, for example, estimating the orientation, brightness, divergence, etc., of the illumination source using numerical computation through imaging data (e.g., as obtained at box 314), which may also involve using information about the orientation of the body surface to refine or detail the estimation. For example, numerical computation can be performed by implementing a ray-tracing method.

[0103] Method 300 further includes determining illumination compensation information at box 326 (e.g., as mentioned in box 104 of method 100). The illumination compensation information may be referred to as the normalized graph Nn of the nth image. In one embodiment, box 326 can be implemented to determine the illumination compensation information by identifying any apparent illumination changes from the illumination information Bn or Bn-1.

[0104] As mentioned in box 106 of method 100, method 300 also includes using illumination compensation information to compensate for illumination variations in an image of the body surface.

[0105] In this embodiment, method 300 includes obtaining an image of the body surface in box 328.

[0106] Method 300 also includes using illumination compensation information at box 330 to compensate for illumination variations in the image (e.g., as obtained at box 328). For example, a normalized map Nn can provide a mapping between an illumination compensation factor for a specific spatial location in the map Nn and the corresponding spatial location of the apparent body surface in the image. The illumination compensation factor can then be applied to the illumination parameters at the corresponding spatial location to compensate for any illumination variations in the appearance of the image. Any illumination variations in the image can be normalized so that the image can be used or processed for a specific application, while addressing any problems that may affect the use or processing of the image for a specific application caused by the illumination settings and / or orientation of the body surface.

[0107] The embodiments described herein can enable the recognition of a specific object (e.g., a user's) face (or any other body surface or object surface) and / or the recognition of a specific lighting setting. This recognition can avoid or reduce the need for repeated, unnecessary measurements, such as when the same object is being used with system 200 and / or when the ambient light source remains unchanged between sessions. Certain testing procedures can be implemented to avoid repeating certain measurements. For example, by performing face (or body surface) recognition, a 3D representation of the object (or body / object) can be stored in a database, and the stored 3D representation can be retrieved in subsequent sessions. Furthermore, by obtaining imaging data (e.g., capturing a single frontal image), any changes in ambient lighting can be estimated, and in the absence of significant changes, lighting information can be retrieved from the database (which can save time and / or processing resources). There may be cases where the same object and the same ambient lighting are detected; in such cases, lighting compensation information (e.g., a normalized map) can be retrieved without implementing certain blocks of method 100 (e.g., it may not be necessary to implement certain time / processing-intensive 3D mapping / lighting information determination blocks).

[0108] As previously described, the embodiments described herein enable the determination of certain characteristics of an illumination source. Certain information can be acquired to facilitate this determination. For example, some methods described herein may involve the calibration of system 200 (e.g., to facilitate a more accurate and / or reliable characterization of body surfaces). In one example, a calibration object can be imaged according to certain blocks of methods 100, 300. Imaging device 206 may have certain sensitivity, imaging characteristics, and / or other characteristics that may be affected by the hardware and / or software used to acquire and process imaging data. For example, the imaging characteristics of imaging device 206 may be unknown to processing device 208. By using calibration objects of other shapes, such as spheres or of known sizes, system 200 is able to make improved determinations of three-dimensional representations (e.g., to ensure the three-dimensional representation has the correct dimensions). In another example, the spectral content from illumination source 204 can be determined by using a calibration object with known surface characteristics, such as spectral reflectance. By measuring the spectral intensity content of reflected illumination, system 200 is able to determine the spectral content of the illumination source, which can improve the determination of illumination information.

[0109] Figure 4 An apparatus 400 is shown, which can be used to implement certain methods described herein, such as method 100 and / or method 300. The apparatus 400 may include components having... Figure 2 The module of the function corresponding to the features described in the system 200, such as its processing device 208.

[0110] Device 400 includes processing circuitry 402 (e.g., which can be generated by...). Figure 2The processing device 208 is provided. The processing circuit 402 includes an acquisition module 404 configured to obtain a three-dimensional representation of a body surface. The acquisition module 404 is also configured to acquire lighting information of the three-dimensional representation. The lighting information may indicate the orientation of the body surface relative to a reference axis. The processing circuit 402 also includes a determination module 406 configured to determine lighting compensation information to compensate for apparent lighting variations in the lighting information. The processing circuit 402 also includes a correction module (408) configured to use the lighting compensation information to compensate for lighting variations in the image of the body surface.

[0111] Figure 5 An apparatus 500 is shown, which can be used to implement certain methods described herein, such as method 100 and / or method 300. The apparatus 500 may include components having... Figure 2 The features described in system 200 correspond to functional modules such as its processing unit 208 and / or Figure 4 Device 400.

[0112] The device 500 includes a processing circuit 502, which includes... Figure 4 The processing circuit 402. The device 500 also includes an imaging module 504, which is configured to enable the imaging device (e.g., Figure 2 The imaging device 206 acquires images (or any imaging data mentioned earlier) of the body surface. For example, the imaging module 504 can provide instructions for the imaging device 206 to acquire images or imaging data, for use in certain blocks of the methods described herein and / or functional modules of the apparatus described herein.

[0113] Figure 6 A tangible machine-readable medium 600 is shown storing instructions 602, which, when executed by at least one processor 604, cause at least one processor 604 to implement certain methods described herein (such as method 100 and / or method 300). Instructions 602 include instructions 606 for obtaining a three-dimensional representation of a body surface. Instructions 606 also obtain illumination information of the three-dimensional representation. The illumination information may indicate the orientation of the body surface relative to a reference axis. Instructions 602 also include instructions 608 for determining illumination compensation information to compensate for illumination variations apparent from the illumination information. Instructions 602 also include instructions 610 for using the illumination compensation information to compensate for illumination variations in an image of the body surface.

[0114] As previously stated, the embodiments may have utility beyond facial skin characterization. For example, the embodiments may have utility for characterizing skin on any part of the body.

[0115] Although the invention has been described and illustrated in detail in the accompanying drawings and the foregoing description, such description and illustration should be considered illustrative or exemplary rather than restrictive; the invention is not limited to the disclosed embodiments.

[0116] One or more features described in one embodiment may be combined with or replace features described in another embodiment. For example... Figure 1 and Figure 3 Methods 100 and 300 can be based on... Figure 2 , Figure 4 and Figure 5 The features described in system 200, device 400, and 500 are modified, and vice versa.

[0117] The embodiments in this disclosure may be provided as a method, a system, or as a combination of machine-readable instructions and processing circuitry. Such machine-readable instructions may be included on a non-transient machine (e.g., computer) readable storage medium (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-readable program code thereon or thereon.

[0118] This disclosure is described with reference to flowchart illustrations and block diagrams of methods, apparatus, and systems according to embodiments of this disclosure. Although the flowcharts above show a specific execution sequence, the execution sequence may differ from that described. A block described in one flowchart may be combined with a block in another flowchart. It should be understood that each block in a flowchart and / or block diagram, and combinations of blocks in flowcharts and / or block diagrams, can be implemented by machine-readable instructions.

[0119] Machine-readable instructions can be executed, for example, by a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to implement the functions described in the specification and figures. Specifically, a processor or processing circuitry, or a module thereof, can execute machine-readable instructions. Therefore, the functional modules of the apparatus 400, 500 described herein (e.g., acquisition module 404, determination module 406, correction module 408, and / or imaging module 504) and other devices can be implemented by a processor that executes machine-readable instructions stored in memory, or by a processor that operates according to instructions embedded in logic circuitry. The term "processor" should be interpreted broadly to include CPUs, processing units, ASICs, logic units, or programmable gate arrays, etc. The methods and functional modules may be executed entirely by a single processor or divided among several processors.

[0120] Such machine-readable instructions can also be stored in a computer-readable storage device that can instruct a computer or other programmable data processing device to operate in a specific mode.

[0121] Such machine-readable instructions can also be loaded onto a computer or other programmable data processing apparatus to cause the computer or other programmable data processing apparatus to perform a series of operations to produce a computer-implemented process, such that the instructions executing on the computer or other programmable apparatus implement the functions specified by the boxes in the flowchart and / or block diagram. Furthermore, the teachings herein can be implemented in the form of a computer program product stored in a storage medium and including a plurality of instructions for causing a computer device to implement the methods described in the embodiments of this disclosure.

[0122] Elements or steps described with respect to one embodiment may be combined with or replaced by elements or steps described with respect to another embodiment. By studying the drawings, disclosure, and appended claims, those skilled in the art can understand and implement variations of the disclosed embodiments in practicing the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. A single processor or other unit may implement the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not imply that combinations of these measures cannot be advantageously used. Computer programs may be stored or distributed on suitable media such as optical storage media or solid-state media, provided together with or as part of other hardware, but computer programs may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems. Any reference numerals in the claims should not be construed as limiting the scope.

Claims

1. A computer-implemented method (100) for compensating for illumination variations in an image of a body surface for use in an imaging-based skin sensing system, the method comprising: Received (102): The three-dimensional representation of the body surface; as well as Illumination information indicating the orientation of the body surface relative to a reference axis in the three-dimensional representation; Determine (104) lighting compensation information, the lighting compensation information being configured to normalize apparent lighting variations in the lighting information, wherein determining the lighting compensation information includes: determining a normalized map representing a plurality of spatial locations on the body surface, wherein each spatial location is associated with a lighting compensation factor, the lighting compensation factor being applied to the corresponding spatial location in the image to correct the lighting variations in the image; Based on the analysis of the three-dimensional reconstruction of the body surface derived from the imaging data of the body surface, the orientation of the body surface is determined; and The illumination compensation information (106) is used to compensate for the illumination variation in the image of the body surface to provide a compensated image for use in the imaging-based skin sensing system, wherein the illumination variation is compensated by uniformly distributing illumination across each spatial location on the body surface.

2. The method of claim 1, wherein the compensated image includes information used by the imaging-based skin sensing system to characterize the skin on the body surface.

3. The method of claim 1 or 2, wherein the lighting information comprises a three-dimensional lighting map representing a plurality of spatial locations of the body surface, wherein each spatial location of the lighting map is associated with a lighting parameter indicating at least one of: attributes of a lighting source; and the relative position of the lighting source with respect to the spatial location.

4. The method of claim 3, further comprising determining the lighting information by: Obtain imaging data of the body surface described in (314); and The illumination parameters of the spatial location of the body surface are estimated based on the analysis of the obtained imaging data taking into account the orientation of the body surface.

5. The method of claim 3, wherein the property of the lighting source includes at least one of the following: the brightness of the lighting source; and the divergence of the lighting source.

6. The method according to any one of claims 1, 2, 4 and 5, wherein determining (326) the lighting compensation information comprises: Obtain imaging data of the body surface described in (314); The lighting information is determined based on the obtained imaging data; as well as Identify any apparent changes in lighting in the lighting information to determine the lighting compensation information.

7. The method of claim 6, wherein determining the illumination information from the imaging data comprises: The imaging data is compared with previously acquired imaging data (318) to determine whether the imaging data was acquired in a previous time frame. If the imaging data has not been previously obtained, the method includes generating (320) the illumination information from the imaging data, and causing the illumination information associated with the imaging data to be stored (322) in a memory, and If the imaging data has been previously acquired, the method includes obtaining (324) the illumination information associated with the imaging data from the memory.

8. The method according to any one of claims 1, 2, 4, 5, and 7, wherein obtaining the three-dimensional representation of the body surface comprises: Obtain imaging data of the body surface described in (302); as well as The three-dimensional representation is determined based on the obtained imaging data.

9. The method of claim 8, wherein determining the three-dimensional representation from the imaging data comprises: Whether the body surface (304) can be identified is determined by examining the memory of a database containing any previously identified body surfaces. If the body surface is not identified, the method includes determining (308) the three-dimensional representation from the imaging data, and such that (310) the three-dimensional representation associated with the body surface is stored in a memory, and If the body surface is identified, the method includes obtaining (312) the three-dimensional representation associated with the identified body surface from the memory.

10. The method according to any one of claims 1, 2, 4, 5, 7 and 9, wherein the body surface includes the face of the object, and the three-dimensional representation includes a three-dimensional reconstruction of the face.

11. The method according to any one of claims 1, 2, 4, 5, 7, and 9, wherein using the lighting compensation information to compensate for the lighting changes in the image of the body surface comprises: Obtain the image of the body surface (328); as well as The lighting changes in the image are compensated (330) using the lighting compensation information.

12. An apparatus (400) for compensating for illumination variations in an image of a body surface for use in an imaging-based skin sensing system, the apparatus comprising a processing circuit (402) including: Obtain the module (404), which is configured to obtain: The three-dimensional representation of the body surface; as well as Illumination information indicating the orientation of the body surface relative to a reference axis in the three-dimensional representation; The module (406) is determined as follows: Illumination compensation information is determined, which is configured to normalize apparent illumination variations in the illumination information, wherein the illumination compensation information is determined by determining a normalized map representing multiple spatial locations of the body surface, wherein each spatial location is associated with an illumination compensation factor, which is applied to the corresponding spatial location of the image to correct the illumination variations in the image; The orientation of the body surface is determined based on the analysis of the three-dimensional reconstruction of the body surface derived from the imaging data of the body surface; as well as A correction module (408) is configured to use the illumination compensation information to compensate for the illumination variation in the image of the body surface to provide a compensated image for use in the imaging-based skin sensing system, wherein the illumination variation is compensated by uniformly distributing illumination across each spatial location on the body surface.

13. The apparatus (500) according to claim 12, further comprising: An imaging module (504) is configured to enable an imaging device (206) to acquire the image of the body surface.

14. A tangible machine-readable medium (600) storing instructions (602) for compensating for illumination variations in an image of a body surface for use in an imaging-based skin sensing system, the instructions causing the at least one processor (604) to: Obtained (606): The three-dimensional representation of the body surface; and Illumination information indicating the orientation of the body surface relative to a reference axis in the three-dimensional representation; Determine (608) lighting compensation information, which is configured to normalize apparent lighting variations in the lighting information, wherein the lighting compensation information is determined by determining a normalized map representing a plurality of spatial locations on the body surface, wherein each spatial location is associated with a lighting compensation factor, which is applied to the corresponding spatial location in the image to correct the lighting variations in the image; The orientation of the body surface is determined based on the analysis of the three-dimensional reconstruction of the body surface derived from the imaging data of the body surface; as well as The illumination compensation information (610) is used to compensate for the illumination variation in the image of the body surface to provide a compensated image for use in the imaging-based skin sensing system, wherein the illumination variation is compensated by uniformly distributing illumination across each spatial location on the body surface.

Images