Multispectral / hyperspectral two-dimensional image processing

Abstract

An apparatus (100) comprising one or more processors (102) configured to acquire a multi / hyperspectral two-dimensional image of an object at a respective wavelength is provided. For at least one pixel of the image corresponding to a first point on a surface of the object, a set of intensity values for the at least one pixel is compared to a characteristic curve to determine a similarity measure. A first angle of the first point is estimated from the similarity measure, or a correction is applied to the image at the first point using the similarity measure. The characteristic curve is a difference between a spectrum of at least one second point on the surface of the object at a second angle relative to a plane of the image and a spectrum of at least one third point on the surface of the object at a third angle relative to the plane of the image.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to methods and apparatuses for use in multi / hyperspectral two-dimensional image processing. BACKGROUND

[0002] Conventional RGB cameras capture visible light using only three channels: a red channel, a green channel, and a blue channel. Multi / hyperspectral cameras capture the electromagnetic spectrum, both visible and non-visible to the human eye, at a large number of wavelengths (typically about 10 wavelengths for multispectral imaging, or more than 100 wavelengths for hyperspectral imaging). These multi / hyperspectral cameras can thus reveal properties of the imaged object that are not observable by the human eye.

[0003] In particular, in the field of skin imaging, such multi / hyperspectral cameras can be employed to estimate the concentration of chromophores present in the skin (e.g. melanin, carotenoids, water, lipids, etc.), which is not possible using conventional RGB cameras. The estimated concentration of chromophores in the skin can provide information about skin health, but more generally can be indicative of lifestyle or systemic health. Of particular interest is the processing of multi / hyperspectral images of large skin surfaces, such as a human face. Tracking multi / hyperspectral images over time can reveal specific local changes in chromophore concentration, which can be attributed to, for example, changes in lifestyle. The process of estimating chromophore concentration from the spectrum is called spectral unmixing. For example, Figure 1 Calibrated hyperspectral images acquired at six different wavelengths (448 nm, 494 nm, 610 nm, 669 nm, 812 nm, and 869 nm) are shown. In Figure 1 In the figure, the pixel position is provided by the x- and y-axes, and the grey scale illustrates the reflectance. The whiter the pixel, the higher the intensity, and thus the higher the reflectance.

[0004] In order to allow correct estimation of chromophores of a curved surface (e.g. skin, such as a human face), it is necessary to know the angle at which the curved surface is positioned with respect to the camera. This angle is necessary because light is absorbed, scattered, and reflected in different layers of the curved surface, leading to a strong angle dependence. In order to obtain such angle data, typically one or more three-dimensional (3D) cameras (e.g. time-of-flight cameras) are installed in order to register 3D images and derive an angle map.

[0005] However, the use of 3D cameras in a multi / hyperspectral setup is both cumbersome and expensive, as additional cameras are needed and the two-dimensional (2D) multi / hyperspectral images need to be combined with the 3D images, which requires the use of advanced image registration techniques and accurate mapping of 2D images to 3D images. These limitations are particularly evident when moving to consumer-type implementations. The use of one or more additional cameras hinders miniaturization. SUMMARY

[0006] As noted above, the limitations of the prior art are that in order to acquire information suitable for observing properties of the object (e.g. concentration of chromophores in the skin), additional cameras are needed and 2D multi / hyperspectral images need to be combined with 3D images, which requires advanced image registration techniques and accurate mapping of 2D images to 3D images. Therefore, improvements aimed at addressing these limitations would be valuable.

[0007] According to a first aspect, therefore, there is provided an apparatus for estimating a first angle of a first point on a surface of an object from multi / hyperspectral two-dimensional images of the object at respective wavelengths, or applying a correction to the multi / hyperspectral two-dimensional images. The apparatus comprises one or more processors configured to acquire multi / hyperspectral two-dimensional images of the object at respective wavelengths. The multi / hyperspectral two-dimensional images at the respective wavelengths are formed of a plurality of pixels. Each pixel has a set of intensity values corresponding to a light intensity value for each of a plurality of light wavelengths. The one or more processors are configured to, for at least one of the plurality of pixels corresponding to the first point on the surface of the object, compare the set of intensity values for the at least one pixel to a characteristic curve for the object to determine a similarity measure of the set of intensity values to the obtained characteristic curve. The one or more processors are configured to estimate the first angle of the first point on the surface of the object corresponding to the at least one pixel from the determined similarity measure, or apply a correction to the multi / hyperspectral two-dimensional images at the first point on the surface of the object using the determined similarity measure. The characteristic curve is indicative of a difference between a spectrum of at least one second point on the surface of the object at a second angle relative to a plane of the images and a spectrum of at least one third point on the surface of the object at a third angle relative to the plane of the images.

[0008] In some embodiments, the first angle can be relative to a plane of the images. In some embodiments, the characteristic curve can characterise how the object reflects or absorbs light as a function of angle.

[0009] In some embodiments, the characteristic curve can be indicative of a difference between an average spectrum of at least two second points on the surface of the object at a second angle relative to a plane of the images and a spectrum of at least two third points on the surface of the object at a third angle relative to the plane of the images.

[0010] In some embodiments, the third angle can be a known angle different from or significantly different from the second angle. In some embodiments, the second angle can be approximately 0 degrees and / or the third angle can be an angle in the range from 45 degrees to 90 degrees.

[0011] In some embodiments, at least one second point on the surface of the object may include at least one brightest point on the surface of the object, and / or, at least one third point on the surface of the object may include at least one darkest point on the surface of the object.

[0012] In some embodiments, at least one second point on the surface of an object can be identified by using landmark detection to detect at least one second point on the surface of the object that is at a second angle relative to the plane of the image, and / or, at least one third point on the surface of an object can be identified by using landmark detection to detect at least one third point on the surface of the object that is at a third angle relative to the plane of the image.

[0013] In some embodiments, the characteristic curve can be predetermined using at least one other multi / hyperspectral two-dimensional image of the same type of object, or the characteristic curve can be determined using a multi / hyperspectral two-dimensional image of the object.

[0014] In some embodiments, one or more processors may be configured to: for at least one pixel of a plurality of pixels corresponding to at least one other first point on the surface of an object, compare a set of intensity values ​​for said at least one pixel with a characteristic curve for the object to determine a similarity measure between the set of intensity values ​​and the obtained characteristic curve. In these embodiments, one or more processors may be configured to: estimate at least one other first angle on the surface of the object corresponding to said at least one pixel based on the determined similarity measure, and derive an angle map including the estimated first angle and the estimated at least one other first angle.

[0015] In some embodiments, the characteristic curves for an object may include (or be selected from) a set of characteristic curves for corresponding sets of second and third angles.

[0016] In some embodiments, the spectrum of at least one second point may include a reflection spectrum or an absorption spectrum, the reflection spectrum indicating the portion of light reflected from the object at at least one second point on the surface of the object at a second angle relative to the plane of the image, and the absorption spectrum indicating the portion of light absorbed by the object at at least one second point on the surface of the object at a second angle relative to the plane of the image; and / or, the spectrum of at least one third point may include a reflection spectrum or an absorption spectrum, the reflection spectrum indicating the portion of light reflected from the object at at least one third point on the surface of the object at a third angle relative to the plane of the image, and the absorption spectrum indicating the portion of light absorbed by the object at at least one third point on the surface of the object at a third angle relative to the plane of the image.

[0017] In some embodiments, the object may be skin, and one or more processors may be configured to determine the concentration of chromophores in the skin based on a multi / hyperspectral two-dimensional image of the skin, or based on a multi / hyperspectral two-dimensional image with correction applied, using an estimated first angle.

[0018] According to a second aspect, a method is provided for estimating a first angle of a first point on the surface of an object or applying a correction to the multi / hyperspectral two-dimensional image of the object at a corresponding wavelength. The method includes acquiring a multi / hyperspectral two-dimensional image of the object at a corresponding wavelength, wherein the multi / hyperspectral two-dimensional image at the corresponding wavelength is formed by a plurality of pixels, each pixel having a set of intensity values ​​corresponding to light intensity values ​​for each of a plurality of light wavelengths. The method includes, for at least one pixel of the plurality of pixels corresponding to the first point on the surface of the object, comparing the set of intensity values ​​for said at least one pixel with a characteristic curve for the object to determine a similarity measure between the set of intensity values ​​and the obtained characteristic curve. The method includes estimating the first angle of the first point on the surface of the object corresponding to said at least one pixel based on the determined similarity measure, or applying a correction to the multi / hyperspectral two-dimensional image at the first point on the surface of the object using the determined similarity measure. The characteristic curve indicates the difference between the spectrum of at least one second point on the surface of the object at a second angle relative to a plane of the image and the spectrum of at least one third point on the surface of the object at a third angle relative to a plane of the image.

[0019] In some embodiments, the first angle may be relative to a plane of the image. In some embodiments, the characteristic curve may characterize how an object reflects or absorbs light according to an angle. In some embodiments, the third angle may be a known angle that is different from or significantly different from the second angle.

[0020] According to a third aspect, an apparatus is provided for determining a characteristic curve for use when estimating a first angle of a first point on the surface of an object based on a multi / hyperspectral two-dimensional image of the object at a corresponding wavelength or wavelength, or when applying correction to the multi / hyperspectral two-dimensional image. The apparatus includes one or more processors configured to acquire a first spectrum of at least one second point on the surface of the object at a second angle relative to a plane of the image, acquire a second spectrum of at least one third point on the surface of the object at a third angle relative to a plane of the image, and determine the characteristic curve as the difference between the first and second spectra.

[0021] In some embodiments, the first angle may be relative to a plane of the image. In some embodiments, the characteristic curve may characterize how an object reflects or absorbs light according to an angle. In some embodiments, the third angle may be a known angle that is different from or significantly different from the second angle.

[0022] According to a fourth aspect, a method is provided for determining a characteristic curve used when estimating a first angle of a first point on the surface of an object based on a multi / hyperspectral two-dimensional image of the object at a corresponding wavelength or wavelength, or when applying correction to a multi / hyperspectral two-dimensional image. The method includes acquiring a first spectrum of at least one second point on the surface of the object at a second angle relative to a plane of the image, acquiring a second spectrum of at least one third point on the surface of the object at a third angle relative to a plane of the image, and determining the characteristic curve as the difference between the first and second spectra.

[0023] In some embodiments, the first angle may be relative to a plane of the image. In some embodiments, the characteristic curve may characterize how an object reflects or absorbs light according to an angle. In some embodiments, the third angle may be a known angle that is different from or significantly different from the second angle.

[0024] According to a fifth aspect, a computer program product including a computer-readable medium is provided. The computer-readable medium has computer-readable code embodied therein. The computer-readable code is configured to cause the computer or processor to perform the previously described methods when executed by a suitable computer or processor.

[0025] According to the aspects and embodiments described above, the limitations of the prior art are resolved. Specifically, according to the aspects and embodiments described above, the need for additional cameras and advanced image registration techniques for combining 2D multi / hyperspectral images with 3D images to obtain information suitable for the properties of the observed object (e.g., the concentration of chromophores in the skin) is overcome. A 3D imaging modality in multi / hyperspectral imaging is no longer required. According to the aspects and embodiments described above, angular information can be reliably derived directly from multi / hyperspectral images without the use of additional camera signals. This is achieved by (deriving and) utilizing characteristic curves. Alternatively, the multi / hyperspectral images are preprocessed using characteristic curves to compensate for the influence of angle on the spectrum.

[0026] Therefore, a useful technique is provided for estimating the angles of points on the surface of an object or applying corrections to multi / hyperspectral two-dimensional images of the object based on multi / hyperspectral two-dimensional images of the object at corresponding wavelengths. A useful technique is also provided for determining characteristic curves for use in such estimations or corrections.

[0027] These and other aspects will become apparent from the embodiments described below and will be illustrated with reference to the embodiments(s) described below. Attached Figure Description

[0028] Exemplary embodiments will now be described by way of example only with reference to the following figures, wherein:

[0029] Figure 1 Here are examples of hyperspectral images acquired at six different light wavelengths;

[0030] Figure 2 This is a schematic diagram of the device according to an embodiment;

[0031] Figure 3 This is a flowchart illustrating a method according to an embodiment;

[0032] Figure 4 This is a flowchart illustrating a method according to an embodiment;

[0033] Figure 5 This is an example schematic diagram of a hyperspectral image along with hyperspectral plots of a slice from that image;

[0034] Figure 6 Here is an example of how a calibrated hyperspectral image changes with distance;

[0035] Figure 7 Here are examples of characteristic curves; and

[0036] Figure 8 This is an example of an exported angle plot. Detailed Implementation

[0037] As noted above, this document provides a technique for estimating the angles of points on the surface of an object from a multi / hyperspectral two-dimensional image of the object at corresponding wavelengths, or for applying corrections to a multi / hyperspectral two-dimensional image. In this document, the object can be of any type. In some embodiments, the object can be any object having a curved surface. In some embodiments, for example, the object can be the skin of an object or the skin of a face.

[0038] Figure 2 The illustration depicts an apparatus 100 according to an embodiment for estimating a first angle of a first point on the surface of an object from a multi / hyperspectral two-dimensional image of the object at a corresponding wavelength, or for applying correction to the multi / hyperspectral two-dimensional image. In some embodiments, the apparatus 100 may be a device (e.g., a consumer device) or an accessory (or add-on) to a device. For example, the device may be a telephone (e.g., a smartphone), a tablet computer, or any other device. Figure 2 As illustrated, device 100 includes one or more processors 102.

[0039] One or more processors 102 may be implemented in various ways, using software and / or hardware, to perform the various functions described herein. In a particular implementation, one or more processors 102 may include multiple software and / or hardware modules, each configured to perform or be used to perform one or more steps of the methods described herein. One or more processors 102 may include, for example, one or more microprocessors, one or more multi-core processors and / or one or more digital signal processors (DSPs), one or more processing units and / or one or more controllers (e.g., one or more microcontrollers) that can be configured or programmed (e.g., using software or computer program code) to perform the various functions described herein. One or more processors 102 may be implemented as a combination of dedicated hardware (e.g., amplifiers, preamplifiers, analog-to-digital converters (ADCs) and / or digital-to-analog converters (DACs)) for performing some functions and one or more processors (e.g., one or more programmed microprocessors, DSPs, and associated circuitry) for performing other functions.

[0040] In short, one or more processors 102 of device 100 are configured to acquire a multi / hyperspectral two-dimensional image of an object at a corresponding wavelength. The multi / hyperspectral two-dimensional image at the corresponding wavelength is formed by a plurality of pixels. Each pixel has a set of intensity values ​​corresponding to a light intensity value for each of the plurality of light wavelengths. The one or more processors 102 of device 100 are also configured to: for at least one pixel among the plurality of pixels corresponding to a first point on the surface of the object, compare the set of intensity values ​​for said at least one pixel with a characteristic curve for the object to determine a similarity measure between the set of intensity values ​​and the obtained characteristic curve. The one or more processors 102 of device 100 are also configured to estimate a first angle of the first point on the surface of the object corresponding to said at least one pixel based on the determined similarity measure, or to apply a correction to the multi / hyperspectral two-dimensional image at the first point on the surface of the object using the determined similarity measure.

[0041] The characteristic curve referred to in this article indicates the difference between the spectrum of at least one second point on the surface of the object at a second angle relative to the plane of the image and the spectrum of at least one third point on the surface of the object at a third angle relative to the plane of the image.

[0042] In this document, the spectrum of at least one second point may include a reflectance spectrum or an absorption spectrum, wherein the reflectance spectrum indicates the portion of light reflected from the object at at least one second point on the surface of the object at a second angle relative to the plane of the image, and the absorption spectrum indicates the portion of light absorbed by the object at at least one second point on the surface of the object at a second angle relative to the plane of the image. Alternatively or additionally, in this document, the spectrum of at least one third point may include a reflectance spectrum or an absorption spectrum, wherein the reflectance spectrum indicates the portion of light reflected from the object at at least one third point on the surface of the object at a third angle relative to the plane of the image, and the absorption spectrum indicates the portion of light absorbed by the object at at least one third point on the surface of the object at a third angle relative to the plane of the image. In this document, a reflectance spectrum may generally be understood to refer to the relative amount of reflected light according to wavelength. Similarly, in this document, an absorption spectrum may generally be understood to refer to the relative amount of absorbed light according to wavelength.

[0043] In some embodiments, one or more processors 102 of the device 100 may be configured to acquire multi / hyperspectral two-dimensional images of an object at corresponding wavelengths from an imaging sensor 104. The imaging sensor may be, for example, a camera, or more specifically, a multi / hyperspectral camera. Figure 2 As illustrated, in some embodiments, device 100 may include imaging sensor 104. Alternatively or additionally, in some embodiments, imaging sensor 104 may be external to device 100 (e.g., detached from or remote from it). For example, according to some embodiments, another device (or apparatus, such as a capture device) may include imaging sensor 104.

[0044] like Figure 2As illustrated herein, in some embodiments, device 100 may include at least one memory 106. Alternatively or additionally, in some embodiments, at least one memory 106 may be external to device 100 (e.g., separate from or remote from it). For example, according to some embodiments, another device may include at least one memory 106. In some embodiments, a hospital database may include at least one memory 106, and at least one memory 106 may be a cloud computing resource, etc. One or more processors 102 of device 100 may be configured to communicate with and / or connect to at least one memory 106. At least one memory 106 may include any type of non-transitory machine-readable medium, such as cache or system memory including volatile and non-volatile computer memory, such as random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), and electrically erasable PROM (EEPROM). In some embodiments, at least one memory 106 may be configured to store program code that can be executed by one or more processors 102 of device 100 to cause device 100 to operate in the manner described herein.

[0045] Alternatively or additionally, at least one memory 106 may be configured to store information required or generated by the methods described herein. For example, at least one memory 106 may be configured to store a multi / hyperspectral two-dimensional image of an object, a determined similarity measure, an estimated first angle of a first point on the surface of the object, a multi / hyperspectral two-dimensional image of the object with corrections applied, or any other information or any combination of information required or generated by the methods described herein. One or more processors 102 of the apparatus 100 may be configured to control at least one memory 106 to store information required or generated by the methods described herein.

[0046] like Figure 2 As illustrated herein, in some embodiments, device 100 may include at least one user interface 108. Alternatively or additionally, in some embodiments, at least one user interface 108 may be external to device 100 (e.g., detached from or remote from it). One or more processors 102 of device 100 may be configured to communicate with and / or connect to at least one user interface 108. In some embodiments, one or more processors 102 of device 100 may be configured to control at least one user interface 108 to operate in the manner described herein.

[0047] User interface 108 may be configured to render (or output, display, or provide) information required or generated by the methods described herein. For example, in some embodiments, one or more user interfaces 108 may be configured to render (or output, display, or provide) any one or more of a multi / hyperspectral two-dimensional image of an object, a determined similarity metric, an estimated first angle of a first point on the surface of the object, a multi / hyperspectral two-dimensional image of the object with corrections applied, or any other information or combination of information required or generated by the methods described herein. Alternatively or additionally, one or more user interfaces 108 may be configured to receive user input. For example, one or more user interfaces 108 may allow a user to manually input information or instructions, interact with device 100, and / or control device 100. Thus, one or more user interfaces 108 may be any one or more user interfaces that enable information rendering (or output, display, or provide) and / or enable a user to provide user input.

[0048] Therefore, user interface 108 may include one or more components. For example, one or more user interfaces 108 may include one or more switches, one or more buttons, keypads, keyboards, mice, displays or screens, graphical user interfaces (GUIs) such as touchscreens, applications (e.g., on smart devices such as tablets, smartphones, or any other smart devices), or any other visual component, one or more speakers, one or more microphones or any other audio component, one or more lights (e.g., one or more light-emitting diodes (LEDs), components for providing haptic or tactile feedback (e.g., vibration functionality or any other haptic feedback component), smart devices (e.g., smart mirrors, tablets, smartphones, smartwatches, or any other smart devices), or any other user interface or combination of user interfaces. In some embodiments, one or more user interfaces controlled to render information may be the same as one or more user interfaces that enable a user to provide user input.

[0049] like Figure 2As illustrated, in some embodiments, device 100 may include at least one communication interface (or communication circuitry) 110. Alternatively or additionally, in some embodiments, at least one communication interface 110 may be external to device 100 (e.g., detached from or remote from it). The communication interface 110 may be used to enable device 100 or components of device 100 (e.g., one or more processors 102, one or more sensors 104, one or more memories 106, one or more user interfaces 108, and / or any other component of device 100) to communicate with and / or connect to each other and / or communicate with and / or connect to one or more other components. For example, one or more communication interfaces 110 may be used to enable one or more processors 102 of device 100 to communicate with and / or connect to one or more sensors 104, one or more memories 106, one or more user interfaces 108, and / or any other component of device 100.

[0050] Communication interface 110 enables device 100 or its components to communicate and / or connect in any suitable manner. For example, one or more communication interfaces 110 may enable device 100 or its components to communicate and / or connect wirelessly, via a wired connection, or via any other communication (or data transmission) mechanism. In some wireless embodiments, for example, one or more communication interfaces 110 may enable device 100 or its components to communicate and / or connect using radio frequency (RF), Bluetooth, or any other wireless communication technology.

[0051] Figure 3 The illustration depicts a method 200 according to one embodiment, used to estimate a first angle of a first point on the surface of an object, or to apply a correction to the multi / hyperspectral two-dimensional image, based on a multi / hyperspectral two-dimensional image of the object at corresponding wavelengths. More specifically, Figure 3 The diagram illustrates the operation from earlier reference. Figure 2 The described apparatus 100 is used to method 200 for estimating a first angle of a first point on the surface of an object or applying correction to the multi / hyperspectral two-dimensional image of the object based on a multi / hyperspectral two-dimensional image of the object at a corresponding wavelength.

[0052] Figure 3 The method 200 illustrated is a computer-implemented method. (See earlier references.) Figure 2 The device 100 includes one or more processors 102. Figure 3 The method 200 illustrated in the figure can usually be derived from earlier references. Figure 2 The described device 100 is executed by one or more processors 102 or under their control.

[0053] refer to Figure 3At box 202, a multi / hyperspectral two-dimensional image of the object at the corresponding wavelength is acquired. As mentioned earlier, the multi / hyperspectral two-dimensional image at the corresponding wavelength is formed by multiple pixels. Each pixel has a set of intensity values ​​that correspond to the light intensity value for each of the multiple light wavelengths.

[0054] exist Figure 3 At box 204, for at least one pixel among a plurality of pixels corresponding to a first point on the surface of the object, the set of intensity values ​​for said at least one pixel is compared with a characteristic curve (or contour) for the object to determine a similarity measure between the set of intensity values ​​and the obtained characteristic curve. As mentioned earlier, the characteristic curve referred to herein indicates the difference between the spectrum of at least one second point on the surface of the object at a second angle relative to the plane of the image and the spectrum of at least one third point on the surface of the object at a third angle relative to the plane of the image. The characteristic curve can be predetermined. In some embodiments, the characteristic curve may be a predetermined characteristic curve stored in memory (e.g., memory 106 of device 100 or any other memory). This can be advantageous because it reduces computational complexity and avoids (manual) errors, thereby improving accuracy.

[0055] A characteristic curve can be a curve that characterizes how an object reflects or absorbs light depending on the angle. In this way, the degree to which a characteristic curve exists (or is visible) at each spatial location (x, y) can be determined. The degree to which a characteristic curve exists is a measure of angle.

[0056] More specifically, in some embodiments, the characteristic curve can characterize how an object reflects (or absorbs) light according to an angle, based on the following equation:

[0057] Reflectivity [λ,α] = Reflectivity [λ,0] + α * Characteristic curve [λ].

[0058] Where λ represents wavelength and α represents angle. Therefore, if the reflectance spectrum of an object is measured at different angles, it can be observed that the reflectance spectrum can be decomposed into the reflectance spectrum at 0 degrees and a portion linearly related to that angle. The extent to which the characteristic curve is available is then a measure of the angle. For the purpose of applying correction to multi / hyperspectral two-dimensional images, this can include approximating the reflectance [λ, 0] according to the above equation.

[0059] Therefore, in Figure 3At box 204, for at least one pixel among a plurality of pixels corresponding to a first point on the surface of the object, the set of intensity values ​​of the at least one pixel is compared with a characteristic curve for the object to determine a similarity measure between the set of intensity values ​​and the characteristic curve. In some embodiments, the set of intensity values ​​for the at least one pixel can be directly compared with the characteristic curve to determine the similarity measure. In other embodiments, one or more features can be derived from the set of intensity values ​​of the at least one pixel, and the one or more derived features can be compared with one or more corresponding features of the characteristic curve to determine the similarity measure. Examples of one or more features include, but are not limited to, the slope of the curve, the standard deviation of the curve, etc.

[0060] In this paper, the spectrum of at least one second point on the surface of the object, forming a second angle with respect to the plane of the image, can be referred to as the baseline spectrum. Furthermore, the spectrum of at least one third point on the surface of the object, forming a third angle with respect to the plane of the image, can be referred to as the edge spectrum. Therefore, the characteristic curve can be defined as the difference between the baseline spectrum and the edge spectrum.

[0061] In some embodiments, Figure 3 The similarity measure determined at box 204 may include a measure of correlation. In some embodiments, regression analysis may be used to determine the similarity measure (e.g., a measure of correlation). In one example, linear regression (or correlation) may be applied on a pixel-by-pixel basis (e.g., for each x,y pair of pixels). This can be represented using a vector representation, as follows:

[0062] a=(p T p) -1 p T d,

[0063] Where p is the characteristic curve (represented as a column vector), d is the input spectrum at position x,y (represented as a column vector), and a is the regression coefficient at position x,y, which establishes the degree to which the characteristic curve exists in the input spectrum. Therefore, the regression coefficient a is a similarity measure in this example. The operator T denotes transpose. The input spectrum at position x,y is the aforementioned set of intensity values ​​for the at least one pixel, where this set of intensity values ​​corresponds to the light intensity value for each of the multiple wavelengths λ of light.

[0064] In some embodiments, for a given x, y coordinate and multiple wavelengths λ, the input spectrum d can be directly a vector s = s[λ, x, y]. Alternatively, the input spectrum d can be baseline compensated by subtracting a baseline spectrum so that for a given x, y coordinate of multiple wavelengths λ, d[λ] = s[λ, x, y] - s[λ, x, y]. baseline[λ]. This can lead to improved robustness.

[0065] Return to Figure 3 At box 206, a first angle of a first point on the surface of an object corresponding to the at least one pixel is estimated based on a determined similarity metric, or a correction is applied to the multi / hyperspectral two-dimensional image at the first point on the object's surface using the determined similarity metric. The first angle may be relative to a plane of the image. Therefore, angle information can be derived from the multi / hyperspectral image using a characteristic curve, or the multi / hyperspectral image can be preprocessed using a characteristic curve. In some embodiments where the regression coefficient α is a similarity metric, the regression coefficient α may be a direct estimate of the first angle of the first point on the surface of the object corresponding to the at least one pixel. For example, in some embodiments, the first angle of the first point on the surface of the object corresponding to the at least one pixel may be estimated as follows:

[0066] α=min(max(c·a,0),π / 2),

[0067] Where c is a predetermined constant, and the min and max operators prevent the angle from being estimated outside the range [0, pi / 2].

[0068] In some embodiments where correction is applied to a multi / hyperspectral two-dimensional image at a first point on the surface of an object, the multi / hyperspectral image may be preprocessed to compensate for the effects of angle. In some embodiments, correction may include flattening the image. In some embodiments, preprocessing for a spectrum s[λ, x, y] of a given coordinate x, y and multiple wavelengths λ can be achieved by establishing the residuals after regression, as follows:

[0069] S preprocessed [λ,x,y]=s[λ,x,y]-0[x,y]·s characteristic [λ].

[0070] In some embodiments, the set of characteristic curves may be stored in memory (e.g., memory 106 of device 100 or any other memory). In some of these embodiments, in Figure 3 At box 204, for at least one pixel among a plurality of pixels corresponding to a first point on the surface of the object, the set of intensity values ​​for said at least one pixel can be compared with each characteristic curve in the set of characteristic curves to determine a corresponding similarity measure between the set of intensity values ​​and the characteristic curves. The characteristic curve in the set of characteristic curves that is most similar to the set of intensity values ​​(i.e., best match) is selected. For example, a best-fit technique can be used. In some embodiments, each characteristic curve in the set of characteristic curves can be stored together with its corresponding angle, for example, in the form of a lookup table. Therefore, in these embodiments, inFigure 3 At box 206, the first angle can be determined as the angle stored along with the selected characteristic curve. Alternatively, in Figure 2 At box 206, the selected characteristic curve can be used to correct the set of intensity values.

[0071] This document also provides an apparatus for determining a characteristic curve used, as described herein, when estimating a first angle of a first point on the surface of an object based on a multi / hyperspectral two-dimensional image of the object at a corresponding wavelength, or when applying correction to a multi / hyperspectral two-dimensional image, as described herein. The apparatus includes components as previously referenced. Figure 2 The description includes one or more processors, and may also include those referenced above. Figure 2 Any one or more other components of the described device 100.

[0072] In some embodiments, the means for determining the characteristic curve may be the same as the one referenced above. Figure 2 The device described is the same as the device 100. Therefore, in some embodiments, the foregoing reference is used. Figure 4 The described apparatus 100 can also be used to determine the characteristic curve referred to herein. Alternatively, in some embodiments, the apparatus for determining the characteristic curve referred to herein may be an apparatus for estimating a first angle of a first point on the surface of an object based on a multi / hyperspectral two-dimensional image of the object at a corresponding wavelength, as described herein, or for applying corrections to the multi / hyperspectral two-dimensional image, as described herein, different from the apparatus 100 for applying corrections to the multi / hyperspectral two-dimensional image.

[0073] In short, one or more processors of the apparatus for determining the characteristic curve are configured to acquire a first spectrum at at least one second point on the surface of the object at a second angle relative to a plane of the image, acquire a second spectrum at at least one third point on the surface of the object at a third angle relative to a plane of the image, and determine the characteristic curve as the difference between the first spectrum and the second spectrum.

[0074] Figure 4 The illustration depicts a method 300 for determining a characteristic curve according to an embodiment, which is used when estimating a first angle of a first point on the surface of an object based on a multi / hyperspectral two-dimensional image of the object at a corresponding wavelength, or when applying correction to the multi / hyperspectral two-dimensional image. More specifically, Figure 4 The illustration shows a method 300 for operating a device used to determine a characteristic curve.

[0075] Figure 4 The method 300 illustrated is a computer-implemented method. As mentioned earlier, the apparatus for determining the characteristic curve includes one or more processors. Figure 4The method 300 illustrated herein can typically be performed by or under the control of one or more processors of the device.

[0076] refer to Figure 4 At box 302, the first spectrum of at least one second point on the surface of the object, at a second angle relative to the plane of the image, is acquired. Figure 4 At frame 304, acquire the second spectrum of at least one third point on the object's surface at a third angle relative to the plane of the image. Figure 4 At box 306, the characteristic curve is determined as the difference between the first and second spectra.

[0077] In some embodiments, the method may include Figure 3 Boxes 302 to 306 and Figure 4 Boxes 202 to 206. For example, the method may include... Figure 3 Boxes 302 to 306, followed by Figure 3 Boxes 202 to 206. Alternatively, for example, the method may include... Figure 4 Box 202, followed by Figure 3 Boxes 302 to 306, followed by Figure 5 Boxes 204 to 206, or any other suitable box order.

[0078] As mentioned above, the characteristic curve referred to herein indicates the difference between the spectrum of at least one second point on the surface of the object (e.g., a plane) at a second angle relative to the plane of the image and the spectrum of at least one third point on the surface of the object (e.g., a plane) at a third angle relative to the plane of the image.

[0079] The second angle referred to herein can be redefined as a second angle relative to the plane of the optical lens of the imaging sensor. Similarly, the third angle referred to herein can be redefined as a third angle relative to the plane of the optical lens of the imaging sensor. In some embodiments, the characteristic curve may indicate the difference between the average spectrum of at least two second points on the surface of the object (e.g., a plane) at a second angle relative to the plane of the image and the spectrum of at least two third points on the surface of the object (e.g., a plane) at a third angle relative to the plane of the image. In some embodiments, the third angle may be a known angle that is different from or significantly different from the second angle.

[0080] In some embodiments, the second angle referred to herein may be approximately 0 degrees. That is, in some embodiments, at least one second point on the surface of the object may be approximately 0 degrees (or parallel or substantially / approximately parallel) relative to the plane of the image. In other words, in some embodiments, at least one second point on the surface of the object may be approximately 0 degrees (or parallel or substantially / approximately parallel) relative to the plane of the optical lens of the imaging sensor. Therefore, in some embodiments, the second angle may be such that the surface of the object (e.g., a plane) at the second point is parallel or substantially / approximately parallel to the plane of the image (or the plane of the optical lens of the imaging sensor). Therefore, the aforementioned baseline spectrum may be a representative spectrum of multiple locations where the plane of the image (or the plane of the optical lens of the imaging sensor) is parallel or substantially / approximately parallel to the surface (e.g., a plane) of the object.

[0081] Alternatively or additionally, in some embodiments, the third angle referred to herein may be an angle in the range of 45 degrees to 90 degrees, such as an angle in the range of 50 degrees to 85 degrees, such as an angle in the range of 55 degrees to 80 degrees, such as an angle in the range of 60 degrees to 75 degrees. For example, in some embodiments, the third angle referred to herein may be an angle selected from any integer or non-integer value selected from 45 degrees, 50 degrees, 55 degrees, 60 degrees, 65 degrees, 70 degrees, 75 degrees, 80 degrees, 85 degrees, 90 degrees, or any of these values.

[0082] Therefore, in some embodiments, at least one third point on the surface of the object may be at 45 to 90 degrees (or parallel or substantially / approximately perpendicular) to the plane of the image. In other words, in some embodiments, at least one third point on the surface of the object may be at 45 to 90 degrees (or perpendicular or substantially / approximately perpendicular) to the plane of the optical lens of the imaging sensor. Therefore, in some embodiments, the third angle may be such that the surface of the object (e.g., the plane) at the third point is perpendicular or substantially / approximately perpendicular to the plane of the image (or the plane of the optical lens of the imaging sensor). Therefore, the aforementioned edge spectrum may be a representative spectrum of multiple locations where the plane of the image (or the plane of the optical lens of the imaging sensor) is perpendicular or substantially / approximately perpendicular to the surface (e.g., the plane) of the object.

[0083] In some embodiments, the second angle and the third angle can be switched. For example, in some embodiments, the second angle referred to herein may be an angle in the range of 45 degrees to 90 degrees and / or the third angle referred to herein may be approximately 0 degrees. In these embodiments, the difference between the spectrum of at least one second point on the surface of the object (e.g., a plane) at a second angle relative to the plane of the image and the spectrum of at least one third point on the surface of the object (e.g., a plane) at a third angle relative to the plane of the image will have a different sign, such as a negative sign. Therefore, in these embodiments, a negative sign is present in subsequent calculations.

[0084] In some embodiments, at least one second point on the surface of the object may include at least one brightest point (or average brightest point) on the surface of the object and / or at least one third point on the surface of the object may include at least one darkest point (or average darkest point) on the surface of the object. At least one brightest point on the surface of the object may, for example, be at least one point on the surface of the object with a second angle of approximately 0 degrees. At least one darkest point on the surface of the object may, for example, be at least one point on the surface of the object with a third angle of approximately 90 degrees. In some embodiments where the object is the face of the object, at least one second point on the surface of the object may be at least one second point on the surface of the tip of the nose on the face of the object and / or at least one third point on the surface of the object may be at least one third point on the surface of the edge of the face of the object.

[0085] In some embodiments, at least one second point on the surface of an object can be identified by detecting at least one second point on the surface of the object that forms a second angle relative to the plane of the image using landmark detection. Alternatively or additionally, at least one third point on the surface of an object can be identified by detecting at least one third point on the surface of the object that forms a third angle relative to the plane of the image using landmark detection. Those skilled in the art will appreciate established techniques that can be used in this manner for landmark (e.g., facial landmark) detection.

[0086] In some embodiments, a plurality of second points on the surface of the object can be identified to detect at least one second point on the surface of the object forming a second angle with respect to the plane of the image, and / or a plurality of third points on the surface of the object can be identified to detect at least one third point on the surface of the object forming a third angle with respect to the plane of the image. In other words, measurements can be taken at multiple locations. Alternatively or additionally, filtering can be employed. In this way, a more robust characteristic curve can be determined.

[0087] Therefore, in the manner described above, at least one second point on the surface of an object can be identified and a baseline spectrum can be obtained. Similarly, in this manner, at least one third point on the surface of an object can be identified and an edge spectrum can be obtained. In some embodiments, the baseline spectrum can be obtained by: obtaining a baseline from another device or apparatus via one or more processors 102 of device 100, obtaining a baseline from memory (such as the memory of device 100 or another memory) via one or more processors 102 of device 100, or determining the baseline spectrum via one or more processors 102 of device 100. Similarly, in some embodiments, the edge spectrum can be obtained by: obtaining a baseline from another device or apparatus via one or more processors 102 of device 100, obtaining a baseline from memory (e.g., the memory of device 100 or another memory) via one or more processors 102 of device 100, or determining the edge spectrum via one or more processors 102 of device 100.

[0088] As an example, baseline spectra baseline [λ] and edge spectra edge [λ] can be determined by weighting the multi / hyperspectral spectra s[λ, x, y], where x is the x-coordinate in the multi / hyperspectral two-dimensional image, y is the y-coordinate in the multi / hyperspectral two-dimensional image, and λ is the wavelength index indicating the light wavelength:

[0089]

[0090]

[0091] In this example, the sum of the weight matrices is one:

[0092]

[0093]

[0094] The weight matrix in this example can be derived by using fixed processing steps to transform the output of landmark detection (e.g., face detection), ensuring that for w baseline [x, y], has a high weight at at least one second point (e.g., the tip of the nose), and for w edge [x, y], with high weights around at least one third point (e.g., around the edges of the face, such as the chin line).

[0095] Continuing from this example, the characteristic curve can be determined as follows:

[0096] s characteristic [λ]=s edge [λ]-s baseline [λ].

[0097] Figure 5 This is an example schematic diagram of a hyperspectral image (left) and a hyperspectral image of a slice from that image (right). In this example, the hyperspectral image can be a calibrated reflectance image at some (arbitrary) wavelength. The hyperspectral image is formed by multiple pixels, each pixel having a set of intensity values ​​corresponding to the light intensity value for each of a plurality of light wavelengths. Arrow 400 in the hyperspectral image indicates the location of the hyperspectral slice. That is, arrow 400 in the image indicates the corresponding hyperspectral cube (x, y, wavelength) along the line where it is sliced.

[0098] like Figure 5 As illustrated in the diagram, in this example, at least one second point on the surface of the object forming a second angle with respect to the plane of the image referred to herein includes a point on the facial surface near the nose (which is the object in this example). This is a point where the skin (e.g., substantially or approximately) is parallel to the plane of the image or the plane of the optical lens of the imaging sensor. This point may also be referred to as the starting point of a hyperspectral slice. Also as... Figure 6 As illustrated in the diagram, in this example, at least one third point on the surface of the object forming a third angle with respect to the plane of the image referred to herein includes a point on the surface of the face at the edge of the face (just below the ear) (which is the object in this example). This is a point on the skin (e.g., substantially or approximately) perpendicular to the plane of the image or the plane of the optical lens of the imaging sensor. This point may also be referred to as the endpoint of a hyperspectral slice.

[0099] The hyperspectral plot of a slice from the image indicates the distance from the starting point of the hyperspectral slice (or more specifically, the pixel distance) (which is plotted on the vertical axis) and the wavelength (λ) index (on the horizontal axis). In this example, the wavelength index covers wavelengths ranging from 428 nm to 1063 nm. The starting point of the hyperspectral slice is indicated by zero. The hyperspectral plot of a slice from the image shows the light intensity value of each of the multiple light wavelengths at different distances from the starting point. The whiter the pixel, the higher the light intensity value, and therefore the higher the reflectivity. As can be seen from the hyperspectral plot of a slice from the image, individual wavelengths start at the starting point (zero) in different ways. This spectral dependence based on distance and therefore angle becomes clearer after observing the actual differences from the first row.

[0100] Figure 6 The illustration shows the variation of the calibrated hyperspectral image based on the distance from the start point of the hyperspectral slice (or more specifically, pixel distance) and the wavelength (λ) index. More specifically, in Figure 6In the diagram, relative reflectance is plotted as a function of pixel location and the λ exponent. The λ exponent covers wavelengths ranging from 428 nm to 1063 nm. The starting point of the hyperspectral slice is indicated by zero. Relative reflectance is relative to this starting point. Figure 7 As shown in the figure, increasingly distinct patterns can be observed as the distance from the starting point of the hyperspectral slice increases.

[0101] Figure 7 This is an example of a characteristic curve. In this example, the characteristic curve shown is the characteristic curve at the maximum (angular) distance from the start of the hyperspectral slice. Figure 7 The diagram illustrates the unfiltered version of characteristic curve 500 and the filtered version of characteristic curve 502. Figure 7 The horizontal axis in the figure shows the wavelength (λ) index, which in this example is indexed for wavelengths sampled in a non-uniform range from 428 nm to 1063 nm. Figure 8 The vertical axis in the diagram illustrates the difference in the spectrum (or more specifically, the amplitude of the characteristic curve) compared to the original position. That is, the vertical axis shows the difference between the spectrum of at least one second point on the object's surface at a second angle relative to the plane of the image (or the plane of the optical lens of the imaging sensor) and the spectrum of at least one third point on the object's surface at a third angle relative to the plane of the image (or the plane of the optical lens of the imaging sensor). The characteristic curve indicates how the spectrum changes with distance and angle.

[0102] In some embodiments, the characteristic curve referred to herein can be predetermined (e.g., pre-calculated) using at least one other multi / hyperspectral two-dimensional image of the same type of object. For example, in embodiments where the object is object skin, the characteristic curve referred to herein can be predetermined (e.g., pre-calculated) using at least one other multi / hyperspectral two-dimensional image of the object skin and / or at least one other multi / hyperspectral two-dimensional image of the skin of one or more other objects (e.g., with different skin types). In some embodiments of these embodiments, the characteristic curve referred to herein may be stored in memory (e.g., memory 106 of the device or another memory), for example, in the form of a value table. In some embodiments, the type of object (e.g., skin type in the case of the object being skin) may be determined first based on the spectrum, and then the type of object may be indexed in memory, such as in the form of a table. In some embodiments of these embodiments, each type of object may have a predetermined (e.g., pre-calculated) characteristic curve.

[0103] In other embodiments, the characteristic curve referred to herein can be determined (e.g., calculated) using multi / hyperspectral two-dimensional images of the object. For example, in an embodiment where the object is the object's skin, a multi / hyperspectral two-dimensional image of the object's skin can be used to determine (e.g., calculate) the characteristic curve referred to herein. Therefore, in some embodiments, the characteristic curve can be determined based on actual image data.

[0104] In some embodiments, the characteristic curves described herein may include a set of characteristic curves for corresponding sets of second and third angles. In some embodiments, the characteristic curves described herein may be selected from the set of characteristic curves for corresponding sets of second and third angles. In these embodiments, once an image is acquired, a characteristic curve for an object in the image can be selected from the set of characteristic curves.

[0105] As mentioned above, in some embodiments, the characteristic curves for an object may be predetermined characteristic curves stored in memory (e.g., memory 106 of device 100 or any other memory) in the form of a lookup table. Therefore, in some embodiments involving a set of characteristic curves, this set of characteristic curves may be a predetermined set of characteristic curves stored in memory (e.g., memory 106 of device 100 or any other memory) in the form of a lookup table. In some embodiments, this set of characteristic curves may have been predetermined (e.g., in a laboratory setting). In some embodiments, this set of characteristic curves may be determined using machine learning or deep learning. Therefore, in some embodiments employing machine learning or deep learning, a set of characteristic curves based on angles may be used instead of a single characteristic curve. In some embodiments involving deep learning, deep learning may be applied by feeding a training difference spectrum and the corresponding angle.

[0106] Multiple characteristic curves can be beneficial for different objects, such as different people. For example, different people may have different skin types, and each skin type may have a different characteristic curve, so a suitable characteristic curve for a specific skin type can be selected from the set of characteristic curves. Therefore, in some embodiments, the set of characteristic curves may include characteristic curves for different objects. For example, in the case of a person, the set of characteristic curves may include characteristic curves for each of several different skin types (e.g., categorized according to the Fitzpatrick scale). Therefore, characteristic curves can be selected from the set of characteristic curves based on a person's skin type. That is, a characteristic curve corresponding to a person's skin type can be selected. In this way, the correction determined or applied in the first angle can be more accurate.

[0107] In some embodiments, a person's skin type can be determined using any existing skin type determination techniques known to those skilled in the art. In other embodiments, for example via communication interface 110 of device 100, a user of device 100 (e.g., a person or another user) can input a person's skin type. While skin type is used as an example, characteristic profiles can be selected based on any other attribute of the object.

[0108] In some embodiments, one or more processors 102 of the apparatus 100 described herein may be configured to, for at least one pixel of a plurality of pixels corresponding to at least one other first point on the surface of an object, compare a set of intensity values ​​for said at least one pixel with a characteristic curve for the object to determine a similarity measure between the set of intensity values ​​and the obtained characteristic curve. In these embodiments, one or more processors 102 of the apparatus 100 described herein may also be configured to estimate at least one other first angle on the surface of the object corresponding to said at least one pixel based on the determined similarity measure. In some embodiments of these embodiments, one or more processors 102 of the apparatus 100 may also be configured to derive an angle map including the estimated first angle and the estimated at least one other first angle. Thus, according to some embodiments, a first angle can be estimated for more than one point, and then an angle map can be derived.

[0109] Figure 8 The illustration shows an example of a derived angle plot. In this example, the object is the face of a subject. For privacy reasons, the eyes and the area around the eyes are excluded from the plot. Figure 8 In the image, the x and y axes on the upper left represent pixel positions, and the vertical axis on the upper right represents the estimated angle (in degrees). Figure 8 In the derived angle diagram illustrated in the figure, the darkest part of the angle diagram represents at least one second point on the surface of the object that is approximately 0 degrees (or parallel or substantially / approximately parallel) to the plane of the image or the plane of the optical lens of the imaging sensor. Similarly, in ​ In the angle plot illustrated herein, the brightest portion represents at least one third point on the surface of the object that is approximately 90 degrees (or perpendicular or substantially / approximately perpendicular) to the plane of the image or the plane of the optical lens of the imaging sensor. In some embodiments of the derived angle plot, the regression coefficient “a” described above may be calibrated (e.g., scaled and / or cropped) to represent the actual angle using a single global gain parameter, for example, making the angle of the region near the edge close to 90 degrees.

[0110] In some embodiments of the derived angle map, one or more processors 102 of the apparatus 100 described herein may be configured to use the derived angle map to estimate a depth map. Therefore, in some embodiments, the angle map can be transformed into a 3D image. For example, the angle map can be converted into a depth map (or 3D image) by starting from a position where the angle is substantially or approximately 0 degrees (i.e., at a baseline position) and expanding outwards to estimate how far neighboring points have been translated in depth according to the angle. Alternatively, in other embodiments, the angle map may be used directly during the spectral decomposition process.

[0111] In some embodiments where the object is skin, one or more processors 102 of the apparatus 100 described herein may be configured to determine the concentration of chromophores in the skin based on a multi / hyperspectral two-dimensional image of the skin, or based on a multi / hyperspectral two-dimensional image with corrections applied, using an estimated first angle. Those skilled in the art will appreciate techniques that can be used to determine the concentration of chromophores in the skin.

[0112] However, one example is a decomposition algorithm that consists of a model function f(). The model function f() describes the theoretical (reflection or absorption) spectrum as a frequency function for a given chromophore concentration vector c. Therefore, f(c) maps to a wavelength λ. Then, as an example, for each pixel location in a multi / hyperspectral 2D image, the following least-squares error is minimized using nonlinear least-squares optimization:

[0113]

[0114] Where s[λ, x, y] represents the multi / hyperspectral spectrum, x is the x-coordinate in the multi / hyperspectral two-dimensional image, y is the y-coordinate in the multi / hyperspectral two-dimensional image, and λ is the wavelength index indicating the wavelength of light.

[0115] This produces the chromophore concentration vector c that best matches the input spectrum. The input spectrum is the set of intensity values ​​of at least one pixel described earlier, where the set of intensity values ​​corresponds to the light intensity value for each of the multiple light wavelengths λ. More advanced models can incorporate angles, meaning that the function f() takes not only the chromophore concentration c as input, but also the angle a. This leads to the minimization of the following least-squares error:

[0116]

[0117] Where s[λ, x, y] represents the multi / hyperspectral spectrum, x is the x-coordinate in the multi / hyperspectral two-dimensional image, y is the y-coordinate in the multi / hyperspectral two-dimensional image, and λ is the wavelength index indicating the wavelength of light.

[0118] A computer program product comprising a computer-readable medium is also provided. The computer-readable medium has computer-readable code embodied therein. The computer-readable code is configured to cause the computer or processor, when executed by a suitable computer or processor, to perform the methods described herein. For example, the computer-readable medium can be any entity or device capable of carrying the computer program product. For example, the computer-readable medium can include a data storage device, such as a ROM (such as a CD-ROM or semiconductor ROM) or a magnetic recording medium (such as a hard disk). Furthermore, the computer-readable medium can be a transmissible carrier, such as an electrical or optical signal, which can be transmitted via cable or optical fiber or by radio or other means. When the computer program product is embodied in such a signal, the computer-readable medium can be constituted by such a cable or other device or component. Alternatively, the computer-readable medium can be an integrated circuit in which a computer program product is embedded, the integrated circuit being adapted to perform or be used to perform the methods described herein.

[0119] Therefore, this document provides an apparatus 100, a method 200, and a computer program product for estimating a first angle of a first point on the surface of an object or applying corrections to a multi / hyperspectral two-dimensional image of the object at a corresponding wavelength, which addresses the limitations associated with the prior art. This document also provides an apparatus 300, a method 300, and a computer program product for determining a characteristic curve used when estimating a first angle of a first point on the surface of an object or applying corrections to a multi / hyperspectral two-dimensional image of the object at a corresponding wavelength.

[0120] By studying the accompanying drawings, the disclosure, and the appended claims, those skilled in the art can understand and implement variations of the disclosed embodiments in practicing the principles and techniques described herein. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plural. A single processor or other unit can perform the functions of several items recited in the claims. The fact that certain measures are recited in mutually different dependent claims does not mean that a combination of these measures cannot be used advantageously. Computer programs can be stored or distributed on suitable media, such as optical storage media or solid-state media provided with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. An apparatus (100) for estimating a first angle of a first point on the surface of an object based on a multi / hyperspectral two-dimensional image of the object at a corresponding wavelength, or for applying a correction to the multi / hyperspectral two-dimensional image, the apparatus (100) comprising one or more processors (102) configured to: Acquire a multi / hyperspectral two-dimensional image of an object at a corresponding wavelength, wherein the multi / hyperspectral two-dimensional image at the corresponding wavelength is composed of multiple pixels, each pixel having a set of intensity values, the set of intensity values ​​corresponding to the light intensity value for each of the multiple light wavelengths; For at least one pixel among the plurality of pixels corresponding to a first point on the surface of the object, the set of intensity values ​​for the at least one pixel is compared with a characteristic curve for the object to determine a similarity measure between the set of intensity values ​​and the obtained characteristic curve. as well as Based on the determined similarity metric, estimate a first angle of the first point on the surface of the object corresponding to the at least one pixel, wherein the first angle is relative to a plane of the image; or apply a correction to the multi / hyperspectral two-dimensional image at the first point on the surface of the object using the determined similarity metric. The characteristic curves therein characterize how the object reflects or absorbs light according to an angle, and indicate the difference between the spectrum of at least one second point on the surface of the object at a second angle relative to the plane of the image and the spectrum of at least one third point on the surface of the object at a third angle relative to the plane of the image, wherein the third angle is a known angle different from the second angle.

2. The apparatus (100) according to claim 1, wherein: The characteristic curve indicates the difference between the average spectrum of at least two second points on the surface of the object at the second angle relative to the plane of the image and the spectrum of at least two third points on the surface of the object at the third angle relative to the plane of the image.

3. The apparatus (100) according to any one of claims 1-2, wherein: The second angle is 0 degrees; and / or The third angle is an angle in the range of 45 degrees to 90 degrees.

4. The apparatus (100) according to any one of claims 1-2, wherein: The at least one second point on the surface of the object includes at least one brightest point on the surface of the object; and / or The at least one third point on the surface of the object includes at least one darkest point on the surface of the object.

5. The apparatus (100) according to any one of claims 1-2, wherein: The at least one second point on the surface of the object is identified by using landmark detection to detect at least one second point on the surface of the object that forms a second angle relative to the plane of the image; and / or The at least one third point on the surface of the object is identified by using landmark detection to detect at least one third point on the surface of the object that forms a third angle relative to the plane of the image.

6. The apparatus (100) according to any one of claims 1-2, wherein: The characteristic curve is predetermined using at least one other multi / hyperspectral two-dimensional image of the same type of object; or The characteristic curve is determined using the multi / hyperspectral two-dimensional image of the object.

7. The apparatus (100) according to any one of claims 1-2, wherein the one or more processors (102) are configured to: For at least one pixel among the plurality of pixels corresponding to at least one other first point on the surface of the object, the set of intensity values ​​for the at least one pixel is compared with the characteristic curve for the object to determine a similarity measure between the set of intensity values ​​and the obtained characteristic curve; as well as Based on the determined similarity metric, at least one other first angle of the at least one other first point on the surface of the object corresponding to the at least one pixel is estimated; as well as Derive an angle diagram including the estimated first angle and at least one other estimated first angle.

8. The apparatus (100) according to any one of claims 1-2, wherein: The characteristic curve for the object is selected from the characteristic curve set of the corresponding sets for the second and third angles.

9. The apparatus (100) according to any one of claims 1-2, wherein: The spectrum of the at least one second point includes: a reflection spectrum indicating the portion of light reflected from the object at the at least one second point on the surface of the object at a second angle relative to the plane of the image; or an absorption spectrum indicating the portion of light absorbed by the object at the at least one second point on the surface of the object at a second angle relative to the plane of the image; and / or The spectrum of the at least one third point includes: a reflection spectrum indicating the portion of light reflected from the object at the at least one third point on the surface of the object at the third angle relative to the plane of the image; or an absorption spectrum indicating the portion of light absorbed by the object at the at least one third point on the surface of the object at the third angle relative to the plane of the image.

10. The apparatus (100) according to any one of claims 1-2, wherein the object is skin, and the one or more processors are configured to: The concentration of chromophores in the skin is determined using the estimated first angle based on the multi / hyperspectral two-dimensional image of the skin, or based on the multi / hyperspectral two-dimensional image with the correction applied.

11. A method (200) for estimating a first angle of a first point on the surface of an object based on a multi / hyperspectral two-dimensional image of the object at a corresponding wavelength, or for applying a correction to the multi / hyperspectral two-dimensional image, the method comprising: Acquire a multi / hyperspectral two-dimensional image of an object at a corresponding wavelength, wherein the multi / hyperspectral two-dimensional image at the corresponding wavelength is composed of multiple pixels, each pixel having a set of intensity values, the set of intensity values ​​corresponding to the light intensity value for each of the multiple light wavelengths; For at least one pixel among the plurality of pixels corresponding to a first point on the surface of the object, the set of intensity values ​​for the at least one pixel is compared with a characteristic curve for the object to determine a similarity measure between the set of intensity values ​​and the obtained characteristic curve. as well as Based on the determined similarity metric, estimate a first angle of the first point on the surface of the object corresponding to the at least one pixel, wherein the first angle is relative to a plane of the image; or apply a correction to the multi / hyperspectral two-dimensional image at the first point on the surface of the object using the determined similarity metric. The characteristic curves therein characterize how the object reflects or absorbs light according to an angle, and indicate the difference between the spectrum of at least one second point on the surface of the object at a second angle relative to the plane of the image and the spectrum of at least one third point on the surface of the object at a third angle relative to the plane of the image, wherein the third angle is a known angle different from the second angle.

12. An apparatus (100) for determining a characteristic curve for use in estimating a first angle at a first point on the surface of an object based on a multi / hyperspectral two-dimensional image of the object at a corresponding wavelength or in applying a correction to the multi / hyperspectral two-dimensional image, wherein the first angle is relative to a plane of the image, the apparatus (100) comprising one or more processors (102) configured to: Obtain the first spectrum of at least one second point on the surface of the object at a second angle relative to the plane of the image; Obtain a second spectrum of at least one third point on the surface of the object at a third angle relative to the plane of the image, wherein the third angle is a known angle different from the second angle; and The characteristic curve is defined as the difference between the first spectrum and the second spectrum, wherein the characteristic curve characterizes how the object reflects or absorbs light depending on the angle.

13. A method (300) for determining a characteristic curve, the characteristic curve being used when estimating a first angle at a first point on the surface of the object based on a multi / hyperspectral two-dimensional image of the object at a corresponding wavelength or when applying a correction to the multi / hyperspectral two-dimensional image, wherein the first angle is relative to a plane of the image, the method (300) comprising: Obtain the first spectrum of at least one second point on the surface of the object at a second angle relative to the plane of the image; Obtain a second spectrum of at least one third point on the surface of the object at a third angle relative to the plane of the image, wherein the third angle is a known angle different from the second angle; and The characteristic curve is defined as the difference between the first spectrum and the second spectrum, wherein the characteristic curve characterizes how the object reflects or absorbs light depending on the angle.

14. A computer program product comprising a computer-readable medium having computer-readable code embodied therein, the computer-readable code being configured to cause the computer or processor, when executed by a suitable computer or processor, to perform the method according to claim 11 or 13.

Images