Method and apparatus for performing a spectral analysis of a subject's skin - Patents.com

Abstract

According to one aspect, a method of aligning image frames in a sequence of image frames of an object is provided, comprising the steps of controlling 101 one or more light sources to emit multispectral light for illuminating the object, the multispectral light having at least a first light component having a first illumination wavelength and a second light component having a second illumination wavelength, and controlling 103 a multichannel imager to acquire two or more sequences of image frames of the object, each sequence of image frames being acquired using a separate light detection channel of the multichannel imager, the multichannel imager including a first light detection channel responsive to light in a first spectral band to generate the first sequence of frames of the object, and a second light detection channel responsive to light in a second spectral band to generate the first sequence of frames of the object. and a second light detection channel responsive to light in a second spectral band for generating a separate second sequence, wherein the first illumination wavelength is in the first spectral band and the second illumination wavelength is in the second spectral band; a step 105 of processing the first sequence of frames to calculate a motion vector for each frame, each motion vector representing motion of the object between the frames of the first sequence; and a step 107 of aligning frames in the second sequence of frames based on corresponding determined motion vectors, wherein the object is aligned across the frames in the second sequence of frames.

Description

[Technical field]

[0001] The present disclosure relates to performing spectral analysis of a subject's skin, including registering image frames in a sequence of image frames, and in particular to registering image frames in a sequence of image frames acquired using multispectral light. [Background technology]

[0002] Multispectral imaging provides a rich amount of information that is not easily obtainable from single or triple spectral channel (e.g., red-green-blue (RGB)) imaging. The advantages are more pronounced in imaging complex objects such as human skin, where the detected backscattered light is modulated by multiple chromophores in the skin / body, such as hemoglobin, melanin, water, lipids, carotenoids, and bilirubin.

[0003] Reflectance spectroscopy is the study of light as a function of wavelength. It finds many applications in medical applications, material analysis and soil analysis. Reflectance spectroscopy is made possible by the fact that different substances have different absorption and scattering spectra. This is also true for the chromophores in human skin, such as oxygenated hemoglobin, deoxygenated hemoglobin, carotenoids, melanin, bilirubin, etc. Reflectance spectroscopy of human skin therefore allows a detailed resolution of the chromophores present. It is also useful to relate the detected chromophores to (systemic) health conditions and to specific skin conditions that cannot be resolved by the human eye.

[0004] Reflectance spectroscopy typically only looks at one point or region of the spectrum. When multispectral or hyperspectral imaging is referred to, an entire grid (or image) is looked at at different wavelengths. A particular area of ​​interest is multispectral imaging of the human body and face.

[0005] High speed multispectral imaging can be achieved using single shot multispectral cameras, but these devices are very expensive and not for commercial home use. A cost effective solution is to use multispectral illumination, where each wavelength of light can be captured or scanned sequentially from multiple frames by an imaging device (e.g., a camera). Summary of the Invention [Problem to be solved by the invention]

[0006] A multispectral image is usually composed of a sequence of image frames, where in each frame the object / subject being imaged is illuminated with light of a specific wavelength. In practice, such a multispectral image "scan" can be completed in a few seconds. This has the disadvantage that there is usually small movement by the subject (e.g. a human) during the scan.

[0007] Since the spectrum at a particular spatial location of the object is usually of interest, it is important that all frames are aligned. For example, a point on the face is imaged while illuminated with light of wavelength 440 nm. The point is at coordinates (X,Y) in the image. This exact same point on the face imaged when illuminated with light of, for example, 700 nm should be at exactly the same coordinates (X,Y) in the image. If they are not at the same coordinates, for example due to the movement of the face and / or the movement of the imaging device relative to the face, image alignment (also called image registration) should be achieved so that the image frames are aligned before performing the spectral analysis using the image frames.

[0008] Traditionally, successive image frames with non-varying illumination can be easily aligned or registered using simple algorithms. However, image alignment of objects illuminated with different (time-varying) light, i.e., multispectral light, is hindered by the fact that the reflectance of skin varies spectrally, and therefore images or parts of images at different wavelengths have different spatial contrast and detail.

[0009] Therefore, a need exists for improved methods and apparatus for performing spectral analysis of a subject's skin, including registering image frames of the subject's skin obtained using multispectral light. [Means for solving the problem]

[0010] According to a first specific aspect, there is provided a method for aligning image frames in a sequence of image frames of an object, the method comprising the steps of controlling one or more light sources to emit multispectral light for illuminating the object, the multispectral light having at least a first light component having a first illumination wavelength and a second light component having a second illumination wavelength, and controlling a multi-channel imager to acquire two or more sequences of image frames of the object, each sequence of image frames being acquired using a separate light detection channel of the multi-channel imager, the multi-channel imager comprising a first light detection channel responsive to light in a first spectral band to generate the first sequence of frames of the object, and a second light detection channel responsive to light in a second spectral band to generate the first sequence of frames of the object. and a second light detection channel responsive to light in a second spectral band to generate a second sequence of frames, wherein the first illumination wavelength is in the first spectral band and the second illumination wavelength is in the second spectral band; processing the first sequence of frames to calculate a motion vector for each frame, each motion vector representing motion of the object between frames in the first sequence; and aligning frames in the second sequence of frames based on corresponding determined motion vectors, wherein the object is aligned between frames in the second sequence of frames.

[0011] According to a second aspect, there is provided a computer program product comprising a computer readable medium having computer readable code embodied therein, which when executed by a suitable computer or processor is configured to cause the computer or processor to perform a method according to the first aspect or any embodiment thereof.

[0012] According to a third aspect, there is provided an apparatus configured to align image frames in a sequence of image frames of an object. The apparatus has a processing unit, the processing unit controls one or more light sources to emit multispectral light to illuminate an object, the multispectral light having at least a first light component having a first illumination wavelength and a second light component having a second illumination wavelength, the processing unit controls a multi-channel imaging device to obtain two or more sequences of image frames of the object, each sequence of image frames being obtained using a separate light detection channel of the multi-channel imaging device, the multi-channel imaging device having a first light detection channel responsive to light in a first spectral band to generate the first sequence of frames of the object and a second light detection channel responsive to light in a second spectral band to generate a second sequence of frames of the object, the first illumination wavelength being within the first spectral band and the second illumination wavelength being within the second spectral band, the processing unit processes the first sequence of frames to calculate a motion vector for each frame, each motion vector representing motion of the object between frames in the first sequence, and the processing unit aligns frames in the second sequence of frames based on a corresponding determined motion vector. The result is that the object is registered across frames in the second sequence of frames.

[0013] According to a fourth aspect, there is provided a system configured to align image frames in a sequence of image frames of an object, the system comprising an apparatus according to the third aspect or any embodiment thereof, one or more light sources, and a multi-channel imaging device.

[0014] According to a fifth aspect, there is provided a method of performing a spectral analysis of skin of a subject, the method comprising the steps of: (i) aligning image frames in a sequence of image frames of the subject's skin, the aligning step comprising controlling one or more light sources to emit multispectral light for illuminating the skin, the multispectral light having at least a first light component having a first illumination wavelength and a second light component having a second illumination wavelength, and controlling an RGB imager to obtain two or more sequences of image frames of the skin, each sequence of image frames obtained using a separate light detection channel of the RGB imager, the RGB imager controlling a first light detection channel responsive to light in a first spectral band and a second light detection channel responsive to light in a second spectral band to generate the first sequence of frames of the skin. and a second light detection channel responsive to light in a second spectral band to generate a first illumination wavelength in the first spectral band and a second illumination wavelength in the second spectral band, processing the first sequence of frames to calculate a motion vector for each frame, each motion vector representing a motion of the skin between frames in the first sequence, and aligning frames in the second sequence of frames based on corresponding determined motion vectors, where the skin is aligned across frames in the second sequence of frames; and (ii) performing a spectral analysis of the skin using the aligned second sequence of frames. The first light component is a green light component, the first spectral band is a green spectral band, the second light component is one of a red light component and a blue light component, and the second spectral band is a corresponding one of a red spectral band and a blue spectral band.

[0015] According to a sixth aspect, there is provided a computer program product comprising a computer readable medium having computer readable code embodied therein, the computer readable code being configured, when executed by a suitable computer or processor, to cause the computer or processor to perform a method according to the fifth aspect or any embodiment thereof.

[0016] According to a seventh aspect, there is provided an apparatus configured to perform a spectral analysis of skin of a subject, the apparatus having a processing unit, the processing unit (i) aligning image frames in a sequence of image frames of the subject's skin, the aligning comprising controlling one or more light sources to emit multispectral light to illuminate the skin, the multispectral light having at least a first light component having a first illumination wavelength and a second light component having a second illumination wavelength, and controlling an RGB imager to obtain two or more sequences of image frames of the skin, each sequence of image frames obtained using a separate light detection channel of the RGB imager, the RGB imager controlling a first light detection channel responsive to light in a first spectral band to generate the first sequence of frames of the skin and a second light detection channel responsive to light in a second spectral band to generate the second sequence of frames of the skin. and a second light detection channel responsive to light in a spectral band of a first illumination wavelength in a first spectral band and a second illumination wavelength in a second spectral band, processing the first sequence of frames to calculate a motion vector for each frame, each motion vector representing a movement of the skin between frames in the first sequence, and aligning frames in a second sequence of frames based on a corresponding determined motion vector, where the skin is aligned across frames in the second sequence of frames, and a processing unit (ii) performing a spectral analysis of the skin using the aligned second sequence of frames. The first light component is a green light component, the first spectral band is a green spectral band, the second light component is one of a red light component and a blue light component, and the second spectral band is a corresponding one of a red spectral band and a blue spectral band.

[0017] According to an eighth aspect, there is provided a system configured to perform a spectral analysis of skin of a subject, the system comprising an apparatus according to the seventh aspect or any embodiment thereof, one or more light sources, and an RGB imager.

[0018] These and other aspects will be apparent from and elucidated with reference to the embodiments described hereinafter. [Brief description of the drawings]

[0019] [Figure 1] 1 is a simplified diagram of a system for acquiring and aligning image frames of an object according to one embodiment; [Diagram 2] 4 is a flow chart illustrating a method for aligning image frames according to one embodiment of the technology described herein. [Diagram 3] 4 is a flow chart illustrating a general method for aligning image frames in accordance with the techniques described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Exemplary embodiments will now be described, by way of example only, with reference to the following drawings, in which:

[0021] 1 is a simplified diagram of a system 2 for acquiring and registering image frames of a subject 4, according to one embodiment of the techniques described herein. The subject 4 is typically a human subject, although the system 2 and techniques can be used with non-human subjects and / or objects where motion of the subject in the sequence of image frames may occur as the image frames are acquired. When used with a human subject 4, the techniques described herein are particularly useful where the acquired image frames are spectrally analyzed to determine one or more properties of the skin, for example relating to chromophores present on the skin.

[0022] System 2 generally comprises a device 6, a multi-channel imaging device 8, and one or more light sources 10. In the embodiment shown in Figure 1, device 6 comprises a processing unit 12, and optionally further comprises a memory unit 14 and / or one or more user interface elements 16.

[0023] A processing unit 12 is provided for controlling the operation of the multi-channel imaging device 8 and the light source 10 to illuminate the object 4 with the required multi-spectral light and to acquire a sequence of image frames of the object 4. The processing unit 12 also operates according to the techniques described herein to align the image frames in the acquired sequence.

[0024] Processing unit 12 can be implemented in multiple ways using software and / or hardware to perform the various functions described herein. Processing unit 12 can have one or more microprocessors or digital signal processors (DSPs) that can be programmed using software or computer program code to perform the required functions and / or control elements of processing unit 12 to achieve the required functions. Control unit 12 can be implemented as a combination of dedicated hardware to perform some functions (e.g., amplifiers, preamplifiers, analog-to-digital converters (ADCs) and / or digital-to-analog converters (DACs)) and processors (e.g., one or more programmed microprocessors, controllers, DSPs and associated circuitry) to perform other functions. Examples of elements that may be employed in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, DSPs, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), hardware for implementing neural networks, and / or so-called artificial intelligence (Al) hardware accelerators (i.e., processors or other hardware designed specifically for Al applications that may be used in parallel with a main processor).

[0025] The processing unit 12 can be connected to a memory unit 14 that can store data, information and / or signals used by the processing unit 12 in performing or implementing the techniques described herein. For example, the memory unit 14 can store image frames obtained by the multi-channel imager 8 and / or can store image frames after registration by the processing unit 12. In some implementations, the memory unit stores computer readable code that can be executed by the processing unit 12 such that the processing unit 12 performs one or more functions including the techniques described herein. The memory unit 14 can include any type of non-transitory machine-readable medium such as a cache or system memory including volatile and non-volatile computer memories such as random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM) and electrically erasable PROM (EEPROM). The memory unit 14 may be implemented in the form of a memory chip, an optical disk (such as a compact disk (CD), digital versatile disk (DVD) or Blu-Ray disk), a hard disk, a tape storage solution, or a solid-state device including a memory stick, a solid-state drive (SSD), a memory card, etc.

[0026] User interface elements 16 may be provided including one or more elements that enable a user of system 2 and / or device 6 (e.g., subject 4 or other party) to input information, data and / or commands to system 2 and / or device 6 and / or enable device 6 to output information or data to the user. For example, user interface elements 16 may include a display screen for displaying one or more image frames of the subject before and / or after the image frames have been aligned according to the techniques described herein. The display screen or other visual display element may additionally or alternatively display the results of the spectral analysis on the aligned image frames. User interface elements 16 may have any suitable input elements, including but not limited to a keyboard, a keypad, one or more buttons, switches, dials, a mouse, a trackpad, a touch screen, a stylus, a camera, a microphone, etc., and / or user interface elements 16 may have any suitable output elements, including but not limited to a display screen, one or more lights or light elements, one or more speakers, a vibration element, etc.

[0027] A multi-channel imaging device 8 is provided for obtaining a sequence of image frames of an object 4. The multi-channel imaging device 8 has a plurality of light detection channels, each responsive to light in a separate spectral band. Each detection channel outputs a separate sequence of frames of the object. The multi-channel imaging device 8 of FIG. 1 has a first light detection channel 18 responsive to light in a first spectral band and a second light detection channel 20 responsive to light in a second spectral band. Thus, the first light detection channel 18 outputs a first sequence of image frames formed from incident light in the first spectral band, and the second light detection channel 20 outputs a second sequence of image frames formed from incident light in the second spectral band.

[0028] The first and second spectral bands are non-overlapping, i.e. the first and second spectral bands cover separate wavelength ranges. Optionally, the multi-channel imaging device 8 may comprise at least a third light detection channel responsive to light in a third spectral band (whose wavelengths covered by the third spectral band do not overlap with those of the first and second spectral bands) to output a third sequence of image frames.

[0029] As described below, light having wavelengths in the first spectral band is used as a "reference light" to identify object motion between image frames. The first spectral band is therefore also referred to herein as a "reference band." In preferred embodiments, particularly when the image frames are images of skin, the first spectral band encompasses green light, for example light having wavelengths of about 535 nanometers (nm) and / or 550 nm. Green light is preferred for skin imaging applications because images of skin under green light are useful for tracking skin, i.e., identifying skin features such as wrinkles, age spots, scars, moles, etc. In these embodiments, the second spectral band can encompass red light or blue light. In the case of red light, the second spectral band can include light having wavelengths of about 615 nm, 630 nm, 650 nm, and / or 660 nm. In the case of blue light, the second spectral band can include light having wavelengths of about 450 nm, 465 nm, and / or 480 nm. In an alternative embodiment, for example if image frames acquired using green light are also used for the spectral analysis of the object 4, the first spectral band may include infrared light (IR) instead of green light.

[0030] In embodiments in which the first photodetection channel 18 is responsive to green light and the second photodetection channel 20 is responsive to red or blue light (and optionally, the multichannel imager 8 has a third photodetection channel responsive to the other of red and blue light), the multichannel imager 8 may be in the form of an RGB camera or imaging unit.

[0031] As mentioned above, each light detection channel 18, 20 captures or generates a separate sequence of image frames. The image frames are generated based on a common frame rate. That is, each light detection channel captures / generates separate image frames simultaneously and at the same frame rate. For example, the light detection channels 18, 20 can capture / generate 30 image frames per second (fps), although other frame rates can be used.

[0032] In a typical implementation, multi-channel imager 8 is a "rolling shutter" type imager, i.e., an imager in which each light-detecting channel captures an image frame line-by-line, rather than capturing the entire frame at once.

[0033] The one or more light sources 10 are provided for emitting multispectral light for illuminating the object 4 when an image frame is acquired by the multi-channel imaging device 8. The multispectral light comprises at least a first light component having a first illumination wavelength λ1 and a second light component having a second illumination wavelength λ2. The first illumination wavelength λ1 is a wavelength in a first spectral band, and the second illumination wavelength λ2 is a wavelength in a second spectral band. The one or more light sources 10 are configured to emit the first light component and the second light component at the same time (i.e. simultaneously). According to a preferred embodiment of the multi-channel imaging device 8 described above, the first illumination wavelength λ1 may be a wavelength of green light, and the second illumination wavelength λ2 may be a wavelength of red light or blue light.

[0034] In some embodiments, the multispectral light emitted by the one or more light sources 10 further comprises at least a third light component having a third illumination wavelength λ3 in a third spectral band. For example, if the second illumination wavelength λ2 is a wavelength of red light, the third illumination wavelength λ3 can be a wavelength of blue light, and vice versa.

[0035] In some embodiments, one or more light sources 10 may be configured to selectively emit multiple light components having distinct illumination wavelengths within a particular spectral band. In these embodiments, the light source 10 emits light components having different illumination wavelengths within the particular spectral band at different times (e.g., the light source 10 alternates between emitting two or more illumination wavelengths within the particular spectral band). It should be understood that while the light components within the particular spectral band are alternated, the light source 10 may continue to emit light components in other spectral bands. For example, the light source 10 may emit a first light component (i.e., green light) in a first spectral band having a wavelength of 535 nm, and the light source 10 may alternately emit a red light component in a second spectral band having a wavelength of 615 nm and another red light component having a wavelength of 630 nm.

[0036] In some embodiments, the one or more light sources 10 comprise a separate light source 10 for each emitted light component. Alternatively, the one or more light sources 10 can comprise a single light source that can be controlled to emit multiple light components with separate illumination wavelengths. In some embodiments, the required illumination wavelengths can be achieved using separate color filters.

[0037] The one or more light sources 10 may be, for example, light emitting diodes (LEDs) or diode lasers. As described, each light source 10 may be configured to emit light at a particular wavelength or may be controllable to emit light at varying wavelengths.

[0038] In the embodiment shown in FIG. 1, the device 6, the multi-channel imager 8, and the one or more light sources 10 are separated from each other. Thus, although not shown in FIG. 1, suitable communication circuitry or elements may be provided to enable control signals and other information (e.g., data representing image frames obtained by the multi-channel imager 8) to be communicated to / from the device 6. Thus, the communication circuitry or elements may be provided to enable data connection and / or data exchange between the device 6, the multi-channel imager 8 / imaging unit, and / or the one or more light sources 10 / illumination units, depending on the embodiment. The connection may be direct or indirect (e.g., via the Internet), and thus the communication circuitry or elements may enable a connection between the device 6 and the multi-channel imager 8 / imaging unit and / or the one or more light sources 10 / illumination units via a network (e.g., the Internet), or may enable a connection between the device 6 and the multi-channel imager 8 / imaging unit and / or the one or more light sources 10 / illumination units directly via any desired wired or wireless communication protocol. For example, the communication circuitry or element may operate using WiFi, Bluetooth, Zigbee, or any cellular communication protocol. In the case of a wireless connection, the communication circuitry or element (and thus the device 6, the multi-channel imager 8 / imaging unit, and the one or more light sources 10 / illumination units) may include one or more suitable antennas for transmitting / receiving over a transmission medium (e.g., air). Alternatively, in the case of a wireless connection, the communication circuitry or element may include means (e.g., connectors or plugs) that allow the communication circuitry or element to be connected to one or more suitable antennas external to the device 6, the multi-channel imager 8 / imaging unit, and the one or more light sources 10 / illumination units for transmitting / receiving over a transmission medium (e.g., air).

[0039] It should be understood that a practical implementation of system 2 may include additional elements to those shown in Figure 1. For example, system 2 may include one or more power sources for various elements of system 2, such as batteries, or elements that allow system 2, and / or portions of system 2, to be connected to a mains power source.

[0040] 1, the multi-channel imager 8 and the one or more light sources 10 are shown as separate from the device 6, however, it should be understood that in alternative embodiments, one or both of the multi-channel imager 8 and the one or more light sources 10 may be integrated with or part of the device 6. In embodiments in which the multi-channel imager 8 is separate from the device 6, the multi-channel imager 8 may be considered to be part of an imaging unit that may have one or more other elements or circuits (e.g., power supplies, control circuits, etc.) to enable operation of the multi-channel imager 8. Similarly, in embodiments in which the one or more light sources 10 are separate from the device 6, the one or more light sources 10 may be considered to be part of an illumination unit that may have one or more other elements or circuits (e.g., power supplies, control circuits, etc.) to enable operation of the one or more light sources 10.

[0041] For example, the device 6 can be in the form of or part of an electronic device such as a smartphone, tablet, smartwatch, laptop, computer, server, etc. Embodiments in which the device 6 and the multi-channel imaging device 8 are integrated include cases in which the device 6 is a smartphone, tablet, or other device that typically includes an imaging unit. In these embodiments, the light source(s) 10 can be or correspond to a controllable "camera flash" light source. Alternatively, the light source(s) 10 can be provided in a separate element or device that can be connected to and controlled by the smartphone / tablet / laptop etc. to emit the required multispectral light.

[0042] Briefly, according to the techniques described herein, the processing unit 12 receives from each of the light detection channels 18, 20 a sequence of image frames obtained while the one or more light sources 10 are illuminating the object 4 with light of an appropriate illumination wavelength, processes the first sequence of frames (i.e., provided by the first light detection channel 18) to calculate a motion vector for each frame, and aligns the frames in the second (and any other) sequence of frames based on the calculated motion vectors. Each motion vector represents the motion of the object 4 between frames in the first sequence. Thus, aligning the frames based on the motion vectors means that the object is aligned across the frames in the second (and any other) sequence.

[0043] The flow chart of Figure 2 illustrates a method for aligning image frames according to one embodiment of the techniques described herein. The method begins at step 50. In step 52, the object 4 is illuminated with a first light component (i.e., a reference light). Simultaneously, in step 54, the object 4 is illuminated with a second light component (e.g., red light) and any other desired light components in spectral bands other than the first spectral band. As described further below, the first light component and / or the second light component (and / or any other light components) may be pulsed, so that the particular light components illuminating the object 4 at any particular moment can vary. In certain embodiments, the rate at which the light is pulsed is higher or much higher than the frame rate of the multi-channel imager 8.

[0044] While light is illuminating the object 4 according to steps 52 and 54, in step 56, the multi-channel imager 8 is used to capture one or more image frames, i.e., each light detection channel 18, 20 in the multi-channel imager 8 generates one or more image frames from the incident light.

[0045] Next, in step 58, one or more image frames generated by the first light detection channel 18 in the multi-channel imager 8 are stored for subsequent processing. The image frames generated by the first light detection channel 18 are also referred to herein as "reference frames."

[0046] Concurrent with step 58, in step 60, one or more image frames generated by the second light detection channel 20 (and any further light detection channels) in the multi-channel imaging device 8 are stored for subsequent processing. The image frames generated by the second light detection channel 20 (and any other light detection channel other than the first light detection channel) are also referred to herein as "spectral frames" because, after alignment, these frames can be spectrally analyzed to determine one or more characteristics of the object 4 in the image.

[0047] The method then proceeds to step 62, where the acquired reference frames are processed to calculate a motion vector for each reference frame. The motion vector represents the motion of the object 4 in the reference frame relative to the previous reference frame. Step 62 may include using conventional techniques to compare multiple reference frames to identify landmarks or other important features (e.g., edges of objects / objects) in the reference frames and determine the movement of those landmarks or features between the reference frames. The motion may be represented as a motion vector that indicates how the object 4 has moved between the reference frames. Those skilled in the art will recognize various techniques that may be used to implement step 62, including Farneback's optical flow for motion estimation (e.g., as described in "Two-Frame Motion Estimation Based on Polynomial Expansion" by Gunnar Farneback), and other optical flow techniques, and details will not be provided herein.

[0048] Next, in step 64, the spectral frames (i.e. the sequence of image frames output by the second light detection channel 20) are aligned based on the motion vectors determined in step 62. That is, the motion vector determined for a particular spectral frame relative to a corresponding reference frame (i.e. a reference frame generated by the first light detection channel 18 simultaneously with the particular spectral frame generated by the second light detection channel 20) is applied to the particular spectral frame in order to align it with the previous spectral frame. This is repeated for each spectral frame to provide an aligned sequence of spectral frames in which the object 4 is in the same position in each frame.

[0049] More precisely, the motion estimation in step 62 results in motion vectors and for each reference frame the (local) displacement is known. These displacements need to be registered with the spectral frames. The sequence of spectral frames and the reference frames are acquired simultaneously by the same multi-channel image processor 8, so that they are perfectly mapped to each other. The calculated motion vectors are directly mapped through a mapping function and applied to the spectral images. As a result, the sequence of spectral images is properly aligned.

[0050] The method ends at step 66.

[0051] In some embodiments, the one or more light sources 10 are controlled to provide pulses of light that include light components of different illumination wavelengths. For example, the one or more light sources 10 can be controlled to follow the following light pulse pattern: Light pulse 1: Red, blue and green lights are generated simultaneously; Light pulse 2: Red 2, Blue 2, and Green lights are generated simultaneously; and Light Pulse 3: 3 red, 3 blue and 3 green lights are generated simultaneously.

[0052] The specific illumination wavelengths of these light components are as follows: [Table 1]

[0053] In a specific example, each light pulse (Light Pulse 1, Light Pulse 2, Light Pulse 3, etc.) can have a duration of 2.8 milliseconds (ms) and each light component can have a color temperature and brightness that is set such that the pulsed light is perceived by a user and / or subject 4 as a constant white light without any perceptible flicker or color shift.

[0054] When the multi-channel imager 8 is a rolling shutter RGB imager with a frame rate of 30 fps, the light pulses generate approximately 12 light and dark bands in each image frame output by the red, green and blue light detection channels. The image frames output by the green light detection channel are used to calculate motion vectors for each image frame as described above in step 68, and these motion vectors are applied to the image frames output by the red and blue light detection channels to align the object 4 within the image frames.

[0055] To enable subsequent spectral analysis of the image frames, a "banded" image frame (i.e., an image frame containing 12 light and dark bands) from each light detection channel can be decomposed into seven separate wavelength-specific spectral image frames (i.e., one wavelength-specific spectral image frame for each wavelength of light contained in the light pulse).

[0056] In a further embodiment, the effect of ambient light needs to be removed from the image frames to improve the spectral analysis, in which case the light pulse pattern described above can be modified to include "light pulses" where no light is emitted by one or more light sources 10 and the multi-channel imager 8 generates the image frames from the incident ambient light. For example, one or more light sources 10 can be controlled to follow the following light pulse pattern: Light pulse 1: Red, blue and green lights are generated simultaneously; Light pulse 2: Red 2, Blue 2 and Green lights are generated simultaneously; Light pulse 3: Red 3, Blue 3 and Green light are generated simultaneously; and Light Pulse 4: Light source 10 off, ambient light only.

[0057] Again, each light pulse (Light Pulse 1, Light Pulse 2, Light Pulse 3, Light Pulse 4) may have a duration of 2.8 ms. Additionally, each light component may have a color temperature and brightness that is set such that the pulsed light is perceived by the user and / or subject 4 as a constant white light without any perceptible flicker or color shift.

[0058] If the multi-channel imager 8 is a rolling shutter RGB imager with a frame rate of 30 fps, the four light pulses will produce approximately 12 bright and dark bands in each image frame output by the red, green and blue light detection channels. The alignment of the image frames is performed as described above.

[0059] To allow for subsequent spectral analysis of the image frames, the "banded" image frame from each light detection channel is decomposed into an ambient image frame and seven separate wavelength-specific spectral image frames (i.e., one wavelength-specific spectral image frame for each wavelength of light contained in the light pulse). Before any spectral analysis is performed on the spectral images, the effects of ambient light can be removed from the wavelength-specific spectral image frames by linearly subtracting the ambient image frame from each wavelength-specific spectral image frame.

[0060] The flowchart in Figure 3 illustrates a general method for aligning image frames based on the techniques described herein. In some embodiments, the method may be performed by processing unit 12 in device 6, for example, based on processing unit 12 executing appropriate computer-readable code. Such code may be stored in or on memory unit 14, or on another type of computer-readable medium.

[0061] In step 101, one or more light sources 10 are controlled to emit multispectral light for illuminating the skin of a subject 4. The multispectral light comprises at least a first light component having a first illumination wavelength and a second light component having a second illumination wavelength. The first light component is also referred to as the "reference light" or "reference light component."

[0062] In step 103, the multi-channel imaging device 8 is controlled to obtain a sequence of image frames of the skin of the subject 4. Each sequence of image frames is obtained using a separate light detection channel 18, 20 of the multi-channel imaging device 8. Step 103 is performed while the one or more light sources 10 emit multispectral light. The multi-channel imaging device 8 has a first light detection channel 18 responsive to light in a first spectral band for generating a first sequence of frames of the skin of the subject 4 and a second light detection channel 20 responsive to light in a second spectral band for generating a separate second sequence of frames of the skin of the subject 4. The first illumination wavelength is in the first spectral band and the second illumination wavelength is in the second spectral band.

[0063] In step 105, the first sequence of frames is processed to calculate a motion vector for each frame, each motion vector representing the movement of the object 4 between frames in the first sequence.

[0064] In step 107, frames in one or more further sequences of frames are registered based on the corresponding determined motion vectors, such that the skin of the subject is registered across frames in the second sequence of frames.

[0065] In some embodiments, the method further comprises performing a spectral analysis of the skin of the subject 4 using the registered frames.

[0066] In some embodiments, step 101 comprises alternating activation of one or more light sources 10 to emit multispectral light and deactivating one or more light sources 10. In these embodiments, step 103 comprises controlling the multichannel imaging device 8 to acquire a sequence of frames while the one or more light sources are activated and deactivated. These embodiments may further comprise processing the sequence of frames to determine an ambient light contribution to the sequence of frames that corresponds to the ambient light around the object 4 when the one or more light sources are deactivated. Then, after step 107, the aligned frames in the second sequence of frames can be corrected for ambient light using the determined ambient light contribution. In some embodiments, the determined ambient light contribution can be subtracted from one or more or all of the frames in the second sequence, and optionally also from the first sequence of frames. Any spectral analysis is then performed on the aligned frames after correction.

[0067] In some embodiments, the multispectral light can include a third light component having a third illumination wavelength within a third spectral band, and the multichannel imager 8 can include a third light detection channel responsive to light in the third spectral band for generating a third sequence of frames of the object 4. In these embodiments, the frames in the third sequence of frames can also be aligned based on the corresponding determined motion vectors. As a result, the skin is aligned across the frames in the third sequence of frames. In embodiments in which a spectral analysis of the skin is performed, the spectral analysis can be performed on one or both of the aligned second sequence of frames and the third sequence of frames.

[0068] In some embodiments, the multi-channel imager 8 is an RGB imager. In these embodiments, the first photo-detection channel may be a green channel of the RGB imager, the second photo-detection channel is one of the blue and red channels of the RGB imager, and the third photo-detection channel is the other of the blue and red channels of the RGB imager.

[0069] In some embodiments, the multispectral light may include a third (or further) light component with a third (or further) illumination wavelength, both the second illumination wavelength and the third illumination wavelength being in the second spectral band. In this case, controlling the one or more light sources 10 comprises controlling the one or more light sources 10 to emit the second and third light components alternately (i.e. alternating between two colors of light in the second spectral band). In these embodiments, the multichannel imaging device 8 may be a rolling shutter imaging device with a frame rate smaller than the frequency at which the second and third light components are alternated. Preferably, bands of different light components within a frame need to be separated and recombined into a frame for a single light component to allow a spectral analysis of the object 4 to be performed. Thus, in some embodiments, the method further comprises, for the second sequence of aligned frames, decomposing a subset of aligned frames based on the second illumination wavelength and the third illumination wavelength to provide a frame component for the second illumination wavelength and a frame component for the third illumination wavelength. Then, for each illumination wavelength in the second spectral band, the frame components for that illumination wavelength are combined to form a composite frame for that illumination wavelength.

[0070] In the above embodiment using multispectral light including a third (or further) light component with a third (or further) illumination wavelength in the second spectral band, an ambient light correction can also be applied. In this case, step 101 can comprise controlling one or more light sources to alternately emit the second light component, the third light component and no light. The method can further comprise aligning frames in a first sequence of sub-frames based on corresponding motion vectors determined in step 105, decomposing a subset of the aligned frames to provide, for each sequence of aligned frames, ambient light frame components corresponding to time periods when the one or more light sources are not emitting light, and combining the ambient light frame components for all sequences of aligned frames to form a composite frame representative of ambient light around the object 4 when the one or more light sources are not emitting light. Finally, each sequence of aligned frames can be corrected for ambient light using the determined composite frame representative of ambient light.

[0071] Thus, an improved method and apparatus is provided for aligning image frames of an object acquired using multispectral light.

[0072] Variations to the disclosed embodiments can be understood and implemented by those skilled in the art implementing the principles and techniques described herein, from a study of the figures, the disclosure, and the appended claims. In the claims, the word "comprises" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used to advantage. The computer program can be stored or distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless communication systems. Any reference signs in the claims should not be interpreted as limiting the scope of the invention.

Claims

1. 1. A method for performing a spectral analysis of skin of a subject, comprising: (i) registering image frames in a sequence of image frames of a subject's skin, the registering comprising: controlling one or more light sources to emit multispectral light for illuminating the skin, the multispectral light having at least a first light component having a first illumination wavelength and a second light component having a second illumination wavelength; controlling an RGB imager to obtain two or more sequences of image frames of the skin, each sequence of image frames obtained using a separate light detection channel of the RGB imager, the RGB imager having a first light detection channel responsive to light in a first spectral band to generate a first sequence of frames of the skin and a second light detection channel responsive to light in a second spectral band to generate a second separate sequence of frames of the skin, a first illumination wavelength being within the first spectral band and a second illumination wavelength being within the second spectral band; processing a first sequence of frames to calculate a motion vector for each frame, each motion vector representing movement of the skin between frames in the first sequence; and registering frames in the second sequence of frames based on the motion vectors, wherein the skin is registered across frames in the second sequence of frames. (ii) performing a spectral analysis of the skin using the registered second sequence of frames to determine one or more characteristics of the skin; 11. A method according to claim 10, wherein the first light component is a green light component, the first spectral band is a green spectral band, the second light component is one of a red light component and a blue light component, and the second spectral band is a corresponding one of a red spectral band and a blue spectral band.

2. The method of claim 1 , wherein the RGB imager is controlled to capture the sequence of frames while the one or more light sources are emitting the multi-spectral light.

3. 2. The method of claim 1 , wherein controlling the one or more light sources comprises alternating between activating the one or more light sources to emit the multispectral light and deactivating the one or more light sources, and controlling the RGB imager comprises controlling the RGB imager to capture the sequence of frames while the one or more light sources are activated and deactivated.

4. processing the sequence of frames to determine an ambient light contribution for the sequence of frames that corresponds to ambient light around the object when the one or more light sources are deactivated; and correcting the aligned frames in the second sequence of frames for the ambient light using the determined ambient light contribution.

5. The method of claim 1 , wherein the multispectral light further comprises a third light component having a third illumination wavelength.

6. the second light component is a red light component; the second illumination wavelength is in the red spectral band; the third light component is a blue light component; the third illumination wavelength is in the blue spectral band; the RGB imager further comprising a third light detection channel responsive to light in the blue spectral band to generate a third sequence of frames of the skin; Optionally, the step of registering further comprises registering frames in the third sequence of frames based on the corresponding determined motion vectors, wherein the skin is registered across frames in the third sequence of frames; 6. The method of claim 5, optionally wherein performing the spectral analysis comprises performing a spectral analysis of the skin using the aligned second sequence of frames and the aligned third sequence of frames.

7. 6. The method of claim 5, wherein the second illumination wavelength and the third illumination wavelength are both in the second spectral band, and controlling the one or more light sources comprises controlling the one or more light sources to alternately emit the second light component and the third light component.

8. 8. The method of claim 7, wherein the RGB imager is a rolling shutter RGB imager having a frame rate that is less than a frequency at which the second light component and the third light component are alternated.

9. For a second sequence of the aligned frames in the second spectral band, decomposing the subset of aligned frames based on the second illumination wavelength and the third illumination wavelength to provide a frame component for the second illumination wavelength and a frame component for the third illumination wavelength; 9. The method of claim 8, further comprising: for each illumination wavelength in the second spectral band, combining frame components for that illumination wavelength to form a composite frame for that illumination wavelength.

10. wherein controlling the one or more light sources comprises controlling the one or more light sources to alternately emit the second light component, emit the third light component, and emit no light, the method comprising: aligning frames in the first sequence of sub-frames based on the corresponding determined motion vectors; for each sequence of aligned frames, decomposing a subset of the aligned frames to provide an ambient light frame component corresponding to a time period during which the one or more light sources are not emitting light; combining the ambient light frame components for all sequences of aligned frames to form a composite frame representing ambient light around the object when the one or more light sources are not emitting light; 10. The method of claim 9, further comprising: correcting each sequence of the aligned frames for the ambient light using the determined composite frame representative of the ambient light.

11. A computer program comprising computer readable code which, when executed by a suitable computer or processor, causes the computer or processor to perform a method as claimed in any one of claims 1 to 4, or a method as claimed in claim 5 which depends directly on claim 1, or a method as claimed in any one of claims 6 to 10 which depends directly on claim 5 which depends directly on claim 1.

12. 1. An apparatus for performing a spectral analysis of a subject's skin, the apparatus comprising: a processing unit, the processing unit comprising: (i) registering image frames in a sequence of image frames of a subject's skin, the registration comprising: controlling one or more light sources to emit multispectral light for illuminating the skin, the multispectral light having at least a first light component having a first illumination wavelength and a second light component having a second illumination wavelength; controlling an RGB imager to obtain two or more sequences of image frames of the skin, each sequence of image frames obtained using a separate light detection channel of the RGB imager, the RGB imager having a first light detection channel responsive to light in a first spectral band to generate a first sequence of frames of the skin and a second light detection channel responsive to light in a second spectral band to generate a second sequence of frames of the skin, the first illumination wavelength being within the first spectral band and the second illumination wavelength being within the second spectral band; processing a first sequence of frames to calculate a motion vector for each frame, each motion vector representing movement of the skin between frames in the first sequence; and registering frames in the second sequence of frames based on the motion vectors, wherein the skin is registered across frames in the second sequence of frames; The processing unit (ii) performs a spectral analysis of the skin using the registered second sequence of frames to determine one or more characteristics of the skin; the first light component is a green light component, the first spectral band is a green spectral band, the second light component is one of a red light component and a blue light component, and the second spectral band is a corresponding one of a red spectral band and a blue spectral band.

13. 1. A system for performing a spectral analysis of a subject's skin, comprising: An apparatus according to claim 12; one or more light sources; and an RGB imager.

Images