Determine pixel intensity values ​​in imaging

Abstract

In one embodiment, a method (100) is described. The method includes accessing (102) data from a sequence of images of an object, the object being illuminated by ambient light and illumination having a temporal sinusoidal intensity modulation. An imaging device is used to acquire the sequence of images and is configured such that different spatial intensity modulation patterns are apparent in consecutive images of the sequence. The method further includes determining (104) a set of corrected pixel intensity values ​​for generating a corrected image of the object based on a set of measured pixel intensity values ​​in each image in the sequence of images such that a reduced level of ambient illumination is apparent in the corrected image compared to a level of ambient illumination apparent in at least one image in the sequence of images.

Description

Technical Field

[0001] The present invention relates to methods, apparatus, and tangible machine-readable media for imaging in certain contexts. Background Art

[0002] A topic of interest in the field of non-invasive measurement and monitoring relates to skin sensing for personal care and health applications. Skin sensing systems are being developed that promise to quantify the skin and monitor skin features, which can provide users with information that is too small to detect, too faint to notice, or too slow to follow. In order to provide user-acceptable results, such skin sensing systems may need to provide sensitivity and specificity when performing skin sensing. Assuming that the measurements made by such skin sensing systems are proven to be robust and reliable, users can build trust in these skin sensing systems.

[0003] Imaging-based skin sensing systems may need to determine information that may be affected by parameters that are difficult to control, such as changes in ambient lighting. For example, some uncontrolled environments, such as a user's home, may have undefined and / or potentially changing ambient lighting. Such uncontrolled environments may result in erroneous measurements of the user's skin, which in turn may result in unacceptable or untrustworthy results for the user. The imaging performance of some cameras (such as smartphone cameras) used in some imaging-based skin sensing systems may vary, making the imaging data unpredictable or unreliable.

[0004] The performance of other imaging-based sensing systems for acquiring information from surfaces other than the skin may also be adversely affected by certain uncontrolled environments. For example, a user wishing to acquire an image of an object for a particular application in an uncontrolled environment may find that the image may have unacceptable lighting variations, which may affect the way the image is perceived or subsequently processed.

[0005] US 2019 / 068862 A1 describes methods for creating frames captured with a camera that are independent of lighting conditions. In a first method, a modulated light source (e.g., amplitude modulation (AM)) is captured to add background light to illuminate a scene. An algorithm is executed to generate a processed image in which the effects of the background lighting have been eliminated.

[0006] Williams et al., “Simultaneous correction of flat field and nonlinearity response of intensified charge-coupled devices”, Review of Scientific Instruments, Vol. 78, 123702 (2007) describes a general method for flat field image correction.

[0007] JP 2010-119035 A1 describes an imaging device including: an imaging section; a flash detection section for detecting external flash from an output of the imaging section; and an adaptive gamma processing section for correction-converting an entire screen of the output of the imaging section using a predetermined gamma value when no flash is detected. Summary of the invention

[0008] Aspects or embodiments described herein are directed to improving imaging in certain scenarios.Aspects or embodiments described herein may eliminate one or more problems associated with imaging in an uncontrolled environment.

[0009] In a first aspect, a method is described. The method is a computer-implemented method. The method includes accessing data from a sequence of images of an object, the object being illuminated by ambient lighting and illumination having a temporal sinusoidal intensity modulation. The sequence of images is acquired by an imaging device. The imaging device is configured to acquire, within each image of the sequence, a first spatial portion of the image at a different time than a second different spatial portion of the image, wherein a relationship between a frequency of the sinusoidal intensity modulation and a frame rate of the imaging device is such that different spatial intensity modulation patterns are apparent in consecutive images of the sequence. The method further includes determining, based on a set of measured pixel intensity values ​​in each image in the sequence of images, a set of corrected pixel intensity values ​​for generating a corrected image of the object by retrieving the amplitude of the sinusoidal temporal modulated illumination from the sequence of images, such that a reduced level of ambient lighting is apparent in the corrected image compared to a level of ambient lighting apparent in at least one image in the sequence of images.

[0010] Certain embodiments related to the first aspect are described below.

[0011] In some embodiments, the set of modified pixel intensity values ​​is determined such that both the ambient light level and the modulation depth of the spatial intensity modulation pattern are reduced in the modified image compared to at least one image in the sequence of images.

[0012] In some embodiments, determining the set of modified pixel intensity values ​​comprises: for each pixel of the set, calculating a specified combination of measured pixel intensity values ​​for the pixel from each image in the sequence of images. The specified combination is determined based on a sinusoidal intensity modulation of the illumination.

[0013] In some embodiments, determining the set of corrected pixel intensity values ​​or estimated intensity distribution of the corrected image is based on three or four consecutive images from the sequence of images.

[0014] In some embodiments, determining the set of modified pixel intensity values ​​or estimated intensity distribution for the image is based on three consecutive images from a sequence of images, wherein the spatial intensity modulation patterns of a first and a fourth consecutive image in the sequence of images are the same.

[0015] In some embodiments, different spatial intensity modulation patterns are apparent in consecutive images of the sequence by at least one of: the frequency of the sinusoidal intensity modulation and the frame rate of the imaging device; and a phase difference between the sinusoidal intensity modulation and the frame acquisition timing of the imaging device.

[0016] In some embodiments, the frequency of the sinusoidal intensity modulation is not an integer multiple of the frame rate of the imaging device.

[0017] In some embodiments, the method includes causing the illumination source to provide illumination having a temporal sinusoidal intensity modulation.

[0018] In some embodiments, a set of corrected pixel intensity values ​​or estimated intensity distribution A for the corrected image is given by:

[0019]

[0020] Wherein I1(x, y), I2(x, y), I3(x, y) and I4(x, y) respectively represent the pixel intensity values ​​of the first, second, third and fourth consecutive images in the sequence of images.

[0021] In some embodiments, the method includes calculating the phase progression of the spatial intensity modulation pattern in the sequence of images according to

[0022]

[0023] Wherein I1(x, y), I2(x, y), I3(x, y) and I4(x, y) respectively represent the pixel intensity values ​​of the first, second, third and fourth consecutive images in the sequence of images.

[0024] In some embodiments, the method includes compensating for a nonlinear response of the imaging device prior to the steps of: accessing data from the sequence of images; and determining the set of corrected pixel intensity values ​​or estimated intensity distributions. The nonlinear response of the imaging device may be determined by: obtaining an indication of a mean intensity of a kernel, the kernel comprising at least one pixel selected from an image sampled from a set of images of the object acquired by the imaging device; and determining the nonlinear response of the imaging device based on the indication of the mean intensity obtained for each image from the set of images.

[0025] In some embodiments, the method includes causing the illumination source to illuminate the object by modulating the illumination source according to an illumination function. The illumination function may be configured to perform at least one of the following while obtaining an indication of an average intensity of the kernel sampled from the set of images: linearly increasing the intensity of the illumination function over a specified time interval, and linearly decreasing the intensity of the illumination function over a specified time interval.

[0026] In some embodiments, the method includes causing an imaging device to acquire the set of images according to an imaging acquisition function. The imaging acquisition function may be configured to perform one of the following while obtaining an indication of an average intensity of a kernel sampled from the set of images: linearly increasing an exposure time of each subsequent image of the set of images, and linearly decreasing an exposure time of each subsequent image of the set of images.

[0027] In a second aspect, a tangible machine-readable medium is described. The tangible machine-readable medium stores instructions which, when executed by at least one processor, cause the at least one processor to implement a method according to the first aspect or any other embodiment described herein.

[0028] In a third aspect, an apparatus is described. The apparatus includes a processing circuit system. The processing circuit system includes an access module and a determination module. The access module is configured to access data from a sequence of images of an object, the object being illuminated by ambient lighting and illumination having a temporal sinusoidal intensity modulation. The sequence of images is acquired by an imaging device. The imaging device is configured to acquire, within each image of the sequence, a first spatial portion of the image at a different time than a second different spatial portion of the image, wherein a relationship between a frequency of the sinusoidal intensity modulation and a frame rate of the imaging device is such that different spatial intensity modulation patterns are apparent in consecutive images of the sequence. The determination module is configured to determine, based on a set of measured pixel intensity values ​​in each image in the sequence of images, a set of corrected pixel intensity values ​​for generating a corrected image of the object by retrieving the amplitude of the sinusoidal temporal modulated illumination from the sequence of images, such that a reduced level of ambient lighting is apparent in the corrected image compared to a level of ambient lighting apparent in at least one image in the sequence of images.

[0029] Certain embodiments related to the third aspect are described below.

[0030] In some embodiments, the apparatus further comprises an illumination source for providing illumination.

[0031] In some embodiments, the apparatus further comprises an imaging device.

[0032] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Exemplary embodiments of the present invention will now be described, by way of example only, with reference to the following drawings, in which:

[0034] Figure 1 relates to methods of improving imaging in certain scenarios according to embodiments;

[0035] Figure 2 is a schematic diagram of a system for improving imaging in certain scenarios according to an embodiment;

[0036] Figure 3 is a schematic diagram of a process for improving imaging in certain scenarios according to an embodiment;

[0037] Figure 4 is a schematic diagram of a process for improving imaging in certain scenarios according to an embodiment;

[0038] Figure 5 is a schematic diagram of a process for improving imaging in certain scenarios according to an embodiment;

[0039] Figure 6 relates to methods of improving imaging in certain scenarios according to embodiments;

[0040] Figure 7 is a schematic diagram of a machine-readable medium for improving imaging in certain scenarios according to an embodiment; and

[0041] Figure 8 is a schematic diagram of an apparatus for improving imaging in certain scenarios, according to an embodiment. DETAILED DESCRIPTION

[0042] Figure 1A method 100 (e.g., a computer-implemented method) for improving imaging in certain settings is shown. For example, imaging may be affected by certain settings, such as in an uncontrolled environment where the illumination / ambient lighting is undefined and / or may vary. As will be described in more detail below, when imaging an object in an uncontrolled environment, certain methods and apparatus described herein may reduce the effects of, compensate for, or correct for, any undefined and / or potentially varying illumination / ambient lighting.

[0043] The method 100 includes, at block 102, accessing data from a sequence of images of an object. The sequence of images is acquired by an imaging device. The object is illuminated by ambient lighting and by illumination having a temporal sinusoidal intensity modulation. The imaging device is configured such that different spatial intensity modulation patterns are apparent in consecutive images of the sequence.

[0044] The imaging device may acquire a sequence of images while illuminating the object with illumination (and illuminating the object with ambient lighting) (e.g., continuously over a period of time). In some embodiments, the data may be accessed by acquiring the data directly from the imaging device (e.g., during operation of the imaging device). In some embodiments, the data may be accessed by accessing a memory storing the data (e.g., the data may be stored in the memory after being acquired by the imaging device).

[0045] Each image in the sequence may correspond to an individual frame acquired by the imaging device.Thus, the frame rate of the imaging device may be related to the number of frames / images acquired by the imaging device over a period of time.

[0046] Sinusoidal intensity modulation in time means that an object is illuminated with illumination whose intensity varies sinusoidally over time. In other words, the illumination may be modulated such that the variation in intensity follows a sinusoidal pattern.

[0047] The sinusoidal intensity modulation is associated with a modulation frequency (eg, the frequency of the sinusoidal pattern). As will be explained in more detail below, this frequency may be such that different spatial intensity modulation patterns are apparent in consecutive images of the sequence.

[0048] The method 100 includes, at block 104, determining a set of modified pixel intensity values ​​for generating a modified image of the object based on a set of measured pixel intensity values ​​in each image in the sequence of images such that a reduced level of ambient lighting is apparent in the modified image as compared to a level of ambient lighting apparent in at least one image in the sequence of images. In other words, the determined set of modified pixel intensity values ​​may refer to an estimated intensity distribution of the image of the object to reduce ambient lighting variations in the image (i.e., the 'corrected image'). Such a determination may be based on a spatial intensity modulation pattern (or a corresponding 'set of measured pixel intensity values') in each image in the sequence of images.

[0049] As referred to herein, the term 'estimated intensity distribution' may refer to a 'set of modified pixel intensity values'. The term 'spatial intensity modulation pattern' may refer to a corresponding 'set of measured pixel intensity values'.

[0050] Estimating the intensity distribution may refer to the intensity distribution of an image that takes into account any ambient lighting variations that are evident in the image. For example, if the ambient lighting is such that one portion of an object is illuminated by brighter ambient lighting than another portion of the object, such ambient lighting variations may be reduced, compensated for, or otherwise corrected in the image.

[0051] In the examples given above, the estimated intensity distribution may indicate that for a portion of an object that is brighter due to ambient lighting changes, the pixel intensity values ​​corresponding to this portion of the object will be reduced (e.g., so that the 'corrected' image appears less bright in this portion). Similarly, the estimated intensity distribution may indicate that for a portion of an object that is darker due to ambient lighting changes, the pixel intensity values ​​corresponding to this portion of the object will be increased (e.g., so that the 'corrected' image appears brighter in this portion). In other cases, the pixel values ​​of a portion of an image may be adjusted rather than adjusting the pixel values ​​of all portions characterized by ambient lighting changes. Certain embodiments described herein relate to various potential ways to estimate an intensity distribution to reduce ambient lighting changes.

[0052] In some cases, the image for which the estimated intensity distribution is determined may be one image in a sequence of images providing data. In some cases, the image may be a different image (eg, a subsequent image) from the sequence of images.

[0053] In some embodiments, an estimated intensity distribution may be calculated for a particular image to reduce the effect of ambient lighting changes in the scene on the image. In other words, pixel intensity values ​​that take into account (e.g., reduce the effect of) ambient lighting changes in the scene may be calculated for a particular image.

[0054] In some embodiments, the set of modified pixel intensity values ​​is determined such that both the level of ambient lighting and the modulation depth of the spatial intensity modulation pattern are reduced in the modified image compared to at least one image in the sequence of images.

[0055] In some embodiments, the modulation depth may refer to the difference between the maximum and minimum measured pixel intensity values ​​in an image that includes a sequence of spatial intensity modulation patterns. For example, the spatial intensity modulation pattern may correspond to a sinusoidal spatial modulation pattern. In the modified image, the modulation depth of the spatial intensity modulation pattern may be reduced or removed so that the (e.g., applied) illumination appears to be spatially uniformly distributed across the modified image, while the apparent ambient lighting is reduced. Thus, in some embodiments, the apparent illumination in the modified image is due to the illumination other than the ambient lighting (or at least the illumination dominates relative to the ambient lighting).

[0056] In some embodiments, determining the set of modified pixel intensity values ​​includes calculating, for each pixel in the set, a specified combination of measured pixel intensity values ​​for the pixel from each image in the sequence of images. The specified combination may be determined based on a sinusoidal intensity modulation of the illumination.

[0057] In some embodiments, the (temporal) sinusoidal intensity modulation properties of the illumination may be utilized to determine the specified combination. For example, due to the (temporal) sinusoidal intensity modulation properties of the illumination, it is possible to determine a 'specified combination of measured pixel intensity values ​​from a sequence of images' that generates a set of corrected pixel values ​​that corresponds to a reduced level of ambient illumination apparent in the corrected image, while also reducing or removing the 'modulation depth' of the (spatial) intensity modulation pattern apparent in the sequence of images (i.e., due to the (temporal) sinusoidal intensity modulation properties of the illumination and the configuration of the imaging device, such as operating in a rolling shutter mode). Examples of specified combinations are given below.

[0058] In some embodiments, the frequency of the sinusoidal intensity modulation and the frame rate of the imaging device are such that different spatial intensity modulation patterns are apparent in consecutive images of the sequence. For example, the frequency of the sinusoidal intensity modulation may not be an integer multiple of the frame rate of the imaging device.

[0059] In some embodiments, the phase difference between the sinusoidal intensity modulation and the frame acquisition timing of the imaging device is such that different spatial intensity modulation patterns are apparent in consecutive images of the sequence.

[0060] A combination of the two embodiments described above may be implemented such that different spatial intensity modulation patterns are apparent in consecutive images of a sequence.

[0061] As described above, the imaging device is configured such that different spatial intensity modulation patterns are apparent in consecutive images of the sequence. In this regard, the imaging device is configured to acquire, within each image of the sequence, a first spatial portion of the image at a different time than a second different spatial portion of the image such that the different spatial intensity modulation patterns are apparent in consecutive images of the sequence.

[0062] Different spatial portions of an image may exhibit different intensity levels depending on the illumination level provided at a time according to the sinusoidal intensity modulation. In other words, as the illumination level changes, the spatial portion of the imaging device that collects imaging data at a time may collect imaging data corresponding to the illumination level at that time. For example, if the modulated illumination level is at a minimum at a time, the spatial portion of the image that collects imaging data at that time may be dark. However, if the modulated illumination level is at a maximum at a different time, the spatial portion of the image that collects imaging data at that different time may be relatively bright.

[0063] In some embodiments, the imaging device may be configured to operate in a rolling shutter mode. In rolling shutter mode, different spatial portions of an image may be sampled at different times within a single frame.

[0064] An example of a rolling shutter mode may refer to an image acquisition method in which data from pixels of an imaging device sensor is acquired pixel row by pixel row (in some cases, multiple pixel rows may be scanned across the sensor at any time). For example, data is acquired by acquiring pixel values ​​from each pixel row / column of the imaging device sensor while scanning the imaging device sensor vertically or horizontally. Therefore, pixel data is not acquired from all pixels of the imaging device sensor at the same time. Instead, there may be a time delay between pixel data from different pixel rows / columns within the same frame. In the case where the imaged object moves at a certain speed (or the image changes in other ways), depending on the frame rate of the imaging device, the rolling shutter mode may cause a so-called 'rolling shutter effect' in the image.

[0065] For example, the combined effect of a rolling shutter pattern and a sinusoidal intensity modulation of the illumination may give rise to a spatial intensity modulation pattern observed in each image of a sequence of images. Thus, as the illumination intensity is modulated in time, a pixel at a given location on the imaging device sensor detects a corresponding illumination level (i.e., from ambient lighting as well as modulated illumination) at a given time. Within a single image (frame) captured by the imaging device, the modulation of the illumination may be apparent (e.g., due to the rolling shutter pattern) due to pixels (at a certain location on the imaging device sensor) registering intensity values ​​at a certain time, where the illumination level detected at that time depends on the modulation of the illumination. For example, if a peak illumination level (due to the modulation) occurs within a frame at time t1, a pixel providing its reading at that time t1 will register a larger intensity value than a pixel (at a different spatial location on the imaging device sensor) at another time t2 within the frame (e.g., where time t2 refers to a moment when the illumination level is at its lowest level due to the modulation). This effect may give rise to a spatial intensity modulation pattern observed in each image. As will be explained in more detail below, differences in spatial intensity modulation patterns observed in a sequence of images may be exploited to determine an estimated intensity distribution.

[0066] In some examples, certain methods and apparatus described herein may support precise and / or accurate measurement of certain surface characteristics (e.g., color, texture, and / or any other surface characteristics). For example, where the subject is a person (or any animal) and the surface being imaged is their skin (e.g., facial skin), certain methods and apparatus described herein may improve imaging in certain scenarios (such as if the ambient lighting is uncontrolled and / or varying). For certain applications, surface characteristics of the skin that may be of interest include, for example, skin color, skin health, skin texture, and / or any other characteristics of the skin that may be of interest. For certain applications, such as skin sensing for personal care and health applications, imaging of the skin may facilitate, for example, skin feature characterization and / or tracking / monitoring of the skin over time.

[0067] Figure 2 2 is a schematic diagram of a system 200 for improving imaging in certain scenarios. System 200 may implement certain methods described herein (eg, method 100). In this embodiment, subject 202 is in a scenario where there is uncontrolled and / or potentially changing ambient lighting.

[0068] The system 200 includes an imaging device 204 for acquiring an image (eg, a sequence of images) of an object 202 .

[0069] The system 200 also includes an illumination source 206, such as a smartphone flash or other device capable of providing illumination with sinusoidal intensity modulation. For example, a controller (e.g., a controller of the illumination source 206 itself or an external controller) can vary the power (e.g., voltage and / or current) provided to the illumination source 206 according to the sinusoidal modulation over time.

[0070] The system 200 also includes a control module 208 for implementing certain functionality. In some embodiments, the control module 208 can control the imaging device 204 (e.g., cause the imaging device 204 to acquire images, receive those images from the imaging device 204, and / or store data corresponding to those images in a memory of the control module 208 or in a memory accessible to the control module 208). In some embodiments, the control module 208 can control the illumination source 206 (e.g., cause the illumination source 206 to modulate the intensity of the illumination it provides and / or control the power provided to the illumination source 206). In some embodiments, the control module 208 can implement certain methods described herein. For example, the control module 208 can implement Figure 1 Any combination of the above embodiments may be implemented by the control module 208 .

[0071] In some embodiments, imaging device 204, illumination source 206, and control module 208 may be implemented by the same device (eg, a 'smart' device such as a smartphone, tablet, etc.).

[0072] In some embodiments, the imaging device 204 and the illumination source 206 may be implemented by the same device (e.g., a 'smart' device such as a smartphone, tablet computer, etc.), while the control module 208 is implemented by a different device or service (e.g., a user computer, a dedicated processing device, a server, a cloud-based service, or other type of processing device).

[0073] In some embodiments, the imaging device 204 and the illumination source 206 may be implemented by different devices (e.g., one of the imaging device 204 and the illumination source 206 may be implemented by a smart device, while the other of the imaging device 204 and the illumination source 206 may be implemented by a separate device (e.g., a dedicated device). In such embodiments, the control module 208 may be implemented by another separate device (e.g., a user computer, a dedicated processing device, a server, a cloud-based service, or other type of processing device), or the control module 208 may be implemented by one of the imaging device 204 and the illumination source 206.

[0074] In the case of a smart device, an onboard camera may provide the functionality of imaging device 204 , an onboard flash may provide the functionality of illumination source 206 , and / or onboard processing circuitry may provide the functionality of control module 208 .

[0075] Figure 3 A process 300 is depicted for improving imaging in certain scenarios according to an embodiment. The process 300 involves accessing data from a sequence 302 of images (four consecutive images in this example) characterized by the spatial intensity modulation pattern described above. The object in the image 302 is a face, but could be any other object whose surface features are of interest within the scenario. Figure 3 As shown, each consecutive image has a different spatial intensity modulation pattern (in this embodiment, this is due to the imaging device operating in a rolling shutter mode and sinusoidal intensity modulated illumination). Based on this information, it is possible to determine ambient lighting variations in the images in order to estimate an intensity distribution for the images that reduces the effects of, compensates for, or otherwise corrects for ambient lighting variations in the scene.

[0076] The pixel intensity value a(x1,y1) of a pixel at coordinates x1,y1 of an imaging device sensor may be tracked across a sequence 302 of images (ie, 'pixel columns') to determine the intensity value at this pixel as a function of the image (frame) number. Figure 3 As shown in the graph 304 in , a selected pixel a(x1, y1) varies in its intensity value (i.e., the 'modulated amplitude' at the pixel) across the sequence 302 of images (the graph 304 depicts 30 frames) in a relatively repeatable manner. As will be explained in more detail below, a 'corrected' amplitude (i.e., 'intensity value') can be retrieved based on the data in the graph 304. Therefore, by tracking the pixel intensity values ​​of all pixels across the sequence 302 of images, it is possible to reconstruct a 'corrected' image 306 from the data (i.e., the image 306 now takes into account ambient lighting changes in its scene to reduce the effects of, compensate for, or otherwise correct for, the ambient lighting changes).

[0077] Due to the convolution properties of harmonic temporal modulation, the combination of an imaging device operating in, for example, a rolling shutter mode and a sinusoidal modulation of the illumination results in each individual 'pixel data stream' of the resulting sequence of images also exhibiting a strict sinusoidal spatial modulation in the resulting image. Under these conditions, it has been found that relatively few images are required to estimate the intensity distribution of the image of an object in the scene.

[0078] For example, if each pixel in a 'pixel column' is equally spaced in time, four consecutive images may be used to retrieve the amplitude of the temporal modulation.Thus, in some embodiments, determining an estimated intensity distribution for an image may be based on four consecutive images from a sequence of images.

[0079] In another example, if the phase relationship of the sinusoidal modulation of the illumination and the imaging device frame rate is such that every fourth image in the sequence has the same spatial intensity modulation pattern as the first image in the sequence, then the amplitude of the temporal modulation may be retrieved using three consecutive images. Thus, in some embodiments, determining an estimated intensity distribution for an image may be based on three consecutive images from the sequence of images. Furthermore, in some embodiments, determining an estimated intensity distribution for an image may be based on three consecutive images from the sequence of images, wherein the spatial intensity modulation patterns of the first and fourth consecutive images in the sequence of images are the same.

[0080] In scenes where objects are moving (e.g., objects such as people may not be stationary due to voluntary and / or involuntary movement), the effect of such movement can change the apparent ambient lighting changes for a short period of time. Being able to use three or four images (rather than more) can reduce the effect of such movement to some extent. For an imaging device with 30 frames per second (fps), using four images, the images are acquired within 133 milliseconds. This can be considered a sufficiently short time scale for collecting data from a sequence of images while minimizing the chance that voluntary or involuntary movement of the object will adversely affect the estimate of the intensity distribution.

[0081] In some embodiments, the frequency of the sinusoidal intensity modulation is greater than twice the frame rate of the imaging device. In this case, at least two intensity peaks may be evident in each image of the sequence (e.g., Figure 3 shown).

[0082] In some embodiments, the frequency of the sinusoidal intensity modulation is at least 70 Hz (e.g., for flicker-free perception). This is an example of a frequency of the sinusoidal intensity modulation that is greater than twice the frame rate of the imaging device (e.g., for a frame rate of 30 Hz). If the imaging device can have a different frame rate (e.g., a faster frame rate), the frequency of the sinusoidal intensity modulation can be adjusted accordingly (e.g., increased to at least twice the frame rate of the imaging device).

[0083] In some embodiments, there may be a different relationship between the frequency of the sinusoidal intensity modulation and the frame rate of the imaging device. For example, the frequency of the sinusoidal intensity modulation may be approximately 50 Hz, where the frame rate of the imaging device is 30 Hz.

[0084] Certain processes for estimating intensity distribution have been described above. In some embodiments, the estimated intensity distribution A of the image may be given by:

[0085]

[0086] Where I1(x, y), I2(x, y), I3(x, y) and I4(x, y) represent the pixel intensity values ​​of the first, second, third and fourth consecutive images in the sequence of images, respectively. That is, for each pixel value (x, y) in the image, by using data from the sequence of images, the intensity distribution of the image that takes into account changes in ambient lighting can be estimated. The process can be run continuously so that when each new frame is acquired, the data is input into the expression for 'A' while using data from the previous three frames. This allows continuous updating of the image that takes into account changes in ambient lighting.

[0087] In some cases, poor control of the image acquisition process by the imaging device can lead to certain expected behaviors. For example, if the timing of the imaging device is well controlled, it may be difficult to coordinate the sinusoidal modulation of the illumination and the image acquisition process. Therefore, the phase progression of the spatial intensity modulation pattern in the sequence of images can be calculated according to the following formula:

[0088] Where I1(x, y), I2(x, y), I3(x, y), and I4(x, y) represent the pixel intensity values ​​of the first, second, third, and fourth consecutive images in the sequence of images, respectively. This phase progression can be taken into account when estimating the intensity distribution in the image. In some cases, the phase progression can be used to improve the coordination between the sinusoidal intensity modulation of the illumination and the image capture timing of the imaging device.

[0089] Some imaging devices (such as some non-scientific cameras) may have built-in nonlinear (e.g., gamma) corrections (e.g., Examples of Nonlinear Responses of Imaging Devices). If not characterized prior to implementing certain methods described herein, such nonlinear corrections may need to be taken into account when attempting to perform any quantitative measurements using images acquired by the imaging device. Gamma 'de-correction' (i.e., inverse gamma correction) is useful in some cases to estimate the intensity distribution in the case where the imaging device implements gamma correction. In some cases, the gamma correction implemented by a certain imaging device may be available to the user implementing the methods described herein (e.g., if the manufacturer publishes this information) without having to perform any specific measurements using the imaging device.

[0090] Thus, in some embodiments, the non-linear response of the imaging device may be compensated for prior to accessing data from the sequence of images and determining the estimated intensity distribution.

[0091] In some embodiments, the nonlinear response of the imaging device can be determined by obtaining an indication of the average intensity of a kernel (wherein a kernel refers to at least one pixel selected in the image) sampled from a set of images (e.g., a portion of the image can be sampled and / or multiple images can be sampled from the set of images) of an object (e.g., the object described above or another object) acquired by the imaging device. The nonlinear response of the imaging device can then be determined based on the indication. The nonlinear response can be used to identify compensation (e.g., including inverse gamma correction) to be applied to data corresponding to the sequence of images in order to compensate for the nonlinear response of the imaging device.

[0092] For example, compensating for a non-linear response of an imaging device may include performing an inverse gamma correction to compensate for the non-linear response due to a gamma correction applied to an image acquired by the imaging device.

[0093] Figure 4 Depicted is a process 400 for improving imaging in certain scenarios, according to an embodiment. The process 400 involves characterizing the nonlinear response of an imaging device (not shown) in order to determine the characteristics of the nonlinear (e.g., gamma) correction implemented by the imaging device (and thereby enable compensation for the nonlinear response to be applied to a sequence of images prior to implementing certain methods described herein).

[0094] In this embodiment, the illumination source is modulated according to an illumination function to illuminate the object. The illumination function 402 in this embodiment is configured to: obtain the average intensity of the kernel sampled from the set of images (e.g., Figure 4 , while at the same time indicating the pixel intensity values ​​in : a(x1, y1), a(x2, y2), and a(x3, y3)), at least one of the following is performed: linearly increasing the intensity of the illumination function 402 within a specified time interval, and linearly decreasing the intensity of the illumination function 402. This linear increase and decrease described by the illumination function 402 causes the illumination source to increase and decrease the illumination level provided accordingly. An imaging device (not shown) acquires a set of images over a period of time. By tracking the kernel on each set of images, the intensity value of each kernel is recorded to generate a curve graph 404. It can be seen from the curve graph that the linear increase or decrease in illumination does not correspond to a linear imaging device response. Based on the curvature (i.e., nonlinearity) of the imaging device response, appropriate compensation (e.g., inverse gamma correction) can be determined. This can be considered before implementing certain methods described herein. That is, before implementing such methods, the pixel values ​​of certain intensity levels can be appropriately adjusted.

[0095] Figure 5Depicted is a process 500 for improving imaging in certain scenarios, according to an embodiment. Similar to process 400, process 500 involves characterizing the response of an imaging device (not shown) in order to determine the characteristics of nonlinear (e.g., gamma) correction implemented by the imaging device (and thereby enable compensation for the nonlinear response to be applied to a sequence of images prior to implementing certain methods described herein).

[0096] In this embodiment, the imaging device is made to collect a set of images according to the imaging acquisition function. The imaging acquisition function in this embodiment is configured to: obtain the average intensity of the kernel sampled from the set of images (for example, Figure 5 While indicating the pixel intensity values ​​in the image: a(x1,y1), a(x2,y2), and a(x3,y3)), perform one of the following: linearly increase the exposure time of each subsequent image in the group of images, and linearly decrease the exposure time of each subsequent image in the group of images.

[0097] This linear increase and / or decrease in exposure time causes each image in the set of images to register a corresponding increase and / or decrease in intensity level. By tracking the kernel across each set of images, the intensity value of each kernel is registered to generate a graph 502. As can be seen from the graph, a linear increase or decrease in exposure time does not correspond to a linear imaging device response. Based on the curvature (i.e., nonlinearity) of the imaging device response, appropriate compensation (e.g., inverse gamma correction) can be determined. This can be considered before implementing certain methods described herein. That is, the pixel values ​​of certain intensity levels can be appropriately adjusted before implementing these methods.

[0098] Figure 6 A method 600 (e.g., a computer-implemented method) is shown for improving imaging in certain scenarios. For example, imaging may be affected by certain scenarios, such as in an uncontrolled environment where the illumination / ambient lighting is undefined and / or may vary. The method 600 may be implemented in conjunction with the method 100 and / or the reference processes 300, 400, 500, or as part of the method 100. In this example, any of the blocks of the method 600 may be omitted and / or reordered, as appropriate.

[0099] In some embodiments, method 600 includes, at block 602, compensating for a nonlinear response of an imaging device (eg, as described above) prior to: accessing data from a sequence of images; and determining an estimated intensity distribution.

[0100] In some embodiments, method 600 includes, at block 604, causing an illumination source to provide illumination having a temporal sinusoidal intensity modulation.

[0101] In some embodiments, method 600 includes, at block 606, calculating a phase progression of a spatial intensity modulation pattern in an image. (e.g., as described above).

[0102] Figure 7 A tangible machine-readable medium 700 is shown storing instructions 702 that, when executed by at least one processor 704, cause at least one processor 704 to implement certain methods described herein (such as method 100 and / or method 600).

[0103] In this embodiment, the instructions 702 include instructions 706 that cause the at least one processor 704 to implement block 102 of the method 100. The instructions 702 further include instructions 708 that cause the at least one processor 704 to implement block 104 of the method 100.

[0104] Figure 8 An apparatus 800 is shown, which may be used to implement certain methods described herein, such as method 100 and / or method 800. Apparatus 800 may include a device having a corresponding Figure 2 The system 200 depicts certain features of the functional modules, such as its control module 208 .

[0105] Apparatus 800 includes processing circuitry 802. Processing circuitry 802 includes access module 804. Access module 804 is configured to access data from a sequence of images of an object illuminated by ambient lighting and illumination having a temporal sinusoidal intensity modulation. The imaging device is configured such that different spatial intensity modulation patterns are apparent in consecutive images of the sequence.

[0106] The processing circuit system 802 further includes a determination module 806. The determination module 806 is configured to determine a set of modified pixel intensity values ​​for generating a modified image of the object based on the set of measured pixel intensity values ​​in each image in the sequence of images so that a reduced level of ambient lighting is apparent in the modified image compared to a level of ambient lighting apparent in at least one image in the sequence of images. In other words, the determination module 806 may be configured to determine an estimated intensity distribution (or a 'set of modified pixel intensity values') of the image of the object based on the spatial intensity modulation pattern (or 'set of measured pixel intensity values') in each image in the sequence of images to reduce ambient lighting variations in the image.

[0107] In some embodiments, the apparatus 800 further includes an illumination source (e.g., Figure 2 In some embodiments, the apparatus further comprises an imaging device (e.g., Figure 2 imaging device 204).

[0108] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments.

[0109] One or more features described in one embodiment may be combined with or replace features described in another embodiment. Figure 2 , 3 , 4, 5, 8, the system 200, the process 300, 400, 500, and / or the apparatus 800 described features to modify Figure 1 Or 6 methods 100, 600, and vice versa.

[0110] Embodiments of the present disclosure may be provided as a method, a system, or as a combination of machine-readable instructions and processing circuitry. Such machine-readable instructions may be embodied on a non-transitory machine (e.g., computer) readable storage medium (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-readable program code therein or thereon.

[0111] The present disclosure is described with reference to the flowcharts and block diagrams of the methods, devices and systems according to the embodiments of the present disclosure. Although the above flowcharts show a specific execution order, the execution order may be different from the described order. The frame described in one flowchart may be combined with the frame of another flowchart. It should be understood that each frame in the flowchart and / or block diagram and the combination of frames in the flowchart and / or block diagram can be implemented by machine-readable instructions.

[0112] Machine-readable instructions can be executed, for example, by a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to implement the functions described in the specification and the drawings. Specifically, a processor or a processing circuit system or a module thereof can execute machine-readable instructions. Therefore, the functional modules (e.g., control module 208, access module 804, and / or determination module 804) of the system 200 and / or the device 800 and other devices described herein can be implemented by a processor that executes machine-readable instructions stored in a memory or a processor that operates according to instructions embedded in a logic circuit. The term 'processor' should be broadly interpreted to include a CPU, a processing unit, an ASIC, a logic unit, or a programmable gate array, etc. These methods and functional modules can all be executed by a single processor, or divided between several processors.

[0113] Such machine-readable instructions may also be stored in a computer-readable storage device, which may direct a computer or other programmable data processing device to operate in a particular mode.

[0114] Such machine-readable instructions may also be loaded onto a computer or other programmable data processing device to cause the computer or other programmable data processing device to perform a series of operations to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device implement the functions specified by the box(es) in the flowchart and / or block diagram.

[0115] Furthermore, the teachings herein may be implemented in the form of a computer program product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device to implement the methods described in the embodiments of the present disclosure.

[0116] Elements or steps described with respect to one embodiment may be combined with or replaced with elements or steps described with respect to another embodiment. By studying the drawings, the disclosure and the appended claims, those skilled in the art may understand and implement variations to the disclosed embodiments when practicing the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may implement the functions of several items listed in the claims. The fact that certain measures are cited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. The computer program may be stored or distributed on an appropriate medium, such as an optical storage medium or solid-state medium provided together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems. Any reference numerals in the claims should not be interpreted as limiting the scope.

Claims

1. A computer-implemented method (100), comprising: accessing (102) data from a sequence of images of an object, the sequence of images acquired by an imaging device, the object being illuminated by ambient lighting and illumination having a temporal sinusoidal intensity modulation, wherein the imaging device is configured to operate in a rolling shutter mode to acquire, within each image of the sequence, a first spatial portion of the image at a different time than a second different spatial portion of the image, wherein a relationship between a frequency of the sinusoidal intensity modulation and a frame rate of the imaging device is such that different spatial intensity modulation patterns are apparent in consecutive images of the sequence; as well as Based on a set of measured pixel intensity values ​​in each image in the sequence of images, a set of corrected pixel intensity values ​​for generating a corrected image of the object is determined (104) by calculating, for each pixel in the set, a specified combination of the measured pixel intensity values ​​of the pixel from each image in the sequence of images, the specified combination generating the set of corrected pixel intensity values ​​corresponding to a reduced level of ambient illumination and a reduced modulation depth of the spatial intensity modulation pattern apparent in the corrected image compared to a level of ambient illumination and a modulation depth of the spatial intensity modulation pattern apparent in at least one image in the sequence of images, wherein the specified combination is based on a combination of the imaging device operating in a rolling shutter mode and the sinusoidal intensity modulation of the illumination, the combination resulting in each of the measured pixel intensity values ​​of the pixel from each image in the sequence of images exhibiting a sinusoidal intensity variation across the sequence of images due to a convolution characteristic of harmonic time modulation.

2. The method of claim 1 , wherein the set of modified pixel intensity values ​​is determined such that: the level of the ambient lighting and the modulation depth of the spatial intensity modulation pattern are both reduced in the modified image compared to at least one image in the sequence of images.

3. A method according to claim 1 or 2, wherein determining the set of corrected pixel intensity values ​​for the corrected image is based on three or four consecutive images from the sequence of images.

4. The method of claim 1 or 2, wherein the different spatial intensity modulation patterns are apparent in consecutive images of the sequence by at least one of: The frequency of the sinusoidal intensity modulation and the frame rate of the imaging device; and The phase difference between the sinusoidal intensity modulation and the frame acquisition timing of the imaging device.

5. The method of claim 4, wherein the frequency of the sinusoidal intensity modulation is not an integer multiple of the frame rate of the imaging device.

6. The method according to claim 1 or 2, causing (604) an illumination source to provide the illumination with the sinusoidal intensity modulation in time.

7. The method according to claim 1 or 2, wherein the set of corrected pixel intensity values ​​A for the corrected image is given by the specified combination as follows: Wherein I1(x, y), I2(x, y), I3(x, y) and I4(x, y) respectively represent the pixel intensity values ​​of the first continuous image, the second continuous image, the third continuous image and the fourth continuous image in the sequence of images.

8. The method according to claim 1 or 2, comprising calculating (606) the phase progression of the spatial intensity modulation pattern in the sequence of images according to Wherein I1(x, y), I2(x, y), I3(x, y) and I4(x, y) respectively represent the pixel intensity values ​​of the first continuous image, the second continuous image, the third continuous image and the fourth continuous image in the sequence of images.

9. The method of claim 1 or 2, comprising compensating (602) for a non-linear response of the imaging device prior to the steps of: accessing the data from the sequence of images; and determining the set of corrected pixel intensity values, wherein the non-linear response of the imaging device is determined by: obtaining an indication of a mean intensity of a kernel comprising the selected at least one pixel in an image sampled from a set of images of an object acquired by the imaging device; and The non-linear response of the imaging device is determined based on the indication of the mean intensity obtained for each image from the set of images.

10. The method according to claim 9, comprising: Causes an illumination source to illuminate the object by modulating the illumination source according to an illumination function, wherein the illumination function is configured to perform at least one of the following while obtaining the indication of the average intensity of the kernel obtained by sampling the set of images: linearly increase the intensity of the illumination function within a specified time interval, and linearly decrease the intensity of the illumination function within a specified time interval.

11. The method according to claim 9, comprising: The imaging device is caused to acquire the set of images according to an imaging acquisition function, wherein the imaging acquisition function is configured to perform one of the following while obtaining the indication of the average intensity of the kernel sampled from the set of images: linearly increase the exposure time for each subsequent image of the set of images, and linearly decrease the exposure time for each subsequent image of the set of images.

12. A tangible machine-readable medium (700) storing instructions (702) which, when executed by at least one processor (704), cause the at least one processor to implement the method of any one of claims 1-11.

13. An apparatus (800) comprising a processing circuit system (802), the processing circuit system comprising: an access module (804) configured to access data from a sequence of images of an object, the sequence of images acquired by an imaging device, the object being illuminated by ambient lighting and illumination having a temporal sinusoidal intensity modulation, wherein the imaging device is configured to operate in a rolling shutter mode to acquire, within each image of the sequence, a first spatial portion of the image at a different time than a second different spatial portion of the image, wherein a relationship between a frequency of the sinusoidal intensity modulation and a frame rate of the imaging device is such that different spatial intensity modulation patterns are apparent in consecutive images of the sequence; as well as A determination module (806) is configured to determine a set of corrected pixel intensity values ​​for generating a corrected image of the object based on a set of measured pixel intensity values ​​in each image in the sequence of images by calculating, for each pixel in the set, a specified combination of the measured pixel intensity values ​​of the pixel from each image in the sequence of images, the specified combination generating the set of corrected pixel intensity values ​​corresponding to a reduced level of ambient illumination and a reduced modulation depth of the spatial intensity modulation pattern apparent in the corrected image compared to a level of ambient illumination and a modulation depth of the spatial intensity modulation pattern apparent in at least one image in the sequence of images, wherein the specified combination is based on a combination of the imaging device operating in a rolling shutter mode and the sinusoidal intensity modulation of the illumination, the combination resulting in each of the measured pixel intensity values ​​of the pixel from each image in the sequence of images exhibiting a sinusoidal intensity variation across the sequence of images due to a convolution characteristic of harmonic time modulation.

Images