Determining pixel information

Abstract

In one embodiment, a method (100) is described. The method is a computer- implemented method. The method includes receiving (102) pixel information corresponding to at least a portion of a first image, a second image, and a third image of an object illuminated by time-modulated illumination acquired by an imaging system. A time delay between acquisition start times of the first image, the second image, and the third image is such that there is a temporal overlap during acquisition of the first image, the second image, and the third image. The method also includes determining (104) modified pixel information for constructing a modified image of the object. The modified pixel information is determined based on: a difference between the received pixel information of the first image, the second image, and the third image to reduce an effect of ambient illumination in the received pixel information; and combining the received pixel information to cancel a spatial intensity modulation pattern evident in each of the first image, the second image, and the third image. The spatial intensity modulation pattern is due to a timing relationship between an imaging time period in which each image is acquired by the imaging system and a modulation time period of the time-modulated illumination.

Description

TECHNICAL FIELD

[0001] The present invention relates to methods, apparatuses, and tangible machine-readable media for imaging in certain settings. BACKGROUND

[0002] A subject of interest in the field of unobtrusive measurement and monitoring relates to skin sensing for personal care and health applications. Skin sensing systems are being developed that promise skin quantification and monitoring of features in the skin that can provide user information that is too small to detect, too weak to notice, or too slow to follow. To deliver user-acceptable results, such skin sensing systems can need to provide sensitivity and specificity in performing skin sensing. Providing measurements by such skin sensing systems that prove to be robust and reliable, users can build trust in these skin sensing systems.

[0003] Imaging-based skin sensing systems can need to determine information that can be affected by uncontrolled parameters, such as environmental lighting variations. For example, certain uncontrolled environments, such as in a user’s home, can have undefined and / or possibly varying environmental lighting. Such uncontrolled environments can lead to false measurements of a user’s skin, which in turn can lead to results that are unacceptable or untrusted by the user. Imaging performance of some cameras, such as smartphone cameras, used in some imaging-based skin sensing systems can be variable, such that imaging data is unpredictable or unreliable.

[0004] Imaging-based skin sensing systems can implement various image processing techniques in order to determine certain information about a user’s skin and / or environmental lighting conditions. When implementing certain image processing techniques, relative motion between a user and an imaging system can lead to errors.

[0005] US 2015 / 249496 Al describes a method for detecting modulated light, the method comprising: receiving a set of images acquired by a rolling shutter camera having image acquisition settings including a frame rate and a line rate; identifying in successive frames of the images a pattern governed by a ratio between a modulation frequency of a modulated light source and the line rate, and identifying between successive frames a spatial shift of the pattern governed by a ratio between the modulation frequency and the frame rate; and providing an estimate of a modulation light amplitude from the light source based on the pattern and its spatial offset.

[0006] EP 3474727A1 describes an apparatus comprising an illumination unit for illuminating a skin region at a flash rate, an imaging unit for capturing images of the skin region at an imaging rate different from the flash rate, and a processing unit for processing the captured images and detecting an amount of shine in the skin region from at least one partial image of the skin region when illuminated by the illumination unit and at least one partial image of the skin region when not illuminated by the illumination unit.

[0007] WO 2013 / 171612A1 describes a motion sensing system comprising an imaging module adapted to capture at least one first image of a sensing region when a light emitting module is activated and at least one second image of the sensing region when the light emitting module is deactivated, wherein at least three images are captured during respective, possibly non-overlapping, time intervals.

[0008] US 2020 / 154024A1 describes an imaging system comprising a rolling shutter sensor capturing a plurality of images of a scene, a time-varying illumination source illuminating the scene, and a processor receiving the plurality of images from the rolling shutter sensor and separating the plurality of images into a plurality of feeds. SUMMARY

[0009] Aspects or embodiments described herein relate to improving imaging in certain settings. Aspects or embodiments described herein can eliminate one or more problems associated with certain image processing techniques where there is relative motion between a subject (e.g., a user) and an imaging system.

[0010] In a first aspect, a method is described. The method is a computer-implemented method. The method includes receiving pixel information corresponding to at least a portion of a first image frame, a second image frame, and a third image frame of an object illuminated by time-modulated illumination. The first image frame, the second image frame, and the third image frame are captured by an imaging system including at least two imaging devices for capturing the first image frame, the second image frame, and the third image frame. A time delay between a capture start time of the first image frame and the second image frame is such that there is a temporal overlap during capture of the first image frame and the second image frame. A time delay between a capture start time of the second image frame and the third image frame is such that there is a temporal overlap during capture of the second image frame and the third image frame. The time delay between the capture start time of the first image frame and the second image frame and the time delay between the capture start time of the second image frame and the third image frame are based on a modulation time period of the time-modulated illumination. The time delays correspond to the modulation time period divided by a number of image frames that are partially captured at a time within the overlapping capture times, such that a timing relationship between an imaging time period in which each image frame is captured by the imaging system and the modulation time period of the time-modulated illumination is such that a different spatial intensity modulation pattern is registered in each of the first image frame, the second image frame, and the third image frame. The method further includes determining modified pixel intensity values for constructing a modified image of the object. The determined modified pixel intensity values are based on a difference between the received pixel information of the first image frame, the second image frame, and the third image frame to determine a combined pixel intensity value for each different pair combination. The difference reduces an effect of ambient illumination present when each of the first image frame, the second image frame, and the third image frame is captured. The determined modified pixel information is further based on a combination of the determined pixel information for different pair combinations to cancel out the spatial intensity modulation pattern evident in each of the first image frame, the second image frame, and the third image frame.

[0011] Some embodiments relating to the first aspect are described below.

[0012] In some embodiments, the modified pixel intensity values for constructing the modified image of the object are determined based on subtracting pixel intensity values of respective pixels from each different pair combination of the first image frame, the second image frame, and the third image frame to produce a resulting pixel intensity value for each respective pixel of each different pair combination. The subtraction reduces an effect of ambient illumination present when each of the first image frame, the second image frame, and the third image frame is captured. The modified pixel intensity values for constructing the modified image of the object are determined based on squaring the resulting pixel intensity values of each different pair combination and summing each squared resulting pixel intensity value to cancel out the spatial intensity modulation pattern evident in each of the first image frame, the second image frame, and the third image frame.

[0013] In some embodiments, the phase shifts of the spatial intensity modulation patterns between the first, second, and third image frames are equal. The modified image M AC The modified pixel intensity values are determined based on the following equation:

[0014]

[0015] where Ii is a pixel intensity value in the first image frame, I2 refers to a pixel intensity value in the second image frame, and I3 refers to a pixel intensity value in the third image frame.

[0016] In some embodiments, one of the imaging devices is used to capture at least one of the first, second, and third image frames, and at least another of the imaging devices is used to capture at least another of the first, second, and third image frames.

[0017] In some embodiments, the method further comprises causing the imaging system to capture the first, second, and third image frames.

[0018] In some embodiments, the method further comprises receiving an indication of a modulation parameter of the time-modulated illumination; determining an operating parameter of the imaging system based on the indication; and causing the imaging system to operate in accordance with the operating parameter.

[0019] In some embodiments, a modulation frequency and / or a frame rate of the time-modulated illumination associated with the first, second, and third image frames are configured such that the phase shifts between the spatial intensity modulation patterns of the first, second, and third image frames are the same.

[0020] In some embodiments, the method further comprises, prior to determining the modified pixel intensity values, identifying whether a location of the object within the first, second, and third image frames is the same. In a case where a difference in the object location in at least one of the image frames compared to at least another of the image frames exceeds a threshold, the method further comprises implementing an image shift operation in at least one of the first, second, and third image frames such that the object location in each of the first, second, and third image frames is the same.

[0021] In some embodiments, the imaging system is configured to operate in a pattern according to a timing relationship between an imaging time period in which each image frame is captured by the imaging system and a modulation time period of the time-modulated illumination, wherein the pattern is either a rolling shutter pattern in which the imaging time period of the imaging system is longer than the modulation time period, or a global shutter pattern in which the imaging time period is shorter than the modulation time period.

[0022] In some embodiments, receiving the data further comprises receiving pixel intensity values corresponding to at least a portion of at least one additional image of the object, wherein a time delay between a start time of acquisition of the third image frame and the at least one additional image frame is such that there is a temporal overlap during acquisition of the third image frame and the at least one additional image frame. The time delay between the start time of acquisition of the third image frame and the at least one additional image frame is based on a modulation time period of the time-modulated illumination. The time delay corresponds to the modulation time period divided by a number of image frames that are partially acquired at a time within the overlapping acquisition time, such that a timing relationship between an imaging time period of each image frame acquired by the imaging system and the modulation time period of the time-modulated illumination is such that each of the first image frame, the second image frame, the third image frame, and the at least one additional image frame registers a different spatial intensity modulation pattern. Modifying the pixel intensity values is determined based on a difference between received pixel intensity values for each different pair combination of the first image frame, the second image frame, the third image frame, and the at least one additional image frame to determine pixel intensity values for each different pair combination. The difference reduces an effect of ambient illumination present when each of the first image frame, the second image frame, the third image frame, and the at least one additional image frame is acquired. Modifying the pixel intensity values is further determined based on a combination of the determined pixel intensity values for each different pair combination to cancel out the spatial intensity modulation pattern evident in each of the first image frame, the second image frame, the third image frame, and the at least one additional image frame.

[0023] In a second aspect, a tangible machine-readable medium is described. The tangible machine-readable medium stores instructions that, when executed by at least one processor, cause the at least one processor to implement a method according to the first aspect or any related embodiment.

[0024] In a third aspect, an apparatus is described. The apparatus comprises processing circuitry. The processing circuitry comprises a receiving module and a determining module. The receiving module is configured to receive pixel intensity values corresponding to at least a portion of first, second and third image frames of an object illuminated by time-modulated illumination. The first, second and third image frames are captured by an imaging system comprising at least two imaging devices for capturing the first, second and third image frames. A time delay between a capture start time of the first and second image frames is such that there is a temporal overlap during the capture of the first and second image frames. A time delay between a capture start time of the second and third image frames is such that there is a temporal overlap during the capture of the second and third image frames. The time delay between the capture start time of the first and second image frames, and the time delay between the capture start time of the second and third image frames are based on a modulation time period of the time-modulated illumination. The time delay corresponds to the modulation time period divided by a number of image frames that are partially captured at a time within the overlapping capture time, such that a timing relationship between an imaging time period in which each image frame is captured by the imaging system and the modulation time period of the time-modulated illumination is such that a different spatial intensity modulation pattern is registered in each of the first, second and third image frames. The determining module is configured to determine modified pixel intensity values for constructing a modified image of the object. The determined modified pixel intensity values are based on a difference between pixel information combined for each different pair. The difference reduces an effect of ambient illumination present at the time of each of the first, second and third image frames. The determined modified pixel intensity values are further based on the determined pixel intensity values for each different pair combination to counteract the spatial intensity modulation pattern evident in each of the first, second and third image frames.

[0025] Some embodiments relating to the third aspect are described below.

[0026] In some embodiments, the apparatus further comprises: an imaging system for capturing the pixel intensity values corresponding to at least a portion of the first, second and third image frames; and / or an illumination unit for providing the time-modulated illumination.

[0027] In some embodiments, the imaging system is configured to operate in a mode according to a timing relationship between an imaging time period in which each image frame is captured by the imaging system and a modulation time period of the time-modulated illumination. The mode is: a rolling shutter mode, in which the imaging time period of the imaging system is longer than the modulation time period; or a global shutter mode, in which the imaging time period is shorter than the modulation time period.

[0028] These and other aspects of the application will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0029] Exemplary embodiments of the application will now be described, by way of example only, with reference to the following drawings:

[0030] Figure 1 Refers to a method of improving imaging in certain settings according to an embodiment;

[0031] Figure 2 Is a schematic of a system for improving imaging in certain settings according to an embodiment;

[0032] Figures 3a to 3b Is a schematic of an imaging system according to two embodiments;

[0033] Figure 4 Is a schematic of a system for implementing certain methods described herein according to an embodiment;

[0034] Figures 5a to 5b Is a schematic of an imaging system according to two embodiments;

[0035] Figure 6 Refers to a method of improving imaging in certain settings according to an embodiment;

[0036] Figure 7 Is a schematic of a machine-readable medium for improving imaging in certain settings according to an embodiment; and

[0037] Figure 8 Is a schematic of an apparatus for improving imaging in certain settings according to an embodiment. DETAILED DESCRIPTION

[0038] In certain settings, ambient lighting can affect the performance of certain imaging-based skin sensing systems. Images acquired by the imaging system can suffer from artifacts caused by relative motion between the subject and the imaging system. This relative motion can be due to the subject not being able to easily stay in the same position and / or due to the subject holding a device that includes the imaging system, so the subject can find it difficult to keep the imaging system stable. Examples of devices include smart devices such as a smart phone, a tablet, a smart mirror, or any other device capable of displaying an image or representation of the subject.

[0039] This relative motion can mean that a set of consecutive images of an object can show the object at different positions within the image frames and / or distort the appearance of the object in the set of images. Thus, when performing certain image processing techniques such as ambient illumination correction, normalization, and / or reducing the effect of ambient illumination changes between consecutive images, the embodiments described herein can provide ways to reduce the effect of this relative motion.

[0040] Figure 1 A method 100 (e.g., a computer-implemented method) is shown that improves imaging in certain settings. The method 100 can be implemented by a computer, such as a user device, or a server or cloud-based service (e.g., communicatively coupled to the user device).

[0041] The method 100 includes, at block 102, receiving pixel information (e.g., ‘data’ or ‘intensity data’ such as pixel intensity values) corresponding to at least a portion of a first image, a second image, and a third image of an object illuminated by time-modulated illumination acquired by an imaging system. A time delay between the acquisition start times of the first image, the second image, and the third image is such that there is an overlap in time during the acquisition of the first image, the second image, and the third image.

[0042] During the image acquisition of the first image, the second image, and the third image (e.g., by an imaging system described in more detail below), a portion of these images can be received by the computer implementing the method 100. For example, in embodiments where the images are acquired by an imaging system configured to operate in a rolling shutter mode, a portion of the image can be acquired and sent to the computer for processing before a subsequent portion of the image is acquired and sent to the computer. In other words, the entire image frame need not be acquired in order to facilitate the implementation of the remainder of the method 100. However, in some cases, the entire image can be acquired before the remainder of the method 100 is implemented.

[0043] As will be described in more detail below, an illumination unit provides the time-modulated illumination. The illumination unit can or can not be controlled by the same computer implementing the method 100. The time-modulated illumination provides illumination to the object with an intensity that varies as a function of time. For example, the time-modulated illumination can have a periodically varying illumination intensity level (such as a sinusoidal variation of the illumination intensity level over time).

[0044] The total acquisition time for each image can be defined by the frame rate of the imaging device of the imaging system. For example, some imaging devices can operate at a frame rate of 30 fps, resulting in an acquisition of more than about 30 ms per frame. Some other imaging devices can operate at different frame rates. Increased (e.g., greater than 30 fps) frame rates achieved by certain imaging devices can reduce artifacts in the acquired images caused by relative motion, but acquire less light due to lower exposure times per frame (which can affect the quality of the images), and / or these imaging devices can be relatively more expensive and / or more complex to implement than some other imaging devices with lower frame rates. Conversely, lower (e.g., less than 30 fps) frame rates achieved by certain imaging devices can increase artifacts in the acquired images caused by relative motion, but increased exposure times per frame can provide images of increased quality as compared to higher frame rate imaging devices.

[0045] Assuming the delay between the acquisition start time for each subsequent image is less than the total acquisition time for each image, the imaging system can operate at an effectively increased frame rate while also providing sufficient image quality (as the exposure time for each image is still long enough to acquire sufficient light for the image). For example, if it takes 30 ms to acquire each image, the first image can begin to be acquired at time t = 0 (zero) ms. The second image can begin to be acquired at a time less than 30 ms after t = 0 (e.g., at t = 5 ms, or at another appropriate time). The third image can begin to be acquired at a time less than 30 ms after t = 0 (e.g., at t = 10 ms, or at another appropriate time). Thus, each of the second and third images begins to be acquired at the same time as the first image is being acquired. Further, the third image begins to be acquired at the same time as the second image is being acquired. In one example, if 3 (three) images are acquired within a time frame of 40 ms (following the example above), the effective frame rate is 3 / 40 ms = 75 fps (i.e., instead of 30 fps without overlap in the acquisition of the first, second, and third images). Thus, a higher frame rate can be achieved while ensuring that the exposure time is long enough to achieve sufficient image quality.

[0046] As will be described in greater detail below, the imaging system can be configured to acquire at least a first image, a second image, and a third image with a time delay as described in block 102 to achieve an increased frame rate while also providing sufficient image quality.

[0047] The method 100 also includes, at block 104, determining modified pixel information (e.g., pixel intensity values) for a modified image (referred to herein as an “ambient-corrected image”) of the object under construction. The modified pixel information is based on differences between the received pixel information of the first, second, and third images to reduce the effects of ambient illumination in the received pixel information. The modified pixel information is also based on a combination of the received pixel information to cancel out spatial intensity modulation patterns evident in each of the first, second, and third images. The spatial intensity modulation patterns are due to the timing relationship between the imaging time period in which each image is captured and the modulation time period of the time-modulated illumination. In other words, block 104 determines an estimated intensity distribution (i.e., “modified pixel information”) for an image of the object (i.e., the ‘modified image’) to reduce the effects of ambient illumination caused by the light source. Determining the estimated intensity distribution is based on the spatial intensity modulation patterns in each of the first, second, and third images.

[0048] The spatial intensity modulation patterns can be observed in each of the first, second, and third images. The spatial intensity modulation patterns can be a result of the time-modulated illumination causing variations in illumination levels over time when the images are captured (e.g., in some embodiments, this can be due to a rolling shutter effect causing different spatial portions of the imaging device to detect varying illumination levels as these spatial portions are scanned). For example, a certain spatial portion of the imaging device can record a relatively high illumination level at a certain time when the illumination level provided by the illumination unit is high, while another spatial portion of the imaging device can record a relatively low illumination level at a later time in the same frame when the illumination level provided by the illumination unit is low.

[0049] This means that the entire image can present certain areas with a relatively higher level of illumination than other areas, and this can be considered to represent a "spatial intensity modulation pattern" - an example of which is described below. Depending on the relative timing of the image acquisition (i.e., the 'temporal relationship'), the imaging acquisition rate, and the modulation frequency of the time-modulated illumination, the spatial intensity modulation pattern can vary between subsequent images. Thus, in some cases, the first image, the second image, and the third image can each have different spatial intensity modulation patterns. From this information, an intensity distribution of an image (i.e., an ambient corrected or ambient compensated image constructed based on the first image, the second image, and the third image) can be estimated that reduces the effects of ambient illumination caused by light sources (e.g., light sources different from the illumination unit, such as light bulbs, sunlight, etc.). In other words, the spatial intensity modulation pattern is due to the temporal relationship between the imaging time period in which the imaging system acquires each image and the modulation time period of the time-modulated illumination. As described with respect to block 104, the combination of the received pixel information can cancel out the spatial intensity modulation pattern that is apparent in each of the first image, the second image, and the third image, while the differences between the received pixel information of the first image, the second image, and the third image can reduce the effects of ambient illumination in the received pixel information.

[0050] Certain methods described herein (e.g., method 100) can allow for obtaining an ambient corrected (or normalized) image in a relatively short capture time, which can reduce artifacts from relative motion that can otherwise detrimentally affect the reduction of ambient lighting effects. The amount of light used for each image capture can be sufficient to provide sufficient image quality to facilitate ambient light reduction, correction, or normalization, which will be described in more detail below.

[0051] In the case of ambient light reduction, normalization, or correction using the time-modulated illumination techniques described above, relative motion can affect the quality of the ambient light removal. According to certain embodiments described herein, relatively fast image acquisition times can be achieved, which can result in ambient light reduction, correction, or normalization that is robust to relative motion with image quality that is useful for skin sensing applications.

[0052] Figure 2 An example system 200 for improving imaging according to certain embodiments is shown. System 200 can implement, at least in part, certain methods described herein, such as method 100 above or method 600 below.

[0053] The system 200 is used by a subject 202 and includes an imaging system 204 and an illumination unit 206. The imaging system 204 is used to acquire images involved in the method 100. The illumination unit 206 is used to provide time-modulated illumination. The imaging system 204 and / or the illumination unit 206 can be implemented by a user device as described above. Thus, in some examples, a separate user device can include the imaging system 204 and the illumination unit 206, and in other examples, the same user device can include the imaging system 204 and the illumination unit 206.

[0054] In some embodiments, the imaging system 204 includes at least two imaging devices (e.g., cameras in a rolling shutter mode) capable of detecting a single or multiple light sources interacting with a surface of the subject 202 for capturing images or videos. The frame rates of the at least two imaging devices can be similar to each other. Further, in some embodiments, the imaging axes of the at least two imaging devices can be substantially collinear.

[0055] The system 200 also includes a computer 208 (e.g., including processing circuitry implemented by a user device or a server or a cloud-based service for implementing certain methods described herein). Thus, the computer 208 can be communicatively coupled to the imaging system 204 and the illumination unit 206 to send information to and / or receive information from these components. For example, the computer 208 can receive data corresponding to the first image, the second image, and the third image from the imaging system 204. This data can be processed by the processing circuitry of the computer 208 and / or stored in a memory (e.g., of the computer 208 or accessible to the processing circuitry of the computer 208). In some embodiments, the computer 208 can control the operation of the imaging system 204 and / or the illumination unit 206. In some embodiments, the computer 208 includes a controller for controlling illumination parameters (e.g., operational parameters for the illumination unit 206) and / or detection parameters (e.g., operational parameters for the imaging system 204) and storing and / or processing captured images or videos.

[0056] Figures 3a to 3b Two different (e.g., imaging systems 304a and 304b of the system 200 shown) imaging systems for acquiring the first image, the second image, and the third image are schematically depicted according to two embodiments. Figure 2

[0057] Figure 3a ​The imaging system 304a includes first and second imaging devices 310a and 310b. These two imaging devices 310a-b can be triggered to begin acquiring images with a time delay between them, while still providing temporal overlap between at least two of the first, second, and third images. The acquisition timing of the first, second, and third images (labeled '1', '2', and '3' in the figure, respectively) is determined by... Figure 3a The timeline is depicted in the diagram. At the start of acquisition, a first image '1' is acquired using the first imaging device 310a. The acquisition time is less than the total acquisition time of the first image (e.g., see [link]). Figure 3a After a time delay (the length of frame '1' in the image), the second image '2' is acquired using the second imaging device 310b. This is followed by another time delay less than the total acquisition time of the second image (see, for example, see...). Figure 3a After the “length” of frame '2’ in the first image is determined, the third image '3’ is acquired using the first imaging device 310a (i.e., after the acquisition of the first image '1' is completed).

[0058] Using two imaging devices 310a and 310b in the above embodiments allows for the acquisition of the first, second, and third images in a shorter time than using a single imaging device. Therefore, the effective frame rate is increased by using imaging system 304a. In this example, the start time for acquiring the second image '2' is midway through the acquisition of the first image '1'. For example, if the acquisition time for the first image '1' is 30 ms, the acquisition of the second image '2' begins 15 ms after the acquisition of the first image '2', and similarly for acquiring the third image '3' relative to the second image '2'. In other embodiments, this timing configuration can be different. For example, the second image '2' can be acquired with a different time delay relative to the start of acquisition of the first image '1' (e.g., before or after the midway point), and similarly for acquiring the third image '3' relative to the second image '1'.

[0059] Figure 3b The 304b imaging system can be similar to Figure 3a The imaging system 304a is implemented in this manner. However, the imaging system 304b includes first, second, and third imaging devices 310a, 310b, and 310c for acquiring the first image, the second image, and the third image, respectively. Therefore, the third imaging device 310c can begin acquiring the third image 3 before the first imaging device 310a has finished acquiring the first image 1. Figure 3a Compared to the imaging system 304a, the imaging system 304b can provide a faster effective frame rate.

[0060] Accordingly, in some embodiments, the imaging system comprises at least two imaging devices for acquiring the first, second, and third images. In some embodiments, the at least two imaging devices comprise imaging devices configured to operate in a rolling shutter mode (i.e., rolling shutter cameras). These imaging devices can be placed in close proximity to each other, similar to what can be found in some devices such as smartphones. In some embodiments, the at least two imaging devices comprise imaging devices configured to operate in a global shutter mode.

[0061] Further, in some embodiments, one of the imaging devices is used to acquire at least one of the first, second, and third images, and at least another of the imaging devices is used to acquire at least another of the first, second, and third images.

[0062] Figure 4 A system for implementing certain methods described herein based on an imaging system 304b implementing Figure 3b is schematically depicted, wherein the first, second, and third images ‘1’, ‘2’, and ‘3’ are acquired using three imaging devices 310a, 310b, and 310c. Figure 4 The first, second, and third images shown feature the above-mentioned spatial intensity modulation pattern. Based on the data corresponding to these three images, an “ambient corrected” image ‘4’ can be determined (e.g., in accordance with block 104 of method 100).

[0063] As mentioned above, the determined modified pixel information is based on the differences between the received pixel information of the first, second, and third images to reduce the influence of ambient lighting in the received pixel information. The determined modified pixel information is further based on a combination of the received pixel information to counteract the spatial intensity modulation pattern apparent in each of the first, second, and third images.

[0064] In some embodiments, the modified pixel information used to construct a modified image of the object is determined by subtracting the different pairs of combined pixel information of the first, second, and third images to reduce the influence of ambient lighting in the received pixel information, and combining the received pixel information to counteract the spatial intensity modulation pattern apparent in each of the first, second, and third images.

[0065] In some embodiments, the imaging system is configured to operate in a mode according to a timing relationship between an imaging time period in which each image is acquired by the imaging system and a modulation time period of the time-modulated illumination. The mode can be a rolling shutter mode, in which the imaging time period of the imaging system is longer than the modulation time period, or a global shutter mode, in which the imaging time period is shorter than the modulation time period. In either mode, the spatial intensity modulation pattern can be observed in each acquired image.

[0066] Assuming equal phase shifts between the spatial intensity modulation patterns in the three images (e.g. 120 degrees for the above embodiment), the intensity M of the ambient corrected image can be estimated according to the example formula AC :

[0067]

[0068] where Ii refers to the (pixel) intensity value in the first image, I2 refers to the (pixel) intensity value in the second image, and I3 refers to the (pixel) intensity value in the third image. As indicated by the above formula, the ambient corrected image (i.e. the modified pixel information used to construct such an image) is determined based on a‘difference’ between the received pixel information of the first, second, and third images (i.e. (Ii-I2), etc.) and a‘combination’ of the received pixel information (i.e. (Ii-I2)2+ (I2-I3)2+ (Ii-I3)2). In some embodiments, each expression (Ii-I2), (I2-I3), (Ii-I3) refers to a‘combination’. Some embodiments described herein relate to at least one‘additional image’ (in addition to the first, second, and third images). Thus, the example formula can be modified according to how many images are acquired.

[0069] In some embodiments, the modulation time period of the time-modulated illumination (i.e. the‘illumination flicker’) is approximately 15 ms (i.e. for a modulation frequency of 70 Hz). By triggering the capture of the images with a delay of 5 ms for the second imaging device and a delay of 10 ms for the third imaging device, the resulting spatial intensity modulation pattern in the captured first, second, and third images can have a phase shift of 120 degrees.

[0070] Thus, the time delay between the acquisition start time of the first, second, and third images is based on the modulation time period of the time-modulated illumination. For example, if the modulation time period is shorter than the above embodiment due to the illumination unit providing the time-modulated illumination at a frequency larger than 70 Hz, the time delay between the first, second, and third images can be smaller than the above embodiment.

[0071] The time delay between the acquisition start times (i.e., the acquisition start times of the first, second, third images) corresponds to the modulation time period divided by the number of images. Thus, in the above embodiment, the modulation time period is approximately 15 ms, and three images are acquired. Thus, for this embodiment, the time delay used is approximately 5 ms.

[0072] In some embodiments, the modulation frequency and / or frame rate of the time-modulated illumination associated with the first, second, and third images is configured such that the phase shift between the spatial intensity modulation patterns of the first, second, and third images is the same. In other words, the modulation frequency and / or frame rate of the imaging system can be configured such that the phase shift between the first, second, and third images is the same.

[0073] In some embodiments, the received (e.g., of block 102) data further includes data corresponding to at least a portion of at least one additional image (e.g., a fourth image, a fifth image, etc. images acquired by one of the at least two imaging devices) of the object. The time delay between the acquisition start times of the first, second, third, and at least one additional images is such that there is a temporal overlap during the acquisition of the first, second, third, and at least one additional images. The determining (e.g., of block 104) of the modified pixel information is based on: a difference between the received pixel information of the first, second, third, and at least one additional images to reduce the effect of ambient illumination in the received pixel information; and a combination of the received pixel information to cancel out the spatial intensity modulation pattern apparent in each of the first, second, third, and at least one additional images. Some imaging devices can be configured to acquire more than three images (e.g., if such an imaging system is properly configured and / or if it includes more than two or more than three imaging devices).

[0074] In some embodiments, the time delay between the acquisition start times of the first, second, third, and at least one additional images is based on the modulation time period of the time-modulated illumination. The time delay between the acquisition start times can correspond to the modulation time period divided by the number of images. For example, if the imaging system is capable of acquiring a larger number of images (e.g., at least four images) than the above embodiment, then for the same modulation time period (i.e., 15 ms), the time delay between the acquisition times of the first, second, third, fourth (or more) images can be less than the time delay in the above embodiment.

[0075] Figures 5a to 5b The acquisition of the first, second, and third images by the imaging devices 510a, 510b, 510c (e.g., similar to the imaging devices 110a, 110b, 110c of FIG. 1) is schematically depicted in FIG. 5A. The acquisition of the first, second, and third images is performed in a time-modulated manner, as described above. The acquisition of the first, second, and third images is performed in a time-modulated manner, as described above. Figure 3bpossible orientations of the rolling shutter scan direction implemented by the imaging devices 310a, 310b, 310c. The rolling shutter scan direction is the same for each imaging system, such that the spatial intensity modulation pattern in each image is "scanned" in the same direction (as indicated by the arrows). Figures 5a to 5b Figure 5a Three imaging devices 510a, 510b, 510c are shown positioned at a horizontal separation location, with the rolling shutter scan direction being in the vertical direction. Figure 5b Three imaging devices 510a, 510b, 510c are shown positioned at a vertical separation location, with the rolling shutter scan direction being in the horizontal direction. In some embodiments, the imaging devices are aligned in the same (i.e., parallel) direction as the rolling shutter scan direction. In some embodiments, the imaging devices are aligned in a direction perpendicular to the rolling shutter scan direction.

[0076] In cases where the positions of the imaging devices are in different directions of the rolling shutter scan direction, an adjustment of the time delay can be used such that the phase shift of the pattern corresponds to 120 degrees with respect to the target object. In some embodiments, the scan direction of the rolling shutter sensors is different for each sensor.

[0077] Figure 6 A method 600 (e.g., computer-implemented method) of improving imaging in certain setups is shown. The method 600 can be implemented by a computer, such as a user device, or a server or cloud-based service (e.g., communicatively coupled to the user device). The method 600 includes blocks 102 and 104 of the method 100. Certain blocks of the method 600 can be omitted and / or Figure 6 The blocks shown can be implemented in a different order than depicted. For convenience, reference is made to the system 200 of Figure 2

[0078] In some embodiments, the method 600 further includes causing the imaging system 204 to acquire the first image, the second image, and the third image at block 602. In some embodiments, more than three images can be acquired, in which case the imaging system 204 can be caused to acquire the first image, the second image, the third image, and at least one additional image.

[0079] ​​In some embodiments, the method 600 includes, at block 604 and prior to determining to modify the pixel information, identifying whether the location of the object within the first image, the second image, and the third image is the same. In cases where the location of the object in at least one image differs from at least one other image by more than a threshold (e.g., a threshold number of pixels for a certain feature of the image to be shifted between images), block 604 of the method 600 includes implementing an image shift operation in at least one of the first image, the second image, and the third image so that the object location is the same in each of the first image, the second image, and the third image. For example, due to the distance between the imaging devices and / or due to relative motion between the object and the imaging system, image shifting can be needed to properly realign the images prior to performing ambient light correction (e.g., at block 104).

[0080] However, in some embodiments, to avoid image offset issues due to the distance between the imaging devices, a beamsplitter or other optical arrangement (not shown) can be used to image the exact same image, but different imaging devices are configured to receive the image using the same optical arrangement.

[0081] In some embodiments, the method 600 includes, at block 606, receiving an indication of a modulation parameter (e.g., a modulation frequency or a modulation period) of the time-modulated illumination; determining, based on the indication, an operating parameter of the imaging system 204 (e.g., a frame rate, an image acquisition start time, etc.); and causing the imaging system 204 to operate in accordance with the operating parameter. For example, the system 200 can determine or receive an indication of a modulation frequency of the time-modulated illumination, and adjust the frame rate and a time delay between the imaging devices of the imaging system 204 accordingly.

[0082] Figure 7 A tangible machine-readable medium 700 that stores instructions 702 is illustratively depicted, which when executed by at least one processor 704, cause the at least one processor 704 to implement certain methods described herein, such as the method 100 or 600.

[0083] In this embodiment, the instructions 702 include instructions 706 that cause the at least one processor 704 to implement block 102 of the method 100. The instructions 702 also include instructions 708 that cause the at least one processor 704 to implement block 104 of the method 100.

[0084] Figure 8 An apparatus 800 is shown, which can be used to implement certain methods described herein, such as the method 100 and / or the method 600. The apparatus 800 can include modules (such as its computer 208) having functionality corresponding to certain features of the system 200 described with respect to Figure 2 the system 200.

[0085] The apparatus 800 comprises processing circuitry 802. The processing circuitry 802 comprises a receiving module 804 configured to receive pixel information corresponding to at least a portion of a first image, a second image and a third image of an object illuminated by time-modulated illumination acquired by an imaging system. A time delay between acquisition start times of the first image, the second image and the third image is such that there is a temporal overlap during acquisition of the first image, the second image and the third image.

[0086] The processing circuitry 802 further comprises a determining module 806 configured to determine, for constructing a modified image of the object, modified pixel information based on: differences between the received pixel information of the first image, the second image and the third image to reduce an effect of ambient illumination in the received pixel information; and combining the received pixel information to cancel a spatial intensity modulation pattern apparent in each of the first image, the second image and the third image. The spatial intensity modulation pattern is due to a timing relationship between an imaging time period in which the imaging system acquires each image and a modulation time period of the time-modulated illumination.

[0087] In some embodiments, the apparatus 800 further comprises an imaging system (e.g., the imaging system 204 of Figure 2 ) for acquiring the pixel information corresponding to at least a portion of the first image, the second image and the third image.

[0088] In some embodiments, the apparatus 800 further comprises an illumination unit (e.g., the illumination unit 206 of Figure 2 ) for providing the time-modulated illumination.

[0089] In some embodiments, the imaging system 204, 304 comprises at least two imaging devices (e.g., the imaging devices 310a, 310b, 310c of Figure 3b ) wherein a scanning direction of a rolling shutter implemented by the at least two imaging devices is the same.

[0090] In some embodiments, the imaging system 304 comprises at least two imaging devices 310a, 310b, 310c.

[0091] In some embodiments, the imaging system 304 is configured to operate in a pattern according to a timing relationship between an imaging time period in which the imaging system acquires each image and a modulation time period of the time-modulated illumination. The pattern can be: a rolling shutter pattern, wherein the imaging time period of the imaging system is longer than the modulation time period; or a global shutter pattern, wherein the imaging time period is shorter than the modulation time period.

[0092] In some cases, any of the above modules (e.g., receiving module 804 and / or determining module 806) may include at least one dedicated processor (e.g., application-specific integrated circuit (ASIC) and / or field-programmable gate array (FPGA) etc.) for implementing the module's functions.

[0093] In some cases, the modules described above (e.g., receiving module 804 and / or determining module 806) may include at least one processor for implementing instructions that cause the at least one processor to perform the functions of the modules described above. In these examples, the instructions may be stored in a machine-readable medium (not shown) accessible by at least one processor. In some examples, the module itself includes a machine-readable medium. In some examples, the machine-readable medium may be separate from the module itself (e.g., at least one processor of the module may be provided to communicate with the machine-readable medium to access the instructions stored therein).

[0094] Although the invention has been described and illustrated in detail in the accompanying drawings and the foregoing description, such description and illustration should be considered illustrative or exemplary rather than restrictive; the invention is not limited to the disclosed embodiments.

[0095] One or more features described in one embodiment may be combined with or replace features described in another embodiment. For example... Figure 1 or Figure 6 Methods 100 and 600 can be based on system 200 (see reference). Figure 2 The features described in Figure 5), the machine-readable medium 700 and / or the device 800 are modified, or vice versa.

[0096] The embodiments in this disclosure may be provided as a method, a system, or as a combination of machine-readable instructions and processing circuitry. Such machine-readable instructions may be included on a non-transitory machine (e.g., computer) readable storage medium (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-readable program code thereon or thereon.

[0097] This disclosure is described with reference to flowchart illustrations and block diagrams of methods, apparatus, and systems according to embodiments of this disclosure. Although the flowcharts above show a specific execution sequence, the execution sequence may differ from that described. A block described in one flowchart may be combined with a block in another flowchart. It should be understood that each block in a flowchart and / or block diagram, and combinations of blocks in flowcharts and / or block diagrams, can be implemented by machine-readable instructions.

[0098] The machine readable instructions may, for example, be executed by a general purpose computer, a special purpose computer, an embedded processor, or other programmable data processing apparatus to implement the functions described in the specification and diagrams. In particular, a processor or processing circuitry or modules thereof can execute machine readable instructions. The functional modules of the system 200 and / or apparatus 800 (e.g., the receiving module 806 and / or the determining module 808), as well as other devices described herein, can be implemented by a processor executing machine readable instructions stored in memory or a processor operating in accordance with instructions embedded in logic circuitry. The term “processor” is to be interpreted broadly to include a CPU, processing unit, ASIC, logic unit, or programmable gate array, etc. The methods and functional modules can be performed by a single processor or divided among several processors.

[0099] Such machine readable instructions can also be stored in a computer readable storage that can direct a computer or other programmable data processing apparatus to operate in a specific mode.

[0100] Such machine readable instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operations to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which operate on the computer or other programmable apparatus provide processes for implementing the functions specified in the flowchart block or blocks and / or in the block or blocks of the block diagrams.

[0101] Furthermore, the teachings herein can be implemented in the form of a computer program product, which is stored in a storage medium and comprises a plurality of instructions for making a computer device implement the methods described in the embodiments of the disclosure.

[0102] Elements or steps described in relation to one embodiment can be combined or replaced with elements or steps described in relation to another embodiment. Variations to the disclosed embodiments can become apparent to those of ordinary skill in the art upon reading the description and drawings and are intended to be within the scope of the claims. In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite articles “a” or “an” do not exclude a plurality. A single processor or other unit can fulfil the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. A computer program can be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state storage medium supplied together with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A computer-implemented method (100) for determining modified pixel intensity values ​​for constructing an image of an object, comprising: Receive (102) pixel intensity values ​​corresponding to at least a portion of a first image frame, a second image frame, and a third image frame of an object illuminated by time-modulated illumination, acquired by an imaging system, the imaging system comprising at least two imaging devices for acquiring the first image frame, the second image frame, and the third image frame, wherein a time delay between the acquisition start times of the first image frame and the second image frame causes a temporal overlap during the acquisition of the first image frame and the second image frame, and wherein a time delay between the acquisition start times of the second image frame and the third image frame causes a temporal overlap during the acquisition of the second image frame and the third image frame, wherein: The time delay between the acquisition start times of the first image frame and the second image frame, and the time delay between the acquisition start times of the second image frame and the third image frame, are determined based on the modulation time period of the time-modulated illumination; and The time delay corresponds to the modulation time period divided by the number of image frames partially acquired at a certain time point within the overlapping acquisition time, such that the temporal relationship between the imaging time period in which the imaging system acquires each image frame and the modulation time period of the time-modulated illumination allows different spatial intensity modulation patterns to be registered in each of the first, second, and third image frames; and The modified pixel intensity values ​​used to construct the modified image of the object are determined (104) based on the following: The difference between the received pixel intensity values ​​of each different pair of combinations of the first image frame, the second image frame, and the third image frame is used to determine the pixel intensity value of each different pair of combinations, wherein the difference reduces the influence of ambient lighting present when acquiring each of the first, second, and third image frames; and The determined combinations of pixel intensity values ​​for different pairs are used to counteract the spatial intensity modulation pattern that is evident in each of the first, second, and third image frames.

2. The method of claim 1, wherein the modified pixel intensity value for constructing the modified image of the object is determined based on the following: Pixel intensity values ​​of corresponding pixels are subtracted from each different pair of combinations of the first, second, and third image frames to produce a resulting pixel intensity value for each corresponding pixel of each different pair of combinations, wherein the step of subtracting pixel intensity values ​​reduces the influence of ambient lighting present during the acquisition of each of the first, second, and third image frames; and The resulting pixel intensity values ​​for each different pair of combinations are squared, and the squared resulting pixel intensity values ​​are summed to cancel out the spatial intensity modulation pattern that is evident in each of the first, second, and third image frames.

3. The method according to any one of the preceding claims, wherein the spatial intensity modulation pattern has equal phase shifts between the first image frame, the second image frame, and the third image frame, and wherein the modified image M used to construct the object... AC The modified pixel intensity value is determined based on the following formula: Wherein I1 refers to the pixel intensity value in the first image frame, I2 refers to the pixel intensity value in the second image frame, and I3 refers to the pixel intensity value in the third image frame.

4. The method according to claim 1 or 2, wherein one imaging device in the imaging device is used to acquire at least one image frame among the first image frame, the second image, and the third image frame, and at least another imaging device in the imaging device is used to acquire at least another image frame among the first image frame, the second image frame, and the third image.

5. The method according to claim 4, further comprising: The imaging system acquires (602) the first image frame, the second image frame, and the third image frame.

6. The method according to any one of claims 1, 2, and 5, comprising: Receive an indication of the modulation parameters of the time-modulated lighting described in (606); The operating parameters of the imaging system are determined based on the indication; And to enable the imaging system to operate according to the operating parameters.

7. The method according to any one of claims 1, 2 and 5, wherein the modulation frequency and / or frame rate of the time-modulated illumination associated with the first image frame, the second image frame and the third image frame are configured such that the phase shifts between the spatial intensity modulation patterns of the first image frame, the second image frame and the third image frame are the same.

8. The method according to any one of claims 1, 2, and 5, comprising: Before determining the modified pixel intensity value, it is identified (604) whether the position of the object is the same in the first image frame, the second image frame and the third image frame; And if the difference between the position of the object in at least one image frame and at least another image frame in the image frame exceeds a threshold, an image shifting operation is performed in at least one of the first image frame, the second image frame, and the third image frame, such that the position of the object is the same in each of the first image frame, the second image frame, and the third image frame.

9. The method according to any one of claims 1, 2, and 5, wherein the imaging system is configured to operate in a mode according to a temporal relationship between the imaging time period in which each image frame is acquired by the imaging system and the modulation time period of the time-modulated illumination, wherein the mode is: Rolling shutter mode, wherein the imaging time period of the imaging system is longer than the modulation time period; or Global shutter mode, wherein the imaging time period is shorter than the modulation time period.

10. The method according to any one of claims 1, 2, and 5, wherein the method further comprises: Receive pixel intensity values ​​corresponding to at least a portion of at least one additional image frame of the object, wherein the time delay between the acquisition start time of the third image frame and the at least one additional image frame causes a temporal overlap during the acquisition of the third image frame and the at least one additional image frame, wherein: The time delay between the acquisition start time of the third image frame and the at least one additional image frame is determined based on the modulation time period of the time-modulated illumination; and The time delay corresponds to the modulation time period divided by the number of image frames partially acquired at a certain time point within the overlapping acquisition time, such that the temporal relationship between the imaging time period for each image frame acquired by the imaging system and the modulation time period of the time-modulated illumination allows different spatial intensity modulation patterns to be registered in each of the first image frame, the second image frame, the third image frame, and at least one additional image frame, and wherein the modified pixel intensity value is determined based on the following: The difference between the received pixel intensity values ​​of each different pair of combinations of the first image frame, the second image frame, the third image frame, and the at least one additional image frame is used to determine the pixel intensity value for each different pair of combinations, wherein the difference reduces the influence of ambient lighting present when acquiring each of the first image frame, the second image frame, the third image frame, and the at least one additional image frame; and For each different combination of the determined pixel intensity values, the spatial intensity modulation pattern is counteracted in each of the first image frame, the second image frame, the third image frame, and the at least one additional image frame.

11. A tangible machine-readable medium (700) storing instructions (702) that, when executed by at least one processor (704), cause the at least one processor to implement the method according to any one of claims 1 to 10.

12. An apparatus (800) for determining modified pixel intensity values ​​for constructing an image of an object, the apparatus comprising a processing circuit (802) comprising: A receiving module (804) is configured to receive pixel intensity values ​​corresponding to at least a portion of a first image frame, a second image frame, and a third image frame of an object illuminated by time-modulated illumination, acquired by an imaging system. The imaging system includes at least two imaging devices for acquiring the first image frame, the second image frame, and the third image frame, wherein a time delay between the acquisition start times of the first image frame and the second image frame causes a temporal overlap during the acquisition of the first image frame and the second image frame, and wherein a time delay between the acquisition start times of the second image frame and the third image frame causes a temporal overlap during the acquisition of the second image frame and the third image frame, wherein: The time delay between the acquisition start times of the first image frame and the second image frame, and the time delay between the acquisition start times of the second image frame and the third image frame, are determined based on the modulation time period of the time-modulated illumination; and The time delay corresponds to the modulation time period divided by the number of image frames partially acquired at a certain time point within the overlapping acquisition time, such that the temporal relationship between the imaging time period in which the imaging system acquires each image frame and the modulation time period of the time-modulated illumination allows different spatial intensity modulation patterns to be registered in each of the first, second, and third image frames; and The determining module (806) is configured to determine the modified pixel intensity values ​​for constructing the modified image of the object based on the following: The difference between the received pixel intensity values ​​of each different pair of combinations of the first image frame, the second image frame, and the third image frame, wherein the difference reduces the influence of ambient lighting when each of the first image frame, the second image frame, and the third image frame is present; and The determined pixel intensity values ​​for each different pair combination are used to counteract the obvious spatial intensity modulation pattern in each of the first, second, and third image frames.

13. The apparatus of claim 12, further comprising: The imaging system (204) and / or the illumination unit (206), the imaging system (204) being used to acquire the pixel intensity values ​​corresponding to at least a portion of the first image frame, the second image frame and the third image frame; The lighting unit (206) is used to provide the time-modulated lighting.

14. The apparatus of claim 12 or 13, wherein the imaging system is configured to operate in a mode according to the temporal relationship between the imaging time period in which each image frame is acquired by the imaging system and the modulation time period of the time-modulated illumination, wherein the mode is: Rolling shutter mode, wherein the imaging time period of the imaging system is longer than the modulation time period; or Global shutter mode, wherein the imaging time period is shorter than the modulation time period.

Images