Optical Flow Sensor

Abstract

The optical device (1) comprises a sensor (3), a mask structure (10) configured to provide a first mask (4, 11a) covering at least a portion of the sensor (3) and a second mask (5, 11b) covering at least a portion of the sensor (3), the second mask (5, 11b) being an anti-mask of the first mask (4, 11a), and the sensor (3) configured to provide a first output signal for light transmitted through the first mask (4, 11a) and a second output signal for light transmitted through the second mask (5, 11b).

Description

[Technical Field]

[0001] The present disclosure relates to optical devices, such as optical flow sensors for gesture recognition, and methods of optical flow sensing. [Background technology]

[0002] Optical flow, or optic flow, is the pattern of apparent motion of an object caused by the relative motion between the observer / sensor and the scene. Optical flow can also be defined as the distribution of the apparent speed of movement of luminance patterns in a sequence of images.

[0003] Optical flow is obtained by analyzing a sequence of images. In lensless systems, an intermediate step exists because the acquired images do not directly represent the observed scene and require a reconstruction stage. Reconstruction algorithms can require significant computational power, which results in delayed detection.

[0004] In the field of gesture recognition using interactive displays, it is important to identify the movement of an observed object (e.g., a finger) but not the object itself. To achieve a flat device that does not obstruct the displayed information, a mask pattern can be used. An image encoded by an optical mask is then projected onto the sensor surface.

[0005] Extracting the object's motion from the projected pattern requires analysis of the image sequence, where the images are dependent on the specific pattern. A problem with this approach is that the reconstruction depends on the object irradiance variations and the specific mask, which can complicate the extraction / determination of motion. Summary of the Invention

[0006] To solve this problem, the present disclosure proposes a patterned aperture-based solution that can provide direct object movement reconstruction without the need for a separate image reconstruction step and independent of the optical mask used to provide the pattern.

[0007] To solve this further problem, it has been proposed to use two optical masks, one of which is the antimask of the other. Antimasks are also referred to herein as "complementary masks." These masks are complementary in that transparent (or non-zero intensity) areas of one mask correspond to opaque (or zero intensity) areas of the other mask. The mask pattern is typically transparent to visible light but blocks infrared (IR) radiation. For a binary mask A, the antimask is A' = (1 - A). This cancels the dependency from the mask, allowing direct information of the object movement to be obtained.

[0008] According to a first aspect of the present disclosure, there is provided an optical device including a sensor and a mask structure. The mask structure is configured to provide a first mask covering at least a portion of the sensor and a second mask covering at least a portion of the sensor (which may be the same portion as or a different portion from the portion covered by the first mask). The second mask is an anti-mask of the first mask, and the sensor is configured to provide a first output signal in response to light (e.g., infrared light) transmitted through the first mask and a second output signal in response to light transmitted through the second mask.

[0009] Thus, the optical device captures a sequence of images projected onto a sensor by a first mask and a second sequence of images projected onto the sensor by a second mask, and these two image sequences can be used to determine the velocity and / or trajectory of an object moving in front of the sensor. No image reconstruction is required. By summing the time derivatives of the two sequences, an image (referred to herein as a "box-blur image") is obtained from which the object's movement can be estimated without requiring knowledge of the specific mask structure.

[0010] Complementary masks can be projected alternately (separated in time) or spatially separated to allow independent photodetector measurements. In the latter case, a delay line memory can be used to match the subtraction of measurements from the same image cell.

[0011] In one embodiment, the first mask covers substantially a first half of the area of ​​the sensor and the second mask covers substantially a second half of the area, preferably, if the sensor has a shorter dimension, the area is divided along the shorter dimension.

[0012] Typically, each of the first and second masks is a binary mask. That is, the masks include patterns that are substantially transparent and substantially opaque to light having wavelengths within a target range (e.g., IR light). Alternatively, gradient masks can be used. The individual mask features (typically squares) can range in size from 16 μm to 48 μm, depending on the target wavelength and target application. For example, for infrared light with a wavelength of 850 nm, squares with a width of approximately 32 μm can be used. The first mask may include a checkerboard pattern including multiple squares. The second mask may include the same checkerboard pattern but be shifted vertically or horizontally by an odd number of squares relative to the sensor. For example, the second mask may be shifted by a single row or column of the checkerboard pattern relative to the first mask so that the first and second masks cover approximately the same portion of the underlying sensor area.

[0013] Each of the first mask and the second mask may include a uniform redundant array (URA) pattern. Using the URA pattern allows an image of the object to be reconstructed from the sensor output. Alternatively, the first mask may include a random or pseudorandom pattern, which may be advantageous for covering arbitrary device shapes. Furthermore, the image filter required to reconstruct the image of the object is the same as that of the first mask.

[0014] The first and second masks can include patterned polymer layers that block light having wavelengths in the range of at least 900 nm to 1200 nm. The polymer layers may block additional light having wavelengths outside the range. Preferably, the polymer layers are transparent to visible light. The polymer layers can be patterned by lithography. Mask structures including polymer layers can be advantageous compared to masks based on alternating high and low refractive indices because the refractive indices depend on the light path, and therefore the angle of incidence. Polymer layer masks can also be relatively cheaper, thinner, and easier to design.

[0015] The mask structure may be reconfigurable to change the first and second masks depending on the application of the optical device. The mask structure may include a switching unit for switching between complementary masks. The mask structure may include an adjustable mask, such as a liquid crystal display (LCD). In this case, the first and second masks are provided by activating or deactivating cells of the LCD so that the first and second masks are temporally separated. In this way, parallax is not generated, but the temporal resolution may be reduced by half.

[0016] Alternatively, the mask structure can include a layer of transistors. For example, vanadium dioxide-based transistors can be configured to block infrared radiation in a controlled manner while allowing visible light radiation to pass through. The transistors can be organized on a flat surface to form a grid, which can be modulated to create a mask pattern. Different types of coded apertures can be generated, followed by mask-antimask pairs, without mask displacement and with dynamic adaptation of the mask (e.g., scene or motion reconstruction) relevant to the application.

[0017] In other embodiments, a layer of embedded emitters may be used as a blocking mask. For example, LEDs (e.g., micro LEDs or OLEDs) may be placed in the darker areas and used to illuminate the subject. Reflected light passes through the transparent areas onto the sensor.

[0018] The optical device may further include a processing unit for receiving and processing output signals from the sensor. The processing unit may be configured to calculate a first derivative with respect to time from the first output signal, calculate a second derivative with respect to time from the second output signal, and sum the first and second derivatives to form a summed derivative. The processing unit may be configured to output the summed derivative (box blur image) to an external device / unit, such as an application of the integrated device. The external device / unit may then use the output to determine object movement. In other embodiments, the processing unit may be further configured to determine the velocity and / or trajectory of the object based on the summed derivative.

[0019] According to a second aspect of the present disclosure, there is provided an optical device for optical flow sensing, the device comprising a sensor and a mask structure (10) configured to provide a mask covering the sensor, the mask having a single opening covering 20% ​​to 90% of the sensor, the sensor (3) configured to provide an output signal in response to light transmitted through the mask.

[0020] The single aperture, which is relatively large but smaller than the sensor area, is a combination of a mask and its anti-mask. Thus, instead of using two masks to provide the mask and anti-mask as in the first embodiment, a single mask is used that is a combination of the two masks. This provides a low-complexity solution to optical flow detection. Furthermore, it is only necessary to acquire the sensor signal and calculate the first derivative. Advantageously, parallax and misalignment do not occur. In addition to the mask structure, the optical device of the second embodiment may include any suitable features included in the optical device of the first embodiment.

[0021] According to a third aspect of the present disclosure, there is provided a method for determining movement of an object (i.e., a method for optical flow detection). The method can use the optical device according to the first aspect. The method includes receiving, using a sensor, light from the object that has passed through a first mask to provide a first output signal, and receiving, using a sensor, light from the object that has passed through a second mask to provide a second output signal, the second mask being an anti-mask of the first mask. The method further includes determining a velocity and / or a trajectory of the object from the first output signal and the second output signal.

[0022] The determining step may include calculating a first derivative of the first output signal, calculating a second derivative of the second output signal, and summing the first and second derivatives to provide a summed derivative. The velocity and / or trajectory are then determined from the summed derivative. For example, the determining step may include training and using artificial intelligence to analyze the summed derivative.

[0023] Specific embodiments of the present disclosure will now be described with reference to the accompanying drawings. [Brief explanation of the drawings]

[0024] [Figure 1] 1 shows a schematic diagram of an integrated device including an optical device according to an embodiment. [Figure 2a] 10 shows an optical device according to another embodiment with a mask structure when providing a first mask. [Figure 2b] 10 shows an optical device where the mask structure provides a second complementary mask. [Figure 3] An object with irradiance O(t) and velocity V(t) following a trajectory r(t) relative to the sensor is shown. [Figure 4] The irradiance, summed derivatives, and vertical and horizontal profiles of an object moving in a circle relative to the sensor are shown. [Figure 5]The irradiance, summed derivatives, and vertical and horizontal profiles of a triangular object moving parallel to the sensor plane are shown. [Figure 6] FIG. 5 shows the vertical profile of the summed derivatives of the moving object. [Figure 7] The irradiance, summed derivatives, and vertical and horizontal profiles of an object moving perpendicular to the sensor plane are shown. [Figure 8] 1 shows a schematic diagram of an algorithm for optical flow detection according to an embodiment; [Figure 9] 1 shows a mask structure that includes two masks, each covering half of the sensor area. [Figure 10] 10 shows another mask structure embodiment in which the mask is subdivided. [Figure 11] 1 shows a mask structure including a checkerboard pattern mask pattern. [Figure 12] 10 shows an optical device for optical flow sensing according to another embodiment having a mask structure with a single large opening. [Figure 13] 1 shows a mask structure with a single large opening. DETAILED DESCRIPTION OF THE INVENTION

[0025] FIG. 1 shows a schematic diagram of an optical device 1 in an integrated device 2 (e.g., a smartphone) according to an embodiment. The device 1 includes a sensor 3 for detecting incident light. The sensor 3 may include an array of photodiodes or other light-sensitive elements. The device further includes a mask structure 10 including two optical masks 4 and 5 in front of the sensor 3. The first optical mask 4 is complementary to the second mask 5 (i.e., if the pattern of the first mask 4 is opaque, the pattern of the second mask 5 is transparent, and vice versa). Each mask 4 and 5 covers half of the sensor 3. Light 6 from a moving object 7 incident on the sensor 3 is transmitted through the optical masks 4 and 5, which modulate the light 6 to project an image onto the sensor 3. The sensor 3 provides a first output signal in response to the light transmitted through the first optical mask 4 and a second output signal in response to the light transmitted through the second optical mask 5. A processing unit 8 is configured to receive and process the output signal from the sensor 3. For example, the processing unit 8 may be configured to calculate a summed derivative of the first output signal and the second output signal from which the movement (e.g., velocity and / or trajectory) of the object 7 may be determined.

[0026] The mask structure 10 may comprise a dye-based transparent polymer (a polymer that is transparent in the visible range) that blocks infrared radiation regardless of the angle of incidence. Masks 4 and 5 are lithographically patterned and placed in front of the sensor 3.

[0027] The subject 7 may be, for example, a finger moving in front of a display of the integrated device 2, such as the display of a smartphone or computer tablet incorporating the optical device 1. The optical device 1 can then be used to determine the movement of the subject 7 in front of the display, which movement can be used by an application 9 of the integrated device, for example, for gesture recognition.

[0028] Instead of having two spatially separated masks, a single adjustable mask can be used. For example, a liquid crystal display (LCD) can be used to generate dynamic patterns according to the application's requirements. The LCD can also directly generate mask A and then generate complementary mask A'. In this way, parallax can be eliminated or reduced, at the expense of reduced temporal resolution. Alternatively, transistors based on vanadium dioxide, for example, can be used to block infrared radiation in a controlled manner while allowing visible light radiation to pass through. Transistors can be organized on a flat surface to form a grid, which can be modulated to provide the desired mask pattern. This embodiment can provide flexibility in mask generation. Different types of coded apertures can be generated, followed by mask-antimask pairs, without mask displacement and with dynamic adaptation of the mask relevant to the application.

[0029] 2a and 2b show schematic diagrams of an optical device 1 according to an embodiment having a mask structure 10 for providing two complementary masks 11a and 11b, where one mask 11a is an anti-mask of the other mask 11b. The mask structure 10 includes a switching unit 12 for switching between the first mask 11a and the second mask 11b. For example, the mask structure 10 can include a liquid crystal display or a transistor array, and individual elements of the mask structure can be controlled by the switching unit 12 to provide different masks. The two masks 11a and 11b are then sampled successively in time. The optical device 1 includes a first mask 11a configured to receive light transmitted through the first mask 11a with a sensor 3. The switching unit is then configured to switch the mask structure 10 to provide the second mask 11b, and the sensor 3 is configured to receive light transmitted through the second mask 11b. A processing unit 8 is connected to the sensor 3 to process the output from the sensor 3. The processing unit 8 may be configured to determine the movement of an object in front of the sensor 3 based on the output from the sensor 3 .

[0030] Figure 3 shows the observed irradiance O(t) and velocity V(t) of an object at a distance r from an optical system (e.g., a mask and a sensor). The image captured on the sensor over a period T (T = t - t).

number

number

[0031] The image perceived by the sensor in the mask projection associated with mask A is:

number

number

[0032] The image perceived by the sensor in the mask projection relative to antimask A' is:

number

number

number

number

[0033] Combining equations 1 and 6, we get:

number

number

[0034] Box blur can be thought of as a spatial low-pass filter whose kernel consists solely of ones. The "1"s appearing in Equation 3 transform ones into zeros and vice versa, so that "1" is a matrix with the same dimensions as mask A itself.

[0035] For applications where the object shape is relatively simple or known a priori, the operation is simplified. For example, in the field of finger movement recognition, the fingertip is known and the typical movement, and therefore the generated pattern, can be simulated and predicted in advance. The boundaries of the box blur image depend on the object's variability.

[0036] Figure 4 shows the object irradiance of a circularly moving Gaussian spot, the resulting box blur image, and the corresponding horizontal and vertical profiles of the box blur image. The arrows in the box blur image indicate the direction of movement. The object's velocity can be determined from the box blur image, and its trajectory can be determined from multiple consecutive box blur images. For example, the vertical and horizontal profiles, specifically the profile maxima and / or minima, can be used to determine the object's velocity parallel to the sensor plane (for planar motion). The object's velocity is proportional to the peak values. The distance between the peaks can be used to determine the object's center coordinates.

[0037] Figure 5 shows the object irradiance at two different times (t1 and t2), the time derivative of the projected mask image, and their sum. As shown, by summing the derivatives, high-pass information is filtered out, leaving only a low-pass box-blur image. From this box-blur image, the object's velocity can be determined. For example, in the vertical profile, the minimum peak occurs "first," followed by the maximum peak, indicating that the object is moving vertically downward.

[0038] Figure 6 shows a plot of the normalized vertical profile of the box-blur image of Figure 5. The object (i.e., the triangle) moves vertically by 5 pixels between time t1 and time t2, which is reflected in the slope of the peak of the vertical profile.

[0039] Figure 7 shows the irradiance O(t) of an object moving away from or towards the screen at times t1 and t2, the resulting box-blur image, and the vertical and horizontal profiles. As shown in the figure, the vertical (out-of-plane) movement of the object is also detected by the proposed solution. The first two rows of the figure show the situation where the object is moving towards the sensor and away from the sensor (in the z-direction perpendicular to the sensor plane), respectively. The third row shows the situation where there is both movement along the sensor and movement perpendicular to the sensor. The levels of the horizontal and vertical profiles between the peaks provide information on the velocity perpendicular to the sensor.

[0040] Figure 8 shows a schematic diagram of an algorithm for implementing an embodiment of flow detection. The algorithm is based on finite differencing of a series of images. The images are organized in sets representing sensor signal readouts related to masks and antimasks. Four sets of image data R n , R' n , R n+1 and R' n+1 The first R of image data is acquired. n and the second R n+1 The set is associated with a first mask and a third R' of image data.n and a fourth R' n+1 The set of image data is associated with a second mask, the second mask being an anti-mask of the first mask. The first set of image data is subtracted from the second set of image data to obtain a first difference in image data ΔR associated with the first mask. Similarly, the third set of image data is subtracted from the fourth set of image data to obtain a second difference in image data ΔR' associated with the second mask. The first and second differences in image data are summed to obtain translation data

number

[0041] 9 shows a schematic diagram of an embodiment of a mask structure 10 including a first mask 4 and a second mask 5, where the second mask 5 is an antimask of the first mask 4. The mask structure 10 is split along its shorter dimension. If the active area (sensor area) is more extended in one direction, the parallax between the mask-antimask pair will be smaller if this split is along the orthogonal direction as shown.

[0042] Figure 10 shows another embodiment of a mask structure 10 in which the first mask 4 and the second mask 5 are subdivided into smaller regions. This configuration has similar properties to the configuration of Figure 9, but in this case, A-A' is the anti-mask of A'-A, so other directions can also be used. This configuration therefore offers different ways to manipulate the information obtained. For example, derivatives of parts of the sensor plane can be compared in real time to better evaluate the subject trajectory.

[0043] Each mask provided by the mask structure contains a pattern, and although any known pattern can be used for the proposed solution, some patterns may be particularly suited to the proposed solution.

[0044] FIG. 11 shows a mask structure 10 for providing a mask with a checkerboard pattern. Small areas of the checkerboard pattern can be used as mask-antimask pairs. Therefore, by repeatedly applying this technique to selected portions of the sensor, the object trajectory can be more accurately reconstructed and mask displacement differences can be reduced. By shifting the mask pattern by a single mask feature (a single square in the checkerboard pattern) in any direction, the resulting mask becomes complementary. Images from the mask-antimask pairs are obtained by analyzing the checkerboard region together with the shifted mask pattern. This reduces the mask displacement to only a single mask feature corresponding to the mask resolution.

[0045] Another pattern family can be advantageously used: uniformly redundant arrays (URAs). These patterns are advantageous for object reconstruction and can achieve a high signal-to-noise ratio (derived from noise introduced by the mask itself), theoretically representing a perfect imaging system. Therefore, URAs can have the added benefit of providing the dual function of reconstructing motion and reconstructing the object. For this type of mask, the complementary mask is also a URA. The matched filter G' (the filter required to obtain the reconstruction) is equal to -G, where G is directly related to the mask A. By displacing the mask-antimask on the sensor area, different object viewpoints (3D views, ranging, etc.) can be obtained.

[0046] Random patterns can also be used, but they are not perfect imagers. The signal-to-noise ratio depends on the amount of mask geometry. They are easy to design and are not bound to specific prime numbers like URAs; any geometry can be devised. Also, the matched filter is the mask itself (A=G). If the noise introduced by the mask is acceptable, even the object can be reconstructed.

[0047] Instead of using two separate masks to provide a mask (A) and an anti-mask (A'), a single mask that is the combination (A+A') may be used for optical flow detection. For example, a single large aperture can be used. The aperture (an area of ​​substantially 100% transparency) can be conceptually thought of as the combination of a mask and its anti-mask, i.e., A+(1-A)=1.

[0048] Therefore, the sensor signal is R+R'=O*A+O*(1-A)=O*1=>box blur object.

[0049] This allows the mask structure to provide a simple way to implement the described box blur technique by simply taking the sensor signal and taking the first derivative (no parallax or misalignment occurs).

[0050] 12 shows a schematic diagram of an optical device 1 for optical flow detection (but not for image reconstruction) comprising a sensor 3 (e.g., a CMOS image sensor) with a mask structure 10 covering the sensor 3. The mask structure 10 provides a mask 13 with a single large opening 14. The opening 14 needs to be smaller than the sensor 3 but may cover, for example, 80% of the sensor 3. The optical device 1 further comprises a processor 8 for processing the output from the sensor to determine the motion of the object 7.

[0051] FIG. 13 shows a schematic diagram of a front view of a mask structure 10 containing a single opening 14.

[0052] Although specific embodiments have been described above, the scope of the claims is not limited to these embodiments. Each feature disclosed can be incorporated into any of the described embodiments, either alone or in any suitable combination with other features disclosed herein. [Explanation of symbols]

[0053] 1 Optical Devices 2 Integrated Devices 3 sensors 4. The First Mask 5 The Second Mask 6 light 7. Subject 8 Processing Unit 9 Applications 10 Mask structure 11a First Mask 11b Second Mask 12 Switching Unit 13 Mask 14 Openings

Claims

1. An optical device (1), A sensor (3), a mask structure (10) configured to provide a first mask (4, 11 a) covering at least a portion of the sensor (3) and a second mask (5, 11 b) covering at least a portion of the sensor (3), the second mask (5, 11 b) being an anti-mask of the first mask (4, 11 a); the sensor (3) is configured to provide a first output signal for light transmitted through the first mask (4, 11 a) and a second output signal for light transmitted through the second mask (5, 11 b); The optical device (1) further comprises a processing unit (8) for processing an output signal from the sensor (3), the processing unit (8) comprising: calculating a first derivative with respect to time from the first output signal; calculating a second derivative with respect to time from the second output signal; summing the first derivative and the second derivative to form a summed derivative; determining a velocity and / or a trajectory of the object (7) based on the summed derivatives.

2. 2. The optical device (1) of claim 1, wherein the first mask (4, 11a) covers substantially a first half of the area of ​​the sensor (3) and the second mask (5, 11b) covers substantially a second half of said area.

3. 2. The optical device (1) according to claim 1, wherein each of the first mask (4, 11a) and the second mask (5, 11b) is a binary mask.

4. 2. The optical device (1) of claim 1, wherein the first mask (4, 11a) comprises a checkerboard pattern comprising a plurality of squares.

5. 5. The optical device (1) according to claim 4, wherein the second mask (5, 11b) contains the same checkerboard pattern shifted vertically or horizontally by an odd number of squares.

6. 2. The optical device (1) of claim 1, wherein each of the first mask (4, 11a) and the second mask (5, 11b) comprises a uniformly redundant array (URA) pattern.

7. 2. The optical device (1) of claim 1, wherein the first mask (4, 11a) comprises a random or pseudo-random pattern.

8. 2. The optical device (1) of claim 1, wherein the first mask (4) and the second mask (5) comprise patterned polymer layers that block light having wavelengths in the range of at least 900 nm to 1200 nm.

9. 2. The optical device (1) of claim 1, wherein the mask structure (10) comprises a liquid crystal display (LCD), and the first mask (4, 11a) and the second mask (5, 11b) are provided by activating or deactivating cells of the LCD such that the first mask (4, 11a) and the second mask (5, 11b) are separated in time.

10. The optical device (1) according to claim 1, wherein the mask structure (10) comprises a layer of a transistor.

11. The optical device (1) according to claim 1, wherein the mask structure (10) comprises a layer of an emitter.

12. 2. The optical device (1) of claim 1, wherein the mask structure (10) is reconfigurable to change the first mask (4, 11a) and the second mask (5, 11b) depending on the application of the optical device (1).

13. A method for determining the movement of an object (7), comprising: receiving light from the object (7) that has passed through a first mask (4, 11a) using a sensor (3) to provide a first output signal; receiving light from the object (7) that has passed through a second mask (5, 11 b) using the sensor (3) to provide a second output signal, the second mask (5, 11 b) being an anti-mask of the first mask (4, 11 a); calculating a first derivative with respect to time from the first output signal; calculating a second derivative with respect to time from the second output signal; summing the first derivative and the second derivative to form a summed derivative; determining a velocity and / or a trajectory of the object (7) based on the summed derivatives.

14. The method of claim 13 , wherein the determining step includes training and using artificial intelligence to analyze the summed derivatives.

15. An optical device (1), A sensor (3), a mask structure (10) configured to provide a first mask (4, 11 a) covering at least a portion of the sensor (3) and a second mask (5, 11 b) covering at least a portion of the sensor (3), the second mask (5, 11 b) being an anti-mask of the first mask (4, 11 a); the sensor (3) is configured to provide a first output signal for light transmitted through the first mask (4, 11 a) and a second output signal for light transmitted through the second mask (5, 11 b); The optical device (1) further comprises a processing unit (8) for processing an output signal from the sensor (3), the processing unit (8) comprising: calculating a first derivative with respect to time from the first output signal; calculating a second derivative with respect to time from the second output signal; summing the first derivative and the second derivative to form a summed derivative; and outputting the summed derivatives to an external device to determine a velocity and / or a trajectory of an object based on the summed derivatives.

Images