Light source for structured light, structured light projection device, and system

Abstract

A light source for structured light comprising a plurality of light source elements arranged in an array, the light source elements being capable of operating in two modes: a calibration mode in which only a portion of the light source elements are adapted to be activated; a normal mode in which the rest of the light source elements are adapted to be driven; a light source configured to be driven by

Description

Technical Field

[0001] The present invention relates to structured light hardware, specifically, a light source for structured light, a structured light projection device, and a system.

Background Art

[0002] In a structured light system, a projection device projects a pattern such as a grid or horizontal / vertical bars onto a scene, which reflects the pattern towards a camera such as a frame-based or event-based sensor, and the camera acquires the pattern. By finding specific portions of the pattern in the output of the camera, it is possible to triangulate points in space from the position of the pattern in the camera and the position of the pattern in the projection device. Usually, by using structured light having a wavelength outside the visible spectrum, interference with other computer vision tasks that would make it difficult to distinguish from the projected pattern can be avoided.

[0003] Structured light systems have many applications. For example, they can be used to photograph fingerprints in a 3D scene. Previously, tape for collecting fingerprints was used and it was copied onto a flat surface, but now, by using a camera to digitally copy the fingerprint onto a flat surface, physical contact with the fingerprint and damage to the fingerprint are prevented. In addition, structured light systems can also be used to recognize a user's face in order to handle identification and authentication at a high security level.

[0004] Due to the many technical characteristics and advantages of structured light systems, there are many trade-offs made in the exact pattern projected, from the perspective of projector technology and the performance of the global system, such as the density of the output depth map.

[0005] Therefore, in structured light systems in the art, when the system needs to be calibrated not only at the factory but also throughout the life of the product, especially during construction, the calibration of such structured light systems becomes a very complex problem, because de-calibration means that its output will lose accuracy and / or precision. In this regard, the cost associated with this calibration occupies a large part of the overall price of the structured light system, to the extent that it becomes a limitation to the popularization of such systems. Summary of the Invention Problems to be Solved by the Invention

[0006] An object of the present invention is to improve these drawbacks. Means for Solving the Problems

[0007] Therefore, according to one aspect of the present invention, there is provided a light source for structured light, comprising a plurality of light source elements arranged in an array, the light source elements being configured to be driven in the following two modes, namely, - a calibration mode adapted such that only a part of the light source elements are driven, and - a normal mode adapted such that the remaining light source elements are driven so as to be driven.

[0008] In such an arrangement according to the present invention, without external steps performed by the user or external devices used by the user, when specific external elements are added for the calibration process or automatic partial recalibration of the system, it is possible not only to reconstruct light structured at high density in a single projector, but also to have patterns that enable calibration of the system.

[0009] In an embodiment according to the present invention, in the normal mode, some of the light source elements driven in the calibration mode are deactivated.

[0010] In an embodiment according to the present invention, the array is a two-dimensional array including rows and columns of light source elements.

[0011] In addition, the rows and columns of the light source elements extend in different directions perpendicular to each other. In particular, the rows or columns of the light source elements are spaced apart at a fixed interval.

[0012] Furthermore, in the calibration mode, only one row or one column of the light source elements is driven. Alternatively, in the calibration mode, only the light source elements around the two-dimensional array are driven, for example, a single light source element at the corner.

[0013] Alternatively, the two-dimensional array can also be arranged in the form of a hexagonal lattice. In another embodiment according to the present invention, a drive circuit electrically connected to the light source elements is further provided, including first and second sub-drive circuits. In the calibration mode, the first sub-drive circuit drives a part of the light source elements, the second sub-drive circuit is deactivated and the rest of the light source elements are not driven, and in the normal mode, the first sub-drive circuit drives a part of the light source elements and the second sub-drive circuit drives the rest of the light source elements.

[0014] In yet another embodiment according to the present invention, in the calibration mode, only a part of the light source elements is adapted to generate a calibration pattern, and in the normal mode, the rest of the light source elements are adapted to generate a normal pattern, where some of the light sources driven by the calibration pattern may no longer need to be driven so that the calibration pattern can be a subset of the normal pattern, or something that shares or does not share the part of the light elements of the normal pattern.

[0015] In yet another embodiment according to the present invention, the light source is a vertical cavity surface emitting laser (VCSEL) array light source or an edge emitting laser (EEL) light source.

[0016] According to a second aspect of the present invention, a pattern design method for a light source according to the above-described light source is provided. - generating a calibration pattern in the calibration mode; - Steps of generating a normal pattern in the normal mode and including, The calibration pattern is, for example, a subset of the normal pattern.

[0017] According to a third aspect of the present invention, - A light source as described above, and - At least one optical element configured to receive and reshape the light beam emitted from the light source A projection device comprising is provided.

[0018] For example, in a projection device, at least one optical element is a diffractive optical element (DOE), and the diffractive optical element (DOE) diffracts light with very little loss into a desired pattern and has a microstructure surface that applies each light source element to several things projected onto the scene.

[0019] According to a fourth aspect of the present invention, - A projection device as described above, configured to emit a structured light pattern towards a target in space, and - An image acquisition device configured to observe a target in space that reflects the structured light pattern so as to obtain a structured light image of the target, and - A processor configured to calculate a depth image of the target according to the principle of triangulation A structured light system comprising is provided.

[0020] In one embodiment according to the present invention, the image acquisition device is an event-based camera or a frame-based camera. In the case of an event-based camera, the projection device and the event-based camera can be synchronized in less than 1 ms.

[0021] Compared with systems in the prior art such as expensive and complex fully programmable projectors such as DLP / DMD projectors, the present invention is simpler, cheaper, smaller, and has better light-emitting capabilities.

[0022] Other features and advantages of the present invention will become apparent in the following description with reference to the accompanying drawings.

Brief Description of the Drawings

[0023]

Figure 1

Best Mode for Carrying Out the Invention

[0024] As described above, various applications use structured light patterns to achieve 3D depth maps. In the present invention, the structured light system - a projection device according to the present invention, configured to emit a structured light pattern towards a target object in space, - a light source according to the present invention, which will be described in detail later, - optionally, at least one optical element configured to receive and reshape the light rays emitted from the light source, such as a diffractive optical element (DOE) and a projection device that can include - an image acquisition device configured to observe a target object in space that reflects the structured light pattern so as to obtain a structured light image of the target object, the image acquisition device being an event-based camera or a frame-based camera (an event-based camera is used for illustration purposes) - optionally, a processor configured to calculate a depth image of the target object according to the principle of triangulation, such as a dedicated logic circuit (ASIC, FPGA,...) or a chip and includes.

[0025] FIG. 1 shows an exemplary embodiment of a light source for structured light according to the present invention. This embodiment uses an array of vertical cavity surface emitting lasers (VCSELs) as the light source, where the lasers in the array are modulated individually or in groups. The individual lasers or groups can be modulated statically or dynamically, for example, to provide and modify a structured light pattern as needed to form the IDs of different light sources. Alternatively, an edge-emitting laser (EEL) can be used as the light source according to the present invention.

[0026] In particular, in this embodiment of the light source according to the present invention, the plurality of light source elements are an array of VCSEL dots arranged directly in the projection space or using a DOE element. In the former case, the laser from the VCSEL dots can directly irradiate the scene. In the latter case, the array of VCSEL dots creates a basic pattern, and then the DOE element regenerates this basic pattern across the entire space to cover the entire field of view (FOV) required by the application.

[0027] The array of VCSEL dots can be arranged in various types of one-dimensional, two-dimensional, or three-dimensional arrays. For example, in the case of a two-dimensional array, the VCSEL dots can be arranged in rows and / or columns that include a single row or column (line), which can be deformed by the DOE into multiple lines or irregular dotted lines. Alternatively, they can also be arranged in the form of a hexagonal lattice. In addition, these dots can have a pseudo-random configuration across the projector, either used as themselves or deformed, for example, into a vertical line with dots having a regular horizontal spacing.

[0028] In FIG. 1, an exemplary embodiment of a light source for structured light comprises, for example, 100x100 VCSEL dots distributed over a flat two-dimensional panel along rows perpendicular to the columns, where the columns and rows are spaced at a fixed interval. For the sake of description, in FIG. 1, the number of VCSEL dots has been reduced to 10x10. Those VCSEL dots are electrically connected, and all the VCSEL dots along columns C1, C2, …, C10 are driven in the same way. In this case, the pins of the VCSELs can be coupled together to reach the periphery of the array along the column. Thus, in this exemplary embodiment, the first upper row L1 and the remaining rows LC are each coupled to a power source. Thus, the light source element according to the present invention can be partially driven. For example, only the VCSEL dots in the first upper row are driven in calibration mode, and all the VCSEL dots are driven in normal mode.

[0029] Such an arrangement is designed to generate vertical stripes, which facilitates the path designation / layout of the array. This is because it can use some VCSEL dots instead of a DOE to create stripes, or use some VCSEL dots to create dotted lines that are blurred into lines by the DOE, thereby enabling a large light output with a smaller individual power of the VCSEL dots. All the connections in FIG. 1 should only be logically considered as being driven in the same way, but the actual drive circuits may be different for electrical reasons. In addition, other arrangements and layouts can also be envisioned based on the present invention. For example, the arrangement can be set as the first left column, and the remaining columns are driven separately.

[0030] The present invention will be discussed in more detail here. In the present invention, geometric constraints are used to reduce the complexity of the projection pattern in order to reconstruct a high-density depth map.

[0031] First, under the condition that the camera and the projection device are arranged on the horizontal axis, the epipolar constraint is used between the camera and a projection device capable of generating vertical stripes of light. Therefore, only a given pattern can be found in the pixels of a given range of the camera. This makes it possible to horizontally repeat a small pattern of vertical stripes. The DOE will address regenerating this line pattern across the system FOV in the column and row directions, i.e., the X and Y directions.

[0032] On the other hand, the VCSEL dots can generate a pattern that enables the re-calibration or accurate calibration of the system from the estimation of calibration obtained from the physical dimensions and positions in the assembly of the camera and the projection device. This initial constraint / calibration estimation can be carried out with a lattice of repeating dots. This is because no alignment error can occur when considering the constraints brought about by that initial calibration estimation. This pattern can be generated by either path-designating in other dimensions (as shown in FIG. 1, a path-designation where the lines of a column and a given path-designation of another one, where L1 drives the first upper line and LC drives the rest of them) or by independently path-designating each line of the VCSEL dot array either by path-designating all the columns and the last one independently. Here, by enabling only this independent line, a lattice of dots will be generated with a sufficiently independent pattern of IDs to be aligned in order to calibrate the system.

[0033] In the exemplary embodiment of FIG. 1, in the normal mode, rows L1 and LC can be coupled to the power supply, and columns C1 - C10 will be driven to the ground with signals necessary to create different light patterns for each column of the array. In the calibration mode, while row L1 can be coupled to the power supply, rows LC and columns C2 - C10 can be coupled to the ground (or left in a floating state), thus deactivating most of the array. By driving C1, it becomes possible to activate only the upper left point of the array to activate a single light source element such as an LED.

[0034] In the calibration mode, it is also possible to select to drive columns C1 to C10 in order to obtain lines of dots where each dot has a different ID. To generate further inputs for calibration, C1 to C10 can be driven in the same way here in both the normal mode and the calibration mode, provided that only the row LC has to be switched between power and ground.

[0035] In this case, the drive circuit can be electrically connected to a light source element that includes first and second sub-drive circuits, where in the calibration mode, the first sub-drive circuit drives a portion of the light source element, the second sub-drive circuit is deactivated, and the remaining light source elements are not driven. In the calibration mode, the first sub-drive circuit drives a portion of the light source element and the second sub-drive circuit drives the remaining light source elements.

[0036] Alternatively, it is also possible to use the drive circuit without sub-drive circuits when different columns C1 to C10 are modulated to create a temporal pattern, where there is one drive circuit per column or one drive circuit that can select which column to drive in each cycle of the projection. In that case, by transitioning from the normal to the calibration mode, the drive sequence of the columns in the temporal pattern changes.

[0037] Thus, during the calibration mode period, the fundamental matrix of the system can be estimated, where the fundamental matrix can be linked to the relative pose (translation and rotation) between the camera and the projection device. Several methods already exist for this purpose, such as the eight-point algorithm, which relies on finding "point correspondences" between two viewpoints (here, the projection device and the camera), and can be easily achieved by a system according to the present invention that enables the generation of calibration patterns using portions of the light source, and those correspondences can be easily found.

[0038] During the normal mode period, the normal pattern that can be switched from the calibration mode can include, or can overlap with, the calibration pattern used in the calibration mode by driving all the light source elements. The normal pattern is used in structured light applications such as 3D reconstruction by some advanced algorithms in the art.

[0039] Accordingly, the present invention proposes a single hardware for generating two fixed patterns, one of which may be a subset of the other, one optimized for 3D reconstruction and the other optimized for calibration that can enable obtaining point correspondences.

[0040] The embodiments described above are illustrative of the present invention. Various modifications can be made to them without departing from the scope of the present invention arising from the appended claims.

Description of Reference Numerals

[0041] Column C1 Column C2 Column C10 Row L1 Row LC

Claims

1. A light source for structured light comprising a plurality of light source elements arranged in an array, wherein the light source elements are adapted to be driven in the following two modes, namely, A calibration mode in which only a part of the light source elements are adapted to be driven to generate a calibration pattern and the remaining light source elements are deactivated; and A normal mode in which the remaining light source elements are adapted to be driven to generate a normal pattern, The light source being configured to be driven in.

2. The light source according to claim 1, wherein the array is a two-dimensional array including rows and columns of the light source elements.

3. The light source according to claim 2, wherein the rows and the columns of the light source elements extend in different directions perpendicular to each other.

4. The light source according to claim 2, wherein the rows or the columns of the light source elements are spaced apart at a fixed interval.

5. The light source according to any one of claims 2 to 4, wherein in the calibration mode, only one row or one column of the light source elements is driven.

6. The light source according to any one of claims 2 to 4, wherein in the calibration mode, only the light source elements at the periphery of the two-dimensional array are driven.

7. The light source according to claim 2, wherein the two-dimensional array is arranged in the form of a hexagonal lattice.

8. The light source according to claim 1, further comprising a drive circuit electrically connected to the light source elements, including first and second sub-drive circuits, wherein in the calibration mode, the first sub-drive circuit drives the part of the light source elements, the second sub-drive circuit is deactivated, and the remaining light source elements are not driven, and in the normal mode, the first sub-drive circuit drives the part of the light source elements and the second sub-drive circuit drives the remaining light source elements.

9. The light source according to claim 1, wherein the calibration mode is a subset of the normal pattern.

10. The light source according to claim 1, which is a vertical cavity surface emitting laser array light source or an edge emitting laser light source.

11. A pattern design method for the light source according to claim 1, comprising: Generating a calibration pattern in the calibration mode; and Generating a normal pattern in the normal mode, Including The method, wherein the calibration pattern is a subset of the normal pattern.

12. A projection device comprising the light source according to claim 1.

13. The projection device according to claim 12, further comprising at least one optical element configured to receive and reshaping the light beam emitted from the light source, wherein the at least one optical element is a diffractive optical element.

14. A projection device according to claim 12, wherein the projection device is configured to emit a structured light pattern towards a target in space, a projection device, An image acquisition device configured to observe the target in the space that reflects the structured light pattern so as to obtain a structured light image of the target, A processor configured to calculate a depth image of the target according to the principle of triangulation, A structured light system comprising:

15. The structured light system according to claim 14, wherein the image acquisition device is an event-based camera.

Images