Light source, structured light projection device and system for structured light

Abstract

A light source for structured light, the light source comprising a plurality of light source elements arranged in an array, wherein the light source elements are configured to be driven in two modes: - a calibration mode, wherein only a part of the light source elements are suitable to be driven; and - a normal mode, wherein the remaining light source elements are suitable to be driven.

Description

Technical Field

[0001] This invention relates to structured light hardware, and more specifically, to light sources, structured light projection devices, and systems for structured light. Background Technology

[0002] In structured light systems, a projection device projects a pattern, such as a grid or horizontal / vertical bars, onto a scene that reflects the pattern toward a camera (e.g., a frame-based or event-based sensor) acquiring the pattern. By locating specific portions of the pattern in the camera's output, triangulation can be performed on points in space based on the pattern's position in the camera and its position in the projection device. Typically, structured light with wavelengths outside the visible spectrum is used to avoid interfering with other computer vision tasks where the projected pattern might be confusing.

[0003] Structured light systems have numerous applications. For example, they can be used to capture fingerprints in 3D scenes. Previously, they would use tape to extract fingerprints and flatten them; now, they can use a camera to digitally flatten the fingerprints, thus preventing physical contact and damage. Furthermore, it can be used to recognize a user's face for higher-level security authentication.

[0004] Because structured light systems have many technical features and advantages, there are many trade-offs to be made in the accurate pattern used for projection, from the perspective of projector technology and overall system performance (e.g., the density of the output depth map).

[0005] Therefore, calibration is a highly complex issue for structured light systems in this field. Specifically, the system requires calibration not only at build time but also throughout its entire lifespan, as decalibration means a loss of accuracy and / or precision in its output. In this respect, the cost associated with calibration constitutes a significant portion of the global price of structured light systems, thus limiting the widespread adoption of such systems.

[0006] The present invention aims to improve upon these shortcomings. Summary of the Invention

[0007] To this end, according to one aspect of the invention, a light source for structured light is provided, the light source comprising a plurality of light source elements arranged in an array, wherein the light source elements are configured to be driven in two modes:

[0008] - Calibration mode, in which only a portion of the light source element is suitable for being driven; and

[0009] - Normal mode, in which the remaining light source elements are suitable for being driven.

[0010] With this arrangement according to the invention, if specific external components are added for the system's calibration process or automatic recalibration, without requiring the user to take or use external steps or devices, a single projector may not only have a pattern for achieving dense structured light reconstruction, but also a pattern for achieving system calibration.

[0011] In an embodiment of the invention, in normal mode, some light source elements that are driven in calibration mode are deactivated.

[0012] In an embodiment of the invention, the array is a two-dimensional array of rows and columns accommodating light source elements.

[0013] In addition, the rows and columns of light source elements extend in different directions perpendicular to each other. Specifically, the rows or columns of light source elements are separated by fixed intervals.

[0014] Furthermore, in calibration mode, only one row or column of the light source elements is driven. Alternatively, in calibration mode, only the light source elements on the periphery of the two-dimensional array are driven, such as a single light source element at a corner.

[0015] Alternatively, the two-dimensional array can also be arranged in the form of a hexagonal grid. In another embodiment of the invention, it further includes a driving circuit electrically connected to the light source element and includes a first sub-driving circuit and a second sub-driving circuit, wherein in a calibration mode, the first sub-driving circuit drives a portion of the light source element and deactivates the second sub-driving circuit and does not drive the remaining light source elements; in a normal mode, the first sub-driving circuit drives a portion of the light source element and the second sub-driving circuit drives the remaining light source elements.

[0016] In yet another embodiment of the invention, in calibration mode, only a portion of the light source elements is suitable for generating a calibration pattern; and in normal mode, the remaining light source elements are suitable for generating a normal pattern, wherein some light sources driven in the calibration pattern may no longer be driven, such that the calibration pattern may be a subset of the normal pattern, or a different pattern that shares its light elements with or does not share its light elements with the normal pattern.

[0017] In yet another embodiment of the invention, the light source is a vertical cavity surface-emitting laser (VCSEL) array light source or an edge-emitting laser (EEL) light source.

[0018] According to a second aspect of the present invention, a pattern design method for a light source based on the above-described light source is provided, comprising:

[0019] - Generate a calibration pattern in calibration mode;

[0020] - Generate normal patterns in normal mode;

[0021] The calibration pattern is, for example, a subset of the normal pattern.

[0022] According to a third aspect of the present invention, a projection device is provided, comprising:

[0023] - The light source mentioned above;

[0024] - At least one optical element configured to receive and reshape a light beam emitted from a light source.

[0025] For example, in a projection device, at least one optical element is a diffractive optical element (DOE) having a microstructured surface that diffracts light into a desired pattern with minimal loss, and multiplies each light source element into multiple light source elements projected onto the scene. According to a fourth aspect of the invention, a structured light system is provided, comprising:

[0026] - As mentioned above, the projection device is configured to emit a structured light pattern toward a target object in space;

[0027] - An image acquisition device configured to observe a target object in space reflecting a structured light pattern in order to obtain a structured light image of the target object; and

[0028] - A processor configured to calculate a depth image of a target object based on the principle of triangulation.

[0029] In one embodiment of the invention, the image acquisition device is either an event-based camera or a frame-based camera. In the case of an event-based camera, the projection device and the event-based camera can be synchronized in less than 1 ms.

[0030] Compared to systems in the art, such as expensive and complex fully programmable projectors similar to DLP / DMD, the present invention is simpler, cheaper, smaller, and has better light emission capabilities. Attached Figure Description

[0031] Other features and advantages of the invention will become apparent in the following description with reference to the accompanying drawings, wherein:

[0032] Figure 1 An exemplary embodiment of the light source for structured light according to the present invention will be described. Detailed Implementation

[0033] As mentioned above, different applications use structured light patterns to achieve 3D depth mapping. In this invention, the structured light system includes:

[0034] - According to the projection device of the present invention, the projection device is configured to emit a structured light pattern toward a target object in space, wherein the projection device according to the present invention may include:

[0035] -The light source according to the invention, which will be described in detail later;

[0036] - Optionally, at least one optical element, the optical element being configured to receive and reshape a light beam emitted from a light source, such as a diffractive optical element (DOE);

[0037] - An image acquisition device configured to observe a target object reflecting a structured light pattern in space in order to obtain a structured light image of the target object, wherein the image acquisition device is either an event-based camera or a frame-based camera. For illustrative purposes, an event-based camera is used;

[0038] as well as

[0039] - Optionally, a processor is configured to calculate a depth image of a target object based on triangulation principles, such as dedicated logic circuits (ASIC, FPGA, ...) or chips.

[0040] Figure 1 An exemplary embodiment of a light source for structured light according to the present invention is shown. This embodiment uses a vertical-cavity surface-emitting laser (VCSEL) array as the light source, wherein the lasers of the array are modulated individually or in groups. The individual lasers or groups can be modulated statically or dynamically to provide and change the structured light pattern as needed, for example, to form IDs from different sources. Alternatively, edge-emitting lasers (EELs) can also be used as the light source according to the present invention.

[0041] Specifically, in this embodiment of the light source according to the invention, multiple light source elements are VCSEL point arrays that tile the projection space directly or by using DOE elements. For the former, laser light from the VCSEL points can directly illuminate the scene. For the latter, the array of VCSEL points forms a basic pattern, which is then replicated throughout the space by DOE elements to cover the entire field of view (FOV) required for the application.

[0042] VCSEL point arrays can be arranged in various one-dimensional, two-dimensional, or three-dimensional arrays. For example, in the case of a two-dimensional array, VCSEL points can be arranged in rows and / or columns, including single rows or columns (lines) that can be transformed into multiple rows or irregular dashed lines using a DOE. Alternatively, they can be arranged in the form of a hexagonal grid. Furthermore, these points can have a pseudo-random organization on the projector, either used as is or transformed into, for example, vertical lines with regularly spaced horizontal points.

[0043] exist Figure 1In this example, an exemplary embodiment of the light source for structured light includes, for instance, 100 x 100 VCSEL points distributed along columns and in rows perpendicular to the columns in a planar two-dimensional panel, wherein the rows and columns are separated by a fixed interval. For ease of description, in... Figure 1 In this embodiment, the number of VCSEL points has been reduced to 10x10. These VCSEL points are electrically connected, and all VCSEL points along columns C1, C2, ... C10 are driven in the same manner. In this case, the pins of the VCSELs can be connected together to reach the periphery of the array along the columns. However, in this exemplary embodiment, the first top row L1 and the remaining rows LC are respectively connected to the power supply. Therefore, the light source element according to the invention can be driven partially, for example, only the VCSEL points in the first top row are driven in calibration mode and all VCSEL points are driven in normal mode.

[0044] This arrangement is designed to produce vertical stripes, which simplifies the array wiring / layout because it uses several VCSEL points instead of DOEs to form stripes, or uses several VCSEL points to blur the dashed lines into a single line with the help of DOEs. This allows for large light output from individual VCSEL points with lower power. Figure 1 All connections are logically considered as points driven in the same way, but the actual driving circuits may differ for electrical reasons. Furthermore, other arrangements and layouts can be envisioned based on this invention, for example, arrangements could be configured to drive the first left column and the remaining columns separately.

[0045] The invention will now be discussed in more detail. In this invention, geometric constraints are used to reduce the complexity of the projection pattern in order to obtain a dense depth map reconstruction.

[0046] First, assuming the camera and projection device are positioned on the horizontal axis, epipolar constraints are used between the camera and the projection device, which produces vertical light stripes. Therefore, a given pattern can only be found within a given pixel range on the camera. This allows for the horizontal repetition of smaller vertical stripe patterns. The DOE will then focus on reproducing this line pattern on the system's FOV in the column and row directions (i.e., the X and Y directions).

[0047] Simultaneously, VCSEL points can generate patterns to enable system recalibration or precise calibration based on a calibration guess, which can be derived from the physical dimensions and positions within the camera and projection device assembly. This initial constraint / guessing of calibration can be achieved using a grid of repeating points because matching errors are impossible when considering the constraints imposed by the initial calibration guess. This can be achieved by making one line of the VCSEL point array independent, or by wiring in another dimension (such as...). Figure 1The pattern can be generated either by routing the given lines and another column (where L1 drives the first top line and LC drives the remaining lines), or by routing all columns and the last column independently. Enabling only this independent line will then generate a dot grid with a sufficient number of independent ID patterns to match the calibration system.

[0048] exist Figure 1 In an exemplary embodiment, in normal mode, rows L1 and LC can be connected to a power source, and columns C1 to C10 are driven to ground by the signals required to form different light patterns on each column of the array. In calibration mode, row L1 can be connected to a power source, while rows LC and columns C2 to C10 can be connected to ground (or levitated), thereby deactivating a large portion of the array. Driving C1 will then be able to drive only the top left point of the array, thereby activating a single light source element, such as an LED.

[0049] In calibration mode, you can also select to drive columns C1 through C10 to obtain dotted lines, each with a different ID. To generate more inputs for calibration, C1 through C10 can then be driven in the same way in normal and standard modes, with the row LC only switching between power and ground.

[0050] In this configuration, the driving circuit can be electrically connected to the light source element. The driving circuit includes a first sub-driving circuit and a second sub-driving circuit. In calibration mode, the first sub-driving circuit drives a portion of the light source element and deactivates the second sub-driving circuit, thus not driving the remaining light source elements. In calibration mode, the first sub-driving circuit drives a portion of the light source element and the second sub-driving circuit drives the remaining light source elements.

[0051] Alternatively, when modulating different columns C1 to C10 to form a time pattern, a drive circuit without sub-drive circuits can be used, where each column has its own drive circuit, or a drive circuit that can select which column to drive in each period of projection. In this case, moving from normal mode to calibration mode changes the driving order of the columns in the time pattern.

[0052] Therefore, during calibration mode, the fundamental matrix of the system can be estimated, which can be related to the relative pose (translation and rotation) between the camera and the projection device. Several methods already exist for this purpose, which rely on finding “point correspondences” between two views (here, the projection device and the camera), such as the eight-point algorithm, and this can be readily achieved by the system according to the invention, which is capable of generating a calibration pattern using a portion of the light source to easily find these correspondences.

[0053] During normal mode, the normal pattern, switched from calibration mode, can contain or overlap the calibration pattern used in calibration mode by driving all light source elements. The normal pattern is used for structured light applications, such as 3D reconstruction, by leveraging several sophisticated algorithms in the art.

[0054] Therefore, the present invention proposes a single hardware for generating two fixed patterns, one of which can be a subset of the other, one optimized for 3D reconstruction and the other for calibration, which can allow point correspondences to be obtained.

[0055] The embodiments described above are illustrative of the present invention. Various modifications may be made thereto without departing from the scope of the invention as derived from the appended claims.

Claims

1. A light source for structured light, the light source comprising a plurality of light source elements arranged in an array and a driving circuit electrically connected to the light source elements, the driving circuit including a first sub-driving circuit and a second sub-driving circuit, wherein the light source elements are configured to be driven in two modes: - Calibration mode, wherein the first sub-driving circuit drives only a portion of the light source elements to generate a calibration pattern, and deactivates the second sub-driving circuit and does not drive the remaining light source elements; and - Normal mode, wherein the first sub-driving circuit drives a portion of the light source elements and the second sub-driving circuit drives the remaining light source elements to generate a normal pattern.

2. The light source according to claim 1, characterized in that, The array is a two-dimensional array of rows and columns that accommodates the light source elements.

3. The light source according to claim 2, characterized in that, The rows and columns of the light source elements extend in different directions perpendicular to each other.

4. The light source according to claim 2, characterized in that, The rows or columns of the light source elements are separated by a fixed interval.

5. The light source according to any one of claims 2 to 4, characterized in that, In the calibration mode, only one row or one column of the light source element is driven.

6. The light source according to any one of claims 2 to 4, characterized in that, In the calibration mode, only the light source elements on the periphery of the two-dimensional array are driven.

7. The light source according to claim 2, characterized in that, The two-dimensional array is arranged in the form of a hexagonal grid.

8. The light source according to claim 1, characterized in that, In the calibration mode, a portion of the light source elements is adapted to generate a calibration pattern, and in the normal mode, all light source elements are adapted to generate a normal pattern, wherein the calibration pattern is a subset of the normal pattern.

9. The light source according to claim 1, characterized in that, The light source is a vertical cavity surface-emitting laser array (VCSEL) or an edge-emitting laser.

10. A pattern design method for a light source according to claim 1, comprising: - Generate a calibration pattern in the calibration mode; - Generate a normal pattern in the normal mode; The calibration pattern is a subset of the normal pattern.

11. A projection device, the projection device comprising the light source according to claim 1.

12. The projection device according to claim 11, characterized in that, It further includes at least one optical element configured to receive and reshape a light beam emitted from the light source, and the at least one optical element is a diffractive optical element.

13. A structured light system, comprising: - The projection device according to claim 11, wherein the projection device is configured to emit a structured light pattern toward a target object in space; - An image acquisition device configured to observe the target object reflecting the structured light pattern in the space in order to obtain a structured light image of the target object; as well as - A processor configured to calculate a depth image of the target object based on the principle of triangulation.

14. The structured light system according to claim 13, characterized in that, The image acquisition device is an event-based camera.

Images