A solid-state structured light three-dimensional imaging system and method

Abstract

A solid-state structured light three-dimensional imaging system and method. The system includes a solid-state structured light projector arranged above a target object and a camera system for image information acquisition. The method includes: Step 1, building a three-dimensional imaging system based on solid-state structured light projection technology and determining the light-emitting mode; Step 2, projecting and acquiring solid-state structured light image information; Step 3, solving the phase using the structured light image information acquired in Step 2; Step 4, reconstructing the depth map and three-dimensional point cloud using the phase information obtained in Step 3. By using a vertical cavity surface laser emitter array of addressable VCSELs in cooperation with a metasurface modulator that responds to parameters such as spatial position, incident angle, incident wavelength, or polarization direction, the present invention modulates the phase and amplitude of light waves to achieve structured light modulation and projection in a fully solid-state manner, thereby realizing the recovery of high-precision three-dimensional information, with the characteristics of high system reliability, small volume, high speed, and low cost.

Description

Technical Field

[0001] The present invention relates to the technical field of optoelectronic measurement in three-dimensional vision, and particularly relates to a solid-state structured light three-dimensional imaging system and method. Background Art

[0002] Due to the characteristics of high precision and good robustness, structured light three-dimensional imaging technology is widely used in fields such as three-dimensional modeling, robot navigation, and quality inspection. A classic structured light three-dimensional imaging system consists of a structured light projection system and a camera with an optical system. The structured light projector projects a pre-designed structured light image or image sequence onto the surface of an object, and the camera with the optical system captures the structured light pattern on the object surface from another angle. Due to the difference in viewing angles between the two, the camera captures a distorted structured light pattern. The distorted structured light pattern is related to the height distribution of the object surface. By designing corresponding calibration and demodulation algorithms, the height distribution of the object surface can be calculated, and then three-dimensional information can be reconstructed.

[0003] Common structured light can be roughly divided into dynamic structured light and static structured light. Static structured light projects random dot matrices, grids, and other morphological features with local or global uniqueness onto the object surface to help restore depth. However, static structured light has the characteristic of single-frame imaging. Although the imaging speed is fast, the accuracy is relatively low. Dynamic structured light uses multiple frames of images to encode the spatial positions of pixels to complete the restoration of depth information. However, since dynamic structured light uses multiple-frame imaging, although the accuracy is high, there is a certain restriction on speed.

[0004] Technicians usually use liquid crystal display (LCD), liquid crystal on silicon (LCoS), and digital light technology (DLP) to project dynamic structured light. LCD and LCoS use electromagnetic fields to drive the rotation angle of liquid crystals to control the brightness and darkness of light. DLP generates changes in brightness and darkness through the integration effect of the reflection of light by each pixel in the mirror array over time. Since LCD and LCoS have strong nonlinearity, they are not widely used in structured light. DLP has good linearity and can reflect light with higher brightness, so it is widely used in the field of structured light. In addition, programmable patterns can also be generated by using mechanical galvanometers and MEMS micromirrors through one-dimensional or two-dimensional scanning methods. However, since DLP, mechanical galvanometers, and MEMS micromirrors all have moving parts, the reliability of the device will decrease after long-term operation, and due to the relatively complex manufacturing process, the cost will also be relatively expensive.

[0005] In the projection technology of dynamic structured light, devices represented by LCD and LCoS belong to traditional pure solid-state devices, which do not contain any moving parts and thus have high reliability. However, their pixel arrays have high requirements for processing technology and are relatively complex to control, requiring a dedicated chip, so the entire solution has high costs, a complex system, and a large volume. In addition, the non-linearity inherent in the method of modulating pixel grayscale through semiconductors in LCD and LCoS has a serious impact on the accuracy of three-dimensional imaging, restricting the improvement of system accuracy. On the other hand, for LCD and LCoS projection systems, limited by semiconductor devices and circuit control systems, the response speed of the projection system is at the millisecond level, and the frame rate is limited to dozens to hundreds of frames, making it difficult to further improve.

[0006] The invention with the publication number CN116222433A and the name "A Structured Light Three-Dimensional Imaging System and Method Based on Metasurface" uses a metasurface-based modulation device to perform high-quality modulation on a laser beam, uses a MEMS micromirror for scanning projection of the laser beam, and cooperates with relevant structured light design and demodulation methods to achieve high-precision reconstruction of the three-dimensional information on the surface of an object; since the MEMS micromirror used still belongs to a semi-solid-state device, its volume and cost are still relatively high, and its working reliability will decrease as the usage time increases. Summary of the Invention

[0007] In order to overcome the above-mentioned disadvantages of the prior art, the purpose of the present invention is to provide a solid-state structured light three-dimensional imaging system and method. By the cooperative work of an array of vertical-cavity surface-emitting lasers of addressable VCSEL and a metasurface modulator, the modulation and projection of structured light can be achieved without any moving parts, thereby realizing the recovery of high-precision three-dimensional information, and having the characteristics of high reliability, high speed, small volume, and low cost.

[0008] In order to achieve the above purpose, the technical solution adopted by the present invention is:

[0009] A solid-state structured light three-dimensional imaging system is arranged above a target object 003 and at least includes a solid-state structured light projector 002 and a camera system 001 for image information acquisition. The solid-state structured light projector 002 is used to modulate and project an encoded two-dimensional structured light light field 205 and form a structured light pattern 206 with brightness modulation on the surface of the target object 003.

[0010] The solid-state structured light projector 002 at least includes an array 201 of vertical-cavity surface-emitting lasers (VCSELs) that can be addressed. In front of the array 201 of vertical-cavity surface-emitting lasers that can be addressed, there is a metasurface modulator 204, and nanostructures 2041 are arranged on the surface of the metasurface modulator 204. The array 201 of vertical-cavity surface-emitting lasers that can be addressed is composed of multiple laser emission units 207, and each laser emission unit 207 is controlled individually or in groups. The entire solid-state structured light projector 002 is encapsulated by a packaging structural member 202.

[0011] The optical axis of the camera system 001 is parallel to the solid-state structured light projector 002.

[0012] Or the optical axis of the camera system 001 and the solid-state structured light projector 002 form an angle within the plane of the optical center connection line and the optical axis, and the value range of the angle is [5°, 45°].

[0013] The metasurface modulator 204 is a device that responds to incident angle, position, wavelength, polarization state, or spatial frequency parameters to achieve the switching of different structured light patterns. The nanostructures 2041 on its surface are fabricated by lithography, electron beam processing, or nanoimprinting of one or more layers of structures. The nanostructures 2041 are composed of discrete nano-unit structural materials, where Ge or Si is used as the unit structural material in the infrared band; Ge or Si is used as the unit structural material in the infrared band; TiO2 or GaN is selected as the unit structural material in the visible band; and HfO2 or AlN is selected as the unit structural material in the ultraviolet band.

[0014] A solid-state structured light three-dimensional imaging method includes the following steps:

[0015] Step 1: Build a solid-state structured light three-dimensional imaging system and determine the light-emitting mode.

[0016] Step 2: The solid-state structured light projector 002 in the solid-state structured light three-dimensional imaging system projects a structured light pattern on the surface of the target object 003, and the camera system 001 collects the structured light image information.

[0017] Step 3: Use the structured light image information collected in Step 2 to solve for the phase.

[0018] Step 4: Use the phase information obtained in Step 3 to reconstruct the depth map and three-dimensional point cloud.

[0019] The method for determining the light-emitting mode in Step 1 is as follows:

[0020] Step 1.1: Determine the coding scheme.

[0021] According to the principle of structured light three-dimensional imaging, each point in the field of view is spatially encoded using binary encoding, Gray encoding, sine-cosine encoding, or a hybrid encoding method to obtain M target projection patterns.

[0022] Step 1.2: Determine the emission pattern of the VCSEL:

[0023] Based on the M target projection patterns obtained in Step 1.1 Determine the corresponding M emission patterns P1, P2,..., P M , where the emission pattern is a matrix corresponding to the address of the addressable VCSEL, and the emission pattern changes two-dimensionally or one-dimensionally; if the light source is a two-dimensional addressable VCSEL, then the emission pattern is a two-dimensional matrix, and P i (j, k) represents the brightness coefficient of the j-th row and k-th column of the i-th projection pattern, where i = 1, 2,..., M; P i (j, k) is a real number with a value in [0, 1], and P i (j, k) = 0 indicates that the light-emitting point at this address is not activated, and P i (j, k) = 1.0 indicates that the light-emitting point at this address emits light at the rated maximum power, and an intermediate value indicates that it operates at a partial power; if the light source is a one-dimensional addressable VCSEL, then P1, P2,..., P M is a one-dimensional matrix. At this time, P i (j) = 0 indicates that the light-emitting point in the j-th row or j-th column is not activated, and P i (j) = 1.0 indicates that the light-emitting point in the j-th row or j-th column emits light at the rated maximum power, and if it is an intermediate value between 0 and 1, it indicates that it operates at a partial power;

[0024] When determining the M emission patterns, two conditions should be met: P1, P2,..., P M Differ from each other to reduce interference between different patterns; P1, P2,..., P M The difference in the emission power of the emission patterns is controlled within 10%;

[0025] The emission pattern in Step 1.2 realizes the projection of the M projection patterns by adjusting the metasurface modulator 204. When there are multiple one-dimensional coded projection patterns , the coding direction is perpendicular to the light source arrangement direction.

[0026] When the metasurface modulator 204 is an incident light angle response device, the light rays emitted by different light emission modes of the metasurface modulator 204 are deflected to a specific angle after passing through the first-layer metasurface 2041-1, and then incident on the second-layer metasurface 2041-2, and are projected onto the object surface after being modulated again; the action process of the first-layer metasurface 2041-1 can be regarded as the superposition of two sub-processes, namely the deflection modulation of the light beam and the determination of the off-axis optical system; specifically, it needs to satisfy:

[0027] For sinusoidal and cosinusoidal structured light, determine N T cosinusoidal structured lights with periodic distributions are sufficient, and the phase shift is achieved by adjusting the position of the light source to emit light;

[0028] The projected pattern at this time is:

[0029]

[0030] Among them, The projected image of the i-th step phase-shifted structured light, b(x, y) and a(x, y) are the background and contrast, x, y are spatial coordinates, T is the period, and the projected phase φ p (x, y) = 2πx / T;

[0031] For the determination of the metasurface modulator 204, according to the generalized Snell's law:

[0032]

[0033] Among them, n t is the refractive index around the refraction interface, n i is the refractive index around the incident interface, λ is the wavelength, is the phase gradient;

[0034] Furthermore, there is:

[0035]

[0036] Among them, n t is the refractive index around the refraction interface, n i is the refractive index around the incident interface;

[0037] For the deflection modulation of the light beam, according to Equation (Equation 1-2), the deflection distribution modulation of the light beam is performed, and the phase distribution that the metasurface needs to provide for the projected light wave is:

[0038]

[0039] Among them, θ i is the incident angle of the first-layer metasurface 2041-1. At this time, n i = n0≈1, and n0 is the refractive index of air; nt = n R , n R is the refractive index of the medium between the two layers of metasurfaces, x and y are the position coordinates on the interface, and θ r is the deflection angle by s; the deflection angle θ r ensures that there is no crosstalk under different emission modes. The number of values of θ r is the number of frequencies in the designed coding scheme, that is

[0040] For an off-axis optical system, its phase distribution should satisfy:

[0041]

[0042] where f is the focal length;

[0043] Superimposing the above processes, the phase transformation provided by the first layer of metasurface 2041-1 is:

[0044]

[0045] Therefore, the first layer of metasurface 2041-1 is determined according to the phase distribution in the above (Equation 1-6);

[0046] The input of the second layer of metasurface 2041-2 is parallel light with an incident angle of θ r . It is necessary to expand the above parallel light and modulate the spatial intensity to form a cosine phase distribution:

[0047] The incident optical field distribution is expressed as:

[0048]

[0049] where is the wave number, λ is the wavelength, and E0 is the amplitude of the electric field.

[0050] After passing through the second layer of metasurface, the optical field is:

[0051]

[0052] In order to obtain the intensity distribution of Equation (1-2), then:

[0053]

[0054] It is obtained that:

[0055]

[0056] The surface structure of the second layer of metasurface 2041-2 enables its phase modulation to satisfy Equation (1.9);

[0057] When the metasurface modulator 204 is a device that responds to the incident position, it satisfies the following:

[0058] Condition 1: Partition the modulator 204. First, according to the emission modes P1, P2,..., P M The number of which is M, evenly divide the light-emitting units 207 of the addressable VCSEL array 201 into M regions. G(j, k) represents the light-emitting region in the j-th column and the k-th row. Each region has the same light-emitting power. The corresponding metasurface modulator 204 is also partitioned and divided into M regions. S(j, k) represents the modulation region in the j-th column and the k-th row; S(j, k) and G(j, k) are spatially aligned to ensure that when S(j, k) emits light, the light irradiates on G(j, k). Here, one of j and k can take the value of 1, and at this time, it is a one-dimensional row distribution or column distribution;

[0059] Then, limit G(j, k) to generate the corresponding projection pattern I(j, k), where I(j, k) represents the k-th projection pattern in the j-th group;

[0060] For sinusoidal and cosinusoidal structured light, the projected pattern is:

[0061]

[0062] Among them, The i-th step phase-shifted structured light projection image, b(x, y) and a(x, y) are the background and contrast, x, y are spatial coordinates, T is the period, and the projection phase φ p (x, y) = 2πx / T;

[0063] For each structured light The corresponding metasurface region S(j, k) only requires a single-layer metasurface to complete.

[0064] When the metasurface modulator 204 is a polarization direction encoder, it needs to satisfy:

[0065] Partition the metasurface modulator 204. First, according to the emission modes P1, P2,..., P M The number of which is M, evenly divide the light-emitting units 207 of the addressable VCSEL array 201 into M regions. G(j, k) represents the light-emitting region in the j-th column and the k-th row. Each region has the same light-emitting power. The corresponding metasurface modulator 204 is also partitioned and divided into M regions. S(j, k) represents the modulation region in the j-th column and the k-th row;

[0066] S(j, k) and G(j, k) are spatially aligned to ensure that when S(j, k) emits light, the light irradiates on G(j, k); when one of j and k takes the value of 1 here, at this time, it is a one-dimensional row distribution or column distribution;

[0067] Let G(j, k) generate the corresponding projection pattern I(j, k), where I(j, k) represents the k-th projection pattern in the j-th group;

[0068] Regard S(j, k) as a combination of nano-units that periodically transform the polarization angle of the incident linearly polarized light. The deflection angle α should satisfy:

[0069]

[0070] where x is the coordinate value and T is the period;

[0071] Condition 2: After satisfying Condition 1, when in use, add a polarizer with the same polarization direction as the light source in front of and behind the metasurface modulator (204), and then cosine structured light can be obtained.

[0072] The specific process of Step 2 is as follows:

[0073] Step 2.1: According to the determined light-emitting mode, drive the laser emission unit 207 of the vertical cavity surface laser emitter array 201 of the corresponding addressable VCSEL to emit light, and project the preset structured light pattern 206 onto the surface of the target object 003;

[0074] Step 2.2, the camera system 001 starts to expose and collect the structured light pattern 206;

[0075] Step 2.3, repeat Step 2.1 - Step 2.3 until N T *N times of structured light projection and collection.

[0076] The specific steps of Step 3 are as follows:

[0077] Describe the structured light pattern 206 captured by the camera system 001 as:

[0078]

[0079] where, represents the n-th step phase-shifted structured light acquisition image, B(x, y) represents the background, A(x, y) represents the contrast, represents the wrapped phase collected;

[0080] The absolute phase collected by the camera system (001) satisfies the following formula:

[0081]

[0082] where K(x, y) is the order and is an integer;

[0083] Calculate the wrapped phase from the following formula:

[0084]

[0085] Then, the wrapped phase is successively expanded from low to high by using the multi-frequency expansion method. The absolute phase φ c (x, y) collected by the camera is obtained.

[0086] The specific process of step 4 is as follows:

[0087] Step 4.1, perform epipolar correction on the absolute phase map;

[0088] Step 4.2, perform phase matching;

[0089] For a monocular structure, the projected absolute phase φ p (x, y) and the collected absolute phase φ c (x, y) are selected, searched, interpolated, and matched row by row to obtain the disparity of the phase matching points, calculate the disparity map, and then use the calibration parameter matrix to reconstruct the depth map to obtain the depth map D(u, v);

[0090] For a binocular structure, the absolute phases φ p (x, y) collected by the left and right cameras are selected, searched, interpolated, and matched row by row to obtain the disparity of the phase matching points, calculate the disparity map, and then use the calibration parameter matrix to reconstruct the depth map to obtain the depth map D(u, v);

[0091] Step 4.3, use the depth map D(u, v) in step 4.2 to reconstruct the three-dimensional information to obtain the three-dimensional point cloud X w =[x w , y w , z w T , thus realizing the projection of pure solid-state structured light and the reconstruction of three-dimensional information.

[0092] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0093] 1. In the present invention, the vertical cavity surface laser emitter array of the addressable VCSEL is used in cooperation with the designed metasurface modulator to realize the projection of dynamic structured light; compared with the traditional semiconductor pixel array devices such as LCD and LCoS that can realize solid-state structured light projection, the system is more simplified and highly integrated, does not require a special optical path, so the volume is smaller, the cost is lower, and the nonlinearity of semiconductor devices is not introduced, so the system accuracy is higher and more stable;

[0094] ​2. Compared with semi-solid state methods such as MEMS and DLP, the present invention does not include moving devices and semiconductor modulation devices, which improves the reliability of the system, extends the service life, avoids the failure of the modulation device due to long-term use, and has better performance in terms of vibration resistance and shock resistance of the system. At the same time, this method has high integration, small volume and low cost.

[0095] 3. Compared with the traditional dynamic structured light modulation system, the present invention realizes structured light projection through the transformation of the light source in cooperation with optical devices. The structured light projection speed is determined by the light source switching speed, and its projection frame rate is above 10 9 fps, and ultra-high-speed structured light three-dimensional reconstruction can be achieved, which is incomparable to traditional systems.

[0096] In summary, by using an addressable light source and the designed metasurface optical system, the present invention can realize the modulation and projection of structured light without any moving parts, thereby realizing the recovery of high-precision three-dimensional information, and has the characteristics of high system reliability, small volume and low cost. Brief Description of the Drawings

[0097] Figure 1 It is a structural diagram of the three-dimensional imaging system of the present invention.

[0098] Figure 2 It is a solid-state structured light projection system of the present invention.

[0099] Figure 3 It is a schematic structural diagram of the solid-state structured light projector 002 of the present invention.

[0100] Figure 4 Design scheme 1 of different light emission modes.

[0101] Figure 5 Design scheme 2 of different light emission modes.

[0102] Figure 6 Schematic diagram of the design scheme of the metasurface modulator.

[0103] Figure 7 Schematic diagram of the design scheme of the polarization-encoded metasurface modulator.

[0104] Among them, 001 is the camera system, 002 is the solid-state structured light projector, 003 is the target object, 201 is the vertical cavity surface laser emitter array of the addressable VCSEL, 202 is the packaging structure, 203 is the encoded laser, 204 is the metasurface modulator, 2041 is the nanostructure, 2041-1 is the first layer of metasurface, 2041-2 is the second layer of metasurface, 205 is the two-dimensional structured light optical field, 206 is the structured light pattern, and 207 is the laser emission unit. Detailed Embodiment

[0105] The following will describe the present invention in detail with reference to the drawings.

[0106] See Figure 1 Figure 1 , the three-dimensional imaging system in step 1 includes at least one solid-state structured light projector 002 and a camera system 001 for collecting image information. Refer to Figure 2 Figure 2 , the solid-state structured light projector 002 is used to regulate and project an encoded two-dimensional structured light field 205, and form a structured light pattern 206 with brightness modulation on the object surface;

[0107] See Figure 3 Figure 3 , the solid-state structured light projector 002 includes at least one vertical cavity surface laser emitter array 201 of addressable VCSELs. A metasurface modulator 204 is arranged in front of the projection of the vertical cavity surface laser emitter array 201 of addressable VCSELs, and nanostructures 2041 are arranged on the surface of the metasurface modulator 204; the vertical cavity surface laser emitter array 201 of addressable VCSELs is composed of multiple laser emission units 207, and each laser emission unit 207 is controlled individually or in groups; the entire solid-state structured light projector 002 is encapsulated by a packaging structure member 202.

[0108] The metasurface modulator 204 is a device that responds to incident angle, position, wavelength, polarization state or spatial frequency parameters to realize the switching of different structured light patterns. The nanostructures 2041 on its surface are fabricated by lithography, electron beam processing or nanoimprinting of one or more layers of structures; the nanostructures 2041 are composed of discrete nano-unit structure materials, where Ge or Si is used as the unit structure material in the infrared band; Ge or Si is used as the unit structure material in the infrared band; TiI2 or GaN is selected as the unit structure material in the visible band; HfI2 or A1N is selected as the unit structure material in the ultraviolet band.

[0109] The optical axis of the camera system 001 is parallel to the solid-state structured light projector 002;

[0110] As an alternative, the optical axis of the camera system 001 and the solid-state structured light projector 002 form an angle in the plane where the optical center connection line and the optical axis are located, and the value range of the angle is [5°, 45°].

[0111] A solid-state structured light three-dimensional imaging method includes the following steps:

[0112] Step 1: Build a solid-state structured light three-dimensional imaging system and determine the light emission mode;

[0113] Step 2: The solid-state structured light projector 002 in the solid-state structured light three-dimensional imaging system projects a structured light pattern on the surface of the target object 003, and the camera system 001 collects the solid-state structured light image information;

[0114] Step 3: Solve the phase using the structured light image information collected in Step 2;

[0115] Step 4: Reconstruct the depth map and 3D point cloud using the phase information obtained in Step 3.

[0116] See Figure 4 and Figure 5 For the determination of the emission mode in Step 1, the specific method is as follows:

[0117] The emission modes of the addressable VCSEL vertical cavity surface laser emitter array 201 (such as 201-1, 201-2, 201-3, 201-4, …) are jointly determined by the encoding method of the structured light pattern 206 and the characteristics of the metasurface modulator 204.

[0118] During operation, the encoded laser 203 formed by the laser beam emitted by the selected laser emission unit 207, after being incident on the metasurface modulator 204, is modulated by the nanostructures 2041 on the surface of the metasurface modulator 204 to form a two-dimensional structured light optical field 205 with a specific spatial intensity distribution. After the structured light pattern 206 on the object surface is collected by the camera system 001, a deformed structured light pattern is obtained. By controlling the emission mode of the vertical cavity surface laser emitter array (VCSEL) 201, different deformed structured light patterns can be collected, and then high-precision three-dimensional information reconstruction can be performed.

[0119] The specific working method is as follows:

[0120] Step 1.1: Determine the encoding scheme:

[0121] According to the principle of structured light three-dimensional imaging, each point in the field of view is spatially encoded using a one-dimensional encoding method such as binary encoding, Gray encoding, sine-cosine encoding, or hybrid encoding to obtain M target projection patterns

[0122] As an alternative, the system uses multi-frame two-dimensional dot matrix encoding to obtain M target projection patterns Or a multi-frame one-dimensional and two-dimensional hybrid encoding scheme to obtain M target projections. The encoding method in Step 1.1 selects sine-cosine encoding for higher accuracy and anti-motion performance, and adopts the multi-frequency N-step phase shift method, which can be expressed as:

[0123]

[0124] where The i-th step phase-shifted structured light projection image, b(x, y) and a(x, y) are the background and contrast, x, y are the spatial coordinates, and T is the period;

[0125] Changing the value of the period T can obtain cosine encodings of different frequencies, and the value of T is N T , usually N T ≥ 3, and N T is a natural number;

[0126] The light source is a one-dimensional addressable VCSEL, and P1, P2,..., P M is a one-dimensional matrix. Assuming that T has 3 values and N = 3, at this time M = 9, and 9 different light-emitting modes are designed to ensure that the light-emitting power is close under different light-emitting modes as much as possible.

[0127] When designing multiple one-dimensional coded projection patterns , it is preferred that the coding direction is perpendicular to the light source arrangement direction to reduce the different projection frame alignment errors introduced by possible processing errors.

[0128] Step 1.2: Determine the light-emitting mode of the VCSEL:

[0129] According to the M target projection modes obtained in Step 1.1 determine the corresponding M light-emitting modes P1, P2,..., P M , the light-emitting mode is a matrix corresponding to the address of the addressable VCSEL, and the light-emitting mode changes two-dimensionally or one-dimensionally; if the light source is a two-dimensional addressable VCSEL, then the light-emitting mode is a two-dimensional matrix, and P i (j, k) represents the brightness coefficient of the j-th row and k-th column of the i-th projection mode, where i = 1, 2,..., M; P i (j, k) is a real number with a value in [0, 1], and P i (j, k) = 0 means that the light-emitting point at this address is not activated, and P i (j, k) = 1.0 means that the light-emitting point at this address emits light at the rated maximum power, and the intermediate value means that it operates at partial power; if the light source is a one-dimensional addressable VCSEL, then P1, P2,..., P M is a one-dimensional matrix. At this time, P i (j) = 0 means that the light-emitting point in the j-th row or j-th column is not activated, and P i (j) = 1.0 means that the light-emitting point in the j-th row or j-th column emits light at the rated maximum power, and if it is an intermediate value between 0 and 1, it means that it operates at partial power;

[0130] When determining the M light-emitting modes, two conditions should be met: P1, P2,..., P M should be as different from each other as possible to reduce the interference between different modes; P1, P2,..., P MThe luminous power of the light-emitting modes is as close as possible, and the difference is preferably controlled within 10%, to prevent overexposure or underexposure when the camera uses the same exposure parameters.

[0131] For the light-emitting mode in step 1.2, the projection of M projection patterns is achieved by adjusting the metasurface modulator 204. When there are multiple one-dimensional encoded projection patterns, the encoding direction is vertically distributed perpendicular to the light source arrangement direction.

[0132] There are multiple schemes to implement the metasurface modulator 204:

[0133] Scheme 1: When the metasurface modulator 204 is an incident light angle response device, referring to Figure 6 , the light rays emitted by different light-emitting modes of the metasurface modulator 204 are deflected to a specific angle after passing through the first-layer metasurface 2041-1, and then incident on the second-layer metasurface 2041-2, and are projected onto the object surface after being modulated again; the action process of the first-layer metasurface 2041-1 can be regarded as the superposition of two sub-processes, namely the deflection modulation of the light beam and the determination of the off-axis optical system; it needs to satisfy:

[0134] For the sine-cosine structured light, it is only necessary to design N T types of cosine structured light with periodic distribution, and the phase shift is achieved by adjusting the light-emitting position of the light source.

[0135] At this time, the projected pattern is:

[0136]

[0137] Among them, the projected image of the i-th step phase-shifted structured light, b(x, y) and a(x, y) are the background and contrast, x, y are the spatial coordinates, T is the period, and the projected phase φ p (x, y) = 2πx / T;

[0138] For the determination of the metasurface modulator 204, according to the generalized Snell's law:

[0139]

[0140] Among them, n t , n i are the refractive indices around the refraction and incident interfaces respectively, λ is the wavelength, is the phase gradient;

[0141] Furthermore:

[0142]

[0143] Among them, n t is the refractive index around the refraction and incident interfaces, n i is the refractive index around the incident interface, and different values need to be substituted in different cases. The outgoing light of one interface is the incident light of the next interface.

[0144] For the deflection modulation of the light beam, according to Equation (Equation 1-2), for the deflection distribution modulation of the light beam, the phase distribution that the metasurface needs to provide for the projected light wave is:

[0145]

[0146] Among them, θ i is the incident angle of the first metasurface 2041-1. At this time, n i = n0≈1, where n0 is the refractive index of air; n t = n R , n R is the refractive index of the medium between the two metasurfaces. x and y are the position coordinates on the interface, and θ r is a specific deflection angle;

[0147] The deflection angle θ r is a specially designed value to ensure that there is no crosstalk in different light-emitting modes. The number of values of θ r is the number of frequencies in the designed coding scheme, that is

[0148] For an off-axis optical system, its phase distribution should satisfy:

[0149]

[0150] Among them, f is the focal length;

[0151] Superimposing the above process, the phase transformation provided by the first metasurface is:

[0152]

[0153] Design the first metasurface 2041-1 according to the above phase distribution;

[0154] The input for the design of the second metasurface 2041-2 is parallel light with an incident angle of θ r . It is necessary to expand the above parallel light and modulate the spatial intensity to form a cosine phase distribution.

[0155] The incident optical field distribution can be expressed as

[0156]

[0157] Among them, is the wave number, λ is the wavelength, and E0 is the amplitude of the electric field.

[0158] After passing through the second metasurface, the optical field is:

[0159]

[0160] In order to obtain the intensity distribution of Equation 1-2, then:

[0161]

[0162] It is obtained that:

[0163]

[0164] Design the surface structure of the second metasurface 2041-2 so that its phase modulation satisfies Equation (1-9).

[0165] Solution 2: When the metasurface modulator 204 is used as a device responsive to the incident position, it satisfies the following:

[0166] See Figure 5 As shown, the metasurface modulator 204 is designed in zones. First, according to the number of light emission modes P1, P2,..., P M , that is, M. The light-emitting units 207 of the addressable VCSEL array 201 are evenly divided into M regions. G(j, k) represents the light-emitting region in the j-th column and the k-th row. Each region has the same light-emitting power. The corresponding metasurface modulator 204 is also designed in zones and is also divided into M regions. S(j, k) represents the modulation region in the j-th column and the k-th row. S(j, k) and G(j, k) are spatially aligned to ensure that the light irradiates G(j, k) when S(j, k) emits light. Here, one of j and k can take the value of 1, and at this time, it is a one-dimensional row distribution or column distribution.

[0167] Then, design G(j, k) to generate the corresponding projection pattern I(j, k). Among them, I(j, k) represents the j-th group and the k-th projection pattern.

[0168] For sinusoidal and cosinusoidal structured light, the projected pattern is:

[0169]

[0170] Among them, The projected image of the i-th step phase-shifted structured light, b(x, y) and a(x, y) are the background and contrast, x, y are spatial coordinates, T is the period, and the projected phase φ p (x, y) = 2πx / T.

[0171] For each structured light The corresponding metasurface region S(j, k) can be completed with only a single layer of metasurface. For the specific design, refer to Equations 1-8 and 1-9, and at this time, θ r = 0.

[0172] In Solution 3, when the metasurface modulator 204 serves as a polarization direction encoder, it needs to satisfy:

[0173] Refer to Figure 5 as shown. Condition 1: Perform a partition design on the metasurface modulator 204. First, according to the number of emission modes P1, P2,..., P M , that is, M. Evenly divide the light-emitting units 207 of the addressable VCSEL array 201 into M regions. G(j, k) represents the light-emitting region in the j-th column and k-th row. Each region has the same light-emitting power. The corresponding metasurface modulator 204 also undergoes a partition design and is also divided into M regions. S(j, k) represents the modulation region in the j-th column and k-th row. S(j, k) and G(j, k) are spatially aligned to ensure that when S(j, k) emits light, the light irradiates onto G(j, k). Here, one of j and k can take the value of 1, and at this time, it is a one-dimensional row distribution or column distribution.

[0174] Then, design G(j, k) to generate the corresponding projection pattern I(j, k). Among them, I(j, k) represents the k-th projection pattern in the j-th group.

[0175] Refer to Figure 7 , S(j, k) is designed as a combination of nano-units that periodically transform the polarization angle of the incident linearly polarized light. The deflection angle α should satisfy:

[0176]

[0177] where x is the coordinate value along the x direction, and T is the period.

[0178] Condition 2: After satisfying Condition 1, when in use, add a polarizer with the same polarization direction as the light source in front of and behind the metasurface modulator (204), and a cosine structured light can be obtained.

[0179] In addition to Solution 1, Solution 2, and Solution 3, the metasurface modulator 204 is a device that responds to the incident wavelength or polarization state, and different structured light patterns can be switched by using light sources with different wavelengths or different polarization states.

[0180] The specific process of Step 2 is as follows:

[0181] Step 2.1: Drive the light-emitting holes of the vertical cavity surface laser emitter array 201 of the corresponding addressable VCSEL to emit light according to the determined emission mode, and project the preset structured light pattern 206 onto the object surface;

[0182] Step 2.2, the camera system 001 starts to expose and collect the structured light pattern 206;

[0183] Step 2.3, repeat steps 2.1 - 2.3 until N T * N times of structured light projection and collection.

[0184] The specific steps of step 3 are as follows:

[0185] Describe the structured light pattern 206 captured by the camera system 001 as:

[0186]

[0187] Where, represents the image of the n - th step phase - shifted structured light collection, B(x, y) represents the background, A(x, y) represents the contrast, represents the wrapped phase collected;

[0188] The absolute phase collected by the camera system (001) satisfies the following formula:

[0189]

[0190] Where, K(x, y ) is a series and is an integer;

[0191] Calculate the wrapped phase from the following formula:

[0192]

[0193] Then use the multi - frequency expansion method to expand the wrapped phase from low to high in turn to obtain the absolute phase φ c (x, y) collected by the camera.

[0194] As an alternative, the wrapped phase in step 2.3 can also be solved by the random phase algorithm to eliminate the possible errors caused by processing and assembly. For the specific method, refer to the reference Du H, Yan J, Wang J. Random phase - shifting algorithm by constructing orthogonal phase - shifting fringe patterns[J]. Applied Optics, 2017, 56(11): 3071 - 3076.

[0195] The specific process of step 4 is as follows:

[0196] Step 4.1, first, according to the system calibration parameters, perform epipolar correction on the phase diagram;

[0197] For the specific solution, please refer to the literature: Feng S, Zuo C, Zhang L, et al. Calibration of fringe projection profilometry: A comparative review[J]. Optics and lasers in engineering, 2021, 143: 106622.

[0198] Step 4.2: Perform phase matching;

[0199] For a monocular structure, select the projected absolute phase φ p (x, y) and the unwrapped phase φ c (x, y) collected. Search, interpolate, and match row by row to obtain the disparity of the phase matching points, calculate the disparity map, and then use the calibration parameter matrix to reconstruct the depth map to obtain the depth map D(u, v);

[0200] For a binocular system, select the unwrapped phases φ p (x, y) collected by the left and right cameras. Search, interpolate, and match row by row to obtain the disparity of the phase matching points, calculate the disparity map, and then use the calibration parameter matrix to reconstruct the depth map to obtain the depth map D(u, v);

[0201] Step 4.3: Use the depth map D(u, v) obtained in Step 4.2 to reconstruct the three-dimensional information to obtain the three-dimensional point cloud X w = [x w , y w , z w T , thereby realizing the projection of structured light in a pure solid state and the reconstruction of three-dimensional information.​

Claims

1. A solid-state structured light three-dimensional imaging system is disposed above a target object (003), characterized in that, It includes at least one solid-state structured light projector (002) and a camera system (001) for image information acquisition. The solid-state structured light projector (002) is used to regulate and project an encoded two-dimensional structured light light field (205), and form a structured light pattern (206) with brightness modulation on the surface of the target object (003). The solid-state structured light projector (002) includes at least one vertical cavity surface laser emitter array (201) of addressable VCSELs. A metasurface modulator (204) is arranged in front of the vertical cavity surface laser emitter array (201) of addressable VCSELs, and nanostructures (2041) are arranged on the surface of the metasurface modulator (204). The vertical cavity surface laser emitter array (201) of addressable VCSELs is composed of multiple laser emission units (207), and each laser emission unit (207) is controlled individually or in groups. The entire solid-state structured light projector (002) is encapsulated by a packaging structural member (202). The optical axis of the camera system (001) is parallel to the solid-state structured light projector (002); or the optical axis of the camera system (001) forms an angle with the solid-state structured light projector (002) in the plane where the optical center connection line and the optical axis are located, and the value range of the angle is [5°, 45°]. The metasurface modulator (204) is a device that responds to incident angle, position, wavelength, polarization state or spatial frequency parameters to realize the switching of different structured light patterns. The nanostructures (2041) on its surface are fabricated by lithography, electron beam processing or nanoimprinting of one or more layers of structures; the nanostructures (2041) are composed of discrete nano-unit structural materials, and Ge or Si is used as the unit structural material in the infrared band. TiO2 or GaN is selected as the unit structural material in the visible light band; HfO2 or AlN is selected as the unit structural material in the ultraviolet band.

2. A solid-state structured light three-dimensional imaging method for the three-dimensional imaging system according to claim 1, characterized in that, It includes the following steps: Step 1: Build a solid-state structured light three-dimensional imaging system and determine the light emission mode. Step 2: The solid-state structured light projector (002) in the solid-state structured light three-dimensional imaging system projects a structured light pattern on the surface of the target object (003), and the camera system (001) acquires the solid-state structured light image information. Step 3: Solve the phase using the structured light image information collected in Step 2. Step 4: Reconstruct the depth map and three-dimensional point cloud using the phase information obtained in Step 3. The specific method for determining the light emission mode in Step 1 is as follows: Step 1.1: Determine the encoding scheme. According to the principle of structured light three-dimensional imaging, each point in the field of view is spatially encoded, and M target projection patterns are obtained using binary coding, Gray coding, sine-cosine coding, or hybrid coding methods. ; Step 1.2: Determine the light emission mode of VCSEL. The M target projection patterns obtained according to Step 1.1 , determine the corresponding M emission patterns . The emission pattern is a matrix corresponding to the address of the addressable VCSEL, and the emission pattern changes two-dimensionally or one-dimensionally; if the light source is a two-dimensionally addressable VCSEL, then the emission pattern is a two-dimensional matrix. represents the brightness coefficient of the j-th row and k-th column of the i-th projection pattern, where i = 1, 2, …, M; is a real number with a value in [0, 1]. indicates that the light-emitting point at this address is not activated. indicates that the light-emitting point at this address emits light at the rated maximum power, and the intermediate value indicates that it operates at partial power; if the light source is a one-dimensionally addressable VCSEL, then is a one-dimensional matrix. At this time, represents the -th row or the -th column where the light-emitting point is not activated. represents the -th row or the -th column where the light-emitting point emits light at the rated maximum power, and if it is an intermediate value between 0 and 1, it indicates that it operates at partial power. When determining M light-emitting modes, two conditions should be met: Have a difference from each other to reduce the interference of different modes; Control the difference in the luminous power of the light-emitting modes within 10%; The light-emitting mode in Step 1.2 realizes the projection of M projection patterns by adjusting the metasurface modulator (204). When there are multiple one-dimensional coded projection patterns, the coding direction is vertically distributed perpendicular to the light source arrangement direction.

3. The three-dimensional imaging method of solid structured light according to claim 2, wherein When the metasurface modulator (204) is an incident light angle response device, the vertical cavity surface laser emitter array (201) of the addressable VCSEL emits light in different emission modes. After passing through the first-layer metasurface (2041-1), the light is deflected to a specific angle and then incident on the second-layer metasurface (2041-2). After being modulated again, it is projected onto the object surface. The action process of the first-layer metasurface (2041-1) is regarded as the superposition of two sub-processes, namely the deflection modulation of the light beam and the determination of the off-axis optical system. Specifically, it needs to satisfy: For sinusoidal and cosinusoidal structured light, it is only necessary to determine a cosinusoidal structured light with a periodic distribution. The phase shift is achieved by adjusting the position of the light source The projected pattern at this time is: Formula (1-2) Among them, The phase-shifted structured light projection image of the i-th step, and are the background and contrast, x and y are spatial coordinates, T is the period, and the projection phase ; For the determination of the metasurface modulator (204), according to the generalized Snell's law: Formula (1-3) Among them, is the refractive index around the refraction interface, is the refractive index around the incident interface, is the wavelength, is the phase gradient; Furthermore: Formula (1-4) Among them, is the refractive index around the refraction interface, is the refractive index around the incident interface; For the deflection modulation of the light beam, according to Equation (1-2), for the deflection distribution modulation of the light beam, the phase distribution that the metasurface needs to provide for the projected light wave is: Formula (1-5) Among them, is the incident angle of the first-layer metasurface (2041-1), at this time , is the refractive index of air; , is the refractive index of the medium between the two layers of metasurfaces, x and y are the position coordinates on the interface, is the deflection angle; the deflection angle ensures that there is no crosstalk in different emission modes, The number of values of is the number of frequencies in the designed coding scheme, that is ; For the off-axis optical system, its phase distribution should satisfy: Formula (1-6) wherein, is the focal length; Superimposing the above processes, the phase transformation provided by the first-layer metasurface (2041-1) is: Formula (1-7) Therefore, the first-layer metasurface (2041-1) is determined according to the phase distribution of Equation (1-6) above; The input of the second-layer metasurface (2041-2) is parallel light with an incident angle of . It is necessary to expand the above parallel light and modulate the spatial intensity to form a cosine phase distribution: The incident optical field distribution is expressed as: wherein, is the wave number, is the wavelength, is the amplitude of the electric field; After passing through the second-layer metasurface, the optical field is: In order to obtain the intensity distribution of Equation (1-2), then: Formula (1-8) Obtained: Formula (1-9) Design the surface structure of the second-layer metasurface (2041-2) so that its phase modulation satisfies Equation (1-9).

4. The three-dimensional imaging method of solid structured light according to claim 2, wherein When the metasurface modulator (204) is a device that responds to the incident position, it satisfies the following: Condition 1: Partition the modulator (204). First, according to the number of light emission modes which is M, evenly divide the laser emission units (207) of the vertical cavity surface laser emitter array (201) of the addressable VCSEL into M regions. G(j, k) represents the light emission region in the j-th column and the k-th row. Each region has the same light emission power. The corresponding metasurface modulator (204) is also partitioned and is also divided into M regions. S(j, k) represents the modulation region in the j-th column and the k-th row; S(j, k) and G(j, k) are spatially aligned to ensure that when S(j, k) emits light, the light irradiates on G(j, k). Here, one of j and k can take the value of 1, and at this time, it is a one-dimensional row distribution or column distribution; Then, limit G(j,k) to generate the corresponding projected pattern I(j,k), where I(j,k) represents the k-th projected pattern in the j-th group; For the sine-cosine structured light, the projected pattern is: Formula (1-2) Among them, The structured light projection image of the i-th step phase shift, and are the background and contrast, x and y are spatial coordinates, T is the period, and the projected phase ; Condition 2: After satisfying Condition 1, when in use, add a polarizer with the same polarization direction as the light source in front of and behind the metasurface modulator (204), and cosine structured light can be obtained.

5. The three-dimensional imaging method of solid structured light according to claim 2, characterized in that When the metasurface modulator (204) is a polarization direction encoder, it needs to satisfy: Partition the metasurface modulator (204). First, according to the number of emission modes , which is M, evenly divide the laser emission units (207) of the vertical cavity surface emitting laser (VCSEL) array (201) of the addressable VCSEL into M regions. G(j, k) represents the emission region in the j-th column and the k-th row. Each region has the same emission power. The corresponding metasurface modulator (204) is also designed with partitions, which are also divided into M regions. S(j, k) represents the modulation region in the j-th column and the k-th row; S(j,k) and G(j,k) are spatially aligned to ensure that the light irradiates G(j,k) when S(j,k) emits light; when one of j and k takes the value of 1, it is a one-dimensional row distribution or column distribution at this time; Let G(j,k) generate the corresponding projected pattern I(j,k), where I(j,k) represents the k-th projected pattern in the j-th group; Regarding S(j, k) as a combination of nano-units that periodically transform the polarization angle of incident linearly polarized light, the deflection angle should satisfy: Where x is the coordinate value and T is the period; When in use, add a polarizer with the same polarization direction as the light source in front of and behind the metasurface modulator, and cosine structured light can be obtained.

6. The three-dimensional imaging method of solid structured light according to claim 2, characterized in that The specific process of Step 2 is as follows: Step 2.1, according to the determined emission mode, drive the laser emission unit (207) of the vertical cavity surface laser emitter array (201) of the corresponding addressable VCSEL to emit light, and project the preset structured light pattern (206) onto the surface of the target object (003); Step 2.2, the camera system (001) starts to expose and collect the structured light pattern (206); Step 2.

3. Repeat steps 2.1 - 2.3 until completion Sub - structured light projection and acquisition.

7. A method for solid-state structured light three-dimensional imaging according to claim 2, characterized in that, The specific steps of Step 3 are as follows: Describe the structured light pattern (206) captured by the camera system (001) as: Formula (1-10) Among them, represents the image acquired by the structured light with phase shift at the n-th step, represents the background, represents the contrast, represents the wrapped phase acquired. The absolute phase collected by the camera system (001) satisfies the following equation: = Formula (1-11) Among them, is the number of stages and is an integer; The wrapped phase is calculated by the following equation: Formula (1-12) Then, the wrapped phase is unfolded from low to high in turn using the multi-frequency unfolding method , and the absolute phase collected by the camera system is obtained .

8. A method for three-dimensional imaging of structured light in a solid state according to claim 2, characterized in that, The specific process of step 4 is as follows: Step 4.1: Perform epipolar correction on the absolute phase map; Step 4.2: Perform phase matching; For a monocular structure, the projected absolute phase is selected and the collected absolute phase , search, interpolate, and match row by row to obtain the disparity of the phase matching points, calculate the disparity map, and then use the calibration parameter matrix to reconstruct the depth map to obtain the depth map ; For the binocular structure, the absolute phase collected by the left and right cameras is selected , searched, interpolated, and matched row by row to obtain the disparity of the phase matching points, calculate the disparity map, and then use the calibration parameter matrix to reconstruct the depth map to obtain the depth map ; Step 4.3, use the depth map obtained in Step 4.2 to reconstruct three-dimensional information to obtain a three-dimensional point cloud , thereby realizing the structured light projection of a pure solid state and the reconstruction of three-dimensional information.

Images