Multispectral structured light three-dimensional imaging system and method

Abstract

The invention discloses a multispectral structured light three-dimensional imaging system and method, and the system is arranged above a target object, and at least comprises a multispectral multispectral structured light projector and a camera system used for collecting multispectral structured light information. The imaging method comprises the following steps: light emitted by the laser of the multispectral structured light projector passes through the beam combiner and is further irradiated on the MEMS micromirror; the light is reflected by the MEMS micromirror to irradiate on a projection plane, and multispectral structured light is collected; determining a coding method, and demodulating the multispectral structured light information; performing depth information and three-dimensional information reconstruction; according to the method, parallel projection of multiple channels by different structured light can be realized, the imaging speed can be increased by several times to dozens of times on the premise of ensuring high precision, and the method has the advantages of high imaging speed, high imaging precision, wide application range and low cost.

Description

Technical Field

[0001] The present invention relates to the field of photoelectric measurement and three-dimensional vision technology, and in particular to a multi-spectral structured light three-dimensional imaging system and method. Background Art

[0002] Structured light 3D imaging technology is widely used in 3D modeling, robot navigation, morphology measurement and other fields due to its high precision and good robustness. The classic structured light 3D imaging system consists of a structured light projector and a camera with an optical system. The structured light projector projects a pre-designed structured light image or image sequence onto the surface of the object. The camera with an optical system captures the structured light pattern on the surface of the object from another angle. Due to the difference in viewing angles between the two, the camera captures a deformed structured light pattern. The deformed structured light and pattern are related to the height distribution of the surface of the object. By designing corresponding calibration and demodulation algorithms, the height distribution of the surface of the object can be calculated, and then the 3D information can be reconstructed. Common structured light can be divided into dynamic structured light and static structured light. Static structured light helps to restore depth by projecting random dots, grids, and other morphological features with local or global uniqueness onto the surface of the object. Dynamic structured light encodes the spatial position of pixels by using multiple frames of images, thereby completing the recovery of depth information. Static structured light usually has the characteristics of single-frame imaging and fast imaging speed, but the disadvantage is that the accuracy is often low. Dynamic structured light uses multiple frames for imaging, so the imaging speed is relatively slow.

[0003] Invention patent application CN202310286922.2 discloses a metasurface-based structured light three-dimensional imaging system and method, which uses a metasurface-based control device to perform high-quality control of the laser beam, uses a MEMS micromirror to scan and project the laser beam, and cooperates with relevant structured light design and demodulation methods to achieve high-precision reconstruction of the three-dimensional information of the object surface; however, since the invention still needs to project and collect multiple frames of images at different times for three-dimensional reconstruction, the imaging speed is relatively slow. Summary of the invention

[0004] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a multi-spectral structured light three-dimensional imaging system and method. By designing a multi-spectral structured light projector, multiple channels of structured light of different wavelengths can be projected in parallel, and multi-frame structured light can be projected and collected at the same time, and the multi-spectral structured light information is demodulated to achieve fast three-dimensional imaging. The present invention has the advantages of fast imaging speed, high imaging accuracy, wide application range and low cost.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A multi-spectral structured light three-dimensional imaging system is arranged above a target object 003, and comprises at least a multi-spectral multi-spectral structured light projector 001 and a camera system 002 for collecting multi-spectral structured light information; the multi-spectral structured light projector 001 comprises a laser 101 as a light source, and the light emitted by the laser 101 passes through a beam combiner 102 and is further irradiated onto a MEMS micromirror 104; and is then reflected by the MEMS micromirror 104 and irradiated onto a projection plane 105.

[0007] The camera system 002 is a monocular system composed of one camera, or a binocular system composed of two cameras, including a metasurface multi-spectral spectroscopic lens 202 and an imaging element 203. After the multi-spectral reflected light field 201 passes through the metasurface multi-spectral spectroscopic lens 202, light of different wavelengths is imaged in different areas.

[0008] The optical axis of the camera system 002 is parallel to the multi-spectral structured light projector 001 .

[0009] There is a certain angle between the optical axis of the camera system 002 and the multi-spectral structured light projector 001 in the plane formed by the optical center line and the optical axis, and the angle ranges from [5°, 45°].

[0010] The MEMS micromirror 104 can rotate in a direction parallel to the beam expansion direction.

[0011] The laser 101 is composed of a plurality of separate lasers with different wavelengths connected in parallel, each laser emits a laser beam, which is combined into a collimated laser beam by a beam combiner 102; the laser is selected from edge emitting lasers EELs or vertical cavity surface emitting laser arrays VCSELs.

[0012] The beam combiner 102 uses a metasurface flat optical element with a nanostructure on the surface.

[0013] The beam expander 103 is a cylindrical mirror, a Powell prism, or a super-surface flat optical element with a nanostructure on the surface.

[0014] The metasurface multi-spectral beam splitting lens 202 is a metasurface flat plate optical element having a nanostructure on its surface.

[0015] The nanostructure is composed of discrete nano unit structural materials, Ge or Si is used as the unit structural material in the infrared band; TiO2 or GaN is selected as the unit structural material in the visible light band; HfO2 or AlN is selected as the unit structural material in the ultraviolet band.

[0016] A multi-spectral structured light three-dimensional imaging method comprises the following steps:

[0017] S1: Determine system parameters and design a corresponding optical system, that is, set a multi-spectral structured light projector 001 based on a metasurface and a camera system 002 for collecting structured light above the target object 003; and adjust the parameters of the multi-spectral structured light projector 001 and the camera system 002;

[0018] S2: Calibrate the camera system 002;

[0019] S3: Determine the encoding method, control the projection and collect the multi-spectral structured light;

[0020] S4: demodulate the multi-spectral structured light information;

[0021] S5: Reconstruct depth information and three-dimensional information using the calibration parameter matrix.

[0022] The step S1 adjusts the parameters of the multi-spectral structured light projector 001 and the camera system 002, and the specific steps are:

[0023] S1-1: The resolution of camera system 002 is [R x ,R y [, the resolution of the depth map is [r x ,r y [, then the resolution of the camera system 002 is selected as follows:

[0024]

[0025] S1-2: The beam combiner 102 in the multi-spectral structured light projector 001 needs to meet the following requirements:

[0026]

[0027] Among them, θ t and θ i denote the diffraction angle and the incident angle, n t and n i They represent the refractive index of the refracting medium and the refractive index of the incident medium, respectively, λ represents the wavelength of the incident light, represents the lens phase gradient;

[0028]

[0029] Among them, θ i2 ,θ i1 They represent the incident angles of two incident lights with wavelengths λ1 and λ2 respectively;

[0030] The phase of the metasurface multi-spectral spectroscopic lens 202 satisfies the following equation:

[0031]

[0032] Wherein, λ represents the wavelength of the incident light, f represents the focal length, x and y represent the horizontal and vertical coordinates of the position of the metasurface multi-spectral spectroscopic lens 202, respectively. Due to different focusing positions, the coordinates of the positions of x and y will be transformed as follows due to the coordinates of different channels:

[0033]

[0034] Among them, x′, y′ are the coordinates of the metasurface multi-spectral spectroscopic lens 202 in the coordinate system, whose zero point is at the center of the lens, and a, b are the principal point positions of different imaging areas in the lens coordinate system, which are determined by the position of the focusing point.

[0035] The specific steps of step S2 are:

[0036] S2-1, using the n The light source is used as the illumination during calibration to ensure that each channel can obtain a valid image. Each imaging area is treated as a camera, and the angle of the calibration plate is changed to take multiple groups of pictures to obtain a group of pictures.

[0037] S2-2, calibrate the calculation system parameters;

[0038] Camera coordinate system X c and multi-spectral structured light projector 001 coordinate system X p The transformation relationship to the world coordinate system is expressed as follows:

[0039]

[0040] Among them, M c =[R c ,t c ] and M p =[R p ,t p ], R and t are rotation and translation matrices respectively; X c =[x c ,y c ,z c ] T , X p =[x p ,y p ,z p ] T ;X w =[x w ,y w ,z w ] T is the coordinate of the corresponding point in the world coordinate system;

[0041] And there are:

[0042]

[0043] in, is the pixel coordinate of the corresponding point in the camera coordinate system, is the pixel coordinate of the corresponding point in the multi-spectral structured light projector 001 coordinate system, A c and A p is the transformation matrix under the pinhole model, and its form is:

[0044]

[0045] Where (u0, v0) is the principal point position, α and β are the focal lengths along the u-axis and v-axis of the image plane, and γ is a parameter describing the skewness of the two image axes;

[0046] then:

[0047]

[0048] The transformation matrix A is obtained by calibration c M c , A p M p .

[0049] The step S3 is specifically as follows:

[0050] S3-1, place the structured light codes of the same frequency in the same channel. When the number of channels is greater than 4, the channel with the principal point closer to the center is assigned to the high-frequency code;

[0051] Assume there are k projection frequencies, and the phase shift steps of each frequency are j1, j2...j k , the total number of structured light projections is:

[0052]

[0053] Among them, N p When it is an integer multiple of the number of channels C, the utilization rate is the highest;

[0054] S3-2, control projection and collect multi-spectral structured light

[0055] S3-2-1, driving the MEMS to run until steady state; after receiving the frame synchronization signal, applying sinusoidal current signals of different frequencies to the lasers 101 of different channels respectively, the phase of the sinusoidal current signal determines the phase of the structured light, and when the frame synchronization signal is received, the camera starts exposure until the exposure of the preset time is completed;

[0056] S3-2-2, after receiving the next frame synchronization signal, the phase of the laser drive signal is changed; at the same time, the camera is exposed and the image is collected;

[0057] S3-2-3, crop the acquired image to ensure that the multi-channel images are point-aligned.

[0058] The step S4 comprises the following specific steps:

[0059] S4-1, use the phase shift method to extract the phase of medium frequency and low frequency, specifically:

[0060] For the N-step phase shift method, the projected structured light satisfies the following formula:

[0061]

[0062] in, represents the n-th phase-shifted structured light projection image, b(x, y) and a(x, y) represent the background and contrast respectively, φ p (x, y) represents the absolute phase of the transmitted structured light;

[0063] The multi-spectral structured light image collected by the camera system 002 satisfies the following formula:

[0064]

[0065] in, represents the image collected by the phase-shifted structured light at the nth step, B(x,y) and A(x,y) represent the background and contrast respectively. Indicates the collected package phase;

[0066] Furthermore, the absolute phase of the camera end acquired by the camera system 002 satisfies the following formula:

[0067]

[0068] Where K(x,y) represents the series;

[0069] The wrapping phase is calculated as follows:

[0070]

[0071] Substitute the segmented and aligned structured light image into the wrapping phase calculation formula to calculate the wrapping phase

[0072] S4-2, Wrapping phase of high frequency fringe pattern using Fourier transform method Perform extraction;

[0073] S4-3, using the multi-frequency expansion method, the wrapped phase is expanded from low to high Get the absolute phase φ of the camera after expansion c (x,y).

[0074] The step S5 comprises the following specific steps:

[0075] S5-1, firstly, epipolar correction is performed on the phase image according to the calibration parameters so that the phase matching points are line-aligned, and interpolation and clipping are performed so that the resolution of the images is the same;

[0076] S5-2, perform phase matching, specifically:

[0077] When the camera system 002 adopts a monocular structure, the absolute phase φ of the projection is selected. p (x,y) and the absolute phase φ at the camera end c (x, y), search, interpolate, and match row by row to obtain the disparity of the phase matching point, calculate the disparity map, and then use the calibration parameter matrix to reconstruct the depth map to obtain D(u, v);

[0078] When the camera system 002 adopts a binocular structure, the absolute phase φ of the camera end is obtained by selecting the left and right cameras to collect and calculate respectively. c (x, y), search, interpolate, and match row by row to obtain the disparity of the phase matching point, calculate the disparity map, and then use the calibration parameter matrix to reconstruct the depth map to obtain D(u, v);

[0079] SS5-3: Reconstruct the three-dimensional information to obtain the three-dimensional space coordinate X w =[x w ,y w ,z w ] T , the calculation method is as follows:

[0080] X w =(M c ) -1 X c

[0081] Among them, X c =[u,v,D(u,v)] T .

[0082] Compared with the prior art, the beneficial effects of the present invention are:

[0083] 1. The present invention achieves isolation of different structured light projection channels through the design of the multi-spectral structured light projector 001. In conjunction with the optical system and imaging algorithm of the acquisition end of the multi-channel regional imaging, the collected multi-spectral structured light information can be demodulated, and depth information and three-dimensional information can be reconstructed, thereby overcoming the defect of the existing structured light projection technology that the multi-channel mutual coupling interference leads to a decrease in accuracy. Therefore, the present invention has the advantage of high accuracy.

[0084] 2. The present invention conveniently realizes more channels by building a multi-spectral structured light three-dimensional imaging system. In contrast, the existing multi-channel structured light projection technology usually uses three channels, RGB, with a small number of channels and poor resistance to ambient light interference. This method greatly speeds up the imaging speed while maintaining a relatively narrow bandwidth and reducing the interference of ambient light.

[0085] 3. The system of the present invention only includes a multi-spectral structured light projector 001 and a camera system 002 for collecting multi-spectral structured light information, and they are both highly integrated and have a compact system structure. Compared with the traditional multi-channel technology that requires the introduction of multiple imaging devices or multiple projection light source systems, this method is convenient for mass production and has the characteristics of high integration and low cost.

[0086] In summary, the present invention can realize the parallel projection of multiple channels of different structured light, and can increase the imaging speed by several times to dozens of times while ensuring high precision. It has the advantages of fast imaging speed, high imaging precision, wide application range and low cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 It is a schematic diagram of the structure of the multi-spectral structured light three-dimensional imaging system of the present invention.

[0088] Figure 2 It is a schematic structural diagram of the multi-spectral structured light projector 001 of the present invention.

[0089] Figure 3 002 is a schematic diagram of the structure of the camera system of the present invention, wherein: Figure 3 (a) is a schematic diagram of the working principle of the camera system. Figure 3 (b) is a schematic diagram of imaging of the camera system on the imaging element.

[0090] Figure 4 It is a flowchart of the steps of the multi-spectral structured light three-dimensional imaging method of the present invention.

[0091] Figure 5 It is a timing diagram of system signals in the present invention.

[0092] Figure 6 Schematic diagram of the channel arrangement of the multi-spectral system in the present invention.

[0093] In the figure, 001, multi-spectral structured light projector; 002, camera system; 003, target object; 101, laser; 102, laser beam combiner; 103, unidirectional beam expander; 104, MEMS micromirror; 105, projection plane; 201, multi-spectral reflection square; 202, metasurface multi-spectral spectroscopic lens; 203, imaging element. DETAILED DESCRIPTION

[0094] The present invention will be described in detail below in conjunction with the accompanying drawings.

[0095] Reference Figure 1 , a multi-spectral structured light three-dimensional imaging system, arranged above a target object 003, comprising at least one multi-spectral structured light projector 001 and a camera system 002 for collecting multi-spectral structured light information; Figure 2 The spectral multi-spectral structured light projector 001 includes a laser 101 as a light source. The light emitted by the laser 101 is combined into a collimated laser beam by a beam combiner 102, and then incident on a unidirectional beam expander 103. After beam expansion by 103, the cross-section of the light beam becomes a very thin "straight line" and has a certain divergence angle in the direction of beam expansion; then it is further irradiated onto a MEMS micromirror 104; then it is reflected by the MEMS micromirror 104 and irradiated onto a projection plane 105.

[0096] The optical axis of the camera system 002 is parallel to the multi-spectral structured light projector 001 .

[0097] There is a certain angle between the optical axis of the camera system 002 and the multi-spectral structured light projector 001 in the plane formed by the optical center line and the optical axis, and the angle ranges from [5°, 45°].

[0098] The MEMS micromirror 104 can rotate parallel to the beam expansion direction to drive the "straight line" laser to scan on the projection plane 105. The laser drive signal and the MEMS micromirror drive signal are synchronously regulated to achieve the projection of multi-spectral structured light.

[0099] The laser 101 is composed of a plurality of separate lasers with different wavelengths connected in parallel, each of which emits a laser beam, which is combined into a collimated laser beam by the beam combiner 102. Therefore, the laser 101 can emit lasers of multiple wavelengths, and each wavelength of laser can be driven and controlled separately. In this embodiment, the laser 101 can emit lasers of at least 4 wavelengths. The number of wavelengths, i.e., the number of channels C, is preferably c 2 (c 2 ≥4, such as 4, 9, 16) to facilitate the design of the acquisition system. The center wavelengths of lasers of different wavelengths are spaced more than 20nm apart to avoid mutual interference between different wavelengths.

[0100] The laser 101 is selected from edge emitting lasers (EELs) or vertical cavity surface emitting laser arrays (VCSELs), which will help improve the integration of the system and reduce costs.

[0101] The beam combiner 102 uses a metasurface flat optical element with a nanostructure on the surface, which modulates the light field through the nanostructure on the surface so as to combine and collimate laser beams of multiple wavelengths into one laser beam, which is more compact than traditional optical systems.

[0102] The beam expander 103 uses a traditional cylindrical mirror, or a Powell prism, or a super-surface flat optical element with a nanostructure on the surface, which is more compact.

[0103] Reference Figure 3 The camera system 002 is a monocular system composed of one camera, or a binocular system composed of two cameras, including a metasurface multi-spectral spectroscopic lens 202 and an imaging element 203. After the multi-spectral reflected light field 201 passes through the spectroscopic lens 202, light of different wavelengths is imaged in different areas, thereby obtaining structured light information of different wavelengths respectively.

[0104] The beam splitter lens 202 is a super-surface flat optical element with a nanostructure on the surface. The beam splitter lens 202 can focus the incident light at different wavelengths on different areas of the surface of the imaging element 203 for imaging. Figure 3 shown.

[0105] The nanostructure is composed of discrete nano unit structural materials, Ge or Si is used as the unit structural material in the infrared band; TiO2 or GaN is selected as the unit structural material in the visible light band; HfO2 or AlN is selected as the unit structural material in the ultraviolet band; by changing its characteristic parameters, the phase of the light wave can be regulated, thereby realizing the control of the light field. The light wave is modulated by changing the scale ratio of the nanocolumns in different directions. The light wave is regulated by the angle arm length parameter of the V-shaped antenna.

[0106] The MEMS scanning micromirror 104 is a micromirror prepared by MEMS technology, and its driving principle is usually electrothermal, electromagnetic, and electrostatic effects. It can perform simple harmonic vibration or controlled motion under the action of electrical signals, thereby realizing reflection and scanning of incident light. The surface of the MEMS micromirror has a reflective coating, and a suitable coating is selected according to the wavelength of the incident light. Common coatings are gold and aluminum. Gold film is selected for wavelengths between [550nm, 1500nm], and aluminum film is selected for wavelengths between [350nm, 550nm).

[0107] Reference Figure 4 , a multi-spectral structured light three-dimensional imaging method, comprising the following steps:

[0108] S1: Determine system parameters and design a corresponding optical system, that is, set a multi-spectral structured light projector 001 based on a metasurface and a camera system 002 for collecting structured light above the target object 003; and adjust the parameters of the multi-spectral structured light projector 001 and the camera system 002;

[0109] S2: Calibrate the camera system 002;

[0110] S3: Determine the encoding method, control the projection and collect the multi-spectral structured light;

[0111] S4: demodulate the multi-spectral structured light information;

[0112] S5: Reconstruct depth information and three-dimensional information using the calibration parameter matrix.

[0113] The step S1 adjusts the parameters of the multi-spectral structured light projector 001 and the camera system 002, and the specific steps are:

[0114] S1-1: Select wavelengths according to application scenarios. For high-precision working scenarios, wavelengths in the ultraviolet and blue bands are preferred; for scenarios that are resistant to ambient light or where the target object is a human body or a biological body, the near-infrared band is preferred. Therefore, the number of channels C of the camera system 002 and the working wavelengths λ1, λ2…λ of each channel are determined. C , since the selection of the resolution of the camera system 002 needs to balance the cost, and the cost is related to the number of channels C, the resolution of the camera system 002 is [R x ,R y ], the resolution of the depth map is [r x ,r y ], the resolution of the camera system 002 is selected as follows:

[0115]

[0116] S1-2: The arrangement of components in the multi-spectral structured light projector 001 will have an impact on the design of the system. Therefore, according to the selected laser arrangement, limit and fixed connection, and the heat dissipation method, the consistency of the input light wave at different time points and different batches should be ensured.

[0117] The beam combiner 102 in the multi-spectral structured light projector 001 needs to meet the following requirements:

[0118] Regarding the design of the beam combiner 102, in order to make different incident lights converge to the same direction, according to Snell's law, the following equation exists:

[0119]

[0120] Among them, θ t and θi denote the diffraction angle and the incident angle, n t and n i They represent the refractive index of the refracting medium and the refractive index of the incident medium, respectively, λ represents the wavelength of the incident light, represents the lens phase gradient;

[0121] In order to make different light beams converge to the same diffraction direction, that is, the diffraction angles θ of different incident light beams t The same, while taking the refractive index n of the incident medium air t =1, so:

[0122]

[0123] Among them, θ i2 ,θ i1 They represent the incident angles of two incident lights with wavelengths λ1 and λ2 respectively;

[0124] The metasurface multi-spectral spectroscopic lens 202, in order to focus the light waves at the same focal point, has a phase that satisfies the following equation:

[0125]

[0126] Wherein, λ represents the wavelength of the incident light, f represents the focal length, x and y represent the horizontal and vertical coordinates of the position of the metasurface multi-spectral spectroscopic lens 202, respectively. Due to different focusing positions, the coordinates of the positions of x and y will be transformed as follows due to the coordinates of different channels:

[0127]

[0128] Among them, x′, y′ are the coordinates of the metasurface multi-spectral spectroscopic lens 202 in the coordinate system, whose zero point is at the center of the lens, and a, b are the principal point positions of different imaging areas in the lens coordinate system, which are determined by the position of the focusing point.

[0129] The design scheme that meets the above objectives can be manufactured by one of the following technologies: photolithography, electron beam lithography, femtosecond laser direct writing lithography, and nanoimprinting.

[0130] The specific steps of step S2 are:

[0131] S2-1, using the n The light source is used as the illumination during calibration to ensure that each channel can obtain a valid image. Then the light source is used to illuminate the calibration plate for image acquisition. Multiple areas on the imaging element are imaged, which can be regarded as multiple cameras. Each imaging area is treated as a camera. The angle of the calibration plate is changed, and multiple groups of pictures are taken to obtain a group of pictures.

[0132] S2-2, calibrate the calculation system parameters;

[0133] Camera coordinate system X c and multi-spectral structured light projector 001 coordinate system X p The transformation relationship to the world coordinate system is expressed as follows:

[0134]

[0135] Among them, M c =[R c ,t c ] and M p =[R p ,t p ], R and t are rotation and translation matrices respectively. c =[x c ,y c ,z c ] T , X p =[x p ,y p ,z p ] T .X w =[x w ,y w ,z w ] T is the coordinate of the corresponding point in the world coordinate system;

[0136] And there are:

[0137]

[0138] in, is the pixel coordinate of the corresponding point in the camera coordinate system, is the pixel coordinate of the corresponding point in the multi-spectral structured light projector 001 coordinate system, A c and A p is the transformation matrix under the pinhole model, and its form is:

[0139]

[0140] Among them, (u0,v0) is the position of the principal point, α and β are the focal lengths along the u-axis and v-axis of the image plane, and γ is a parameter describing the skewness of the two image axes.

[0141] then:

[0142]

[0143] The transformation matrix A is obtained by calibration c M c , Ap M p .

[0144] For specific calibration methods, please refer to mature solutions in the industry (such as: Zhang S, Huang P S. Novel method for structured light system calibration [J]. Optical Engineering, 2006, 45 (8): 083601-083601-8.).

[0145] As an alternative, the system includes two cameras and a projection device to form a binocular system. In this case, the two cameras can be calibrated directly to obtain the internal and external parameters of the two cameras. Here, the images of the corresponding bands need to be calibrated as one more set of binocular cameras to obtain n sets of camera parameters. For details, please refer to Zhang Zhengyou's calibration method (Zhang Z. A flexible new technique for camera calibration [J]. IEEE Transactions on pattern analysis and machine intelligence, 2000, 22 (11): 1330-1334.).

[0146] The step S3 specifically includes:

[0147] The present invention adopts sine-cosine phase structured light encoding to have higher accuracy and anti-motion characteristics. Specifically, the following steps are included.

[0148] S3-1, determine the coding scheme based on the number of channels C and the system accuracy target, and place the structured light coding of the same frequency in the same channel to reduce the phase shift error caused by the calibration error of different channels. When the number of channels is greater than 4, place the channel with the principal point closer to the center (i.e., a 2 +b 2 Smaller) is allocated to high-frequency encoding to ensure the highest system accuracy.

[0149] As an alternative, structured light of the same frequency but different brightness is encoded in different channels to perform high dynamic range imaging to accommodate curved surfaces with large variations in surface reflectivity.

[0150] Assume there are k projection frequencies, and the phase shift steps of each frequency are j1, j2...j k , the total number of structured light projections is:

[0151]

[0152] Among them, N pWhen it is an integer multiple of the number of channels C, the utilization rate is highest.

[0153] In this embodiment, a 9-channel multi-spectral system is selected, and the 9 wavelengths are: 792nm, 830nm, 850nm, 878nm, 888nm, 905nm, 915nm, 940nm, 975nm, so that all 9 channels work in the near-infrared band.

[0154] like Figure 6 As shown in the figure, the distribution of the focusing positions of ch1-ch9 channels is shown in the figure. The wavelengths of ch1-ch9 are: 792nm, 830nm, 850nm, 878nm, 888nm, 905nm, 915nm, 940nm, 975nm. This ensures that the short wavelength is located at a better imaging position.

[0155] An example encoding method given in this implementation case is: ch1—high-frequency structured light, ch2, ch3, ch4, ch5—four-step phase-shifted medium-frequency structured light, ch6, ch7, ch8, ch9—four-step phase-shifted low-frequency structured light.

[0156] S3-2, control projection and collect multi-spectral structured light

[0157] S3-2-1, refer to Figure 5 , use square wave signal S404 to drive MEMS to run until steady state; after receiving frame synchronization signal S406, add sinusoidal current models S401, s402, s403 of different frequencies to lasers 101 of different channels respectively, the period of sinusoidal current determines the spatial frequency of projected stripe structured light, the amplitude of current determines the contrast of stripe pattern, the phase of sinusoidal signal determines the phase of structured light, and when frame synchronization signal S406 is received, the camera starts exposure until the preset exposure time is completed.

[0158] It should be noted that the photosensitive element may have different sensitivities to light of different wavelength bands, so the power of the laser driving signal of different channels needs to be appropriately adjusted to ensure that different areas of the imaging element are in a unified working range.

[0159] S3-2-2, after receiving the next frame synchronization signal S406, start the preset phase shift, that is, change the phase of the laser drive signal S401-1, s402-1, s403-1. At the same time, the camera is exposed and the image is collected;

[0160] It should be noted that a unified clock model s405 is used to align different signals; if the MEMS micromirror is in simple harmonic oscillation, the laser drive signal needs to be corrected to ensure that the projected structured light has good sinusoidal properties.

[0161] S3-2-3, cropping the acquired image, the specific method is to crop and correct according to the principal point position of the calibration parameters in S2 above, so as to ensure that the multi-channel images are point-aligned.

[0162] The step S4 comprises the following specific steps:

[0163] S4-1, use the phase shift method to extract the phase of medium frequency and low frequency, specifically:

[0164] For the N-step phase shift method, the projected structured light satisfies the following formula:

[0165]

[0166] in, represents the n-th phase-shifted structured light projection image, b(x, y) and a(x, y) represent the background and contrast respectively, φ p (x, y) represents the absolute phase of the transmitted structured light;

[0167] The multi-spectral structured light image collected by the camera system 002 satisfies the following formula:

[0168]

[0169] in, represents the image collected by the phase-shifted structured light at the nth step, B(x,y) and A(x,y) represent the background and contrast respectively. Indicates the collected package phase;

[0170] Furthermore, the absolute phase of the camera end acquired by the camera system 002 satisfies the following formula:

[0171]

[0172] Among them, K(x,y) represents the series;

[0173] The wrapping phase is calculated as follows:

[0174]

[0175] Substitute the segmented and aligned structured light image into the wrapping phase calculation formula to calculate the wrapping phase

[0176] S4-2, Wrapping phase of high frequency fringe pattern using Fourier transform method Extraction; (For specific methods, refer to: Mao X, Chen W, Su X. Improved Fourier-transform profilometry [J]. Applied optics, 2007, 46 (5): 664-668.)

[0177] S4-3, using the multi-frequency expansion method, the wrapped phase is expanded from low to high Get the absolute phase φ of the camera after expansion c (x,y).

[0178] The step S5 comprises the following specific steps:

[0179] S5-1, firstly, the phase image is subjected to epipolar correction according to the calibration parameters so that the phase matching points are line-aligned, and interpolation and cropping are performed so that the resolution of the images is the same.

[0180] S5-2, phase matching is performed, specifically:

[0181] When the camera system 002 adopts a monocular structure, the absolute phase φ of the projection is selected. p (x,y) and the absolute phase of the camera c (x, y), search, interpolate, and match by row to obtain the disparity of the phase matching point, calculate the disparity map, and then use the calibration parameter matrix to reconstruct the depth map to obtain D(u, v).

[0182] When the camera system 002 adopts a binocular structure, the absolute phase φ of the camera end is obtained by collecting and calculating the left and right cameras respectively. c (x, y), search, interpolate, and match by row to obtain the disparity of the phase matching point, calculate the disparity map, and then use the calibration parameter matrix to reconstruct the depth map to obtain D(u, v).

[0183] S5-3: Reconstruct the three-dimensional information to obtain the three-dimensional space coordinate X w =[x w ,y w ,z w ] T , the calculation method is as follows:

[0184] X w =(M c ) -1 X c

[0185] Among them, X c =[u,v,D(u,v)] T .

[0186] The present invention achieves isolation of different structured light channels through system design and matching use of imaging algorithms, can project and collect structured light within a narrow wavelength bandwidth, demodulate the collected multi-spectral structured light information, and reconstruct depth information and three-dimensional information, thus overcoming the defect of reduced precision caused by multi-channel mutual coupling interference in existing structured light projection technology and improving the speed of the system. In addition, the integrated design also makes the present invention have the characteristics of high integration and low cost.

Claims

1. A multi-spectral structured light three-dimensional imaging system, arranged above a target object (003), characterized in that: The invention comprises at least one multi-spectral structured light projector (001) and a camera system (002) for collecting multi-spectral structured light information; the multi-spectral structured light projector (001) comprises a laser (101) as a light source, the light emitted by the laser (101) passes through a beam combiner (102), and is further irradiated onto a MEMS micromirror (104); and is then irradiated onto a projection plane (105) after being reflected by the MEMS micromirror (104).

2. The multi-spectral structured light three-dimensional imaging system according to claim 1, characterized in that: The camera system (002) is a monocular system composed of one camera, or a binocular system composed of two cameras, and includes a metasurface multi-spectral spectroscopic lens (202) and an imaging element (203). After the multi-spectral reflected light field (201) passes through the spectroscopic lens (202), light of different wavelengths is imaged in different areas. The optical axis of the camera system (002) is parallel to the multi-spectral structured light projector (001); Alternatively, the optical axis of the camera system (002) and the multi-spectral structured light projector (001) form a certain angle in a plane formed by a line connecting the optical centers and the optical axis, and the angle ranges from [5°, 45°].

3. The multi-spectral structured light three-dimensional imaging system according to claim 1, characterized in that: The MEMS micromirror (104) is capable of rotating in a direction parallel to the beam expansion direction; The laser (101) is composed of a plurality of separate lasers with different wavelengths connected in parallel, each laser emits a laser beam, which is combined into a collimated laser beam by a beam combiner (102); the laser is selected from edge emitting lasers (EELs) or vertical cavity surface emitting laser arrays (VCSELs).

4. The multi-spectral structured light three-dimensional imaging system according to claim 1, characterized in that: The beam combiner (102) uses a super-surface flat optical element with a nanostructure on the surface; The beam expander (103) is a conventional cylindrical mirror, a Powell prism, or a super-surface flat optical element with a nanostructure on the surface; The beam splitting lens (202) is a super-surface flat optical element having a nanostructure on its surface; The nanostructure is composed of discrete nano unit structure materials, and Ge or Si is used as the unit structure material in the infrared band; In the visible light band, TiO2 or GaN is selected as the unit structure material; in the ultraviolet band, HfO2 or AlN is selected as the unit structure material.

5. An imaging method based on a multi-spectral structured light three-dimensional imaging system according to claim 1, characterized in that: The following steps are involved: S1: Determine system parameters and design a corresponding optical system, that is, set a multi-spectral structured light projector (001) based on a metasurface and a camera system (002) for collecting structured light above a target object (003); and adjust the parameters of the multi-spectral structured light projector (001) and the camera system (002); S2: Calibrate the camera system (002); S3: Determine the encoding method, control the projection and collect the multi-spectral structured light; S4: demodulate the multi-spectral structured light information; S5: Reconstruct depth information and three-dimensional information using the calibration parameter matrix.

6. The imaging method of a multi-spectral structured light three-dimensional imaging system according to claim 5, characterized in that: The step S1 adjusts the parameters of the multi-spectral structured light projector (001) and the camera system (002), and the specific steps are: S1-1: The resolution of the camera system (002) is [R x ,R y ], the resolution of the depth map is [r x ,r y [, then the resolution of the camera system (002) is selected as follows: S1-2: The beam combiner (102) in the multi-spectral structured light projector (001) needs to meet the following requirements: Among them, θ t and θ i denote the diffraction angle and the incident angle, n t and n i They represent the refractive index of the refracting medium and the refractive index of the incident medium, respectively, λ represents the wavelength of the incident light, represents the lens phase gradient; Among them, θ i2 ,θ i1 They represent the incident angles of two incident lights with wavelengths λ1 and λ2 respectively; The phase of the metasurface multi-spectral spectroscopic lens (202) satisfies the following formula: Wherein, λ represents the wavelength of the incident light, f represents the focal length, x and y represent the horizontal and vertical coordinates of the position of the metasurface multi-spectral spectroscopic lens (202), respectively. Due to different focusing positions, the coordinates of the positions of x and y will be transformed as follows due to the coordinates of different channels: Wherein, x′, y′ are coordinates in the coordinate system of the metasurface multi-spectral spectroscopic lens (202), whose zero point is at the center of the lens, and a, b are principal point positions of different imaging areas in the lens coordinate system, which are determined by the position of the focusing point.

7. The imaging method of a multi-spectral structured light three-dimensional imaging system according to claim 5, characterized in that: The specific steps of step S2 are: S2-1, using the n The light source is used as the illumination during calibration to ensure that each channel can obtain a valid image. Each imaging area is treated as a camera, and the angle of the calibration plate is changed to take multiple groups of pictures to obtain a group of pictures. S2-2, calibrate the calculation system parameters; Camera coordinate system X c and the multi-spectral structured light projector (001) coordinate system X p The transformation relationship to the world coordinate system is expressed as follows: Among them, M c =]R c ,t c ] and M p =]R p ,t p ], R and t are rotation and translation matrices respectively; X c =[x c ,y c ,z c ] T , X p =[x p ,y p ,z p ] T ;X w =[x w ,y w ,z w ] T is the coordinate of the corresponding point in the world coordinate system; And there are: in, is the pixel coordinate of the corresponding point in the camera coordinate system, is the pixel coordinate of the corresponding point in the (001) coordinate system of the multi-spectral structured light projector, A c and A p is the transformation matrix under the pinhole model, and its form is: Where (u0, v0) is the principal point position, α and β are the focal lengths along the u-axis and v-axis of the image plane, and γ is a parameter describing the skewness of the two image axes; then: The transformation matrix A is obtained by calibration c M c , A p M p .

8. The imaging method of a multi-spectral structured light three-dimensional imaging system according to claim 5, characterized in that: The step S3 specifically includes: S3-1, place the structured light codes of the same frequency in the same channel. When the number of channels is greater than 4, the channel with the principal point closer to the center is assigned to the high-frequency code; Assume there are k projection frequencies, and the phase shift steps of each frequency are j1, j2...j k , the total number of structured light projections is: Among them, N p When it is an integer multiple of the number of channels C, the utilization rate is the highest; S3-2, control projection and collect multi-spectral structured light S3-2-1, driving the MEMS to operate until a steady state; after receiving a frame synchronization signal, applying sinusoidal current signals of different frequencies to the lasers (101) of different channels respectively, the phase of the sinusoidal current signal determines the phase of the structured light, and when the frame synchronization signal is received, the camera starts exposure until the exposure of a preset time is completed; S3-2-2, after receiving the next frame synchronization signal, the phase of the laser drive signal is changed; at the same time, the camera is exposed and the image is collected; S3-2-3, crop the acquired image to ensure that the multi-channel images are point-aligned.

9. The imaging method of a multi-spectral structured light three-dimensional imaging system according to claim 5, characterized in that: The step S4 comprises the following specific steps: S4-1, use the phase shift method to extract the phase of medium frequency and low frequency, specifically: For the N-step phase shift method, the projected structured light satisfies the following formula: in, represents the n-th phase-shifted structured light projection image, b(x, y) and a(x, y) represent the background and contrast respectively, φ p (x, y) represents the absolute phase of the transmitted structured light; The multi-spectral structured light image collected by the camera system (002) satisfies the following formula: in, represents the image collected by the phase-shifted structured light at the nth step, B(x,y) and A(x,y) represent the background and contrast respectively. Indicates the collected package phase; Furthermore, the absolute phase of the camera end collected by the camera system (002) satisfies the following formula: Where K(x,y) represents the series; The wrapping phase is calculated as follows: Substitute the segmented and aligned structured light image into the wrapping phase calculation formula to calculate the wrapping phase S4-2, Wrapping phase of high frequency fringe pattern using Fourier transform method Perform extraction; S4-3, using the multi-frequency expansion method, the wrapped phase is expanded from low to high Get the absolute phase φ of the camera after expansion c (x,y).

10. The imaging method of a multi-spectral structured light three-dimensional imaging system according to claim 5, characterized in that: The step S5 comprises the following specific steps: S5-1, firstly, according to the calibration parameters, the phase image is subjected to epipolar correction so that the phase matching points are line-aligned, and interpolation and clipping are performed so that the resolution of the images is the same; S5-2, phase matching is performed, specifically: When the camera system (002) adopts a monocular structure, the absolute phase φ of the projection is selected. p (x,y) and the absolute phase φ at the camera end c (x, y), search, interpolate, and match row by row to obtain the disparity of the phase matching point, calculate the disparity map, and then use the calibration parameter matrix to reconstruct the depth map to obtain D(u, v); When the camera system (002) adopts a binocular structure, the absolute phase φ of the camera end collected by the left and right cameras is selected c (x, y), search, interpolate, and match row by row to obtain the disparity of the phase matching point, calculate the disparity map, and then use the calibration parameter matrix to reconstruct the depth map to obtain D(u, v); S5-3: Reconstruct the three-dimensional information to obtain the three-dimensional space coordinate X w =[x w ,y w ,z w ] T , the calculation method is as follows: X w =(M c ) -1 X c Among them, X c = [u, v, D(u, v)] T .

Images