High-precision MEMS mechanical rotation angle phase correction method and device

Through deep learning to predict and correct the MEMS phase offset value, the problem of phase inaccurate MEMS devices in structured light three-dimensional scanners is solved, which improves the accuracy of the monocular structured light camera and reduces system cost and power consumption.

CN119063655APending Publication Date: 2024-12-03CHINA JILIANG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410757261.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-13
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In the existing structured light three-dimensional scanners, the phase inaccurate calculation of monocular structured light depth based on MEMS devices, resulting in the inability to perform accurate structured light three-dimensional reconstruction.

Method used

Using high-precision phase correction method, the time value of the MEMS device reaches the monitoring position is collected through TDC, the phase offset value is obtained by calibration, and the MEMS phase offset value and time relationship are trained through deep learning methods, and each frame scan is predicted and corrected. Phase offset value during the process.

Benefits of technology

It improves the MEMS phase accuracy, enhances the accuracy of monocular structured optical cameras, and reduces the cost of neural networks and system power consumption, and reduces the volume.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119063655A_ABST
    Figure CN119063655A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of structured light three-dimensional imaging, in particular to a high-precision phase compensation method, a monocular structured light camera device based on MEMS and electronic equipment. The high-precision phase compensation method comprises the following steps: randomly selecting more than two fixed positions in the rotation process of the MEMS galvanometer, and obtaining the time of the MEMS galvanometer passing through the positions in the rotation process; a light spot projected to the MEMS galvanometer by the modulated laser forms a stripe light spot with phase information on the surface of the checkerboard; pixel coordinates and phase information of checkerboard angular points are calculated; and training and deploying the neural network model to obtain phase jitter information of the current rotation process so as to calculate phase information. The monocular structured light device based on the MEMS comprises a laser, an MEMS galvanometer, a cylindrical lens or DOE equipment, a main controller and neural network equipment, the circuit size is reduced, and time delay is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of active three-dimensional computational imaging, and specifically relates to a high-precision phase correction method and device. Background Art

[0002] Structured light 3D scanners can perform pixel-level 3D imaging, providing ultra-high distance resolution and spatial resolution in the fields of industry, consumption, architecture, and cultural relics reproduction. Therefore, structured light 3D scanners are increasingly favored by the market. Among them, ranging range, spatial resolution, and ranging accuracy are the most important performance indicators of structured light 3D scanners.

[0003] The structured light 3D scanner mainly includes a transmitting module, a receiving module, a control module and a computing module. There are two main technical routes for structured light: wired structured light and surface structured light. Surface structured light is achieved by projecting the surface shape onto the target surface. After the reflected echo of the target object is collected by the camera, it enters the signal processing and demodulates the distance information of the target object. At present, the main means of projecting surface spots are DLP devices and MEMS devices. Among them, DLP devices are currently only owned by TI in the world. They are expensive and have high power consumption. MEMS devices have relatively large advantages in terms of size and cost due to their micro-electromechanical technology. Devices based on MEMS are mainly binocular structured light products.

[0004] Compared with DLP, where each micro-mirror is in a fixed position, MEMS oscillates resonantly, so the starting position each time is not easy to predict. This results in inaccurate phase when using the phase shift method to solve the monocular structured light depth calculation, making it impossible to perform accurate structured light 3D reconstruction. Summary of the invention

[0005] The purpose of this application is to protect a high-precision phase correction method and device, which uses TDC to collect the time value of the MEMS device arriving at the monitoring position, obtains the phase offset value of the MEMS device each time it scans through calibration, and uses deep learning methods to train the relationship between the MEMS phase offset value and the time the MEMS device arrives at the monitoring position, so that the phase offset value of the MEMS device during each frame scanning process can be predicted and corrected.

[0006] In order to achieve the above purpose, the technical solution adopted by this application is:

[0007] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution, but are merely further supplements or preferences. Under the premise that there are no technical or logical contradictions, each optional method can be combined with the above-mentioned overall solution separately, and multiple optional methods can also be combined.

[0008] A high-precision phase correction method, the high-precision phase correction method comprising:

[0009] Step S1, the system initializes the working parameters of the laser, MEMS, and camera, and starts the MEMS rotation;

[0010] Step S2, confirm the rotation direction of MEMS. When MEMS reaches the starting position, FPGA starts the camera, the synchronized FPGA controls the laser and modulates its light signal, MEMS projects the laser onto the calibration board, and the camera collects the signal returned from the calibration board;

[0011] Step S21: select three points during the MEMS galvanometer rotation process to monitor the MEMS rotation position and direction, such as Figure 4 , P1, P2, and P3, among which P2 is close to P1, and P1, P2, and P3 are respectively connected to the three interfaces of FPGA. The order of signal pulses is P1, P2, and P3, which rotates from left to right. The order of signal pulses is P3, P2, and P1, which rotates from right to left.

[0012] Step S22, when the MEMS rotates from left to right, when the rotating mirror reaches position P1, the laser is started and the camera is started at the same time; when the rotating mirror reaches position P3, the laser is turned off and the camera is turned off at the same time;

[0013] Step S23, when the MEMS rotates from right to left, when the rotating mirror reaches position P3, the laser is started and the camera is started at the same time; when the rotating mirror reaches position P1, the laser is turned off and the camera is turned off at the same time;

[0014] Step S24, the modulation of the laser adopts sinusoidal modulation, I = A + B * cos (2 * π * xp / w + φ), where A, B represent the background light intensity and the modulation intensity respectively, xp represents the pixel sequence number, and w represents the resolution;

[0015] Step S3, using a high-precision TDC to record the moment when the MEMS rotates to the monitoring position;

[0016] Step S31, when the MEMS mirror rotates from left to right, the time points of collecting P1, P2, and P3 using the TDC chip or FPGA are tl1, tl2, and tl3 respectively;

[0017] Step S32: When the MEMS mirror rotates from right to left, the TDC chip or FPGA is used to collect the time points of P3, P2, and P1, which are tr1, tr2, and tr3 respectively.

[0018] Step S33, tl1==0;

[0019] Step S4, performing phase analysis on the image captured by the camera to obtain the phase information of the marking point on the calibration plate;

[0020] Step S41, obtaining pixel position information of corner points on a chessboard in the captured image by using Zhang Zhengyou calibration method;

[0021] Step S42, determining the phase information at the position of the marking point by using the collected light intensity information, first selecting the effective area of ​​the required image, then calculating the light intensity value of the pixel by column, finding the maximum or minimum point according to the collected light intensity value, and then confirming the phase value of the marking point;

[0022] Step S43, repeat S41 and S42 until enough sample data is collected;

[0023] Step S5, putting the time information and phase information into the neural network for training;

[0024] Step S51, using the time information collected in S3 as the input value of the neural network;

[0025] Step S52, using the phase information of the marking point calculated in S4 as the output value of the neural network;

[0026] Step S53: training the neural network and deploying the trained neural network in the computing module;

[0027] Step S6: When the structured light scanner is working, the phase shift information of the current frame can be predicted by using the deployed neural network by collecting the moment when the MEMS galvanometer reaches the monitoring position, thereby completing the phase correction;

[0028] Another aspect of the present invention is to protect a high-precision phase correction system, which includes a laser, a MEMS rotating mirror, a calibration plate, a high-resolution camera, an FPGA module, and a computing module:

[0029] The laser is used to emit laser light to the surface of the MEMS mirror, and the laser light is obtained by amplitude modulation according to a preset modulation signal;

[0030] The MEMS mirror receives the amplitude modulated signal of the laser, and projects the signal to the target area after passing through the DOE device and the grating. The optional MEMS can be selected to rotate from top to bottom, from bottom to top, from left to right, and from right to left. After the amplitude modulated laser signal passes through the cylindrical lens or DOE, then the grating, and finally is projected to the target area, a striped light spot can be formed.

[0031] The MEMS mirror position monitoring device mainly includes: a hole information indicating the current rotation position of the mirror, a plurality of pairs of photodetectors, the photodetector signals are connected to the FPGA, and the TDC generated by the FPGA or a dedicated TDC chip is optionally used to receive the position signal.

[0032] The computing module is used to receive the collected time information, deploy the trained neural network, and predict the phase shift information.

[0033] Due to the above technical solution, the present invention has the following beneficial effects:

[0034] The high-precision MEMS phase correction method of the present invention uses the phase offset value calculated by the calibration plate through deep learning to predict the relationship between the deflection of the rotating mirror and the phase offset value, which can effectively improve the calculation of MEMS phase accuracy, thereby improving the accuracy of the MEMS-based monocular structured light camera, and at the same time can effectively reduce the volume compared to the binocular structured light camera. Secondly, since the brain-like chip is used as the computing unit of the neural network, the cost of the neural network and the system power consumption are effectively reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments or prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 Flow chart of the high-precision MEMS rotating mirror mechanical phase correction method based on deep learning. The overall architecture includes calibration board, MEMS, laser, FPGA, camera and other equipment.

[0037] Figure 2 Schematic diagram of the MEMS mirror rotation. The mirror options include left to right and right to left. The random positions of the mirror include A1 to A2 at the A end and B1 to B2 at the B end.

[0038] Figure 3 Schematic diagram of the projection device in a MEMS-based structured light camera, where the structured light will be projected onto a checkerboard.

[0039] Figure 4 The neural network deployment process based on deep learning first collects data samples, then trains the neural network, and finally performs phase correction calculations based on the neural network data.

[0040] Figure 5 Schematic diagram of a deep learning neural network, including the neural network process from left to right and from right to left.

[0041] Figure 6 The timing diagram of the brain-like neural network based on FPGA uses a high-precision time digital change technology to accurately measure the time information of the galvanometer passing through the P1, P2, and P3 positions.

[0042] Figure 7 Flowchart of high-precision phase correction method based on MEMS. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0044] The technical solution provided by the embodiment of the present invention is applicable to any phase correction scenario, especially the phase correction scenario with mechanical moving parts, such as the high-precision scenario of a MEMS-based monocular camera. In this scenario, since the starting position of the MEMS rotating mirror is random at each rotation, it is necessary to accurately predict the phase offset value at the starting moment.

[0045] like Figure 4 As shown, the workflow of the high-precision MEMS phase correction method of this embodiment includes four steps: sample data collection, neural network training, neural network deployment, and phase correction.

[0046] The sample data acquisition mainly includes: system initialization, acquisition of the moment when the MEMS mirror reaches three or more different positions when rotating from left to right, or acquisition of the moment when the MEMS mirror reaches three or more different positions when rotating from right to left, the amplitude modulated laser signal is reflected by the MEMS and projected onto the calibration board, and during the rotation of the MEMS mirror from left to right or from right to left, the camera synchronously exposes and acquires the laser signal returned from the calibration board, and based on the acquired image information, uses OpenCV to find the position of the marking point on the calibration board on the pixel, and records the phase information of this marking point. Repeat the acquisition until enough data samples are collected.

[0047] The collected time information and calculated phase information are respectively used as the input and output of the neural network and sent to the neural network for training. The optional neural network can have three or more layers. The number of layers of the neural network and the selection of neural nodes in each layer are based on the evaluation function of the neural network, so as to select a better neural network shape.

[0048] During the rotation of MEMS, the neural network does not need to be called. Optionally, it is called once after the MEMS rotates from left to right or from right to left. Optionally, a brain-like pulse neural network chip is used as the implementation method of the neural network to encode the time information into the input signal required by the brain-like chip, thereby utilizing the low power consumption and low cost of the neural network chip to complete the deployment of the neural network.

[0049] After completing the calibration and deployment of the neural network, during normal operation, the phase offset information of the current MEMS mirror can be predicted through the neural network based on the collected time information, thereby achieving high-precision correction of the MEMS mirror phase.

[0050] like Figure 1 As shown, the high-precision MEMS rotating mirror phase correction system includes a laser, an FPGA, a MEMS rotating mirror module, a calibration board, and a camera, wherein the laser is used to emit a single-frequency continuous point laser signal and modulate it, and the MEMS module reflects the received point laser signal, passes through a cylindrical lens or a DOE device to form a line laser, and then passes through a grating to project it onto the surface of the object to be measured (calibration board). The calibration board can be a chessboard or a dot, with at least one marking point information, and optionally, the calibration board is deployed in the central area of ​​the camera field of view. The resolution of the selected camera can be higher than the resolution during actual work, and the actual work can optionally require a 1080P camera, so a 4K or 8K camera can be selected during the calibration process. The FPGA mainly completes the switch control of the laser, the drive control of the MEMS, the drive control of the camera, and the timing synchronization, time acquisition, image acquisition and other functions. The optional time acquisition of the MEMS rotating mirror position can select a dedicated TDC ASIC chip, or it can select the TDC function implemented by FPGA to collect time information.

[0051] like Figure 2 As shown, ∠MON represents the mechanical angle range of MEMS, and ∠P1OP2 represents the effective optical angle range. Since the MEMS rotating mirror is a mechanical rotating device, the swing of P1 position is between A1A2, and the swing of P2 is between B1B2. In this way, when the same stripe light spot emitted by the MEMS rotating mirror is projected on the target detection area, the completely consistent images before and after cannot overlap 100%. In this way, accurate phase resolution cannot be performed using methods based on phase shift code or Gray code.

[0052] like Figure 2 As shown, this embodiment schematically shows the six position information of tl1, tl2, tl3, tr1, tr2, and tr3 collected when the MEMS mirror rotates from left to right and from right to left through three different positions P1, P2, and P3, among which the three moments tl1, tl2, and tl3 are independent of the three moments tr1, tr2, and tr3.

[0053] like Figure 3 As shown, this embodiment schematically shows the image information of the stripe light spot projected onto the calibration plate after passing through the MEMS module and captured by the camera, thereby calculating the corner point position information and phase information on the calibration plate.

[0054] like Figure 5 As shown, this embodiment schematically shows the optional neural network shape. Optionally, the neural network shape of the MEMS mirror rotating from left to right and the neural network of the MEMS mirror rotating from right to left are independent of each other, and the neural network shapes of the two can be selected to be the same or different. Optionally, the two neural networks can also be merged into the same neural network, and the merged input is 6 time signals and 2 output signals. Optionally, the number of input parameters of the neural network can be 3 or more, and the corresponding input signals merged into the same neural network are 6 or more.

Claims

1. A high-precision MEMS galvanometer phase correction method, characterized in that: The high-precision MEMS galvanometer phase correction method comprises: a) Initialize the system, configure the working parameters of the laser, MEMS, and camera, and write the register list; b) Confirm the rotation direction of the MEMS. When the MEMS reaches the starting position, the FPGA starts the camera. The synchronized FPGA controls the laser and modulates its light-emitting signal. The MEMS projects the laser onto the calibration board, and the camera collects the signal returned from the calibration board. i. Step S21, select three points during the rotation of the MEMS galvanometer to monitor the MEMS rotation position and direction, the selected three points are represented by P1, P2, and P3 respectively, P1 is optionally close to the left position, P3 is optionally close to the right position, P2 is optionally close to P1 or P2 is optionally close to P3, and P1, P2, and P3 are respectively connected to three interfaces of the FPGA, the order of the signal pulses is P1, P2, P3, which means rotation from left to right, and the order of the signal pulses is P3, P2, P1, which means rotation from right to left. ii. Step S22, when the MEMS rotates from left to right, when the rotating mirror reaches position P1, the laser is started and the camera is started at the same time; when the rotating mirror reaches position P3, the laser is turned off and the camera is turned off at the same time; iii. Step S23, when the MEMS rotates from right to left, when the rotating mirror reaches position P3, the laser is started and the camera is started at the same time; when the rotating mirror reaches position P1, the laser is turned off and the camera is turned off at the same time; iv. Step S24, the laser is modulated by sinusoidal modulation, I = A + B * cos (2 * π * x p / w), where A and B represent the background light intensity and modulation intensity respectively, x p represents the pixel serial number, w represents the resolution; c) Step S3, using a high-precision TDC to record the moment when the MEMS rotates to the monitoring position; i. Step S31, when the MEMS mirror rotates from left to right, the time points of P1, P2, and P3 collected by the TDC chip or FPGA are tl1, tl2, and tl3 respectively; ii. Step S32: When the MEMS mirror rotates from right to left, the TDC chip or FPGA is used to collect the time points of P3, P2, and P1, which are tr1, tr2, and tr3 respectively. iii. Step S33, tl1==0; d) Step S4, performing phase analysis on the image captured by the camera to obtain the phase information of the marking point on the calibration plate; i. Step S41, obtaining the pixel position information of the corner points on the chessboard on the captured image by Zhang Zhengyou calibration method; ii. Step S42, determining the phase information at the position of the marking point by collecting the light intensity information; iii. Step S43, repeat S41 and S42 until sufficient sample data is collected; e) Step S5, putting the time information and phase information into the neural network for training; i. Step S51, using the time information collected in S3 as the input value of the neural network; ii. Step S52, the phase information of the marking point calculated in S4 is used as the output value of the neural network; iii. Step S53, training the neural network and deploying the trained neural network in the computing module; f) Step S6: When the structured light scanner is working, by collecting the moment when the MEMS galvanometer reaches the monitoring position, the deployed neural network can be used to predict the phase offset information of the current frame to complete the phase correction.

2. A high-precision MEMS mirror phase correction device, the characteristics of which include: a) Another aspect of the present invention is to protect a high-precision phase correction system, which includes a laser, a MEMS rotating mirror, a calibration plate, a high-resolution camera, an FPGA module, and a computing module: b) The laser is used to emit laser light to the surface of the MEMS mirror, and the laser light is obtained by amplitude modulation according to a preset modulation signal; c) The MEMS mirror receives the amplitude modulated signal of the laser, and projects the signal to the target area after passing through the DOE device and the grating. The optional MEMS can choose to rotate from top to bottom, from bottom to top, from left to right, and from right to left. After the amplitude modulated laser signal passes through the cylindrical lens or DOE, Then it goes to the grating and finally is projected onto the target area to form a striped light spot. d) The MEMS mirror position monitoring device mainly includes: a hole information indicating the current rotation position of the mirror, a plurality of pairs of photodetectors, the photodetector signals are connected to the FPGA, and the TDC generated by the FPGA or a dedicated TDC chip is optionally used to receive the position signal e) The computing module is used to receive the collected time information, deploy the trained neural network, and predict the phase shift information.