Calibration method and system of structured light projector, and medium

Abstract

The invention provides a structured light projector calibration method and system and a medium, and the method comprises the steps: projecting a multi-frequency stripe pattern to a target region based on an MEMS micromirror array module, obtaining a distorted image sequence, and carrying out the phase decoding, and obtaining a phase diagram corresponding to each frequency; resolving the phase diagram based on a phase shift method and an optical path triangulation principle to obtain a preliminary three-dimensional point coordinate; constructing a neural network model and a calibration sample set, obtaining a real three-dimensional coordinate by using a standard calibration plate or a known curved surface, and calculating the deviation between the initial three-dimensional point coordinate and the real three-dimensional coordinate to obtain a training sample set; and training the neural network model according to a training sample set based on an Euclidean distance loss function, outputting a three-dimensional offset compensation amount according to the neural network model, correcting the initial three-dimensional point coordinates, and realizing an end-to-end error self-compensation mechanism through the neural network model, so that deviation compensation is effectively performed on the three-dimensional point coordinates, and the accuracy of the three-dimensional point coordinates is improved. And the three-dimensional reconstruction precision is improved.

Description

Technical Field

[0001] This application relates to the field of parameter calibration technology, and more specifically, to a calibration method, system, and medium for a structured light projector. Background Technology

[0002] In the field of 3D vision inspection and measurement, structured light projectors are widely used in industrial quality control, automated manufacturing, and medical image reconstruction due to their high precision and non-contact characteristics. Structured light 3D reconstruction technology projects a known pattern onto a target surface, and then, based on the distortion of the pattern on the object's surface and combined with images captured by a camera, recovers the 3D information through decoding and reconstruction algorithms. Traditional structured light systems mostly rely on fixed grids or DLP projectors, which have slow update speeds, limited spatial resolution and distortion control capabilities, and significant limitations in projection accuracy and distortion correction when dealing with complex curved surfaces or miniature components.

[0003] With the increasing demand for microscopic precision measurement and real-time 3D surface modeling, existing structured light systems face the following technical bottlenecks:

[0004] Limited pattern generation: Traditional projectors mainly rely on DMD or LCD to generate stripe patterns. The patterns are static and the refresh rate is low, making it impossible to flexibly generate high-frequency or adaptive patterns.

[0005] System distortion modeling is difficult: Optical nonlinear distortion caused by factors such as lens nonlinear distortion and non-uniform response of MEMS devices is difficult to model effectively using traditional polynomial fitting or Zhang calibration method.

[0006] Unable to correct errors in real time: Due to the offline nature of the modeling and calibration process, the system cannot achieve real-time distortion compensation and high-precision dynamic reconstruction during operation;

[0007] High-frequency structure decoding error is large: striped patterns undergo complex deformation after being projected onto curved surfaces, and traditional phase decoding algorithms have poor stability under high-frequency patterns.

[0008] Against this backdrop, there is an urgent need for a novel structured light system with higher spatial frequency control capability, higher projection accuracy, and stronger distortion adaptive capability, coupled with a deep learning-driven real-time calibration algorithm to achieve adaptive compensation for nonlinear errors. Summary of the Invention

[0009] The purpose of this application is to provide a calibration method, system, and medium for a structured light projector. By implementing an end-to-end error self-compensation mechanism through a neural network model, the deviation of three-dimensional point coordinates can be effectively compensated, thereby improving the accuracy of three-dimensional reconstruction.

[0010] This application also provides a calibration method for a structured light projector, including:

[0011] Multi-frequency stripe patterns are projected onto a target area using a MEMS micromirror array module, and image sequences of the multi-frequency stripe patterns after distortion on the target surface are acquired using a camera.

[0012] Phase decoding is performed on the image sequence of the multi-frequency stripe pattern after distortion on the target surface to obtain the phase map corresponding to each frequency.

[0013] Based on the phase shift method and the principle of optical path triangulation, the phase diagram corresponding to each frequency is calculated to obtain the preliminary three-dimensional point coordinates;

[0014] Construct a neural network model and a calibration sample set. Use a standard calibration plate or a known surface to obtain the true 3D coordinates. Calculate the deviation between the preliminary 3D point coordinates and the true 3D coordinates to obtain the training sample set.

[0015] The neural network model is trained based on the Euclidean distance loss function and the training sample set. The initial three-dimensional point coordinates are input into the trained neural network model, and the three-dimensional offset compensation is output. The initial three-dimensional point coordinates are corrected based on the three-dimensional offset compensation to obtain the final three-dimensional coordinates.

[0016] Optionally, in the calibration method for the structured light projector described in the embodiments of this application, the multi-frequency fringe pattern generation method is as follows:

[0017] Assume the MEMS micromirror array is a two-dimensional array M(x,y)∈{0,1}, where each mirror can be independently controlled for deflection angle, simulating a sinusoidal fringe function:

[0018] I p (x,y,t)=A·[1+cos(2πf x x+2πf y y+φ(t))]

[0019] in:

[0020] A: Light intensity amplitude;

[0021] f x ,f y : Horizontal and vertical stripe frequencies;

[0022] φ(t): Phase that changes with time;

[0023] The voltage control vector of the micromirror array is V(t), which satisfies:

[0024] M(x,y,t)=H[cos(2πf x x+2πf y y+φ(t))-θ(x,y,t)];

[0025] The multi-frequency stripe pattern is fused based on a multi-frequency combination strategy, as follows:

[0026] Using different frequencies f1, f2, ..., f n The stripes are projected sequentially, and the phase diagram at each frequency is recorded:

[0027] φ i (u,v)=unwrap(arg(I i (u,v)))

[0028] The CRT uses the remainder theorem to synthesize multi-frequency phase signals, as shown in the following formula:

[0029]

[0030] Where w i These are the weighting coefficients corresponding to the frequencies;

[0031] Where H[·] is the Heaviside step function and θ(x,y,t) is the voltage bias modulation function, used to simulate amplitude or phase modulation.

[0032] It should be noted that the advantages analysis is as follows:

[0033] Each mirror state can achieve a refresh rate of up to 10kHz, which is significantly better than DMD;

[0034] The fringe frequency can be programmed in real time to adapt to the complexity of the surface.

[0035] High energy concentration, suitable for micro-surface detection;

[0036] The pattern control system can be constructed using the following vector control function:

[0037]

[0038] Where a ij (t)∈{0,1} represents the projection switch, forming an adjustable grayscale approximation.

[0039] Optionally, in the calibration method for the structured light projector described in the embodiments of this application, the phase decoding method based on the phase shift method and the optical path triangulation measurement principle is as follows:

[0040] Assuming the projector and camera are at a certain parallax angle, and the pattern is displaced on the target surface, the position (X,Y,Z) of point P in the world coordinate system can be calculated from the phase φ(u,v) of pixel point (u,v):

[0041]

[0042] φ(u,v) represents the phase value decoded at pixel (u,v), x(u,v) represents the horizontal coordinate of pixel (u,v) in the projected image, and f x φ0 represents the spatial frequency of the stripe pattern in the horizontal or x-direction, and φ0 represents the initial phase offset or reference phase value.

[0043] The reconstructed 3D coordinates are calculated by triangulation of the projection model and the camera model:

[0044]

[0045] Triangulate represents the geometric intersection process of inferring world coordinates from image coordinates and phase information.

[0046] Optionally, in the calibration method for the structured light projector described in this application embodiment, the neural network model construction method is as follows:

[0047] Define the 3D coordinates of each pixel (u,v) obtained in the original 3D reconstruction as follows:

[0048]

[0049] X raw (u,v): Represents the original 3D coordinate vector obtained by preliminary 3D reconstruction at pixel (u,v). It is a 3D column vector.

[0050] X(u,v): The X coordinate (usually horizontal) in three-dimensional space corresponding to this pixel.

[0051] Y(u,v): The Y coordinate in three-dimensional space corresponding to this pixel (usually in the vertical direction).

[0052] Z(u,v): The Z coordinate (depth direction) in the three-dimensional space corresponding to this pixel.

[0053] This represents the transpose of a vector, meaning the three-dimensional coordinates are represented as a column vector.

[0054] The input features of the neural network are:

[0055]

[0056] Wherein: F input It is a multimodal feature vector that integrates spatial location, image brightness, edge information, and phase structure to help neural networks learn more accurately and compensate for systematic errors in 3D reconstruction.

[0057] I(u,v): pixel grayscale;

[0058] Image gradient reflects edge intensity;

[0059] φ(u,v): Corresponding phase value, reflecting the local fringe structure;

[0060] The network output is a 3D offset, as shown in the following formula:

[0061]

[0062] The actual calibration point location is X gt The loss function is Euclidean distance loss, and the formula for the loss function is as follows:

[0063]

[0064] This represents the loss function used to train the neural network.

[0065] f θ (F input () represents a trained neural network model used to predict bias;

[0066] X raw This represents the initial measured 3D coordinates; after training, the model is embedded into an online system for real-time correction, as shown in the following formula:

[0067] X corrected =X raw +f θ (F input ).

[0068] Optionally, in the calibration method for the structured light projector described in this application embodiment, the three-dimensional offset compensation amount is analyzed as follows:

[0069] We introduce a neural network model for error regression, as shown in the following formula:

[0070]

[0071] in:

[0072] f θ : Neural network, with parameter θ;

[0073] The input includes image coordinates (u,v), phase φ (u,v), and local gray level I (u,v);

[0074] The output is a three-dimensional offset compensation ΔX;

[0075] The neural network structure uses a residual encoder structure:

[0076] f θ =MLP(ResNet(I(u,v),φ,u,v)).

[0077] Optionally, in the calibration method for the structured light projector described in this application embodiment, the neural network model training method is as follows:

[0078] Use a standard calibration board (such as a black and white checkerboard) to perform preliminary distortion fitting;

[0079] Obtaining the true offset based on MEMS projection structured light patterns;

[0080] Record each pixel and deviation ΔX;

[0081] Constructing the training sample set:

[0082]

[0083] Training a deep regression network using MSE loss:

[0084]

[0085] Secondly, embodiments of this application provide a calibration system for a structured light projector. The system includes a memory and a processor. The memory includes a program for a calibration method of the structured light projector. When the program for the calibration method of the structured light projector is executed by the processor, it performs the following steps:

[0086] Multi-frequency stripe patterns are projected onto a target area using a MEMS micromirror array module, and image sequences of the multi-frequency stripe patterns after distortion on the target surface are acquired using a camera.

[0087] Phase decoding is performed on the image sequence of the multi-frequency stripe pattern after distortion on the target surface to obtain the phase map corresponding to each frequency.

[0088] Based on the phase shift method and the principle of optical path triangulation, the phase diagram corresponding to each frequency is calculated to obtain the preliminary three-dimensional point coordinates;

[0089] Construct a neural network model and a calibration sample set. Use a standard calibration plate or a known surface to obtain the true 3D coordinates. Calculate the deviation between the preliminary 3D point coordinates and the true 3D coordinates to obtain the training sample set.

[0090] The neural network model is trained based on the Euclidean distance loss function and the training sample set. The initial three-dimensional point coordinates are input into the trained neural network model, and the three-dimensional offset compensation is output. The initial three-dimensional point coordinates are corrected based on the three-dimensional offset compensation to obtain the final three-dimensional coordinates.

[0091] Optionally, in the calibration system for the structured light projector described in this application embodiment, the method for generating the multi-frequency fringe pattern is as follows:

[0092] Assume the MEMS micromirror array is a two-dimensional array M(x,y)∈{0,1}, where each mirror can be independently controlled for deflection angle, simulating a sinusoidal fringe function:

[0093] I p (x,y,t)=A·[1+cos(2πf x x+2πf y y+φ(t))]

[0094] in:

[0095] A: Light intensity amplitude;

[0096] f x ,f y : Horizontal and vertical stripe frequencies;

[0097] φ(t): Phase that changes with time;

[0098] The voltage control vector of the micromirror array is V(t), which satisfies:

[0099] M(x,y,t)=H[cos(2πf x x+2πf y y+φ(t))-θ(x,y,t)];

[0100] The multi-frequency stripe pattern is fused based on a multi-frequency combination strategy, as follows:

[0101] Using different frequencies f1, f2, ..., f n The stripes are projected sequentially, and the phase diagram at each frequency is recorded:

[0102] φ i (u,v)=unwrap(arg(I i (u,v)))

[0103] The CRT uses the remainder theorem to synthesize multi-frequency phase signals, as shown in the following formula:

[0104]

[0105] M(x,y,t) represents the control of the micromirror state to generate structured light patterns;

[0106] φ(t) is the phase that changes with time and is used for phase shifting;

[0107] φ i (u,v) is the unwrapped phase at a single frequency;

[0108] unwrap is the operation to remove phase transitions;

[0109] φ global(u,v) is the global phase after multi-frequency fusion, used for high-precision 3D reconstruction;

[0110] Where w i These are the weighting coefficients corresponding to the frequencies;

[0111] Where H[·] is the Heaviside step function and θ(x,y,t) is the voltage bias modulation function, used to simulate amplitude or phase modulation.

[0112] Optionally, in the calibration system for the structured light projector described in this application embodiment, the phase decoding method based on the phase shift method and the optical path triangulation measurement principle is as follows:

[0113] Assuming the projector and camera are at a certain parallax angle, and the pattern is displaced on the target surface, the position (X,Y,Z) of point P in the world coordinate system can be calculated from the phase φ(u,v) of pixel point (u,v):

[0114]

[0115] φ(u,v) represents the phase value decoded at pixel (u,v), x(u,v) represents the horizontal coordinate of pixel (u,v) in the projected image, and f x φ0 represents the spatial frequency of the stripe pattern in the horizontal or x-direction, and φ0 represents the initial phase offset or reference phase value.

[0116] The reconstructed 3D coordinates are calculated by triangulation of the projection model and the camera model:

[0117]

[0118] Triangulate represents the geometric intersection process of inferring world coordinates from image coordinates and phase information.

[0119] Thirdly, embodiments of this application also provide a computer-readable storage medium, which includes a calibration method program for a structured light projector. When the calibration method program for the structured light projector is executed by a processor, it implements the steps of the calibration method for the structured light projector as described in any of the preceding claims.

[0120] As described above, the calibration method, system, and medium for a structured light projector provided in this application involves projecting a multi-frequency fringe pattern onto a target area using a MEMS micromirror array module, acquiring an image sequence of the distorted multi-frequency fringe pattern on the target surface using a camera, performing phase decoding on the distorted image sequence to obtain a phase map corresponding to each frequency, calculating the phase map corresponding to each frequency based on the phase shift method and optical path triangulation principle to obtain preliminary three-dimensional point coordinates, constructing a neural network model and a calibration sample set, obtaining the true three-dimensional coordinates using a standard calibration plate or a known surface, calculating the deviation between the preliminary three-dimensional point coordinates and the true three-dimensional coordinates to obtain a training sample set, training the neural network model based on the Euclidean distance loss function using the training sample set, inputting the preliminary three-dimensional point coordinates into the trained neural network model, outputting a three-dimensional offset compensation amount, correcting the preliminary three-dimensional point coordinates based on the three-dimensional offset compensation amount to obtain the final three-dimensional coordinates, and realizing an end-to-end error self-compensation mechanism through the neural network model, thereby effectively compensating for the deviation of the three-dimensional point coordinates and improving the accuracy of three-dimensional reconstruction. Attached Figure Description

[0121] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0122] Figure 1 A flowchart illustrating the calibration method for a structured light projector provided in this application embodiment;

[0123] Figure 2 A schematic diagram illustrating the MEMS dynamic stripe pattern generation principle of the calibration method for the structured light projector provided in this application embodiment;

[0124] Figure 3 A schematic diagram of triangulation between structured light projection and camera viewpoint, which is a calibration method for a structured light projector provided in an embodiment of this application.

[0125] Figure 4 A schematic diagram of a multi-frequency stripe structured light pattern for a calibration method of a structured light projector provided in an embodiment of this application;

[0126] Figure 5 A flowchart of neural network error correction for the calibration method of the structured light projector provided in this application embodiment;

[0127] Figure 6 Point cloud comparison before and after error correction for the calibration method of the structured light projector provided in the embodiments of this application;

[0128] Figure 7 A schematic diagram of the three-dimensional reconstruction of a micro-part of the calibration method for a structured light projector provided in this application embodiment;

[0129] Figure 8 A block diagram of a calibration system for a structured light projector provided in an embodiment of this application. Detailed Implementation

[0130] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0131] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0132] Please refer to Figures 1-7 As shown, Figure 2 In this context, "static sinusoidal fringes" represents static sinusoidal fringes; "dynamic fringes" represents dynamic fringes; and "dynamic fringes micromirror array" represents a dynamic fringes micromirror array. Figure 5 In this context, "neural network" refers to a neural network; "correction output" refers to the correction output; and "3Dcoordinates" refers to three-dimensional coordinates. Figure 6 In the text, "uncorrected point cloud" represents the point cloud that has not been corrected; "corrected point cloud" represents the point cloud that has been corrected.

[0133] The calibration method for this structured light projector is used in a terminal device. The calibration method includes the following steps:

[0134] S101, based on the MEMS micromirror array module, a multi-frequency stripe pattern is projected onto the target area, and the image sequence of the multi-frequency stripe pattern after distortion on the target surface is obtained based on the camera;

[0135] S102, perform phase decoding on the image sequence of the multi-frequency stripe pattern after distortion on the target surface to obtain the phase map corresponding to each frequency;

[0136] S103, based on the phase shift method and the principle of optical path triangulation, the phase diagram corresponding to each frequency is calculated to obtain the preliminary three-dimensional point coordinates;

[0137] S104, Construct a neural network model and calibration sample set, obtain the real three-dimensional coordinates using a standard calibration plate or a known surface, calculate the deviation between the preliminary three-dimensional point coordinates and the real three-dimensional coordinates, and obtain the training sample set;

[0138] S105 trains the neural network model based on the Euclidean distance loss function and the training sample set. The initial three-dimensional point coordinates are input into the trained neural network model, and the three-dimensional offset compensation is output. The initial three-dimensional point coordinates are corrected based on the three-dimensional offset compensation to obtain the final three-dimensional coordinates.

[0139] According to an embodiment of the present invention, the method for generating a multi-frequency stripe pattern is as follows:

[0140] Assume the MEMS micromirror array is a two-dimensional array M(x,y)∈{0,1}, where each mirror can be independently controlled for deflection angle, simulating a sinusoidal fringe function:

[0141] I p (x,y,t)=A·[1+cos(2πf x x+2πf y y+φ(t))]

[0142] in:

[0143] A: Light intensity amplitude;

[0144] f x ,f y : Horizontal and vertical stripe frequencies;

[0145] φ(t): Phase that changes with time;

[0146] The voltage control vector of the micromirror array is V(t), which satisfies:

[0147] M(x,y,t)=H[cos(2πf x x+2πf y y+φ(t))-θ(x,y,t)];

[0148] The multi-frequency stripe pattern is fused based on a multi-frequency combination strategy, as follows:

[0149] Using different frequencies f1, f2, ..., f n The stripes are projected sequentially, and the phase diagram at each frequency is recorded:

[0150] φ i (u,v)=unwrap(arg(I i (u,v)))

[0151] The CRT uses the remainder theorem to synthesize multi-frequency phase signals, as shown in the following formula:

[0152]

[0153] Where w i These are the weighting coefficients corresponding to the frequencies;

[0154] Where H[·] is the Heaviside step function and θ(x,y,t) is the voltage bias modulation function, used to simulate amplitude or phase modulation.

[0155] According to an embodiment of the present invention, the phase decoding method based on the phase shift method and the optical path triangulation measurement principle is as follows:

[0156] Assuming the projector and camera are at a certain parallax angle, and the pattern is displaced on the target surface, the position (X,Y,Z) of point P in the world coordinate system can be calculated from the phase φ(u,v) of pixel point (u,v):

[0157]

[0158] φ(u,v) represents the phase value decoded at pixel (u,v), x(u,v) represents the horizontal coordinate of pixel (u,v) in the projected image, and f x φ0 represents the spatial frequency of the stripe pattern in the horizontal or x-direction, and φ0 represents the initial phase offset or reference phase value.

[0159] The reconstructed 3D coordinates are calculated by triangulation of the projection model and the camera model:

[0160]

[0161] Triangulate represents the geometric intersection process of inferring world coordinates from image coordinates and phase information.

[0162] According to an embodiment of the present invention, the method for constructing a neural network model is as follows:

[0163] Define the 3D coordinates of each pixel (u,v) obtained in the original 3D reconstruction as follows:

[0164]

[0165] X raw (u,v): Represents the original 3D coordinate vector obtained by preliminary 3D reconstruction at pixel (u,v). It is a 3D column vector.

[0166] X(u,v): The X coordinate (usually horizontal) in three-dimensional space corresponding to this pixel.

[0167] Y(u,v): The Y coordinate in three-dimensional space corresponding to this pixel (usually in the vertical direction).

[0168] Z(u,v): The Z coordinate (depth direction) in the three-dimensional space corresponding to this pixel.

[0169] This represents the transpose of a vector, meaning the three-dimensional coordinates are represented as a column vector.

[0170] The input features of the neural network are:

[0171]

[0172] Wherein: F input It is a multimodal feature vector that integrates spatial location, image brightness, edge information, and phase structure to help neural networks learn more accurately and compensate for systematic errors in 3D reconstruction.

[0173] I(u,v): pixel grayscale;

[0174] Image gradient reflects edge intensity;

[0175] φ(u,v): Corresponding phase value, reflecting the local fringe structure;

[0176] The network output is a 3D offset, as shown in the following formula:

[0177]

[0178] The actual calibration point location is X gt The loss function is Euclidean distance loss, and the formula for the loss function is as follows:

[0179]

[0180] This represents the loss function used to train the neural network.

[0181] f θ (F input () represents a trained neural network model used to predict bias;

[0182] X raw This represents the initial measured 3D coordinates; after training, the model is embedded into an online system for real-time correction, as shown in the following formula:

[0183] X corrected =X raw +f θ (F input )X corrected This represents the final three-dimensional coordinates after correction by the neural network.

[0184] According to an embodiment of the present invention, the three-dimensional offset compensation amount is analyzed as follows:

[0185] We introduce a neural network model for error regression, as shown in the following formula:

[0186]

[0187] in:

[0188] f θ : Neural network, with parameter θ;

[0189] The input includes image coordinates (u,v), phase φ (u,v), and local gray level I (u,v);

[0190] The output is a three-dimensional offset compensation ΔX;

[0191] The neural network structure uses a residual encoder structure:

[0192] f θ =MLP(ResNet(I(u,v),φ,u,v)).

[0193] MLP is a basic feedforward neural network consisting of multiple fully connected (dense layers), each containing multiple neurons, with non-linear transformations between layers via activation functions;

[0194] ResNet is a deep convolutional neural network whose core idea is to introduce "residual blocks" to solve the gradient vanishing and degradation problems in deep network training through skip connections.

[0195] According to an embodiment of the present invention, the neural network model training method is as follows:

[0196] Use a standard calibration board (such as a black and white checkerboard) to perform preliminary distortion fitting;

[0197] Obtaining the true offset based on MEMS projection structured light patterns;

[0198] Record each pixel and deviation ΔX;

[0199] Constructing the training sample set:

[0200]

[0201] Training a deep regression network using MSE loss:

[0202]

[0203] According to embodiments of the present invention, the method further includes: using multi-frequency stripe projection combination to enhance decoding accuracy, as detailed below:

[0204] To avoid decoding ambiguity issues caused by occlusion or interference in the edge regions of high-frequency structured light patterns, this invention introduces the following stripe frequency group:

[0205] f i =f0·α i i = 0, 1, ..., n

[0206] Where α>1 represents the frequency growth rate, and f0 represents the lowest frequency fringe. The phase is calculated for each frequency:

[0207] φ i (u,v)=arg[I i (u,v)]

[0208] By combining the phases corresponding to all frequencies using the remainder theorem, we obtain the composite phase:

[0209]

[0210] The algorithm is equivalent to high-precision phase fusion reconstruction and has the following advantages:

[0211] Strong anti-interference: Low-frequency stripes can still be decoded even when there is uneven lighting in a local area;

[0212] Improved accuracy: A surface accuracy of <10μm can be achieved by doubling the frequency;

[0213] Highly adaptive: Automatically selects frequency combinations based on the target curvature.

[0214] This invention also includes: an end-to-end self-calibration mechanism adapted to microscale size measurement, as detailed below:

[0215] This application supports a self-calibration process that requires no manual intervention, and is suitable for flexible configurations with different installation angles and optical path parameters. The overall process includes:

[0216] Projected multi-frequency stripe pattern;

[0217] Capture image I(u,v) and calculate preliminary 3D points;

[0218] Use a calibration plate or a known surface to mark the error;

[0219] Neural network training and deployment;

[0220] Automatically corrects during real-time operation.

[0221] Meanwhile, for the contour edge processing of tiny parts, this system introduces an edge-preserving filter and a gradient enhancement module:

[0222]

[0223] X final (u,v) represents the final three-dimensional coordinates after all corrections and filtering.

[0224] Bilateral filters are a type of nonlinear filtering method that can smooth noise while preserving edge information.

[0225] Where λ is the edge enhancement weighting factor. Experiments show that in the measurement tasks of metal micro-molds and electronic component packages, the edge size measurement error can be controlled within ±5μm.

[0226] Experimental verification and performance evaluation of this application:

[0227] To verify the effectiveness of the system in actual measurement tasks, the following three types of experiments were conducted:

[0228] Accuracy testing (planar / sloping plane reconstruction):

[0229] Reconstruction is performed on a standard planar target, and the evaluation metrics are as follows:

[0230] RMSE (Root Mean Square Error);

[0231] Max Error (maximum absolute error);

[0232] Fitting deviation (residual)

[0233] The results are shown in Table 1.

[0234] Table 1

[0235]

[0236] Micro-part measurement experiment:

[0237] Miniature screw caps and MEMS components were selected and their dimensions were measured within a 10mm × 10mm field of view. The results are as follows:

[0238] Maximum error: ±4.8μm;

[0239] The smoothness of the contour edges is significantly improved;

[0240] Compared to the reconstruction results from the white light interferometer, the error remains within ±0.2%.

[0241] Surface reconstruction capability test:

[0242] The structured light reconstruction capability was tested in a highly nonlinear region using samples such as hemispherical surfaces, spiral surfaces, and freeform surfaces.

[0243] Curvature retention improved by 62.1%;

[0244] Reconstruction can be achieved on the surface of mirror materials;

[0245] The system frame rate is maintained at 55fps during real-time reconstruction.

[0246] The application prospects and industry expansion are as follows:

[0247] This invention is not limited to scientific research or laboratory use; it possesses strong commercialization potential and industry relevance in the following application areas:

[0248] 3D Surface Inspection and Defect Identification:

[0249] Applications in automotive manufacturing include headlights and freeform surfaces for car bodies;

[0250] Identification of minute defects such as welds, indentations, and cracks.

[0251] 3D Dimension Measurement of Micro-parts:

[0252] Suitable for rapid inspection of chip packages, solder joints, miniature screws, etc.

[0253] It can replace expensive confocal and interferometric microscopy systems.

[0254] Online dynamic 3D modeling:

[0255] Implement simultaneous measurement and compensation in assembly line scenarios;

[0256] Suitable for high-frequency factory testing and high-speed defect screening.

[0257] Reverse modeling and CNC feedback:

[0258] Supports real-time acquisition of workpiece surface models and reverse correction of CNC commands;

[0259] Improve manufacturing consistency and process iteration efficiency.

[0260] In summary, this invention achieves programmable, high-frequency, and high-precision dynamic stripe structured light projection by introducing a MEMS micromirror array. At the same time, it introduces deep learning methods to model and correct nonlinear distortion errors in real time, breaking through the accuracy bottleneck of traditional structured light systems in complex surface reconstruction and microscale measurement.

[0261] This application can achieve the following technical effects:

[0262] Dynamic, adjustable pattern generation: arbitrary frequency and phase modulation are achieved through a micromirror array;

[0263] Deep learning calibration: It can learn multi-factor distortion and automatically correct 3D reconstruction errors;

[0264] Multi-frequency stripe fusion: enhances phase decoding stability and anti-interference capabilities;

[0265] End-to-end measurement process: supports automatic calibration, real-time reconstruction, and high-speed feedback;

[0266] Strong industrial application capability: adaptable to multiple scenarios such as online inspection, 3D modeling and precision measurement.

[0267] Experimental results demonstrate that this application outperforms traditional methods in tasks involving the measurement of micro-component dimensions and the reconstruction of freeform surfaces, exhibiting significantly lower errors. Its modular design of structure and algorithm also lays the foundation for subsequent research on intelligent vision systems such as multi-projector fusion, adaptive pattern generation, and scene self-optimization.

[0268] Dynamic patterns are constructed based on MEMS micromirror arrays. The high-speed driving characteristics of MEMS micromirrors are used to dynamically project structured light patterns. The pattern frequency, phase, and amplitude can be adjusted in real time to adapt to different curvature surfaces or measurement accuracy requirements.

[0269] A neural network-based algorithm for nonlinear system distortion modeling and real-time correction is proposed. This algorithm can learn the nonlinear deformation caused by various non-ideal factors in the optical path, including but not limited to lens distortion, micromirror deviation, and projection angle nonlinearity.

[0270] By combining a structured light 3D reconstruction model with a deep learning calibration network, an end-to-end error self-compensation mechanism is achieved, improving the system's adaptability to small parts and complex curved surfaces.

[0271] By constructing a multi-frequency fringe projection combination strategy, the ability to encode the detailed features of the target surface is effectively enhanced, thereby improving the accuracy of 3D reconstruction.

[0272] This invention can be applied not only to the three-dimensional measurement of tiny parts and the defect detection of complex freeform surfaces, but also to the ability of high frame rate dynamic modeling, providing infrastructure support for online quality inspection in the context of Industry 4.0.

[0273] like Figure 8 As shown in the figure, Controllplatform represents the control platform; MEMS micromirror array represents the MEMS micromirror array; Light source represents the light source; Projection optics represents the projection optics system; Domputionplatform should be interpreted as Computation platform; Target surface represents the target surface; and Computation platform represents the computing platform. Secondly, embodiments of this application provide a calibration system for a structured light projector. This system includes a memory and a processor. The memory includes a program for a calibration method of the structured light projector. When the program for the calibration method of the structured light projector is executed by the processor, it implements the following steps:

[0274] Multi-frequency stripe patterns are projected onto a target area using a MEMS micromirror array module, and image sequences of the multi-frequency stripe patterns after distortion on the target surface are acquired using a camera.

[0275] Phase decoding is performed on the image sequence of the multi-frequency stripe pattern after distortion on the target surface to obtain the phase map corresponding to each frequency.

[0276] Based on the phase shift method and the principle of optical path triangulation, the phase diagram corresponding to each frequency is calculated to obtain the preliminary three-dimensional point coordinates;

[0277] Construct a neural network model and a calibration sample set. Use a standard calibration plate or a known surface to obtain the true 3D coordinates. Calculate the deviation between the preliminary 3D point coordinates and the true 3D coordinates to obtain the training sample set.

[0278] The neural network model is trained based on the Euclidean distance loss function and the training sample set. The initial 3D point coordinates are input into the trained neural network model, and the 3D offset compensation is output. The initial 3D point coordinates are corrected based on the 3D offset compensation to obtain the final 3D coordinates.

[0279] It should be noted that the overall system is divided into five core modules, namely:

[0280] Pattern projection module: a projection system based on MEMS micromirror array;

[0281] Light source module: Uses a stable wavelength narrowband LED / laser with a center wavelength of λ = 532nm;

[0282] Imaging module: Uses an industrial-grade CMOS camera with a resolution of 2048×2048 and a maximum frame rate of 120fps;

[0283] Image acquisition and control module: Based on FPGA, image high-speed buffer and MEMS synchronous control are implemented;

[0284] Computing module: Uses a high-performance GPU platform (such as NVIDIA RTX series) for neural network inference and 3D reconstruction.

[0285] The layered structure of the software system is as follows:

[0286] The bottom driving layer controls the voltage matrix of the MEMS lens and is responsible for pattern timing generation.

[0287] Acquisition synchronization layer: Synchronizes MEMS and camera based on time stamp controller (TSC);

[0288] Image processing layer: includes operations such as multi-frequency phase decoding, image filtering, and gradient enhancement;

[0289] Calibration and neural network inference layer: performing error modeling and compensation;

[0290] Reconstruct the output layer: Perform tasks such as triangulation, point cloud generation, surface fitting, and size calculation.

[0291] The system operation process is as follows:

[0292] The control module sends a trigger signal;

[0293] MEMS dynamic projection pattern I p (x,y,t);

[0294] The camera captures an image I(u,v,t);

[0295] The image processing module decodes the phase φ(u,v);

[0296] The calculation module performs 3D coordinate correction based on the calibration and network model;

[0297] Output point cloud P(X,Y,Z) or reconstruct mesh.

[0298] According to an embodiment of the present invention, the method for generating a multi-frequency stripe pattern is as follows:

[0299] Assume the MEMS micromirror array is a two-dimensional array M(x,y)∈{0,1}, where each mirror can be independently controlled for deflection angle, simulating a sinusoidal fringe function:

[0300] I p (x,y,t)=A·[1+cos(2πf x x+2πf y y+φ(t))]

[0301] in:

[0302] A: Light intensity amplitude;

[0303] f x ,f y : Horizontal and vertical stripe frequencies;

[0304] φ(t): Phase that changes with time;

[0305] The voltage control vector of the micromirror array is V(t), which satisfies:

[0306] M(x,y,t)=H[cos(2πf x x+2πf y y+φ(t))-θ(x,y,t)];

[0307] The multi-frequency stripe pattern is fused based on a multi-frequency combination strategy, as follows:

[0308] Using different frequencies f1, f2, ..., f n The stripes are projected sequentially, and the phase diagram at each frequency is recorded:

[0309] φ i (u,v)=unwrap(arg(I i (u,v)))

[0310] The CRT uses the remainder theorem to synthesize multi-frequency phase signals, as shown in the following formula:

[0311]

[0312] M(x,y,t) represents the control of the micromirror state to generate structured light patterns;

[0313] φ(t) is the phase that changes with time and is used for phase shifting;

[0314] φ i (u,v) is the unwrapped phase at a single frequency;

[0315] unwrap is the operation to remove phase transitions;

[0316] φ global (u,v) is the global phase after multi-frequency fusion, used for high-precision 3D reconstruction;

[0317] Where w i These are the weighting coefficients corresponding to the frequencies;

[0318] Where H[·] is the Heaviside step function and θ(x,y,t) is the voltage bias modulation function, used to simulate amplitude or phase modulation.

[0319] According to an embodiment of the present invention, the phase decoding method based on the phase shift method and the optical path triangulation measurement principle is as follows:

[0320] Assuming the projector and camera are at a certain parallax angle, and the pattern is displaced on the target surface, the position (X,Y,Z) of point P in the world coordinate system can be calculated from the phase φ(u,v) of pixel point (u,v):

[0321]

[0322] φ(u,v) represents the phase value decoded at pixel (u,v), x(u,v) represents the horizontal coordinate of pixel (u,v) in the projected image, and f x φ0 represents the spatial frequency of the stripe pattern in the horizontal or x-direction, and φ0 represents the initial phase offset or reference phase value.

[0323] The reconstructed 3D coordinates are calculated by triangulation of the projection model and the camera model:

[0324]

[0325] Triangulate represents the geometric intersection process of inferring world coordinates from image coordinates and phase information.

[0326] A third aspect of the present invention provides a computer-readable storage medium including a calibration method program for a structured light projector, wherein when the calibration method program for the structured light projector is executed by a processor, the calibration method program for the structured light projector as described in any of the preceding claims implements the steps of the calibration method for the structured light projector.

[0327] This invention discloses a calibration method, system, and medium for a structured light projector. It projects a multi-frequency fringe pattern onto a target area using a MEMS micromirror array module, and acquires an image sequence of the distorted multi-frequency fringe pattern on the target surface using a camera. Phase decoding is performed on the distorted image sequence to obtain a phase map corresponding to each frequency. The phase map corresponding to each frequency is calculated based on the phase shift method and optical path triangulation principle to obtain preliminary 3D point coordinates. A neural network model and calibration sample set are constructed. The true 3D coordinates are obtained using a standard calibration plate or a known surface. The deviation between the preliminary 3D point coordinates and the true 3D coordinates is calculated to obtain a training sample set. The neural network model is trained based on the Euclidean distance loss function using the training sample set. The preliminary 3D point coordinates are input into the trained neural network model, which outputs a 3D offset compensation amount. The preliminary 3D point coordinates are corrected based on the 3D offset compensation amount to obtain the final 3D coordinates. The neural network model achieves an end-to-end error self-compensation mechanism, effectively compensating for deviations in the 3D point coordinates and improving the accuracy of 3D reconstruction.

[0328] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0329] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0330] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0331] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0332] Alternatively, if the integrated units of the present invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

Claims

1. A calibration method for a structured light projector, characterized in that, include: Multi-frequency stripe patterns are projected onto a target area using a MEMS micromirror array module, and image sequences of the multi-frequency stripe patterns after distortion on the target surface are acquired using a camera. Phase decoding is performed on the image sequence of the multi-frequency stripe pattern after distortion on the target surface to obtain the phase map corresponding to each frequency. Based on the phase shift method and the principle of optical path triangulation, the phase diagram corresponding to each frequency is calculated to obtain the preliminary three-dimensional point coordinates; Construct a neural network model and a calibration sample set. Use a standard calibration plate or a known surface to obtain the true 3D coordinates. Calculate the deviation between the preliminary 3D point coordinates and the true 3D coordinates to obtain the training sample set. The neural network model is trained based on the Euclidean distance loss function and the training sample set. The initial 3D point coordinates are input into the trained neural network model, and the 3D offset compensation is output. The initial 3D point coordinates are corrected based on the 3D offset compensation to obtain the final 3D coordinates.

2. The calibration method for a structured light projector according to claim 1, characterized in that, The method for generating multi-frequency stripe patterns is as follows: Assume the MEMS micromirror array is a two-dimensional array M(x,y)∈{0,1}, where each mirror can be independently controlled for deflection angle, simulating a sinusoidal fringe function: I p (x,y,t)=A·[1+cos(2πf x x+2πf y y+φ(t))] in: A: Light intensity amplitude; f x ,f y : Horizontal and vertical stripe frequencies; φ(t): Phase that changes with time; The voltage control vector of the micromirror array is V(t), which satisfies: M(x,y,t)=H[cos(2πf x x+2πf y y+φ(t))-θ(x,y,t)]; The multi-frequency stripe pattern is fused based on a multi-frequency combination strategy, as follows: Using different frequencies f1, f2, ..., f n The stripes are projected sequentially, and the phase diagram at each frequency is recorded: φ i (u,v)=unwrap(arg(I i (u,v))) The CRT uses the remainder theorem to synthesize multi-frequency phase signals, as shown in the following formula: M(x,y,t) represents the control of the micromirror state to generate structured light patterns; φ(t) is the phase that changes with time and is used for phase shifting; φ i (u,v) is the unwrapped phase at a single frequency; unwrap is the operation to unwrap the element and eliminate phase transitions; φ global (u,v) is the global phase after multi-frequency fusion, used for high-precision 3D reconstruction; Where w i These are the weighting coefficients corresponding to the frequencies; Where H[·] is the Heaviside step function and θ(x,y,t) is the voltage bias modulation function, used to simulate amplitude or phase modulation.

3. The calibration method for a structured light projector according to claim 2, characterized in that, The phase decoding method based on the phase shift method and the optical path triangulation principle is as follows: Assuming the projector and camera are at a certain parallax angle, and the pattern is displaced on the target surface, the position (X,Y,Z) of point P in the world coordinate system is calculated by the phase φ(u,v) of pixel (u,v): φ(u,v) represents the phase value decoded at pixel (u,v), x(u,v) represents the horizontal coordinate of pixel (u,v) in the projected image, and f x φ0 represents the spatial frequency of the stripe pattern in the horizontal or x-direction, and φ0 represents the initial phase offset or reference phase value. The reconstructed 3D coordinates are calculated by triangulation of the projection model and the camera model: Triangulate represents the geometric intersection process of inferring world coordinates from image coordinates and phase information.

4. The calibration method for a structured light projector according to claim 3, characterized in that, The neural network model construction method is as follows: Define the 3D coordinates of each pixel (u,v) obtained in the original 3D reconstruction as follows: X raw (u,v) represents the original 3D coordinate vector obtained by preliminary 3D reconstruction at pixel (u,v), which is a 3D column vector; X(u,v) represents the X coordinate in three-dimensional space corresponding to the pixel; Y(u,v) represents the Y coordinate of the pixel in three-dimensional space; Z(u,v) represents the Z coordinate of the pixel in three-dimensional space. This represents the transpose of a vector, meaning the three-dimensional coordinates are represented as a column vector. The input features of the neural network are: Among them, Finput is a multimodal feature vector that integrates spatial location, image brightness, edge information and phase structure to help the neural network learn more accurately and compensate for systematic errors in 3D reconstruction; I(u,v): pixel grayscale; Image gradient reflects edge intensity; φ(u,v): Corresponding phase value, reflecting the local fringe structure; The network output is a 3D offset, as shown in the following formula: The actual calibration point location is X. gt The loss function is Euclidean distance loss, and the formula for the loss function is as follows: This represents the loss function used to train the neural network. f θ (F input () represents a trained neural network model used to predict bias; X raw Represents the three-dimensional coordinates of the preliminary measurement; After training, the model is embedded into an online system for real-time correction, as shown in the following formula: X corrected =X raw +f θ (F input ) X corrected This represents the final three-dimensional coordinates after correction by the neural network.

5. The calibration method for a structured light projector according to claim 4, characterized in that, The three-dimensional offset compensation amount is analyzed as follows: We introduce a neural network model for error regression, as shown in the following formula: in: f θ : Neural network, with parameter θ; The input includes image coordinates (u,v), phase φ (u,v), and local gray level I (u,v); The output is a three-dimensional offset compensation ΔX; The neural network structure uses a residual encoder structure: f θ =MLP(ResNet(I(u,v),φ,u,v)) MLP is a basic feedforward neural network composed of multiple fully connected layers, each containing multiple neurons, with non-linear transformations between layers through activation functions. ResNet is a deep convolutional neural network.

6. The calibration method for a structured light projector according to claim 5, characterized in that, The neural network model training method is as follows: Use a standard calibration board (such as a black and white checkerboard) to perform preliminary distortion fitting; Obtaining the true offset based on MEMS projection structured light patterns; For each pixel, record φ(u,v) and I(u,v). and deviation ΔX; Constructing the training sample set: Training a deep regression network using MSE loss:

7. A calibration system for a structured light projector, characterized in that, The system includes a memory and a processor. The memory contains a program for a calibration method of a structured light projector. When the program for the calibration method of the structured light projector is executed by the processor, it performs the following steps: Multi-frequency stripe patterns are projected onto a target area using a MEMS micromirror array module, and image sequences of the multi-frequency stripe patterns after distortion on the target surface are acquired using a camera. Phase decoding is performed on the image sequence of the multi-frequency stripe pattern after distortion on the target surface to obtain the phase map corresponding to each frequency. Based on the phase shift method and the principle of optical path triangulation, the phase diagram corresponding to each frequency is calculated to obtain the preliminary three-dimensional point coordinates; Construct a neural network model and a calibration sample set. Use a standard calibration plate or a known surface to obtain the true 3D coordinates. Calculate the deviation between the preliminary 3D point coordinates and the true 3D coordinates to obtain the training sample set. The neural network model is trained based on the Euclidean distance loss function and the training sample set. The initial 3D point coordinates are input into the trained neural network model, and the 3D offset compensation is output. The initial 3D point coordinates are corrected based on the 3D offset compensation to obtain the final 3D coordinates.

8. The calibration system for a structured light projector according to claim 7, characterized in that, The method for generating multi-frequency stripe patterns is as follows: Assume the MEMS micromirror array is a two-dimensional array M(x,y)∈{0,1}, where each mirror can be independently controlled for deflection angle, simulating a sinusoidal fringe function: I p (x,y,t)=A·[1+cos(2πf x x+2πf y y+φ(t))] in: A: Light intensity amplitude; f x ,f y : Horizontal and vertical stripe frequencies; φ(t): Phase that changes with time; The voltage control vector of the micromirror array is V(t), which satisfies: M(x,y,t)=H[cos(2πf x x+2πf y y+φ(t))-θ(x,y,t)]; The multi-frequency stripe pattern is fused based on a multi-frequency combination strategy, as follows: Using different frequencies f1, f2, ..., f n The stripes are projected sequentially, and the phase diagram at each frequency is recorded: φ i (u,v)=unwrap(arg(I i (u,v))) The CRT uses the remainder theorem to synthesize multi-frequency phase signals, as shown in the following formula: Where w i These are the weighting coefficients corresponding to the frequencies; Where H[·] is the Heaviside step function and θ(x,y,t) is the voltage bias modulation function, used to simulate amplitude or phase modulation.

9. The calibration system for a structured light projector according to claim 8, characterized in that, The phase decoding method based on the phase shift method and the optical path triangulation principle is as follows: Assuming the projector and camera are at a certain parallax angle, and the pattern is displaced on the target surface, the position (X,Y,Z) of point P in the world coordinate system can be calculated from the phase φ(u,v) of pixel point (u,v): φ(u,v) represents the phase value decoded at pixel (u,v), x(u,v) represents the horizontal coordinate of pixel (u,v) in the projected image, and f x φ0 represents the spatial frequency of the stripe pattern in the horizontal or x-direction, and φ0 represents the initial phase offset or reference phase value. The reconstructed 3D coordinates are calculated by triangulation of the projection model and the camera model: Triangulate represents the geometric intersection process of inferring world coordinates from image coordinates and phase information.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a calibration method program for a structured light projector, which, when executed by a processor, implements the steps of the calibration method for a structured light projector as described in any one of claims 1 to 6.

Images