A point positioning method based on 3-D moire effect

Abstract

The application provides a point positioning method based on 3-D Moire effect, and a point positioning system based on 3-D Moire effect, which utilizes 3-D Moire effect caused by a shadow of a serial mask fixed in front of a bare camera sensor to perform accurate spatial point positioning, and a beacon marked on a target point is illuminated; the beacon is the target point to be positioned; the method adopts a positioning spectrum method to deduce a geometric relationship between the beacon and a Moire pattern; a positioning model of the positioning spectrum method is equivalent to a pinhole model with a periodic extended range, and a frequency of a mask pattern of the positioning model determines an analog focal length; the application is easy to implement, the algorithm is intuitive and simple, and positioning resolution and accuracy are about one order of magnitude higher than those of a benchmark model (including a pinhole model and a single mask model).

Description

TECHNICAL FIELD

[0001] The present application relates to the field of machine vision and precision measurement, and in particular to a point localization method based on 3-D Moiré effect. BACKGROUND

[0002] Vision-based spatial point coordinate estimation, i.e. point localization, is the cornerstone of motion measurement in many fields such as industry, medicine, entertainment, etc. The number of spatial points that a localization method can identify is called resolution, which is an important indicator of the performance of the method and is independent of scale.

[0003] Most point localization methods rely on the pinhole camera model, in which a spatial point is projected along a straight line passing through the optical center. This introduces an upper bound: the resolution of a pinhole-based method cannot be larger than the number of image points it can resolve. Under this constraint, there are two ways to improve. One is achieved by adding markers. The target is marked by markers such as crosses, circles or spheres. The projected target with markers can use more information to solve the uncertainty by marker localization. The other is achieved by multi-view. The target is observed from different angles, the projected target in each view is localized independently, and then the results are fused to minimize the re-projection error. Although both methods can improve the resolution by sub-pixel accuracy, they are still limited by the upper bound. Specifically, their upper bound is the product of the number of pixels and the square of the sub-pixel resolution. Compared with the full potential of the value of the image, i.e. the number of intensity levels is raised to the power of the number of pixels, the resolution still has a great potential for improvement. Therefore, the potential of point localization is still largely untapped.

[0004] The upper bound comes from the pinhole camera model used for imaging. If imaging is not required, it is possible to break through this upper bound and fully tap the potential of spatial point localization. For this reason, localization methods based on 3-D Moiré effect have attracted widespread attention. This method aims to use the sensitivity of the 3-D Moiré pattern produced by the shadow of the serial mask to its mask displacement. This is the basis of the grating ruler and a milestone in the field of optical metrology. However, there are many problems in previous work. These methods lack a clear localization model. The geometric relationship between the Moiré pattern and the measured value is established by enumeration or machine learning. They are only superior to pinhole methods in terms of view estimation, not in terms of point localization. Therefore, the present application uses the 3-D Moiré effect produced by the shadow of the serial mask fixed in front of the bare camera sensor, while the beacon marked on the target point is illuminated. In addition, a new method called localization spectrum is proposed, based on which the localization model of the proposed method can be derived in an easier way than existing 3-D Moiré methods. SUMMARY

[0005] The application provides a point positioning method based on 3-D Moire effect, which is a low-cost, simple-to-produce and simple-to-model spatial point positioning method.

[0006] The application adopts the following technical scheme.

[0007] A point positioning method based on 3-D Moire effect, which adopts a point positioning system based on 3-D Moire effect, uses 3-D Moire effect caused by a shadow of a serial mask fixed in front of a bare camera sensor for accurate spatial point positioning, and uses a beacon marked on a target point for illumination; the beacon is the target point to be positioned; the method adopts a positioning spectrum method to deduce a geometric relationship between the beacon and a Moire pattern; a positioning model of the positioning spectrum method is equivalent to a pinhole model with a periodic extended range, and a frequency of a mask pattern of the pinhole model determines an analog focal length.

[0008] The method comprises the following steps.

[0009] Step one, obtaining a Moire image S required for calibration through a point positioning system;

[0010] Step two, performing phase unwrapping calibration by using the Moire image to obtain unwrapped phases m and n;

[0011] Step three, calculating an image point h in the image;

[0012] Step four, performing calibration of system internal parameters by using h to obtain system internal parameters f, a and b;

[0013] Step five, performing spatial point positioning.

[0014] The form of the beacon comprises a point light source, a laser and a single-mode optical fiber.

[0015] The positioning spectrum method includes a calibration algorithm for phase unwrapping for accurately positioning the phase ambiguity of the decoded periodic pattern; the positioning model of the positioning spectrum method includes a calibration algorithm for obtaining the internal parameters of the system for accurate positioning, specifically a calibration algorithm for phase unwrapping of 3-D Moire through projective geometry, so as to calibrate the internal parameters of the system;

[0016] The calibration algorithm includes three steps: image processing, quadratic surface fitting to estimate the periodic phase, and least squares method;

[0017] In the image processing step, the Moire image is low-pass filtered to smooth the image and reduce signal noise, and the histogram of the image is equalized to improve the contrast of the image and improve the signal-to-noise ratio of the image;

[0018] In the quadratic surface fitting to estimate the periodic phase step, a complete period of the 3-D Moire signal is regarded as a quadratic surface, and the sub-pixel position of the wave peak position of the signal in the image is extracted by quadratic surface fitting, and the period u, v of the Moire and the corresponding phase and ψ are estimated.

[0019] In the least squares method step, a basic model for solving the periodic ambiguity is derived through projective geometry and positioning spectrum, in which the unknown number of unwrapped phases to be solved is fixed and does not change in quantity, and is solved by the least squares method.

[0020] The pinhole model with periodic extension range is calibrated by the chessboard calibration algorithm; specifically, based on the camera imaging principle, the internal and external parameters of the system are calculated by shooting different angle calibration images, to improve the accuracy and expand the application range.

[0021] When shooting images, the beacon is moved in space, and the moving track is a perfect chessboard array, which replaces the traditional chessboard calibration board of the chessboard calibration algorithm.

[0022] The point positioning system includes a bare camera sensor and two masks etched on both sides of a quartz glass substrate; the quartz glass substrate is pasted on the sensor cover glass, the masks and the bare camera sensor are parallel to each other, and the thickness of the cover glass and the substrate is h and s respectively; the transmittance of the mask is denoted as T i , where i=1, 2, respectively, indicating the front and back of the mask;

[0023] T i is designed as a chessboard pattern, and the simplified form is T i (x) = F(Φ i (x)), where F is a two-dimensional cosine function with a period of (1, 1);

[0024] Align the origin and direction of the mask, and define the camera origin O at the vertical projection of the mask's origin C , so that the unwrapped phase of the mask is obtained, which is expressed as:

[0025] Φ1(x) = (u T x, v T x) T

[0026] Φ2(x) = (1 + a) Φ1(x)

[0027] where (1 + a) e [1, 2] represents the frequency scaling of the two masks; u and v are not necessarily aligned with the X, Y axes;

[0028] Etch a control point at the origin of T1, denoted as g = (0, 0, h + s) T , to solve the periodic ambiguity of the Moire pattern;

[0029] Define an image coordinate system O I at the top left of the sensor C ; define a virtual coordinate system O T at O G ; this coordinate system is used to describe the pinhole camera coordinate system that the system regards as; the X, Y axes of O C , O I and O G are parallel to the edges of the sensor;

[0030] The coordinates of the beacon in O C are denoted as P = (p T , z) T , where p represents the in-plane coordinate and z represents the depth; the distance from P to the front mask is denoted as d = z - h - s;

[0031] T i The projected pattern on the sensor is projected by the beacon, denoted as The synthesis of the image T and the Moire pattern S is expressed as

[0032]

[0033] where LP{..} is a low-pass filter, and the image of the Moire pattern is I = Img(S), where Img{..} involves sampling, quantization, and blurring operations;

[0034] The positioning model is expressed as S(x) = F(Φ(x)) and

[0035] where β is inversely proportional to the period length of S; φ and Φ are initial phases in the interval [0, 1).

[0036] The optical path connecting P and p The positioning spectrum on P is S The projection on P is an orthogonal transformation, which is not affected by refraction; then The projection spectrum on P is S ∞ (x) = LP{T1(x)T2(x)}. According to the product and identity, we have Φ ∞ (x) = Φ2(x) - Φ1(x); therefore, Φ ∞ (x) = αΦ1(x); again, because F(Φ(p)) = F(Φ ∞ (p)), we have,

[0037]

[0038] (β-α)v T p+(ψ-n) = 0

[0039] Let h be the (m, n) peak of S, i.e.

[0040]

[0041] βv T h+ψ = n

[0042] and combine it with the above formula. We have,

[0043]

[0044] The in-plane coordinate p of the beacon is bound to a feature peak of the Moire pattern S; let h be the "image point" of p;

[0045] The calibration algorithm of phase unwrapping is as follows: by controlling point g, the direct relationship between the light and the pattern is established as:

[0046]

[0047] Again, because , we have:

[0048]

[0049] (β-λ * )v T c+(ψ-n c ) = 0,

[0050] where λ * = (1 + α)(1 - λ) is an internal parameter, which is obtained by calibration; the periods m c and n c are solved,

[0051] and we get

[0052] Where RoundP..} represents the floor function; m c and n c Both are integers, β, λ * u, v, c, ψ and ψ are values ​​that do not require high precision;

[0053] The calibration algorithm for the localization model is as follows: what is read from I is O. J Pixel coordinates in the coordinate system and The relationship between them is Where w is the pixel size, (a, b) T For O C Pixel coordinates in a coordinate system. Therefore, we have,

[0054]

[0055] This positioning model is equivalent to an optical center at (0, 0, γ*). T The pinhole model with focal length f is considered as a virtual camera fixed on the system, and its coordinate system is O. G =O C -(0, 0, γ*) T The ghost camera and P are calibrated and P is solved using projective geometry techniques via PnP or binocular triangulation.

[0056] In point positioning systems, such as Figure 1 As shown, a photolithographic glass plate (mask) with masks on both sides is attached to the bare camera sensor (sensor in the figure); the side closest to the bare camera sensor is denoted as T1, and the other side as T2; the image is projected onto the sensor through beacon P to generate... Therefore, the Moiré image S is generated by the Moiré effect; the movement of S can be observed by the movement of beacon P, and the spatial position of beacon P can be located based on the movement of S, so P is also called the target point;

[0057] In the localization spectrum method, the hovering theorem of the adjoint localization spectrum is adopted; specifically: Figure 3 In this study, a mask-based vision measurement system is used, which treats the mask as N transmission planes. When there are 2 masks, the transmittance function is denoted as F, where F is a two-dimensional cosine function.

[0058] Each transmission plane is defined by a point (p, p). z ) T =(p x p y p z ) T The transmittance projection of the projection center on the XY plane is denoted as . The integrated transmittance T and the moiré S on it are

[0059]

[0060] As shown in the figure, given a point q on S, the light path from P to q is denoted as Push P along to infinity, i.e. P ∞ = (p, ∞) T , i.e. the projective transformation becomes parallel transformation; the projected transmittance is denoted as Let the light path from P to q remain unchanged, then T and have the following relationship

[0061]

[0062] Then, assume the light path around q is also the same under the projective transformation and parallel transformation, i.e.

[0063]

[0064] where is called the positioning spectrum, q is called the hovering point, the transmittance of the hovering point is called the hovering value, and the above relationship is called the hovering theorem.

[0065] In step two, the phase unwrapping calibration is performed using the Moiré image to obtain the unwrapping phases m and n, as shown in Figure 4 , including the following steps:

[0066] Step A1: low-pass filter the Moiré image to smooth the image and reduce signal noise, and perform histogram equalization on the image to improve the contrast of the image and improve the signal-to-noise ratio of the image, and detect the image coordinates c of the control points; Step A2: use quadratic surface fitting to extract the sub-pixel position of the signal wave peak position in the image, and estimate the Moiré period u, v and the corresponding phase and ψ;

[0067] Step A3: calibrate λ * by least squares method, in order to calibrate λ * , an additional beacon is added on the one-dimensional grating ruler, and the beacon is constantly moved to capture a series of has:

[0068]

[0069] (β j -λ * )v T c j +(ψj -n j )=0

[0070] If the absolute value of m j and n j cannot be determined, the image is continuously taken to monitor its increase and decrease; therefore, m j =m0+Δm j and n j =n0+Δn j , where m0and n0are unknown, and Δm j and Δn j are known;

[0071] In matrix form, there are,

[0072]

[0073] In the case of sufficient samples, λ*is solved by the least square algorithm.

[0074] Step A4: solve m and n by least square method using the formula .

[0075] The pinhole model with periodic extension range, the pinhole model is shown as Figure 6 , the periodic extension is shown as Figure 7 , as shown in Figure 6 , it is equivalent to a virtual camera fixed on the system, that is, expressed as formula

[0076] O G =O C -(0,0,γ * ) T ,

[0077] The virtual imaging plane is between the virtual camera sensor plane O G and the actual system camera sensor plane O C , the distance from the virtual imaging plane to O G is called the focal length f of the pinhole model; then, the virtual camera can be calibrated and P can be solved by PnP or binocular triangulation using projective geometry techniques; specifically:

[0078] Given a WxW sensor and the moving range of P (beacon), in order to optimize the positioning resolution, the camera in the pinhole model needs to cover the range through its field of view, as shown in Figure 7 part a; the 3-D Moiré model sets a large f concentrated in a small range;

[0079] As shown in Figure 8 , the peak points of the blue Moiré pattern in the figure,

[0080] If the obtained Moiré pattern is periodic, its periodicity is extended by de-ambiguating the phase blur, and then its extended range is used to cover other ranges, see [link to relevant documentation]. Figure 7 Part b; most of the resolution cap can improve R = (Wex / W). 2 , where Wex is the side length of the extended range.

[0081] To address the shortcomings of existing technologies, this invention provides a low-cost, simple-to-fabricate, and easy-to-use spatial point localization method. This method embeds the complex 3-D Moiré effect into the camera. Users simply need to fix a beacon to the target, which is much easier than using feature maps, imaging media, or 3-D Moiré markers. The beacon can take various forms, such as point light sources, lasers, single-mode fibers, etc. Previous strategies often lack a well-defined localization model, while the localization model of this invention is explicit and concise. It can be interpreted as a pinhole model with a periodically expanding range, where the frequency of the mask pattern determines the analog focal length. This facilitates the design of localization and calibration algorithms using established projective geometry, making these algorithms simple and intuitive. Furthermore, the localization resolution and accuracy of this invention are approximately an order of magnitude higher than benchmark models (including pinhole and single-mask models).

[0082] This invention provides a point positioning system based on the 3-D Moiré effect, comprising: a novel method called positioning spectrum, which allows for a simpler and clearer understanding of the basic model of this invention; a phase unwrapping calibration algorithm for decoding the phase ambiguity of periodic patterns for accurate positioning; and a positioning model calibration algorithm for obtaining the system's internal parameters for accurate positioning. Attached Figure Description

[0083] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0084] Appendix Figure 1 This is a schematic diagram of the point localization method based on the 3-D Moiré effect provided in an embodiment of the present invention;

[0085] Appendix Figure 2 This is a schematic flowchart of the point localization method based on the 3-D Moiré effect provided in the embodiments of the present invention;

[0086] Appendix Figure 3 This is a schematic diagram of a mask-based vision measurement system provided in an embodiment of the present invention;

[0087] Appendix Figure 4 This is a schematic diagram of the unwrapping phase calibration process provided in the embodiments of the present invention;

[0088] AppendixFigure 5 is a flowchart of the calibration algorithm of the positioning model provided by the embodiment of the present application;

[0089] Figure 2 is a schematic diagram of a pinhole model provided by the embodiment of the present application; Figure 6 is a schematic diagram of a pinhole model provided by the embodiment of the present application;

[0090] Figure 4 is a schematic diagram of a pinhole model provided by the embodiment of the present application with a periodic extension; Figure 7

[0091] Figure 5 is a schematic diagram of a Moire pattern obtained by the system provided by the embodiment of the present application with a periodic extension. Figure 8 DETAILED DESCRIPTION

[0092] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. However, it should be understood that the description is only exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description, many specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it is obvious that one or more embodiments can be implemented without these specific details. In addition, in the following description, the description of well-known structures and techniques is omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0093] All terms used herein (including technical and scientific terms) have meanings commonly understood by one of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings consistent with the context of the present specification, and should not be interpreted in an idealized or overly formal manner.

[0094] In addition, the shapes and sizes of the components in the drawings do not reflect the actual sizes and proportions, but only illustrate the content of the embodiments of the present disclosure. In addition, in the claims, any reference symbols located between parentheses should not be construed as a limitation of the claims.

[0095] In addition, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0096] ​​As shown in the figure, a point positioning method based on 3-D Moire effect, the method adopts a point positioning system based on 3-D Moire effect, which uses the 3-D Moire effect generated by the shadow of the serial mask fixed in front of the bare camera sensor for accurate spatial point positioning, and is illuminated by the beacon marked on the target point; the beacon is the target point to be positioned; the method adopts a positioning spectrum method to derive the geometric relationship between the beacon and the Moire pattern; the positioning model of the positioning spectrum method is equivalent to a pinhole model with a periodic extended range, and the frequency of the mask pattern determines the analog focal length;

[0097] The method comprises the following steps:

[0098] Step one, obtaining the Moire image S required for calibration by the point positioning system;

[0099] Step two, using the Moire image to perform phase unwrapping calibration to obtain unwrapped phases m and n;

[0100] Step three, calculating the image point h in the image;

[0101] Step four, using h to calibrate the system internal parameters to obtain the system internal parameters f, a, and b;

[0102] Step five, performing spatial point positioning.

[0103] The form of the beacon includes a point light source, a laser, and a single-mode optical fiber.

[0104] The positioning spectrum method includes a phase unwrapping calibration algorithm for accurately positioning the phase ambiguity of the decoded periodic pattern; the positioning model of the positioning spectrum method includes a calibration algorithm for obtaining the internal parameters of the system for accurate positioning, specifically, performing phase unwrapping calibration of 3-D Moire through projective geometry to calibrate the system internal parameters of the system;

[0105] The calibration algorithm includes three steps: image processing, quadratic surface fitting to estimate the periodic phase, and least squares method;

[0106] In the image processing step, the Moire image is low-pass filtered to smooth the image to reduce signal noise, and the image is histogram equalized to improve the contrast of the image to improve the signal-to-noise ratio of the image;

[0107] In the quadratic surface fitting to estimate the periodic phase step, a complete period of 3-D Moire signal is regarded as a quadratic surface, and the sub-pixel position of the wave peak position of the signal in the image is extracted by quadratic surface fitting, and the period u, v of the Moire and the corresponding phase And ψ are estimated;

[0108] In the least square step, a basic model for solving the periodic ambiguity is derived by using the projective geometry and the positioning spectrum, in which the unknown number of the unwrapped phase to be solved is fixed and unchangeable, and is solved by using the least square method.

[0109] The pinhole model with a periodically extended range is calibrated by using a chessboard calibration algorithm; specifically, based on the camera imaging principle, the internal parameters and the external parameters of the system are calculated by using the feature points in the images of different angles, so that the precision is improved and the application range is expanded.

[0110] When the image is shot, the beacon is moved in space, and the moving track is a perfect chessboard array, so that the chessboard calibration board of the traditional chessboard calibration algorithm is replaced by the chessboard array.

[0111] The point positioning system comprises a bare camera sensor and two masks etched on two opposite surfaces of a quartz glass substrate; the quartz glass substrate is pasted on the sensor cover glass, the masks and the bare camera sensor are parallel to each other, and the thicknesses of the cover glass and the substrate are h and s respectively; the transmittance of the mask is T i , wherein i = 1, 2, respectively representing the front and back surfaces of the mask;

[0112] T i is designed in a chessboard pattern, and the simplified form is T i (x) = F(Φ i (x)), wherein F is a two-dimensional cosine function with a period of (1, 1);

[0113] The origin and direction of the mask are aligned, and the camera origin O C is defined at the vertical projection of the origin of the mask, so that the unwrapped phase of the mask is obtained, which is expressed in a formula as follows:

[0114] Φ1(x) = (u T x, v T x) T

[0115] Φ2(x) = (1 + α)Φ1(x)

[0116] , wherein (1 + α) ∈ [1, 2] represents the frequency scaling of the two masks; u and v are not necessarily aligned with the X and Y axes; a control point is etched at the origin of T1, and is denoted as g = (0, 0, h + s) T , which is used to solve the periodic ambiguity of the moire pattern;

[0117] An image coordinate system O I is defined at the upper left of the sensor, and a virtual coordinate system O C is defined at O T -(0, 0, t) GThis coordinate system is used to describe the coordinate system used by the system as a pinhole camera coordinate system; O C O I and O G The X and Y axes are parallel to the edge of the sensor;

[0118] beacon in O C The coordinates in the graph are represented as P = (p T ,z) T Where p represents the coordinates in the plane and z represents the depth; the distance from P to the front mask is denoted as d = zhs;

[0119] T i The projected pattern on the sensor is projected by the beacon, denoted as... The composite image T and the Moiré pattern S are expressed by the formula as follows:

[0120]

[0121] Where LP{..} is a low-pass filter, and the image of the Moiré pattern is I = Img(S), where Img{..} involves sampling, quantization, and blurring operations;

[0122] The localization model is expressed as S(x)=F(Φ(x)) and

[0123] β is inversely proportional to the period length of S; φ and ψ are the initial phases in the interval [0, 1).

[0124] Optical path connecting P and p In the positioning spectrum above; let along The projection is an orthogonal transformation and is unaffected by refraction; therefore... The projection spectrum on is S ∞ (x)=LP{T1(x)T2(x)}. According to the product and identity, we have Φ ∞ (x)=Φ2(x)-Φ1(x); Therefore, Φ ∞ (x)=αΦ1(x); and because F(Φ(p))=F(Φ ∞ (p)), therefore,

[0125]

[0126] (β-α)v T p+(ψ-n)=0

[0127] Define h as the (m, n) peaks of S, i.e.

[0128]

[0129] βv T h+ψ=n

[0130] and combine it with the above formula. There are,

[0131]

[0132] The in-plane coordinate p of the beacon is bound to one feature peak of the Moire pattern S; h is regarded as the "image point" of p;

[0133] The calibration algorithm of phase unwrapping is as follows: the direct relationship between the light and the pattern is established by the control point g:

[0134]

[0135] And because , there are:

[0136]

[0137] (β-λ * )v T c+(ψ-n c )=0,

[0138] where λ * =(1+α)(1-λ) is an internal parameter, obtained by calibration; the periods m c and n c are solved, and

[0139] where Round{..} represents the rounding function; m c and n c are integers, β, λ * , u, v, c, and ψ are values that do not require high accuracy;

[0140] The calibration algorithm of the positioning model is as follows: the read-out from I is the pixel coordinate I in the O C coordinate system and The relationship between them is where w is the pixel size, (a, b) T is the pixel coordinate in the O C coordinate system. Therefore, there are,

[0141]

[0142] The positioning model is equivalent to a pinhole model with an optical center at (0, 0, γ * ) T and a focal length f, which is regarded as a virtual camera fixed on the system, and its coordinate system is O G = O C -(0, 0, γ* ) T The ghost camera and P are calibrated and P is solved using projective geometry techniques via PnP or binocular triangulation.

[0143] In point positioning systems, such as Figure 1 As shown, a photolithographic glass plate (mask) with masks on both sides is attached to the bare camera sensor (sensor in the figure); the side closest to the bare camera sensor is denoted as T1, and the other side as T2; the image is projected onto the sensor through beacon P to generate... Therefore, the Moiré image S is generated by the Moiré effect; the movement of S can be observed by the movement of beacon P, and the spatial position of beacon P can be located based on the movement of S, so P is also called the target point;

[0144] In the localization spectrum method, the hovering theorem of the adjoint localization spectrum is adopted; specifically: Figure 3 In this study, a mask-based vision measurement system is used, which treats the mask as N transmission planes. When there are 2 masks, the transmittance function is denoted as F, where F is a two-dimensional cosine function.

[0145] Each transmission plane is defined by a point (p, p). z ) T =(p x p y p z ) T The transmittance projection of the projection center on the XY plane is denoted as . The overall transmittance T and the moiré pattern S formed on it are

[0146]

[0147] As shown in the figure, given a point q on S, the light path from P to q is denoted as . Move P along Extending to infinity, i.e., P ∞ = (p, ∞) T That is, the projective transformation becomes the parallel transformation; the transformed transmittance projection is denoted as... Assuming the optical path remains constant from P to q, then we have T and The following relationship exists

[0148] Then, assuming the nearby optical path The same applies to projective and parallel transformations.

[0149] in, This is called the positioning spectrum, q is called the hovering point, the transmittance of the hovering point is called the hovering value, and the above relationship is called the hovering theorem.

[0150] In step two, the phase unwrapping calibration is performed using the Moire image to obtain the unwrapped phase m and n, as shown in Figure 4 The method comprises the following steps:

[0151] Step A1: The Moire image is low-pass filtered to smooth the image and reduce signal noise, and histogram equalization is performed on the image to improve the contrast of the image and improve the signal-to-noise ratio of the image, and the image coordinates c of the control points are detected; Step A2: The sub-pixel position of the wave peak position of the signal in the image is extracted using quadratic surface fitting, and the period u, v of the Moire and the corresponding phase and ψ are estimated.

[0152] Step A3: The least squares method is used to calibrate λ * In order to calibrate λ * , a beacon is added to the one-dimensional grating ruler, and the beacon is moved continuously to capture a series of There are:

[0153]

[0154] (β j -λ * )v T C j +(ψ j -n j )=0

[0155] If the absolute values of m j and n j cannot be determined, the image is continuously captured to monitor the increase and decrease thereof; therefore, m j =m0+Δm j and n j =n0+Δn j , wherein m0 and n0 are unknown, and Δm j and Δn j are known.

[0156] In matrix form, there are,

[0157]

[0158] In the case of sufficient samples, λ * is solved by the least squares algorithm.

[0159] Step A4: The least squares method is used to solve m and n using the formula .

[0160] The pinhole model with a periodic extension range is as shown in Figure 6 , and the periodic extension is as shown in Figure 7 ; asFigure 6 As shown, the virtual camera is equivalent to a virtual camera fixed on the system, i.e. expressed in formula as O G = O C -(0, 0, γ * ) T ,

[0161] The virtual imaging plane is between the virtual camera sensor plane O G and the actual system camera sensor plane O C , the distance from the virtual imaging plane to O G is called the focal length f of the pinhole model; then, the virtual camera can be calibrated and P can be solved by PnP or binocular triangulation using projective geometry techniques; specifically:

[0162] In order to optimize the positioning resolution, given a WxW sensor and the moving range of P (beacon), the camera in the pinhole model needs to cover the range through its field of view, as shown in part a of Figure 7 ; the 3-D Moiré model sets a large f concentrated in a small range;

[0163] As shown in Figure 8 , the peak points of the Moiré pattern of the blue dots in the figure,

[0164] If the obtained Moiré pattern is periodic, its period is extended by deblurring the phase ambiguity, and then its extended range is used to cover other ranges, see part b of Figure 7 ; most of the upper limit of the resolution can improve R = (Wex / W) 2 , where Wex is the side length of the extended range.

[0165] Embodiment:

[0166] The point positioning method based on 3-D Moiré effect, as shown in Figures 1 to 8 , includes the following steps:

[0167] Obtain the Moiré image required for calibration through the system;

[0168] Use the Moiré image to perform phase unwrapping calibration to obtain unwrapped phases m and n;

[0169] Calculate the image point h;

[0170] Use h to calibrate the system intrinsic parameters to obtain system intrinsic parameters f, a, and b;

[0171] Perform spatial point positioning;

[0172] The system proposed in this example is composed of a bare camera sensor and two masks etched on the opposite side of the quartz glass substrate, and then pasted on the sensor cover glass.

[0173] The mask and the sensor are parallel to each other, and the thickness of the cover glass and the substrate are h and s respectively. The transmittance of the mask is denoted as T i , where i = 1, 2, denote the front and back of the mask respectively. T i is designed in a chessboard pattern, which can be simplified as T i (x) = F(Φ i (x)), where F is a two-dimensional cosine function with period (1, 1). The origin and direction of the mask are aligned, and the camera origin O C is defined at the vertical projection of the origin of the mask, so that the unwrapped phase of the mask is obtained as

[0174] Φ1(x) = (u T x, v T x) T

[0175] Φ2(x) = (1 + α)Φ1(x)

[0176] where (1 + α) ∈ [1, 2] represents the frequency scaling of the two masks. It is worth noting that u and v are not necessarily aligned with the X, Y axes. A control point is etched at the origin of T1, denoted as g = (0, 0, h + s) T , which is used to solve the periodic ambiguity of the Moire pattern. An image coordinate system O I is defined at the top left of the sensor, and a virtual coordinate system O C is defined at O T (0, 0, t) G . This coordinate system is used to describe the pinhole camera coordinate system as seen by the system. The XY axes of O C , O I and O G are parallel to the edges of the sensor.

[0177] The coordinates of the beacon in O C are denoted as P = (p T , z) T , where p represents the in-plane coordinates and z represents the depth. The distance from P to the front mask is denoted as d = z - h - s. T i The projected pattern on the sensor is projected by the beacon, denoted as Their composition T and the Moire pattern S are

[0178]

[0179] where LP{..} is a low-pass filter, and the image of the Moire pattern is I = Img(S), where Img{…} involves sampling, quantization, blurring, etc.

[0180] Thus, the model of the present application can be written as S(x) = F(Φ(x)) and

[0181]

[0182] where β is inversely proportional to the period length of S. φ and ψ are initial phases in the interval [0, 1).

[0183] The present application studies the localization spectrum of the optical path connecting P and p The projection along is an orthogonal transformation, which is not affected by refraction. Therefore, the projection spectrum on ∞ (x) = LP{T1(x)T2(x)}. According to the identity of product, we have Φ ∞ (x) = Φ2(x) - Φ1(x). Thus,

[0184] Φ ∞ (x) = αΦ1(x)

[0185] Since F(Φ(p)) = F(Φ ∞ (p)), we have,

[0186]

[0187] (β - α)v T p + (ψ - n) = 0

[0188] Let h be the (m, n)th peak of S, i.e.

[0189]

[0190] βv T h + ψ = n

[0191] and combine it with the above equation. We have,

[0192]

[0193] This example shows an interesting and simple fact: the in-plane coordinate p of the beacon is tied to a characteristic peak of the moire pattern S. Therefore, h can be regarded as the "image point" of p.

[0194] The calibration algorithm for phase unwrapping: by controlling points g, the direct relationship between the light and the pattern is established as:

[0195]

[0196] Since we have:

[0197]

[0198]

[0199] where λ * =(1+α)(1-λ) is an internal parameter, which can be obtained by calibration. The periods m c and n c can be solved, i.e.

[0200]

[0201] where RoundP..} denotes the rounding function. Since m c and n c are integers, β, λ * , u, v, c, and ψ do not need to be very precise.

[0202] Calibration algorithm of the localization model: what can be read out from I by the present application is O I , the pixel coordinates in the O and , instead of the physical coordinates h and c in the O C coordinate system. The relationship between them is where w is the pixel size, (a, b) T are the pixel coordinates in the O C coordinate system. Therefore, we have

[0203]

[0204] This means that the proposed localization model is equivalent to a pinhole model with the optical center at (0, 0, γ * ) T and the focal length f. One can imagine a virtual camera fixed on the system, whose coordinate system is O G = O C -(0, 0, γ * ) T . Then, one can calibrate the ghost camera and solve P by PnP or binocular triangulation using mature projective geometry techniques.

[0205] The above only describes the preferred embodiments of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments. Any technical solutions falling within the principles of the present application shall be considered as falling within the protection scope of the present application. For those skilled in the art, several improvements can be made without departing from the principles of the present application, and these improvements shall also be considered as falling within the protection scope of the present application.

Claims

1. A point positioning method based on 3-D Moire effect, characterized in that: The method adopts a 3-D Moire effect-based point positioning system, which utilizes a 3-D Moire effect generated by a shadow of a serial mask fixed in front of a bare camera sensor for precise spatial point positioning, and is illuminated by a beacon marked on a target point; the beacon is the target point to be positioned; the method adopts a positioning spectrum method to derive a geometric relationship between the beacon and a Moire pattern; a positioning model of the positioning spectrum method is equivalent to a pinhole model with a periodic extended range, and a frequency of a mask pattern determines an analog focal length; The method comprises the following steps: Step one, obtaining a Moire image S required for calibration by a point positioning system; Step two, performing phase unwrapping calibration by using the Moire image to obtain unwrapped phases m and n; Step three, calculating an image point h in the image; Step four, performing calibration of system internal parameters to obtain system internal parameters f, a, and b by using h; Step five, performing spatial point positioning; The point positioning system comprises a bare camera sensor and two masks etched on two opposite surfaces of a quartz glass substrate; the quartz glass substrate is pasted on a sensor cover glass, the masks and the bare camera sensor are parallel to each other, the thicknesses of the cover glass and the substrate are h and s respectively; the transmittance of the mask is T i wherein i = 1, 2, respectively representing the front surface and the back surface of the mask; T i Designed as a checkerboard pattern, simplified form as T i (x) = F(Φ i (x)), where F is a two-dimensional cosine function with a period of (1, 1); Align the origin and direction of the mask and define the camera origin O at the vertical projection of the origin of the mask C Thus, the unwrapped phase of the mask is obtained, which is expressed in a formula as follows: Φ1(x) = (u T x,v T x) T Φ2(x)=(1+α)Φ1(x) where (1+α) ∈ [1,2] represents the frequency scaling of the two masks; the period of the Moiré u, v, u and v are not necessarily aligned with the X, Y axes; a control point is etched at the origin of T1, denoted as g = (0, 0, h + s) T , to resolve the periodic ambiguity of the Moiré pattern; An image coordinate system O is defined at the top left of the sensor I , O C is defined at the origin (0,0,t) T A virtual coordinate system O G is defined at the origin (0,0,t) C ; this coordinate system is used to describe the pinhole camera coordinate system as seen by the system; the X, Y axes of O I , O G are parallel to the edges of the sensor; The beacon is represented in O C with coordinates P = (p T , z) T where p represents the in-plane coordinates and z represents the depth; the distance of P to the front mask is noted d = z - h - s; T i The projected pattern on the sensor is projected by the beacon, denoted as The synthesis of the pattern T and the Moire pattern S is formulated as Wherein LP{..} is a low-pass filter, an image of the Moire pattern is I=Img(S), and Img{..} involves sampling, quantization, and blurring operations; The positioning model is expressed as S(x) = F(Φ(x)) and where β is inversely proportional to the period length of S; and ψ is an initial phase in the interval [0, 1). The optical path connecting P and p is a projection onto the spectrum of the positioning spectrum; let the projection be an orthogonal transformation, unaffected by refraction; then the projection spectrum on is S ∞ (x) = LP{T1(x)T2(x)}; by the product and identity, Φ ∞ (x) = Φ2(x) - Φ1(x); thus, Φ ∞ (x) = αΦ1(x); and because F(Φ(p)) = F(Φ ∞ (p)), there is,​ (β - a)v T p + (ψ - n) = 0 Define h as (m, n) wave peaks of S, that is, βv T h+ψ=n And combine it with the above formula; that is, An in-plane coordinate p of the beacon is bound to a feature peak of the Moire pattern S; h is regarded as an image point of p; The calibration algorithm for phase unwrapping is specifically: the direct relationship between the light and the pattern is established through the control point g as follows: Also because There are: (β - λ * v T c + (ψ - n c ) = 0, where λ * = (1 + a) (1 - λ) is an internal parameter, obtained by calibration; the period m c and n c are solved, i.e. where Round{..} denotes the rounding function; m c and n c are integers, β, λ * , u, v, c, and ψ are values that do not require high precision; The calibration algorithm of the positioning model is specifically: reading out from I is O I Pixel coordinates in the coordinate system And The relationship between them is Where w is the pixel size, (a, b) T For O C Pixel coordinates under the coordinate system; therefore, The positioning model is equivalent to a pinhole model with an optical center at (0, 0, γ * ) T , and a focal length of f. The pinhole model is equivalent to a virtual camera fixed on the system, whose coordinate system is O G = O C -(0, 0, γ * T The phantom camera is calibrated and P is solved by PnP or binocular triangulation through projective geometry techniques.​ 2. The point positioning method based on 3-D Moire effect according to claim 1, characterized in that: The form of the beacon includes a point light source, a laser, and a single-mode optical fiber.

3. The point positioning method based on 3-D Moire effect according to claim 1, characterized in that: The positioning spectrum method comprises a phase unwrapping calibration algorithm, which is used for precise positioning of phase ambiguity of a decoded periodic pattern; a positioning model of the positioning spectrum method comprises the calibration algorithm, which is used for obtaining internal parameters of a system for precise positioning; specifically, the calibration algorithm is used for phase unwrapping calibration of 3-D Moire to calibrate the system to calibrate system internal parameters; The calibration algorithm comprises three steps: image processing, quadratic surface fitting estimation of a periodic phase, and least squares method; In the image processing step, the Moire image is subjected to low-pass filtering to smooth the image to reduce noise of the signal, and histogram equalization of the image is performed to improve the contrast of the image to improve the signal-to-noise ratio of the image; In the step of quadratic surface fitting to estimate the periodic phase, a complete period of the 3-D Moire signal is regarded as a quadratic surface, and the sub-pixel position of the wave peak in the image is extracted by using the quadratic surface fitting, and the period u, v of the Moire and the corresponding phase are estimated and ψ; In the least squares method step, a basic model for solving periodic ambiguity is derived by projection geometry and positioning spectrum; in the model, unknown numbers of unwrapped phases to be solved are fixed and unchangeable, and are solved by the least squares method.

4. The point positioning method based on 3-D Moiré effect according to claim 3, characterized in that: The pinhole model with a periodic extended range calibrates the system by a chessboard calibration algorithm; specifically, based on a camera imaging principle, internal parameters and external parameters of the system are calculated by shooting calibration images at different angles to improve the precision and expand the application range.

5. A point location method based on 3-D Moiré effect according to claim 4, characterized in that: When shooting the image, the beacon is moved in space, and a moving track is a perfect chessboard array, which replaces a chessboard calibration plate of a traditional chessboard calibration algorithm.

6. The point positioning method based on 3-D Moire effect according to claim 1, characterized in that: In the point positioning system, a photolithography glass plate mask with masks on both sides is pasted on a bare camera sensor; Let the side close to the bare camera sensor be T1 and the other side be T2; the projection on the sensor through the beacon P generates Thus a Moire image S is generated from the Moire effect; the movement of S is observed from the movement of the beacon P, the spatial position of the beacon P is located based on the movement of S, so P is also called the target point; In the positioning spectrum method, a hovering theorem of an accompanying positioning spectrum is adopted; specifically, a mask plate is regarded as N transmission planes in a mask-based visual measurement system. Each of the transmission planes is taken at the point (p, p z ) T = (p x , p y , p z ) T The transmission projection with the projective center on the X-Y plane is denoted by The overall transmission T and the moire S formed thereon are Given a point q on S, the path of a ray from P to q is denoted as . Move P along Extending to infinity, i.e., P ∞ = (p, ∞) T That is, the projective transformation becomes the parallel transformation; The transformed transmittance is denoted by If the optical path from P to q remains unchanged, then T and have the following relationship Then, assuming a nearby optical path The same is true under projective transformations and parallel transformations, with wherein The above relationship is called the hang-up theorem, q is called the hang-up spectrum, the hang-up point is called the hang-up point, and the transmittance of the hang-up point is called the hang-up value.

7. The point positioning method based on 3-D Moire effect according to claim 5, characterized in that: In step two, the phase unwrapping calibration is carried out by using the Moire image to obtain the unwrapping phase m and n, including the following steps: Step A1: The Moire image is low-pass filtered to smooth the image and reduce the noise of the signal, and the histogram equalization of the image is carried out to improve the contrast of the image and improve the signal-to-noise ratio of the image, and the image coordinates c of the control points are detected; Step A2: Extract sub-pixel position of signal wave-peak in image using quadratic surface fitting, and estimate Moiré period u, v and corresponding phase and ψ; Step A3: Calibrate λ by least square method * , in order to calibrate λ * , attach a beacon on the one-dimensional grating ruler and move the beacon constantly to capture a series of There are: (β j -λ * )v T c j +(ψ j -n j ) = 0 If the absolute value of m j and n j cannot be determined, the image is taken successively to monitor its increase or decrease; thus, m j = m0+ Δm j and n j = n0+ Δn j , where m0and n0are unknown, and Δm j and Δn j are known; In the matrix form, there are In case of sufficient samples, λ is solved by least square algorithm * ; Step A4: Use the formula Least squares is used to solve for m and n.

8. The point positioning method based on 3-D Moire effect according to claim 5, characterized in that: The pinhole model with the periodic extension range is regarded as a virtual camera fixed on the system, that is, expressed by the formula O G = O C - (0, 0, γ * ) T , The virtual imaging plane is in the virtual camera sensor plane O G with the actual system camera sensor plane O C The distance between the virtual imaging plane to O G is called the focal length f of the pinhole model; then, using the projective geometry technique, calibrate the virtual camera and solve P by PnP or binocular triangulation; in particular: given a WxW sensor and the moving range of P beacon, in order to optimize the positioning resolution, the camera in the pinhole model needs to cover the range through its field of view range; 3-D Moiré model sets a large f concentrated in a small range; if the obtained Moiré pattern is periodic, then its period is extended by deblurring the phase ambiguity and then its extended range is used to cover other ranges; most of the upper resolution limit can be improved by R = (Wex / W) 2 where Wex is the side length of the extended range.

Images