Three-dimensional measurement method of mirror object and computer-readable storage medium

Abstract

A method for three-dimensional measurement of mirrored objects and a computer-readable storage medium, wherein the method comprises: calibrating the positional relationship between a camera, a projection device, and an object plane; obtaining a grating projection image of the object to be measured; obtaining the coordinates of the screen point and object point corresponding to each pixel in the camera and grating projection images of the object to be measured in the same coordinate system based on the grating projection image of the object to be measured and the positional relationship between the camera, the projection device, and the object plane; calculating the normal vector at the object point corresponding to each pixel based on the geometric relationship between the object point and the screen point corresponding to the pixel in the camera and grating projection images; calculating the gradient at the object point based on the normal vector at the object point; and obtaining the three-dimensional surface shape of the object to be measured based on the gradient at each object point. This method achieves non-destructive measurement of mirrored objects and improves the accuracy and efficiency of three-dimensional measurement.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a three-dimensional measurement method for a mirror object and a computer-readable storage medium. Background Art

[0002] Glass substrates, automotive glass, silicon wafers, etc., as representatives of mirror-reflective objects, are widely used in the production and manufacturing of various related products and have become an indispensable part of people's daily lives. Glass substrates used in flat-panel liquid crystal displays, automotive glass, and silicon wafers used for chip processing have strict requirements on surface flatness and processing accuracy during production, resulting in high production process requirements and high costs. During production and manufacturing, the light beam projected onto the surface of glass substrates, silicon wafers, etc. will undergo mirror reflection like a common mirror, which makes it difficult to non-destructively measure their surface shape during the manufacturing process. There is no doubt that research on surface shape measurement methods for mirror-reflective objects such as glass substrates, silicon wafers, and free-form glass can guide the fine processing and measurement processes during their production, and is of great significance to improving the production and processing efficiency and quality of glass substrates, silicon wafers, free-form glass, etc., and reducing their production costs.

[0003] Manual inspection involves visually inspecting mirrored surfaces under strong lighting conditions. This method is inefficient, poses health risks to workers, and significantly impacts individual performance. However, given the overall speed and accuracy of the quality inspection process, manual inspection is still the dominant method for quality inspection of mirrored surfaces in industrial environments. This situation urgently needs to change.

[0004] Three-dimensional measurement technology can be categorized as contact or non-contact based on the contact method. Coordinate measuring machines (CMMs) are traditional contact-based three-dimensional topography measurement technologies. They can measure objects with complex shapes with high accuracy, but because they require contact with the object's surface, measuring the height of each point can lead to slow measurement speeds, long measurement times, and surface wear. Optical three-dimensional measurement technologies, such as machine vision, have attracted widespread attention due to their non-contact, high-precision, fast measurement speeds, and full-field measurement capabilities, becoming a hot topic in academic research. With the advancement of digital signal processing technology and related devices, three-dimensional measurement technology will develop towards high speed and high precision, and the objects being measured will become larger and have microstructures. Machine vision is a promising inspection method that can overcome the inefficiency of manual inspection methods and often has a larger field of view. However, applying machine vision to three-dimensional measurement of mirror-surface objects still presents challenges. Summary of the Invention

[0005] The present invention provides a three-dimensional measurement method for a mirror object and a computer-readable storage medium, aiming to perform non-destructive three-dimensional measurement of the surface of a mirror object by using a machine vision method.

[0006] According to a first aspect, an embodiment provides a three-dimensional measurement method for a mirror object, which is applied to a three-dimensional measurement system. The three-dimensional measurement system includes a projection device, an object plane, and a camera. The three-dimensional measurement method includes:

[0007] Calibrate the positional relationship between the camera, the projection device, and the object plane;

[0008] Acquire a grating projection image of the object to be measured, wherein the object to be measured is placed on the object plane, and the grating projection image of the object to be measured is an image captured by the camera of the object to be measured when the projection device performs grating projection on the object to be measured;

[0009] Obtaining, based on the grating projection image of the object to be measured and the positional relationship between the camera, the projection device, and the object plane, the coordinates of the screen point and the object point corresponding to each pixel point in the grating projection images of the camera and the object to be measured in the same coordinate system, wherein the screen point is a point on the screen of the projection device, and the object point is a point on the object to be measured that reflects light emitted by the screen point and is imaged in the camera;

[0010] Calculating a normal vector at the object point corresponding to each pixel point based on a geometric relationship between the camera and the object point corresponding to the pixel point in the grating projection image and the screen point;

[0011] Calculate the gradient at the object point based on the normal vector at the object point;

[0012] The three-dimensional shape of the object to be measured is obtained according to the gradient at each object point.

[0013] According to a second aspect, an embodiment provides a computer-readable storage medium, on which a program is stored. The program can be executed by a processor to implement the three-dimensional measurement method as described in the first aspect above.

[0014] According to the above-described embodiment, the three-dimensional measurement method for mirror-surface objects first calibrates the positional relationship between the camera, the projection device, and the object plane. A grating projection image of the object to be measured is then acquired. Based on the grating projection image of the object to be measured and the positional relationship between the camera, the projection device, and the object plane, the coordinates of the screen point and the object point corresponding to each pixel in the camera and grating projection images of the object to be measured are obtained in the same coordinate system. Based on the geometric relationship between the object point and the screen point corresponding to the pixel in the camera and grating projection images, the normal vector at the object point corresponding to each pixel is calculated. The gradient at the object point is calculated based on the normal vector at the object point. Finally, the three-dimensional surface shape of the object to be measured is obtained based on the gradient at each object point, effectively improving the accuracy and efficiency of three-dimensional measurement. This method can overcome the difficulty of reconstructing the topography of mirror-surface objects by actively projecting a grating projection and processing the grating projection image to obtain the surface topography of the mirror-surface object, thereby achieving non-destructive measurement of the object's surface topography. Furthermore, the method is highly adaptable, capable of performing three-dimensional measurement on a variety of mirror-surface objects and having good adaptability to complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Schematic diagram of the structure of a three-dimensional measurement system according to an embodiment;

[0016] Figure 2 Schematic diagram of the transformation of each coordinate system in the pinhole camera model;

[0017] Figure 3 is a schematic diagram of a projected grating image in one embodiment;

[0018] Figure 4 A flowchart of a three-dimensional measurement method for a mirror object according to an embodiment;

[0019] Figure 5 A schematic diagram of a direction vector involved in solving a normal vector at an object point in one embodiment;

[0020] Figure 6 A flowchart of a method for calibrating positional relationships in a three-dimensional measurement system according to an embodiment;

[0021] Figure 7 The present invention is a flowchart of an embodiment of calibrating the intrinsic and extrinsic parameters of a camera based on a calibration plate image. DETAILED DESCRIPTION

[0022] The present invention will be further described in detail below by means of specific embodiments in conjunction with the accompanying drawings. Similar elements in different embodiments are numbered with associated similar elements. In the following embodiments, many detailed descriptions are provided to enable the present application to be better understood. However, those skilled in the art will readily appreciate that some of the features may be omitted in different circumstances, or may be replaced by other elements, materials, or methods. In some cases, some operations related to the present application are not shown or described in the specification. This is to avoid the core portion of the present application being overwhelmed by excessive descriptions, and for those skilled in the art, it is not necessary to describe these related operations in detail. They will fully understand the related operations based on the description in the specification and the general technical knowledge in the art.

[0023] In addition, the features, operations, or characteristics described in the specification may be combined in any appropriate manner to form various embodiments. Furthermore, the steps or actions in the method description may be reordered or adjusted in a manner readily apparent to those skilled in the art. Therefore, the various sequences in the specification and drawings are provided solely for the purpose of clearly describing a particular embodiment and are not intended to be mandatory, unless otherwise specified.

[0024] The serial numbers assigned to components herein, such as "first," "second," etc., are used solely to distinguish the objects being described and do not convey any sequential or technical meaning. References to "connection" and "coupling" herein, unless otherwise specified, include both direct and indirect connections (couplings).

[0025] In industrial production, 3D measurement or reconstruction of products is often necessary. Machine vision methods project objects and use their 2D images to restore 3D information (such as their surface shape), enabling non-destructive 3D measurement. This 3D information can be used to analyze product quality and provide guidance for the production and processing of products.

[0026] Objects with highly reflective or transparent surfaces are widely used in industrial production practices, such as the smooth body of a car and its rearview mirror, smooth glass surfaces, and glass substrates used for flat-panel displays. Light beams projected onto these surfaces will undergo specular reflection, so they are often referred to as specular objects. The specular reflections produced by specular objects make non-destructive measurement of their surface shape difficult during manufacturing. The following are the main issues with non-destructive testing of specular surfaces:

[0027] (1) It is difficult to present defects. It is impossible to present or photograph all surface defects from a certain direction.

[0028] (2) It is difficult to present two-dimensional images of some defects, especially objects with black backgrounds and low-contrast surfaces;

[0029] (3) Certain mirrors and mirror-like objects have unique high light transmittance and invisibility, which become the focus and difficulty in the field of optical three-dimensional detection.

[0030] Currently, machine vision is used to perform three-dimensional surface measurement of mirror objects, mainly using the interferometry method. The interferometry method is generally used to measure objects with regular surface shapes and has very high measurement accuracy. However, it usually requires complex and expensive compensating optical systems and a strictly stable environment, and has a limited measurement range and poor versatility.

[0031] Three-dimensional measurement of the surface of an object based on machine vision is often achieved through a three-dimensional measurement system. Please refer to Figure 1 In one embodiment, the three-dimensional measurement system includes a projection device 1, an object plane 2 and a camera 3. The projection device 1 can be any device that can emit light to form a pattern on the object plane 2, such as an LCD (Liquid Crystal Display), an LED (Light-Emitting Diode) display screen, etc. The object plane 2 is a platform for carrying the object to be measured, such as a stage, etc. The camera 3 can be a CCD (Charge-coupled Device) camera, a CMOS (Complementary Metal Oxide Semiconductor) camera, etc. Assuming that the object plane 2 produces mirror reflection, the light emitted by the projection device 1 is reflected by the mirror of the object plane 2 and then imaged on the imaging plane of the camera 3. Please refer to Figure 1 At this time, only the virtual image of the projection device 1 can be observed on the imaging plane of the camera 3, which is equivalent to a virtual projection device 1'. The projection device 1 and the projection device 1' are mirror-symmetrical about the object plane 2.

[0032] 3D measurement using this 3D measurement system can be performed using methods such as grating projection. This 3D measurement method projects a regular grating pattern onto the surface of an object, analyzes the resulting fringe image as a carrier of 3D information, and uses visual principles to obtain surface information, such as height.

[0033] Based on the above-mentioned 3D measurement system, the present invention provides a 3D measurement method for mirror-surface objects. This method uses a camera to capture and analyze images of the object to be measured, obtaining the object's 3D shape and achieving surface shape measurement or reconstruction. The entire technical solution mainly includes the following parts:

[0034] (1) Calibration of the relative position relationship between the camera, object plane, and projection device;

[0035] (2) Based on the calibrated position relationship, the normal vector is calculated;

[0036] (3) Based on the calculation results of the normal vector, the surface shape of the object is estimated.

[0037] In some embodiments of the present invention, corresponding improvements are made to the above parts for mirror objects, which can adapt to the situation of mirror reflection, more accurately measure the surface morphology of mirror objects, overcome the problem of difficulty in three-dimensional reconstruction of mirror objects, and realize fast and non-destructive three-dimensional measurement of mirror objects.

[0038] In order to more clearly understand the technical solution of the present invention, camera calibration and grating projection are first introduced below.

[0039] The purpose of camera calibration is to obtain the camera's intrinsic parameters, extrinsic parameters, and distortion coefficients. The main methods currently used for camera calibration are designed and calculated based on Zhang Zhengyou's calibration method, which mainly includes the following calculation steps:

[0040] (1) Obtain the homography matrix based on the correspondence between the world coordinates and image coordinates of the feature points in the calibration plate;

[0041] (2) Decomposing the homography matrix and calculating the initial parameters of the internal or external parameters;

[0042] (3) The LM (Levenberg-Marquardt) algorithm is used to perform nonlinear optimization on the initial parameters, and the internal parameters, external parameters and distortion coefficients are iteratively calculated to obtain the final calibration results.

[0043] The projection transformation relationship of each coordinate system in the imaging process of the camera can be expressed by the pinhole camera model, such as Figure 2 Point P in the World Coordinate System (WCS) w Project the point P on the imaging plane through the lens projection center to obtain point P w The image coordinates q projected onto the imaging plane i , you need to first convert it to the Camera Coordinate System (CCS). The x- and y-axes of the camera coordinate system are parallel to the c- and r-axes of the image, respectively. The z-axis is perpendicular to the imaging plane and is oriented so that all points in front of the camera have positive z coordinates. The c-axis of the image is horizontal, and the r-axis is vertical. Figure 2 Medium x c Axis, y caxis and z c The axes represent the x-axis, y-axis, and z-axis of the camera coordinate system. The transformation from the world coordinate system to the camera coordinate system can be expressed as c = c H w p w To express, where p c =(x c ,y c ,z c ) T is the coordinate in the camera coordinate system, p w =(x w ,y w ,z w ) T is the coordinate in the world coordinate system, c H w It can be represented by the rotation matrix R and the translation vector t.

[0044] After converting the world coordinate system to the camera coordinate system, it needs to be converted to the image plane coordinate system. This is a process of converting 3D coordinates to 2D coordinates. This transformation can be expressed as:

[0045]

[0046] Where f represents the focal length of the camera lens, (u,v) T Represents the coordinates in the image plane coordinate system.

[0047] After projection onto the imaging plane, the lens distortion will result in coordinates q c =(u,v) T changes, so that the coordinates formed on the imaging plane are distorted This change can be modeled solely on the imaging plane, meaning no three-dimensional information is required. For most lenses, their distortion can be adequately approximated as radial distortion. There are two common models for describing this distortion: the division model and the polynomial model. The division model is as follows:

[0048]

[0049] The parameter κ represents the magnitude of radial distortion. If κ is negative, it becomes barrel distortion, and if κ is positive, it becomes pincushion distortion. The distortion can be corrected using the following formula:

[0050]

[0051] The polynomial model is as follows:

[0052]

[0053] in k1, k2, k3, p1, p2 are model coefficients. According to the above model, when the distortion coefficients and the distorted coordinates are known, In the case of , the Newton method can be used to solve the undistorted coordinate q c =(u,v) T .

[0054] Finally, the image plane coordinate system is converted to the image coordinate system (ICS), which can be expressed as follows:

[0055]

[0056] where s x and s y are the pixel sizes of the camera in the horizontal and vertical directions, respectively. (c x ,c y ) is the principal optical axis point, generally the center of the image.

[0057] Therefore, the entire transformation above can be expressed as follows if the distortion is not considered:

[0058]

[0059] This is the mathematical model on which camera calibration is based. is the intrinsic parameter part of the camera, the rotation matrix R and the translation vector t are the extrinsic parameters. Further simplification can be expressed as:

[0060] sm=A[R|t]M, (2)

[0061] Where A is the intrinsic parameter matrix of the camera,

[0062] For grating projection, any existing grating projection method can be used. In one embodiment, Gray code images and phase shift images can be projected, see Figure 3 , is a Gray code image and a phase shift image with a width of 32 pixels projected in an embodiment, wherein the image sequence numbered 1-4 is the Gray code image, and the image sequence numbered 5-8 is the phase shift image.

[0063] Because the phase-shift image is periodic, the acquired phase is in the range [0, 2π] and needs to be converted to an absolute phase of 2kπ (k is an integer). After obtaining the absolute phase map, combined with the calibrated positional relationship, 3D data can be generated.

[0064] The phase shift method is widely used in optical measurement. Due to its high measurement accuracy and speed, it is generally used for high-precision three-dimensional measurement of objects. In the phase shift method, the process of obtaining the phase is as follows: (1) First, a sawtooth phase value is obtained using the phase shift formula. The range is [-π, π], which is called the truncated phase; (2) The sawtooth phase value is restored to a continuous phase value, called the absolute phase. This process is called phase solution (or phase unwrapping, phase unwrapping).

[0065] There are many ways to obtain the phase value using the phase shift method. For example, the N-step phase shift method can be used. If the projected light intensity is a standard cosine distribution, the phase shift image moves 2π / N phase each time, generating a new light intensity function I n (x, y), and translate N-1 times to obtain N phase-shifted images, where (x, y) are the coordinates of the pixel in the phase-shifted image. The four-step phase-shift method is commonly used because it can eliminate the nonlinear effects of the detector. The four-step phase-shift method shifts the projected phase-shifted image three times by π / 2 each time. Figure 3 The phase shift image shown is a four-step phase shift image. The intensity function of the four-step phase shift can be expressed as:

[0066]

[0067] Among them I i (i=1, 2, 3, 4) is the fringe grayscale value of the i-th phase-shifted image, I′(x, y) is the background value of the fringe light intensity, that is, the original light intensity emitted by the projection device 1, and I″(x, y) is the modulated light intensity value. is the desired phase value. We can get:

[0068]

[0069] The phase calculated by the phase shift method is only the principal phase value, which contains an inverse tangent function with a range of [-π,π] and is discontinuous. To address this issue, since there is a 2kπ difference between the phase obtained by the phase shift method formula and the true value, k must be calculated to restore the principal phase value to the true absolute phase. Therefore, the complete phase value, that is, the absolute phase formula, should be:

[0070]

[0071] Where k(x,y) is an integer, representing the integer multiple of 2π corresponding to the pixel point (x,y). It can be seen that the key to phase resolution is to determine the decoding period k(x,y). In practical applications, k(x,y) represents the number of cycles of the grating stripe pattern where the pixel point (x,y) is located, that is, which stripe in the grating stripe field the pixel point (x,y) belongs to and in which period the stripe is located. For example, please refer to Figure 3, we can see that the phase-shifted image is arranged periodically, with pixels 0-3 in the first period, pixels 4-7 in the second period, and so on. The number of periods of the grating stripes where a pixel is located can be obtained based on the Gray code information of the grating stripes where it is located.

[0072] The following is an introduction to the three-dimensional measurement method of the mirror object of the present invention. Please refer to Figure 4 In one embodiment, the method includes steps 100 to 600, which are described in detail below.

[0073] Step 100: Calibrate the positional relationship between the camera, the projection device, and the object plane.

[0074] In use Figure 1 Before performing 3D measurement of an object, the illustrated 3D measurement system requires calibrating the positional relationship between the projection device 1, object plane 2, and camera 3. This facilitates subsequent conversion of points on the projection device 1, object plane 2, and camera 3 to the same coordinate system for calculation. Calibration can be performed based on the distance between the two devices and their relative positions (e.g., rotation, translation, etc.).

[0075] Calibration of positional relationships affects the accuracy of subsequent three-dimensional measurements and is a very important step. In one embodiment of the present invention, a method for calibrating positional relationships in the above-mentioned three-dimensional measurement system is provided, wherein the positional relationships are represented by the transformation relationships of coordinate systems, including the transformation relationships between the projection device coordinate system and the camera coordinate system, and the transformation relationships between the object plane coordinate system and the camera coordinate system. Please refer to Figure 1 , the projection device coordinate system is a spatial coordinate system established on the projection device 1, the object plane coordinate system is a spatial coordinate system established on the object plane 2, and the camera coordinate system is a spatial coordinate system established on the camera 3. In order to adapt to mirror reflection, the present invention also introduces a virtual space coordinate system, and calibration is performed with the help of the virtual space coordinate system. The virtual space coordinate system is a spatial coordinate system established on the virtual projection device 1′, that is, the virtual space coordinate system is a spatial coordinate system obtained by mirror symmetry of the projection device coordinate system about the object plane 2. The camera coordinate system is denoted as c, the object plane coordinate system is denoted as w, the projection device coordinate system is denoted as s, and the virtual space coordinate system is denoted as v. Figure 1 The origin position and the direction of the coordinate axis of the central coordinate system are for reference only. In practice, the origin position and the direction of the coordinate axis can be set according to specific needs and are not limited here.

[0076] To determine the positional relationship between the camera and the object plane, the object plane coordinate system can be used as the world coordinate system to calibrate the camera's internal and external parameters. The camera's external parameters represent the transformation relationship between the world coordinate system and the camera coordinate system, thus obtaining the transformation relationship between the object plane coordinate system and the camera coordinate system. To determine the positional relationship between the camera and the projection device, grating projection can be used for calibration. The correspondence between pixels and screen points is obtained from the grating projection image, and the transformation relationship between the projection device coordinate system and the camera coordinate system is then determined. A screen point refers to a point on the projection device's screen.

[0077] The calibration method of the present invention will be described in detail below.

[0078] Step 200: Acquire a grating projection image of the object to be measured.

[0079] Since the positional relationship between the camera and the object plane has been calibrated, the object to be measured is placed on the object plane and calculations are performed using this calibrated positional relationship. A projection device projects a grating fringe pattern onto the object placed on the object plane, and the camera captures the image, producing a grating projection image of the object. The grating fringe patterns, when projected onto the surface of the object, are modulated by the height of the object, resulting in a grating projection image that reflects the surface topography of the object.

[0080] In one embodiment, in order to eliminate interference from overly bright areas on the object to be measured and reduce the amount of data, the original image captured by the camera can be cropped to remove areas with excessively large grayscale values. Specifically, the projection device is first used to perform grating projection on the object to be measured according to the four-step phase shift method, and four grating projection images I1(x,y), I2(x,y), I3(x,y), and I4(x,y) are obtained in sequence, where (x,y) represents the coordinates of the pixel points; according to the surface grayscale modulation function k(x,y) = [I1(x,y) - I3(x,y)] 2 +[I2(x,y)-I4(x,y)] 2 Obtain the modulated grayscale image k(x,y); then obtain the area in the grayscale image k(x,y) where the grayscale value is greater than the preset grayscale threshold as the area to be detected, and use the part of the area to be detected in the grayscale image k(x,y) as the grating projection image of the object to be detected.

[0081] In one embodiment, the grating projection image of the object to be measured may include a grating projection image of the object to be measured in the X direction and a grating projection image of the object to be measured in the Y direction. The projection device displays grating stripes in the X direction (i.e., the horizontal direction) on its screen, and the grating projection image obtained by the camera photographing the object to be measured is referred to as the grating projection image of the object to be measured in the X direction. The projection device displays grating stripes in the Y direction (i.e., the vertical direction) on its screen, and the grating projection image obtained by the camera photographing the object to be measured is referred to as the grating projection image of the object to be measured in the Y direction.

[0082] Step 300: According to the grating projection image of the object to be measured and the positional relationship between the camera, the projection device and the object plane, the coordinates of the screen points and the object points corresponding to each pixel point in the camera and the grating projection image of the object to be measured in the same coordinate system are obtained.

[0083] An object point is the point on the object being measured that reflects light from a screen point and forms an image in the camera. The relationship between pixel points and their corresponding object and screen points is as follows: if light from screen point A is reflected by object point B and forms an image on the camera as pixel C, screen point A is called the screen point corresponding to pixel C, and object point B is called the object point corresponding to pixel C.

[0084] Since the positional relationship between the camera, projection device, and object plane has been calibrated in step 100, the camera, object points on the object to be measured, and screen points on the projection device can be converted to the same coordinate system based on the calibrated positional relationship for coordinate representation. This same coordinate system can be the camera coordinate system, the object plane coordinate system, or another pre-defined coordinate system. Using the grating projection image, the corresponding screen point of each pixel can be determined. The camera's coordinates can be represented by the coordinates of the optical center of its lens.

[0085] In one embodiment, the coordinates of the camera, object points and screen points in the world coordinate system can be obtained. When the object plane coordinate system is used as the world coordinate system, it is equivalent to obtaining the coordinates of the camera, object points and screen points in the object plane coordinate system.

[0086] Specifically, the grating projection image of the object to be measured is first subjected to phase deconvolution to obtain the phase information of each pixel, and the screen point coordinates (x s ,y s ), where the screen point coordinates refer to the two-dimensional coordinates of the screen point on the screen. For phase resolution, please refer to the above introduction on phase resolution or the prior art. If the grating projection image of the object to be measured includes the grating projection image of the object to be measured in the X direction and the grating projection image of the object to be measured in the Y direction, then for each pixel point, the phase of the grating projection image of the object to be measured in the X direction can be resolved to obtain a phase, called the X phase, and the phase of the grating projection image of the object to be measured in the Y direction can also be resolved to obtain a phase, called the Y phase. The X phase and Y phase can be used to resolve the complete screen point coordinates (x s ,y s Specifically, the grating projection image of the object to be measured in the X direction is first phase-processed to obtain the X phase of each pixel. Perform phase de-phase processing on the grating projection image of the object in the Y direction to obtain the Y phase of each pixel Then for each pixel, according to its X phase and Y phase Calculate the corresponding screen point coordinates (x s ,y s ), the calculation formula is as follows:

[0087]

[0088] Where T x Indicates the number of cycles of the grating stripes where the grating projection image of the pixel point in the X direction is located, T y Indicates the number of cycles of the grating stripes where the grating projection image of the pixel point in the Y direction is located.

[0089] Then, according to the transformation relationship between the projection device coordinate system and the camera coordinate system and the external parameters of the camera, the coordinate p s =(x s ,y s ,0) is transformed into the world coordinate system as the world coordinate of the screen point. First, according to the transformation relationship between the projection device coordinate system and the camera coordinate system, the coordinate p is transformed into the world coordinate of the screen point. s =(x s ,y s ,0) is transformed into the camera coordinate system, and then transformed into the world coordinate system according to the camera's external parameters.

[0090] Then, based on the camera’s intrinsic and extrinsic parameters, the world coordinates of the corresponding object point are obtained from the image coordinates of each pixel point in the grating projection image of the object to be measured. For details, please refer to formula (1).

[0091] Step 400: Calculate the normal vector at the object point corresponding to each pixel point based on the geometric relationship between the object point and the screen point corresponding to the pixel point in the camera and grating projection image.

[0092] The geometric relationship may be a geometric relationship between points, lines, and / or vectors, etc. Since the coordinates of the camera, object points, and screen points in the same coordinate system have been obtained, it is easy to calculate the geometric relationship based on their coordinates.

[0093] In one embodiment, the normal vector can be solved based on the direction vector between the camera, the object point and the screen point. Figure 5 For each pixel in the grating projection image, the direction vector l between the camera and the screen point can be calculated based on the coordinates of the camera, the object point corresponding to the pixel, and the coordinates of the screen point. ij and the direction vector s between the camera and the object point ij , and the direction vector r between the object point and the screen point ij From the geometric relationship, we can get The subscripts i and j represent the image coordinates of the pixel, i represents the row coordinate, j represents the column coordinate, and nij Represents the normal vector at the object point corresponding to the pixel point (i, j). Considering the mirror symmetry of the screen point, the following equation can be obtained from the geometric relationship using mirror symmetry:

[0094]

[0095] where ρ ij Represents the distance between the object point and the screen point, σ ij Indicates the distance between the camera and the object point. ij , σ ij and n ij is an unknown quantity, since the normal vector n ij Contains three dimensions, so there are five unknowns in total. According to the above formula, only three equations can be constructed from the three dimensions of the vector, so it is an under-constrained problem. In order to solve this problem, the present invention converts the under-constrained problem into a solvable problem by adding two constraints to realize the normal vector n ij Specifically, the present invention introduces two additional intermediate variables a ij and b ij , respectively expressed as:

[0096] a ij =s ij +r ij =s ij +(l ij -σ ij s ij ) / ||l ij -σ ij s ij ||,

[0097] b ij =l ij ×σ ij s ij .

[0098] According to the above formula, we can know that:

[0099]

[0100] For a ij and n ij have:

[0101]

[0102] Therefore a ij 、b ij and n ij They are orthogonal to each other and can be used as the three axes of a three-dimensional coordinate system, such as Figure 5 As shown, a ij and bij This is equivalent to the expansion of the plane where the object point corresponding to the pixel point (i, j) is located. Equation (3) can be used as two additional constraints, together with Equation (2) to form five equations, thereby solving the normal vector n ij .

[0103] Step 500: Calculate the gradient at the object point according to the normal vector at the object point.

[0104] After obtaining the normal vector, the surface morphology of the object to be measured can be estimated based on the normal vector. ij and direction vector s ij , as well as the camera's internal and external parameters, can be used to calculate the depth information at the object point. Based on this depth information and the normal vector, the surface shape of the object to be measured can be estimated. However, due to the limitations of the camera's own calibration accuracy and noise interference during the measurement process, the calculated depth information is often of limited accuracy. Compared to depth data, the gradient data obtained from the normal vector often has a very small error. Therefore, the present invention abandons the method of estimating surface shape using depth data in 3D measurement, and instead uses gradient data to reconstruct the 3D surface shape of the object to be measured through integration.

[0105] In this step, the gradient at the object point is first calculated based on the normal vector at the object point. If the coordinates are expressed in the object plane coordinate system, the gradient includes an X component and a Y component. In one embodiment, for each pixel point, the X component and Y component of the gradient at the object point can be calculated according to the following formula, thereby obtaining the gradient at the object point:

[0106]

[0107] in Represents the X component of the gradient at the object point corresponding to the pixel point (i, j), Represents the Y component of the gradient at the object point corresponding to the pixel point (i, j), n x (i, j) represents the X component of the normal vector at the object point corresponding to the pixel point (i, j), n y (i, j) represents the Y component of the normal vector at the object point corresponding to the pixel point (i, j), n z (i, j) represents the Z component of the normal vector at the object point corresponding to the pixel point (i, j).

[0108] Step 600: Obtain the three-dimensional shape of the object to be measured according to the gradient at each object point.

[0109] The 3D shape information can include the height of the object surface, which, when expressed in the object's plane coordinate system, is the Z coordinate of the object point. Estimating the 3D shape of the object to be measured can be considered as determining the height of the object point.

[0110] When using gradient data for surface reconstruction, a global integration process can be used. A point near the center of the area to be integrated can be selected as the starting point for integration. Two integration baselines can be drawn along two perpendicular directions. Line integration is performed along these two integration baselines to obtain the height value. If the coordinates are expressed in the object plane coordinate system, it can be expressed as:

[0111] z(x,y)=z(x0,y0)+∫ L F x (x,y)dx+F y (x,y)dy,

[0112] Where z(x,y) is the height value obtained by integration, z(x0,y0) is the height value of the integration starting point (x0,y0), F x (x,y),F y (x, y) are the gradient values ​​in two perpendicular directions calculated by the normal vector, and L is the actual integration path.

[0113] This invention improves the gradient integral and obtains more accurate height values ​​through a cyclic iteration. Assuming that the slope of the object to be measured is well consistent with the reconstructed height, the three-dimensional surface height value can be accurately reconstructed from the gradient data. Based on the above gradient integral, the gradient and height of the object point corresponding to adjacent pixels can be expressed as:

[0114]

[0115] where x i,j Indicates the X coordinate of the object point corresponding to the pixel point (i, j), i,j Indicates the Y coordinate of the object point corresponding to the pixel point (i, j), Z i,j represents the height value of the object point corresponding to the pixel point (i, j), M represents the number of rows of the grating projection image of the object to be measured, and N represents the number of columns of the grating projection image of the object to be measured.

[0116] In one embodiment of the present invention, the height value is calculated by a loop iteration method according to the above formula, specifically including a first calculation step and a second calculation step. In the first calculation step, the height value at each object point is calculated according to formula (4). By sorting out formula (4), the expression DZ=G can be obtained, where D is a coefficient matrix, which is a sparse matrix of size [(M-1)×N+M×(N-1)]×MN, specifically

[0117]

[0118] Z is the height matrix to be solved with a size of MN×1, specifically Z=[z 1,1 ,z2,1 ,…,z M-1,N ,z M,N ] T ; G is the measurement gradient matrix of size [(M-1)×N+M×(N-1)]×1, specifically

[0119]

[0120] Further sorting, we can solve it according to the least square method: Z=(D T D) -1 D T G.

[0121] In the second calculation step, for each object point, its gradient is updated according to the following formula:

[0122]

[0123] Among them F x represents the X component of the gradient at the object point, F y Represents the Y component of the gradient at the object point, x c Indicates the X coordinate of the camera, y c Indicates the Y coordinate of the camera, z c Indicates the Z coordinate of the camera, x h Indicates the X coordinate of the object point, y h Indicates the Y coordinate of the object point, Z h Indicates the height value of the object point, x s Indicates the X coordinate of the screen point corresponding to the object point, y s Indicates the Y coordinate of the screen point corresponding to the object point, z s Indicates the Z coordinate of the screen point corresponding to the object point, d h2s Indicates the distance between the object point and the screen point, d h2c Indicates the distance between the object point and the camera. h is the output result of the first calculation step, that is, the height value obtained in the first calculation step, d h2s d h2c 、x c 、y c 、z c 、x s 、y s 、z s It can be obtained based on the calibration results of the position relationship.

[0124] The first and second calculation steps are repeated until a preset stopping condition is met. The height value obtained when the calculation stops is used as the height measurement result of the object point, completing the three-dimensional measurement of the object to be measured. The preset stopping condition may be that the calculation reaches a preset number of times, the difference between the current calculation result and the previous calculation result is less than a preset threshold, etc.

[0125] The above embodiment only considers the first-order effect when calculating the height value, and does not consider the influence of higher-order noise components. However, in actual industrial production, the measurement environment is often relatively harsh, and the collected gradient data contains various noises, and there may be cases where gradient data is missing. Therefore, higher-order data can be used for processing. By establishing constraints on the position and gradient of the object point corresponding to four adjacent pixels in a row, we can obtain:

[0126]

[0127] In another embodiment of the present invention, according to the above formula, the height value is calculated by a loop iteration method, specifically including a third calculation step and a second calculation step. In the third calculation step, the height value at each object point is calculated according to formula (5). In the second calculation step, the gradient value is updated according to the calculated height value. The second calculation step here is the same as above and will not be repeated here. The third calculation step and the second calculation step are repeated until the preset stop condition is reached, and the height value obtained when the calculation is stopped is used as the measurement result of the height value at the object point to complete the three-dimensional measurement of the object to be measured. The preset stop condition can be that the calculation reaches a preset number of times, the difference between the result of this calculation and the result of the previous calculation is less than a preset threshold, etc.

[0128] The calculation process described above often places high demands on the integrity of the gradient data. However, in actual measurements, the objects being measured often have irregular shapes, making it difficult to ensure that the acquired gradient data has a good geometric boundary, often resulting in invalid or missing data. Therefore, in one embodiment, invalid or missing gradient data can be filled using the average value of valid gradient data within a preset range.

[0129] According to the above-described embodiment, the three-dimensional measurement method for mirror-surface objects calibrates the positional relationship between the camera, the projection device, and the object plane, obtains a grating projection image of the object to be measured, and then obtains the coordinates of the screen point and object point corresponding to each pixel in the grating projection image of the object to be measured and the positional relationship between the camera, the projection device, and the object plane. Based on the grating projection image of the object to be measured and the positional relationship between the camera, the projection device, and the object plane, the coordinates of the screen point and the object point corresponding to each pixel in the grating projection image of the camera and the object to be measured in the same coordinate system are obtained. Based on the geometric relationship between the object point and the screen point corresponding to the pixel in the grating projection image of the camera and the object to be measured, the normal vector at the object point corresponding to each pixel is calculated. Then, the gradient at the object point is calculated based on the normal vector at the object point. Finally, the three-dimensional surface shape of the object to be measured is obtained based on the gradient at each object point, which can effectively improve the accuracy and efficiency of three-dimensional measurement. This method can overcome the difficulty of reconstructing the topography of mirror-surface objects. By actively projecting a grating projection and processing the grating projection image, the surface topography of the mirror-surface object is obtained, achieving non-destructive measurement of the object surface topography without the need for complex and expensive compensation optical systems. It has strong adaptability, can perform three-dimensional measurement on a variety of mirror-surface objects, and has good adaptability to complex environments.

[0130] In one embodiment, when calibrating the positional relationship between the camera, the projection device, and the object plane, a virtual space coordinate system that is mirror-symmetrical to the projection device coordinate system is introduced. Calibration with the help of the virtual space coordinate system can adapt to the situation of mirror reflection, thereby providing a good foundation for the three-dimensional measurement of mirror objects.

[0131] In one embodiment, when calculating the normal vector based on the geometric relationship, two new vectors are constructed and the normal vector is tensor-expanded, thereby adding constraints on the normal vector and converting the under-constrained problem into a solvable problem, thereby achieving the smooth solution of the normal vector of the mirror object surface.

[0132] In one embodiment, the gradient integral is improved. Based on slope consistency, the gradient is used to iteratively calculate the 3D shape of the object to be measured, thereby quickly and accurately estimating the object's 3D shape. In another embodiment, higher-order data is used for 3D shape estimation, suppressing the influence of higher-order noise.

[0133] The following describes the calibration method of the position relationship in the three-dimensional measurement system of the present invention. Figure 6 In one embodiment, the method includes steps 110 to 160, which are described in detail below.

[0134] Step 110: Acquire a calibration plate image of the object plane.

[0135] The calibration plate image of the object plane is the image captured by camera 3 of object plane 2 when projection device 1 projects the calibration plate pattern onto object plane 2. The calibration plate image can be a checkerboard pattern, a circular array pattern, or other similar pattern. To prevent specular reflection from affecting the calibration of camera intrinsic and extrinsic parameters, a piece of white paper can be placed on object plane 2 so that the calibration plate pattern is projected onto the paper to avoid specular reflection.

[0136] Step 120: Using the object plane coordinate system as the world coordinate system, calibrate the camera's intrinsic and extrinsic parameters based on the calibration plate image. Since the extrinsic parameters represent the transformation relationship between the world coordinate system and the camera coordinate system, obtaining the extrinsic parameters also obtains the transformation relationship between the object plane coordinate system and the camera coordinate system.

[0137] Please refer to Figure 7 In one embodiment, step 120 includes steps 121 to 124.

[0138] Step 121: Acquire feature points in the calibration plate image, as well as the image coordinates and corresponding world coordinates of the feature points.

[0139] For a checkerboard, a feature point is a checkerboard corner. For a circular array, a feature point is the center of gravity of a circular feature point within the circular array. A circular feature point is a circular pattern within the circular array. A world coordinate system can be constructed based on the calibration plate pattern's parameter information to obtain the world coordinates corresponding to the feature point. The calibration plate pattern's parameter information may include the calibration plate's dimensions, the checkerboard's dimensions, the radius of the circular feature point, and the spacing between feature points. Acquiring feature points and their image coordinates in the calibration plate image can be accomplished using existing techniques and will not be elaborated upon here.

[0140] Step 122: Calculate the homography matrix based on the image coordinates of the feature points and the corresponding world coordinates. It can be understood that the image coordinates p of multiple feature points are used. i and the corresponding world coordinate p w , and the transformation relationship between image coordinates and world coordinates p i =Hp w , establish the objective function: min∑[p i -Hp w ] 2 , the homography matrix H can be calculated using the least squares method, and the elements in the homography matrix H are represented by h0, h1, h2, h3, h4, h5, h6, h7 and h8, then

[0141] Step 123: Calculate the camera intrinsic parameters using the homography matrix H according to the constraint relationship between the homography matrix H and the camera intrinsic parameters.

[0142] Let the equivalent focal length f x =f / sx , f y =f / s y , then the internal reference part can be expressed as When establishing the world coordinate system, it is usually assumed that the points on the calibration plate are located on the plane z = 0, so the rotation and translation in the z direction can be ignored. Therefore, the external parameter part can be expressed as Where r1, r2, r3, r4, r5 and r6 are the elements of the rotation matrix, t x and t y are the x and y components of the translation vector respectively. So we have

[0143]

[0144] If the origin of the image coordinate system is set to the center of the image, then we can get

[0145]

[0146] From the orthogonality constraints of each vector in the rotation matrix, we can get:

[0147]

[0148] According to the orthogonality and unit constraints, the constraint relationship between the homography matrix H and the camera intrinsic parameters can be obtained:

[0149]

[0150] in According to the above constraints, the equivalent focal length f can be calculated from the homography matrix H: x and f y , camera principal optical axis point coordinates (c x ,c y ) can be found in the camera manual.

[0151] Step 124: Calculate the rotation matrix R and the translation vector t according to the homography matrix H.

[0152] From formula (2), we know that H = A[R|t]. According to orthogonality, we can obtain:

[0153] H=[h1 h2 h3]=A[r1 r2 t],

[0154] Where [r1 r2 t] = [R|t], h1 is the first column vector of the homography matrix H, h2 is the second column vector of the homography matrix H, h3 is the third column vector of the homography matrix H, r1 is the first column vector of the rotation matrix R, and r2 is the second column vector of the rotation matrix R. The parameter matrix A can be calculated according to the following constraints:

[0155]

[0156] According to r1=A -1 h1, r2 = A -1 h2 calculates vectors r1 and r2, then the rotation matrix R = [r1 r2], according to t = A -1 h3 calculates the translation vector t, thereby obtaining the external parameter part.

[0157] Then the transformation relationship between the object plane coordinate system and the camera coordinate system can be expressed by the rotation matrix R and the translation vector t, which can be recorded as and

[0158] Step 130: Transform the object plane into several postures, obtain a calibration plate image or a grating projection image of the object plane in each posture, obtain the coordinates of the pixel points in the calibration plate image or the grating projection image in the camera coordinate system, and the coordinates of the corresponding points in the virtual space coordinate system.

[0159] To achieve more accurate calibration results, the present invention calibrates the transformation relationship between the projection device coordinate system and the camera coordinate system by randomly transforming object plane 2 into multiple poses. Calibration is also performed using a virtual space coordinate system to accommodate specular reflections. In each pose, projection device 1 projects a calibration plate pattern or a grating fringe pattern onto object plane 2, which is captured by camera 3 to obtain an image of the calibration plate or the grating projection of object plane 2.

[0160] As can be seen above, the image coordinates can be transformed into the camera coordinate system using the camera's intrinsic parameters. Therefore, the coordinates of the pixel in the calibration plate image or grating projection image can be obtained based on the image coordinates of the pixel and the camera's intrinsic parameters. The pixel in the calibration plate image can specifically be a feature point in the calibration plate image.

[0161] The point in the virtual space coordinate system corresponding to the pixel point is the point in the virtual space coordinate system observed from the pixel point. For the calibration plate image, the world coordinates corresponding to the pixel point in the calibration plate image can be obtained as the coordinates of the point in the corresponding virtual space coordinate system. The world coordinates corresponding to the pixel point can be obtained by transforming the image coordinates of the pixel point through the intrinsic and extrinsic parameters of the camera. For the grating projection image, the grating projection image is subjected to phase deconvolution to obtain the phase information of the pixel point in the grating projection image. Then, the screen point coordinates (x s ,y s ); Set the coordinate p s =(x s ,y s,0) as the coordinates of the point in the virtual space coordinate system corresponding to the pixel point in the grating projection image. The phase resolution process can refer to the above introduction on phase resolution or existing technology. s ,y s ) can be obtained by referring to step 300, which will not be repeated here.

[0162] Step 140: For each posture of the object plane, calculate the transformation relationship between the virtual space coordinate system and the camera coordinate system under this posture based on the coordinates of the pixel points in the calibration plate image or grating projection image under this posture in the camera coordinate system and the coordinates of the corresponding points in the virtual space coordinate system.

[0163] Using the coordinates of multiple sets of pixel points in the camera coordinate system and the coordinates of the corresponding points in the virtual space coordinate system, the transformation relationship between the virtual space coordinate system and the camera coordinate system can be calculated. In one embodiment, the transformation relationship between the virtual space coordinate system and the camera coordinate system can be expressed as a rotation matrix. and translation vectors To express it, suppose the point in the virtual space coordinate system can be expressed as The point in the camera coordinate system can be expressed as Then the transformation relationship can be established as:

[0164]

[0165] Where k represents the pixel number, m represents the mth posture, and m = 1, 2...N c , N c Indicates the number of postures, Indicates the coordinates of the kth pixel in the camera coordinate system under the mth posture, Represents the coordinates of the point in the virtual space coordinate system corresponding to the k-th pixel in the m-th posture, represents the rotation matrix between the virtual space coordinate system and the camera coordinate system under the m-th posture, Represents the translation vector between the virtual space coordinate system and the camera coordinate system in the mth posture.

[0166] According to the above formula, the following objective function can be established, and the least squares method can be used to calculate the transformation relationship between the virtual space coordinate system and the camera coordinate system in each posture:

[0167]

[0168] Where N represents the number of pixels in the calibration plate image or grating projection image.

[0169] Step 150: Calculate the transformation relationship between the projection device coordinate system and the camera coordinate system based on the geometric relationship between the virtual space coordinate system and the projection device coordinate system and the transformation relationship between the virtual space coordinate system and the camera coordinate system in all postures.

[0170] In one embodiment, the transformation relationship between the projection device coordinate system and the camera coordinate system can be expressed by a rotation matrix and translation vectors Please refer to Figure 1 For a point p on the projection device, it is imaged in the camera after being reflected by a point on the object plane, and its mirror point on the virtual projection device is p′. According to the geometric relationship, the transformation from the virtual space coordinate system to the camera coordinate system can be expressed as:

[0171]

[0172] Where I is the unit matrix, n is the normal vector at the reflection point on the object plane, and d is the distance from the camera to the object plane, specifically the distance from the optical center of the camera lens to the object plane.

[0173] When using the rotation matrix and translation vectors To express the transformation from the virtual space coordinate system to the camera coordinate system, we can get:

[0174]

[0175] In one embodiment of the present invention, the rotation matrix is ​​used according to the formula and translation vectors Rotation Matrix and translation vectors Perform calibration.

[0176] In order to balance calculation accuracy and efficiency, the object plane can be transformed into three postures in step 130, and the transformation relationship between the virtual space coordinate system and the camera coordinate system under these three postures can be used to calculate the transformation relationship between the projection device coordinate system and the camera coordinate system.

[0177] For any p,q∈{1,2,3}, define m p,q =n p ×n q , where n p Represents the normal vector of the object plane in the p-th posture, since Established, so m p,q is a matrix The eigenvector corresponding to the minimum eigenvalue obtained after singular value decomposition. p,q =n p ×n q Can be obtained by m p,qGet the normal vector n p .

[0178] Therefore, in step 150 of one embodiment of the present invention, first, for any matrix p,q∈{1,2,3} Perform singular value decomposition to obtain the eigenvector m corresponding to the minimum eigenvalue p,q ; Then calculate the normal vector n according to the following formula p :

[0179]

[0180] According to the formula The rotation matrix of the projection device coordinate system and the camera coordinate system can be calculated Here n p is the normal vector in any posture, is the rotation matrix between the virtual space coordinate system and the camera coordinate system under this posture;

[0181] Finally, the translation vector between the projection device coordinate system and the camera coordinate system can be calculated according to the following formula:

[0182]

[0183] Where d1, d2 and d3 are the distances from the camera to the object plane in the three postures. The above formula can be expressed as Ax = b, where So the solution is x=(A T A) - 1 A T b.

[0184] At this point, the transformation relationship between the projection device coordinate system and the camera coordinate system is obtained.

[0185] Step 160: Globally optimize the transformation relationship between the projection device coordinate system and the camera coordinate system according to a preset global optimization objective function.

[0186] Since the transformation relationship between the projection device coordinate system and the camera coordinate system obtained in step 150 may be only a local optimal solution, this step performs global optimization to search for the global optimal solution and improve the accuracy of calibration. Global optimization mainly performs iterative optimization of parameters based on a preset global optimization objective function, and the initial value of the iteration is the transformation relationship between the projection device coordinate system and the camera coordinate system obtained in step 150. The global optimization objective function can be designed based on the idea of ​​minimizing the error between the actual value and the estimated value. In one embodiment, if the rotation matrix is ​​used and translation vectors To express the transformation relationship between the projection device coordinate system and the camera coordinate system, the global optimization objective function can be:

[0187]

[0188] where x mk Represents the coordinates of the kth pixel of the calibration plate image or grating projection image of the object plane in the mth posture in the camera coordinate system, N c Indicates the number of postures, z mk =f(p′ mk ), v m =d m n m , n m represents the normal vector of the object plane in the mth posture, d m Indicates the distance between the object plane and the camera in the mth posture, p mk Represents x mk The coordinates of the corresponding point in the projection device coordinate system, p′ mk is the p observed on the imaging plane of the camera in the camera coordinate system mk The coordinates of the virtual image point, z mk Represents point p′ mk The coordinates obtained by the camera's nonlinear mapping f are: is a vector consisting of the parameters to be optimized.

[0189] where x mk It can be obtained based on the image coordinates of the pixel points in the calibration plate image or grating projection image and the intrinsic parameters of the camera. mk It can be obtained based on the transformation relationship between the projection device coordinate system and the camera coordinate system, that is, based on the rotation matrix and translation vectors Get. Click p mk Since the mirror reflection of the object plane forms a virtual image on the camera imaging plane, the real point p cannot be seen from the camera imaging plane. mk , but its virtual image point p′ mk , virtual image point p′ mk With point p mk The p′ calculated here is mirror symmetric about the object plane. mk is the coordinate transformed into the camera coordinate system. The nonlinear mapping f refers to the distortion of the camera lens.

[0190] By iteratively optimizing the parameters to be optimized according to the above global optimization objective function, the final transformation relationship between the projection device coordinate system and the camera coordinate system can be obtained, thereby completing the calibration of the position relationship in the three-dimensional measurement system.

[0191] The method for calibrating positional relationships in a three-dimensional measurement system according to the above-described embodiment is used to calibrate the positional relationships between a projection device, an object plane, and a camera in the three-dimensional measurement system, wherein the positional relationships are represented by the transformation relationships between the projection device coordinate system and the camera coordinate system, and the transformation relationships between the object plane coordinate system and the camera coordinate system. A virtual space coordinate system is introduced during the calibration process. The virtual space coordinate system is a spatial coordinate system obtained by mirroring the projection device coordinate system with respect to the object plane. The transformation relationship between the virtual space coordinate system and the camera coordinate system is first calibrated. The transformation relationship between the virtual space coordinate system and the camera coordinate system is then used to calibrate the positional relationship between the projection device and the camera, thereby accommodating subsequent three-dimensional measurement of mirrored objects. Furthermore, when calibrating the positional relationship between the projection device and the camera, the transformation relationship between the virtual space coordinate system and the camera coordinate system under multiple poses of the object plane is used for calculation, thereby obtaining a more accurate estimate and avoiding falling into a local optimum solution with a large deviation from the true value. Finally, the obtained estimate is used as the initial value for global optimization, thereby improving the calibration accuracy and providing a good foundation for subsequent three-dimensional measurement of the object.

[0192] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer program. When all or part of the functions in the above embodiments are implemented by computer program, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above functions can be implemented. In addition, when all or part of the functions in the above embodiments are implemented by computer program, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and saved in the memory of the local device by downloading or copying, or the system of the local device is updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be implemented.

[0193] The above examples are used to illustrate the present invention, which are only used to help understand the present invention and are not intended to limit the present invention. Those skilled in the art can make several simple deductions, modifications or substitutions based on the concept of the present invention.

Claims

1. A three-dimensional measurement method for a mirror object, applied to a three-dimensional measurement system, wherein the three-dimensional measurement system comprises a projection device, an object plane, and a camera, characterized in that: The three-dimensional measurement method comprises: Calibrate the positional relationship between the camera, the projection device, and the object plane; Acquire a grating projection image of the object to be measured, wherein the object to be measured is placed on the object plane, and the grating projection image of the object to be measured is an image captured by the camera of the object to be measured when the projection device performs grating projection on the object to be measured; Obtaining, based on the grating projection image of the object to be measured and the positional relationship between the camera, the projection device, and the object plane, the coordinates of the screen point and the object point corresponding to each pixel point in the grating projection images of the camera and the object to be measured in the same coordinate system, wherein the screen point is a point on the screen of the projection device, and the object point is a point on the object to be measured that reflects light emitted by the screen point and is imaged in the camera; For each pixel, the direction vector l between the camera and the screen point is calculated based on the coordinates of the camera, the object point corresponding to the pixel, and the screen point. ij and the direction vector s between the camera and the object point ij , where the subscripts i and j represent the image coordinates of the pixel, i represents the row coordinate, and j represents the column coordinate; The normal vector n at the object point corresponding to the pixel point (i, j) is calculated according to the following equations: ij ; in a ij =s ij +r ij =s ij +(l ij -s ij s ij ) / ||l ij -s ij s ij ||, b ij =l ij ×σ ij s ij , ρ ij Represents the distance between the object point and the screen point, σ ij Represents the distance between the camera and the object point; Calculate the gradient at the object point based on the normal vector at the object point; The three-dimensional shape of the object to be measured is obtained according to the gradient at each object point.

2. The three-dimensional measurement method according to claim 1, wherein: The positional relationship includes a transformation relationship between a projection device coordinate system and a camera coordinate system, and a transformation relationship between an object plane coordinate system and a camera coordinate system. Calibrating the positional relationship between the camera, the projection device, and the object plane includes: Acquire a calibration plate image of the object plane, where the calibration plate image of the object plane is an image of the object plane captured by the camera when the projection device projects a calibration plate pattern onto the object plane; Using the object plane coordinate system as the world coordinate system, calibrating the intrinsic and extrinsic parameters of the camera according to the calibration plate image, wherein the extrinsic parameters represent the transformation relationship between the world coordinate system and the camera coordinate system; Transforming the object plane into a plurality of postures, obtaining a calibration plate image or a grating projection image of the object plane in each posture, obtaining coordinates of pixel points in the calibration plate image or the grating projection image of the object plane in a camera coordinate system, and coordinates of corresponding points in a virtual space coordinate system, wherein the virtual space coordinate system is a spatial coordinate system obtained by performing mirror symmetry on the projection device coordinate system about the object plane; For each posture of the object plane, calculate the transformation relationship between the virtual space coordinate system and the camera coordinate system according to the coordinates of the pixel points in the calibration plate image or the grating projection image in the camera coordinate system and the coordinates of the corresponding points in the virtual space coordinate system; According to the geometric relationship between the virtual space coordinate system and the projection device coordinate system, the transformation relationship between the virtual space coordinate system and the camera coordinate system is calculated using the transformation relationship between the virtual space coordinate system and the camera coordinate system in all postures; The transformation relationship between the projection device coordinate system and the camera coordinate system is globally optimized according to a preset global optimization objective function.

3. The three-dimensional measurement method according to claim 2, wherein: The global optimization objective function is: where x mk Indicates the coordinates of the kth pixel in the camera coordinate system under the mth posture, N c Indicates the number of postures, z mk =f(p′ mk ), Represents the rotation matrix of the projection device coordinate system relative to the camera coordinate system, Represents the translation vector of the projection device coordinate system relative to the camera coordinate system, v m =d m n m , n m represents the normal vector of the object plane in the mth posture, d m Indicates the distance between the object plane and the camera in the mth posture, p mk Represents x mk The coordinates of the corresponding point in the projection device coordinate system, p′ mk is the p observed on the imaging plane of the camera in the camera coordinate system mk The coordinates of the virtual image point, z mk Represents point p′ mk The coordinates obtained by the camera's nonlinear mapping f are: is a vector consisting of the parameters to be optimized.

4. The three-dimensional measurement method according to claim 2 or 3, wherein: The method of obtaining coordinates of screen points and object points corresponding to each pixel point in the camera and the grating projection image of the object to be measured in the same coordinate system according to the grating projection image of the object to be measured and the positional relationship between the camera, the projection device, and the object plane includes: The grating projection image of the object to be measured is subjected to phase deconvolution to obtain the phase information of each pixel, and the screen point coordinates (x s ,y s ), where screen point coordinates refer to the two-dimensional coordinates of the screen point on the screen; According to the transformation relationship between the projection device coordinate system and the camera coordinate system and the external parameters of the camera, the coordinate p s =(x s ,y s ,0) transform to the world coordinate system as the world coordinate of the screen point; According to the intrinsic parameters and extrinsic parameters of the camera, the world coordinates of the corresponding object point are obtained from the image coordinates of each pixel point in the grating projection image of the object to be measured.

5. The three-dimensional measurement method according to claim 1, wherein: Calculating the gradient at the object point according to the normal vector at the object point includes: The X and Y components of the gradient at the object point are calculated according to the following formulas: in Represents the X component of the gradient at the object point corresponding to the pixel point (i, j), Represents the Y component of the gradient at the object point corresponding to the pixel point (i, j), n x (i, j) represents the X component of the normal vector at the object point corresponding to the pixel point (i, j), n y (i, j) represents the Y component of the normal vector at the object point corresponding to the pixel point (i, j), n z (i, j) represents the Z component of the normal vector at the object point corresponding to the pixel point (i, j).

6. The three-dimensional measurement method according to claim 5, wherein: The three-dimensional surface shape includes a height value at each object point, and obtaining the three-dimensional surface shape of the object to be measured according to the gradient at each object point includes: The first calculation step is to calculate the height value of each object point according to the following formula: where x i,j Indicates the X coordinate of the object point corresponding to the pixel point (i, j), i,j Indicates the Y coordinate of the object point corresponding to the pixel point (i, j), Z i,j represents the height value of the object point corresponding to the pixel point (i, j), M represents the number of rows of the grating projection image of the object to be measured, and N represents the number of columns of the grating projection image of the object to be measured; In the second calculation step, for each object point, its gradient is updated according to the following formula: Among them F x represents the X component of the gradient at the object point, F y Represents the Y component of the gradient at the object point, x c Indicates the X coordinate of the camera, y c represents the Y coordinate of the camera, z c represents the Z coordinate of the camera, x h Indicates the X coordinate of the object point, y h Indicates the Y coordinate of the object point, Z h Indicates the height value of the object point, x s Indicates the X coordinate of the screen point corresponding to the object point, y s Indicates the Y coordinate of the screen point corresponding to the object point, z s Indicates the Z coordinate of the screen point corresponding to the object point, d h2s Indicates the distance between the object point and the screen point, d h2c represents the distance between the object point and the camera; The first calculation step and the second calculation step are repeatedly performed until a preset stop condition is reached, and the height value obtained when the calculation is stopped is used as the measurement result of the height value at the object point.

7. The three-dimensional measurement method according to claim 5, wherein: The three-dimensional surface shape includes a height value at each object point, and obtaining the three-dimensional surface shape of the object to be measured according to the gradient at each object point includes: The third calculation step is to calculate the height value of each object point according to the following formula: where x i,j Indicates the X coordinate of the object point corresponding to the pixel point (i, j), i,j Indicates the Y coordinate of the object point corresponding to the pixel point (i, j), Z i,j represents the height value of the object point corresponding to the pixel point (i, j), M represents the number of rows of the grating projection image of the object to be measured, and N represents the number of columns of the grating projection image of the object to be measured; In the second calculation step, for each object point, its gradient is updated according to the following formula: Among them F x Represents the X component of the gradient at the object point, F y Represents the Y component of the gradient at the object point, x c Indicates the X coordinate of the camera, y c represents the Y coordinate of the camera, z c represents the Z coordinate of the camera, x h Indicates the X coordinate of the object point, y h Indicates the Y coordinate of the object point, Z h Indicates the height value of the object point, x s Indicates the X coordinate of the screen point corresponding to the object point, y s Indicates the Y coordinate of the screen point corresponding to the object point, z s Indicates the Z coordinate of the screen point corresponding to the object point, d h2s Indicates the distance between the object point and the screen point, d h2c represents the distance between the object point and the camera; The third calculation step and the second calculation step are repeatedly performed until a preset stop condition is reached, and the height value obtained when the calculation is stopped is used as the measurement result of the height value at the object point.

8. The three-dimensional measurement method according to any one of claims 1 to 7, characterized in that: The step of obtaining a grating projection image of the object to be measured comprises: Sequentially obtain grating projection images I1(x, y), I2(x, y), I3(x, y), and I4(x, y) obtained by performing grating projection on the object to be measured according to the four-step phase shift method, where (x, y) represents the coordinates of the pixel point; According to the surface grayscale modulation function k(x,y)=[I1(x,y)-I3(x,y)] 2 +[I2(x,y)-I4(x,y)] 2 Get the modulated grayscale image k(x,y); The area whose grayscale value is greater than the preset grayscale threshold in the grayscale image k(x,y) is obtained as the area to be detected; The part of the area to be detected in the grayscale image k(x,y) is used as the grating projection image of the object to be detected.

9. A computer-readable storage medium, characterized in that The medium stores a program, which can be executed by a processor to implement the three-dimensional measurement method according to any one of claims 1 to 8.

Images