A method for correcting camera and projector distortion based on structured light

Through the calibration algorithm combining affine transformation and three-dimensional data of structured light technology, the problem of difficult to correct distortion of the projector system is solved, significantly improving the distortion correction accuracy of the camera and projector, and improving the three-dimensional detection accuracy.

CN118761942BActive Publication Date: 2024-11-08SICHUAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411245105.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-06
Publication Date
2024-11-08
Estimated Expiration
2044-09-06

AI Technical Summary

Technical Problem

The existing projector calibration methods cannot effectively correct projector system distortion, affecting the three-dimensional detection accuracy, and it is difficult to improve the projector's subsequent distortion correction accuracy and reduce system errors.

Method used

Using a structured light-based distortion correction method, the center pixel coordinates and phases are obtained through affine transformation, combined with three-dimensional data, and a calibration algorithm for specific points is used to introduce three-dimensional data in the calibration process of camera and projector to improve the distortion correction accuracy. .

Benefits of technology

The camera's distortion correction error was reduced from 6.26e-5 to 5.66e-5, a decrease of 10.5%; the projector's distortion correction error was reduced from 8.72e-05 to 8.03e-05, a decrease of 7.9%, improving the camera and projection The distortion correction accuracy of the instrument reduces system errors and improves the three-dimensional detection accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118761942B_ABST
    Figure CN118761942B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for correcting camera and projector distortion based on structured light, which belongs to the calibration and correction of cameras and projectors in the field of optical measurement technology. The purpose is to solve the problem that the distortion of the projector system is difficult to be corrected during the calibration process of the existing projector, thereby affecting the detection accuracy of the projector and its three-dimensional detection system. The steps include obtaining a fringe image, finding the center of a circle by affine transformation, camera calibration, camera distortion correction, phase unwrapping, and projector calibration and distortion correction; and the same innovative calibration algorithm is used in both camera calibration and projector calibration. By finding the center of a circle by affine transformation, more accurate pixel coordinates and phase of the center of a circle can be obtained, and the use of a customized three-dimensional calibration plate can greatly improve the accuracy of subsequent distortion correction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of optical measurement, and relates to the calibration and correction of cameras and projectors, and in particular to a correction method for camera and projector distortion based on structured light. Background Art

[0002] With the rapid development of microelectronics manufacturing technology, various types of micro-components, such as semiconductor devices, optoelectronic components and MEMS, have been mass-produced in industry and widely used in high-end electronic products and precision instruments, such as cameras and projectors.

[0003] In cameras and projectors, they all involve three-dimensional model building, image measurement processes, and machine vision applications, and their requirements for precision and accuracy are particularly strict. Among the existing structured light measurement systems, the structured light three-dimensional measurement system composed of a camera + projector is widely used because of its simple structure and high efficiency in point cloud reconstruction. In a typical structured light three-dimensional measurement system composed of a camera + projector, during the measurement process, the projector projects the coded stripe pattern onto the surface of the object to be measured, and at the same time uses the camera to capture the deformed grating image modulated by the surface of the object to be measured, and then processes the deformed stripe image to calculate the phase information representing the height of the object, and finally reconstructs the three-dimensional point cloud of the surface of the object to be measured based on the phase information and the calibrated system parameters.

[0004] As mentioned above, in the structured light 3D measurement system, it is necessary to calibrate the relevant system parameters of the camera and projector in the system in order to improve the precision and accuracy of the measurement system. There are relatively mature calibration methods for camera calibration, such as the Zhang Zhengyou calibration method for internal calibration of the camera. However, as a projection display device, the projector cannot directly determine the correspondence between the projected image and the calibration plate, so its calibration is difficult. Treating the projector as a reverse camera, establishing the correspondence between the projector image and the camera image, converting the projector calibration into a mature camera calibration, and thus converting the single-camera structured light 3D measurement system into a classic binocular vision system is a popular projector calibration method. Projector calibration is a key link that affects the accuracy of 3D reconstruction. Exploring various projector calibration methods and improving their accuracy and convenience are important research topics in this field.

[0005] The invention patent application with application number 201710293985.5 discloses a projector calibration method, which includes providing a calibration plate; providing a structured light three-dimensional detection system; using a first type of circle to align the calibration plate; within the effective range of the calibration plate, taking a picture of the calibration plate, and having the projector project coded stripes onto the calibration plate and photographing it, and then repeating this step at least twice; performing phase calculation on each group of images with stripes to obtain the horizontal and vertical absolute phase values ​​of the center point position on each group of calibration plate images, using a dot extraction algorithm to calculate the pixel coordinates of the center point on the calibration plate image, and combining them to obtain the absolute phase value of each center point; converting the absolute phase value of the center point of the calibration plate from camera pixels to projector pixels to achieve calibration of the projector. Among them, it first performs a four-step phase shift calculation and obtains the wrapped phase diagram, and then uses a three-frequency heterodyne algorithm to obtain the horizontal and vertical absolute phase values ​​of the center point of the circle; then the center of the circle is calibrated and sorted on the straightened calibration plate, and the center of the circle with the sorting number is affine transformed to obtain the pixel coordinates of the center point of the circle.

[0006] The projector calibration method above calculates the phase of each group of images with stripes to obtain the horizontal and vertical absolute phase values ​​of the center point position on each group of calibration plate images, and uses the circle point extraction algorithm to calculate the pixel coordinates of the center point on the calibration plate image, and combines them to obtain the absolute phase value of each center point; the absolute phase value of the center point of the calibration plate is converted from the camera pixel to the projector pixel to achieve the calibration of the projector. However, although the projector calibration method above can achieve the calibration of the projector, system distortion will occur during the use of the projector, which will have a significant impact on the three-dimensional detection accuracy; and the existing technology has not been able to solve the problem of correcting the system distortion of the projector, and cannot improve the subsequent distortion correction accuracy of the projector, nor can it further reduce the system error and improve the three-dimensional detection accuracy. Summary of the invention

[0007] The purpose of the present invention is to solve the technical problem that the distortion of the projector system in the calibration process of the existing projector is difficult to be corrected, thereby affecting the detection accuracy of the projector and its three-dimensional detection system, and provide a correction method for the distortion of the camera and projector based on structured light. The calibration system based on structured light can obtain more accurate pixel coordinates and phases of the center of the circle by finding the center of the circle through affine transformation, and adopt an innovative calibration algorithm with specific points in the subsequent camera calibration process. Introducing three-dimensional data in the calibration algorithm can greatly improve the accuracy of subsequent distortion correction of the camera. After testing, after adopting the camera calibration method of the present application, the distortion correction error of the camera can be reduced from 6.26e-5e-05 before correction to 5.66e-5 after correction, which is reduced by 10.5%; after calibrating the projector using the same calibration method, the distortion correction error of the projector is reduced from 8.72e-05 before correction to 8.03e-05 after correction, which is reduced by 7.9%. It reduces the difficulty of correcting the system distortion of the camera, especially solves the technical problem that the system distortion of the projector is difficult to correct during the calibration process, greatly improves the accuracy of distortion correction for cameras and projectors, greatly reduces system errors of cameras, projectors, etc., and improves the accuracy of three-dimensional detection.

[0008] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0009] A method for correcting camera and projector distortion based on structured light, the specific steps are:

[0010] Step S1, obtaining a fringe image;

[0011] Using a projector to project sinusoidal stripes on the calibration plate in the vertical direction and the horizontal direction respectively; and using a camera to take a picture of the calibration plate projected with the sinusoidal stripes during the projection, to obtain an image with stripes;

[0012] Step S2, finding the center of the circle by affine transformation;

[0013] Select 8 corner points in the image with stripes, perform affine transformation, and identify the center of the circle on the image after affine transformation to obtain the coordinates of the center of the circle in the camera pixel coordinate system. ;

[0014] Step S3, camera calibration;

[0015] The specific steps of camera calibration are:

[0016] Step S3.1, based on the coordinates of the circle center in the camera pixel coordinate system , the coordinates of the center of the circle in world coordinates , calculate the camera homography matrix Parameters arrive , the calculation formula is:

[0017]

[0018]

[0019] Step S3.2, according to the homography matrix Parameters arrive , solve the camera's internal parameters and the external parameter matrix , the solution formula is:

[0020]

[0021]

[0022]

[0023]

[0024] in, represents the scaling factor, ( , , 1) represents the pixel coordinates under homogeneous coordinates, ( , , , 1) represents the world coordinates under homogeneous coordinates, arrive Represents the homography matrix Parameters within; represents the camera intrinsic parameter matrix, represents the rotation matrix in the camera extrinsic parameters, represents the translation matrix in the camera extrinsic parameters, arrive Represents the parameters of the rotation matrix in the camera extrinsic parameters, arrive Represents the parameters of the translation matrix in the camera extrinsic parameters, , Respectively represent the normalized focal length on the x-axis and y-axis in the camera pixel coordinate system, ( , ) represents the coordinates of the image center in the camera pixel coordinate system, Represents the first row of the rotation matrix in the camera extrinsic parameters, Represents the second row of the rotation matrix in the camera extrinsic parameters, Represents the third row of the rotation matrix in the camera extrinsic parameters, The rotation matrix of the camera extrinsic parameter OK, The rotation matrix of the camera extrinsic parameter OK, The rotation matrix of the camera extrinsic parameter Line List, represents the translation matrix, represents the tilt factor in the camera, Represents the homography matrix The first line of Represents the homography matrix The second line, Represents the homography matrix The third line of Represents the homography matrix No. Line List;

[0025] Step S4, camera distortion correction;

[0026] Using the camera’s intrinsic parameters , external parameter matrix Calculate the radial distortion parameters and tangential distortion parameters of the camera and perform distortion correction;

[0027] Step S5, phase unwrapping;

[0028] The coordinates of the center of the circle obtained in step S2 in the camera pixel coordinate system Perform phase unwrapping and normalize to 0-2π to obtain the absolute phase in the horizontal direction corresponding to the center of the circle and the absolute phase in the vertical direction ;

[0029] Step S6, projector calibration and distortion correction;

[0030] First find the pixel position in the projected image corresponding to the center of the circle ; Then use the camera calibration method in step S3 to calibrate the projector and calculate the homography matrix of the projector Parameters arrive ; Then use the camera calibration method in step S3 to solve the internal parameters of the projector , external parameter matrix ;Finally, using the internal reference of the projector , external parameter matrix Calculate the radial distortion parameters and tangential distortion parameters of the projector and perform distortion correction;

[0031] in, , are the length and width of the projected image, respectively. represents the rotation matrix in the projector extrinsic parameters, Represents the translation matrix in the projector extrinsic parameters.

[0032] Furthermore, in step S1, the encoding of the projected sinusoidal stripes in the horizontal direction is:

[0033]

[0034] The encoding of the projected sinusoidal stripes in the vertical direction is:

[0035]

[0036] in, is the sine wave intensity at the coordinate in the projected image, is the coding pattern coordinate, , are the background light intensity and modulated light intensity of the projection, is the sine wave spatial frequency, , are the length and width of the projected image, respectively. , are the phase shift steps and the total phase shift steps respectively.

[0037] Furthermore, when projecting sinusoidal stripes in the vertical direction, the projector performs multiple projections with a projection frequency of 1 and a phase shift step of 4, a projection frequency of 8 and a phase shift step of 4, and a projection frequency of 64 and a phase shift step of 12, to obtain multiple vertical projection images with stripes;

[0038] When projecting sinusoidal stripes in the horizontal direction, the projector uses the same projection frequency and phase shift steps as those used in the vertical direction to perform multiple projections to obtain multiple horizontal projection images with stripes.

[0039] Further, in step S2, the modulated light intensity of the image with stripes is calculated , and modulate the light intensity As the original image for finding the center of the circle;

[0040] Modulation light intensity The calculation method is:

[0041]

[0042] in, , Represent the phase shift steps and the total phase shift steps respectively, Represents the sine wave light intensity at the camera's center coordinate.

[0043] Furthermore, in step S4, the specific steps of camera distortion correction are:

[0044] Step S4.1, project the pixel coordinate system of the center of the calibration plate to the normalized coordinate system, and project the world coordinate system to the camera coordinate system, which is specifically expressed as:

[0045]

[0046]

[0047] Step S4.2, coordinates of the camera coordinate system ( , , ) is normalized, which is specifically expressed as:

[0048]

[0049] Step S4.3, calculate the radial distortion parameter and tangential distortion parameter of the camera, and perform distortion correction; the calculation formula of the radial distortion parameter and tangential distortion parameter is:

[0050]

[0051]

[0052] in,( , , ) represents the coordinates of the camera coordinate system, ( , ) represents the coordinates of the center of the circle in the normalized coordinate system after distortion in the camera; , Respectively represent the normalized focal length on the x-axis and y-axis in the camera pixel coordinate system, represents the tilt factor, ( , ) represents the coordinates of the image center in the camera pixel coordinate system, arrive Represents the parameters of the rotation matrix in the camera extrinsic parameters, arrive Represents the parameters of the translation matrix in the camera extrinsic parameters, ( , , 1) represents the pixel coordinates under homogeneous coordinates, ( , , , 1) represents the world coordinates under homogeneous coordinates; ( , ) represents the coordinates of the point in the normalized coordinate system without distortion in the camera; , , Both represent radial distortion parameters, , Both represent tangential distortion parameters, It represents the distance from the center of the circle to the origin in the normalized coordinate system without distortion.

[0053] Furthermore, in step S5, before phase unwrapping, the background light intensity of the image with stripes is calculated. , modulated light intensity ;

[0054] Background light intensity The calculation method is:

[0055]

[0056] Modulation light intensity The calculation method is:

[0057]

[0058] in, , Represent the phase shift steps and the total phase shift steps respectively, Represents the sine wave light intensity at the camera's center coordinate.

[0059] Furthermore, the specific steps of phase unwrapping are:

[0060] Step S5.1, respectively calculate the wrapping phase of the projection image with a projection frequency of 1 and a phase shift step of 4, a projection frequency of 8 and a phase shift step of 4, and a projection frequency of 64 and a phase shift step of 12, and obtain the wrapping phase , , , the specific calculation formula is:

[0061]

[0062] Step S5.2, wrapping the phase High frequency phase , Wrap Phase Low frequency phase , wrap the phase Unwrapping to Wrapping Phase , and the unwrapped phase , the specific calculation method is:

[0063]

[0064] Step S5.3, using the unwrapped phase obtained in step S5.2 High frequency phase , Wrap Phase Low frequency phase , the unwrapped phase obtained in step S5.2 Unpack to , and obtain the final unwrapped phase , the specific calculation method is:

[0065]

[0066] Step S5.4, the final unwrapped phase obtained in step S5.3 Normalized to 0-2π, the absolute phase in the horizontal direction corresponding to the center of the circle is obtained and the absolute phase in the vertical direction .

[0067] The beneficial effects of the present invention are as follows:

[0068] 1. In the present invention, the calibration system based on structured light can obtain more accurate pixel coordinates and phases of the center of the circle by affine transformation to find the center of the circle, and adopts an innovative calibration algorithm with specific points in the subsequent camera calibration process. Introducing three-dimensional data in the calibration algorithm can greatly improve the accuracy of subsequent distortion correction of the camera. After testing, after adopting the camera calibration method of the present application, the distortion correction error of the camera can be reduced from 6.26e-5e-05 before correction to 5.66e-5 after correction, which is reduced by 10.5%; after calibrating the projector using the same calibration method, the distortion correction error of the projector is reduced from 8.72e-05 before correction to 8.03e-05 after correction, which is reduced by 7.9%. The difficulty of system distortion correction of the camera is reduced, especially the technical problem that the system distortion of the projector is difficult to correct during the calibration process is solved, the accuracy of distortion correction of the camera and the projector is greatly improved, the system errors of the camera, projector, etc. are greatly reduced, and the three-dimensional detection accuracy is improved.

[0069] 2. In the present invention, by projecting sinusoidal stripes and using the calculated modulated light intensity as the original image processing, the ambient light intensity can be filtered out to avoid its interference with subsequent system calibration and distortion correction, thereby improving the distortion correction accuracy of cameras and projectors, reducing system errors of cameras, projectors, etc., and improving three-dimensional detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a schematic diagram of the structure of the present invention;

[0071] Figure 2 is a projected sinusoidal fringe pattern projected by the projector in the present invention, wherein (a) is a projected sinusoidal fringe pattern in the horizontal direction, and (b) is a projected sinusoidal fringe pattern in the vertical direction;

[0072] Figure 3 are the background light intensity image and the modulated light intensity image of the present invention, wherein (a) is the background light intensity image, and (b) is the modulated light intensity image;

[0073] Figure 4 Schematic diagram of finding the center of a circle by affine transformation in the present invention, wherein (a) is a schematic diagram after selecting 8 corner points in the original image, (b) is a schematic diagram after affine transformation, (c) is a schematic diagram after center recognition, and (d) is a schematic diagram of each center of the circle in the original image after inverse affine transformation;

[0074] Figure 5 Schematic diagram of the absolute phase after unwrapping in the present invention, wherein (a) is the absolute phase diagram after unwrapping of horizontal stripes, and (b) is the absolute phase diagram after unwrapping of vertical stripes;

[0075] Figure 6 It is a schematic diagram of the position of the center of the calibration plate in the projector pixel coordinate system in the present invention. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0077] Therefore, based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0078] Example 1

[0079] This embodiment provides a method for correcting camera and projector distortion based on structured light, which uses structured light and a camera to correct the distortion of the projector. Figure 1 As shown, the specific steps are:

[0080] Step S1, obtaining a fringe image;

[0081] A projector is used to project sinusoidal stripes onto the calibration plate in the vertical direction and the horizontal direction respectively; and a camera is used to take a picture of the calibration plate with the sinusoidal stripes projected thereon during the projection to obtain an image with stripes.

[0082] When generating projection fringes, the encoding of the projected sinusoidal fringes in the horizontal direction is:

[0083]

[0084] The encoding of the projected sinusoidal stripes in the vertical direction is:

[0085]

[0086] in, is the sine wave light intensity at the coordinate in the projected image, is the coding pattern coordinate, , are the background light intensity and modulated light intensity of the projection, is the sine wave spatial frequency, , are the length and width of the projected image, respectively. , are the phase shift steps and the total phase shift steps respectively.

[0087] The fringe image is as follows Figure 2 As shown, Figure 2 (a) is the horizontal projection sinusoidal fringe pattern. Figure 2 (b) is the projected sinusoidal fringe pattern in the vertical direction.

[0088] When the vertical projection fringes are generated by the above method,

[0089] When projecting sinusoidal fringes in the vertical direction, the projector uses projection frequencies of 1, 8, and 64 and phase shift steps of 4, 4, and 12 for multiple projections, and obtains multiple vertical projection images with fringes. That is, the projector projects multiple times with a projection frequency of 1 and a phase shift step of 4, and then projects multiple times with a projection frequency of 8 and a phase shift step of 4, and then projects multiple times with a projection frequency of 64 and a phase shift step of 12, for a total of 20 projections; each time the projector projects, the camera takes a picture of the calibration plate projected with sinusoidal fringes, and obtains 20 images with fringes.

[0090] When projecting sinusoidal fringes in the horizontal direction, the projector uses projection frequencies of 1, 8, and 64 and phase shift steps of 4, 4, and 12 for multiple projections, and obtains multiple vertical projection images with fringes. That is, the projector projects multiple times with a projection frequency of 1 and a phase shift step of 4, and then projects multiple times with a projection frequency of 8 and a phase shift step of 4, and then projects multiple times with a projection frequency of 64 and a phase shift step of 12, for a total of 20 projections; each time the projector projects, the camera takes a picture of the calibration plate projected with sinusoidal fringes, and obtains 20 images with fringes.

[0091] Step S2, finding the center of the circle by affine transformation;

[0092] Select 8 corner points in the image with stripes, perform affine transformation, and identify the center of the circle on the image after affine transformation to obtain the coordinates of the center of the circle in the camera pixel coordinate system. .

[0093] Due to the background light intensity Will be affected by ambient light, so choose to modulate the light intensity As the original image for finding the center of the circle, the modulated light intensity The image is as Figure 3 (b). Therefore, it is necessary to calculate the modulated light intensity first .Right now:

[0094] Modulation light intensity The calculation method is:

[0095]

[0096] in, , Represent the phase shift steps and the total phase shift steps respectively, Represents the sine wave light intensity at the camera's center coordinate.

[0097] like Figure 4 As shown in the figure, the specific method of finding the center of a circle using affine transformation is:

[0098] Step S2.1, first modulate the light intensity Select 8 corner points from the image, such as Figure 4 (a).

[0099] Step S2.2, perform affine transformation based on the selected 8 corner points to obtain the image after affine transformation, such as Figure 4 (b) When performing affine transformation, the existing affine transformation method can be used without any creative work.

[0100] Step S2.3, perform circle center recognition on the image after affine transformation to obtain the image after circle center recognition, such as Figure 4 As shown in (c), the blue represents the recognized circle and the red dot represents the center of the recognized circle. When recognizing the center of the circle, the existing recognition method can be used without creative work.

[0101] Step S2.4, obtain the center of the circle at the modulated light intensity by affine inverse transformation The pixel coordinates in the image ,like Figure 4 (d) When performing an inverse affine transformation, the existing inverse affine transformation method can be used without any creative work.

[0102] Step S3, camera calibration;

[0103] Through the above steps, the coordinates of 112 circle centers in the camera pixel coordinate system have been calculated. , and the world coordinates of the center of the circle are known Next, you can calibrate the camera. The specific steps for camera calibration are:

[0104] Step S3.1, based on the coordinates of the circle center in the camera pixel coordinate system , the coordinates of the center of the circle in world coordinates , calculate the camera homography matrix Parameters arrive , the calculation formula is:

[0105]

[0106]

[0107]

[0108] Step S3.2, according to the homography matrix Parameters arrive , solve the camera's internal parameters and the external parameter matrix , the solution formula is:

[0109]

[0110]

[0111]

[0112]

[0113] in, represents the scaling factor, ( , , 1) represents the pixel coordinates under homogeneous coordinates, ( , , , 1) represents the world coordinates under homogeneous coordinates, arrive Represents the homography matrix Parameters within; represents the camera intrinsic parameter matrix, represents the rotation matrix in the camera extrinsic parameters, represents the translation matrix in the camera extrinsic parameters, arrive Represents the parameters of the rotation matrix in the camera extrinsic parameters, arrive Represents the parameters of the translation matrix in the camera extrinsic parameters, , Respectively represent the normalized focal length on the x-axis and y-axis in the camera pixel coordinate system, ( , ) represents the coordinates of the image center in the camera pixel coordinate system, Represents the first row of the rotation matrix in the camera extrinsic parameters, Represents the second row of the rotation matrix in the camera extrinsic parameters, Represents the third row of the rotation matrix in the camera extrinsic parameters, The rotation matrix of the camera extrinsic parameter OK, The rotation matrix of the camera extrinsic parameter OK, The rotation matrix of the camera extrinsic parameter Line List, represents the translation matrix, represents the tilt factor in the camera, Represents the homography matrix The first line of Represents the homography matrix The second line, Represents the homography matrix The third line of Represents the homography matrix No. Line List.

[0114] Step S4, camera distortion correction;

[0115] Using the camera’s intrinsic parameters , external parameter matrix Calculate the radial distortion parameters and tangential distortion parameters of the camera and perform distortion correction.

[0116] When performing camera distortion correction, the specific steps of camera distortion correction are:

[0117] Step S4.1,

[0118] set up( , , ) is the coordinate of the camera coordinate system, ( , ) is the coordinate of the center of the circle in the normalized coordinate system after distortion in the camera. Project the pixel coordinate system of the center of the calibration plate to the normalized coordinate system, and project the world coordinate system to the camera coordinate system, which can be specifically expressed as:

[0119]

[0120]

[0121] Step S4.2, coordinates of the camera coordinate system ( , , ) is normalized, which is specifically expressed as:

[0122]

[0123] Step S4.3, assuming ( , ) represents the coordinates of the point in the normalized coordinate system without distortion, and the radial distortion parameters and tangential distortion parameters of the camera are calculated, and distortion correction is performed; the calculation formulas for the radial distortion parameters and tangential distortion parameters are:

[0124]

[0125]

[0126] in,( , , ) represents the coordinates of the camera coordinate system, ( , ) represents the coordinates of the center of the circle in the normalized coordinate system after distortion in the camera; , represent the normalized focal length on the x-axis and y-axis respectively, represents the tilt factor, ( , ) represents the coordinates of the image center in the camera pixel coordinate system, arrive Represents the parameters of the rotation matrix in the camera extrinsic parameters, arrive Represents the parameters of the translation matrix in the camera extrinsic parameters, ( , , 1) represents the pixel coordinates under homogeneous coordinates, ( , , , 1) represents the world coordinates under homogeneous coordinates; ( , ) represents the coordinates of the point in the normalized coordinate system without distortion in the camera; , , Both represent radial distortion parameters, , Both represent tangential distortion parameters, It represents the distance from the center of the circle to the origin in the normalized coordinate system without distortion. According to the test, the average reprojection error before distortion correction is 6.26e-5, and the average reprojection error after distortion correction is 5.66e-5, which is reduced by 10.5%.

[0127] Step S5, phase unwrapping;

[0128] The coordinates of the center of the circle obtained in step S2 in the camera pixel coordinate system Perform phase unwrapping and normalize to 0-2π to obtain the absolute phase in the horizontal direction corresponding to the center of the circle and the absolute phase in the vertical direction .

[0129] The two-dimensional imaging model of the camera is:

[0130]

[0131] in, represents the coordinates of the center of the circle in camera image coordinates, Represents the sine wave light intensity at the camera coordinate, is the background light intensity, To modulate the light intensity, is the phase, , Represent the phase shift steps and the total phase shift steps respectively.

[0132] Before phase unwrapping, calculate the background intensity of the image with stripes , modulated light intensity ;

[0133] Background light intensity The calculation method is:

[0134]

[0135] Modulation light intensity The calculation method is:

[0136]

[0137] in, , Represent the phase shift steps and the total phase shift steps respectively, Represents the sine wave light intensity at the camera's center coordinate.

[0138] When performing phase unwrapping, the specific steps of phase unwrapping are:

[0139] Step S5.1, respectively calculate the wrapping phase of the projection image with a projection frequency of 1 and a phase shift step of 4, a projection frequency of 8 and a phase shift step of 4, and a projection frequency of 64 and a phase shift step of 12, and obtain the wrapping phase , , , the specific calculation formula is:

[0140]

[0141] Step S5.2, wrapping the phase High frequency phase , Wrap Phase Low frequency phase , wrap the phase Unwrapping to Wrapping Phase , and the unwrapped phase , the specific calculation method is:

[0142]

[0143] Step S5.3, using the unwrapped phase obtained in step S5.2 High frequency phase , Wrap Phase Low frequency phase , the unwrapped phase obtained in step S5.2 Unpack to , and obtain the final unwrapped phase , the specific calculation method is:

[0144]

[0145] Step S5.4, the final unwrapped phase obtained in step S5.3 Normalized to 0-2π, the absolute phase in the horizontal direction corresponding to the center of the circle is obtained and the absolute phase in the vertical direction .

[0146] The absolute phase diagram after unwrapping is as follows Figure 5 As shown, (a) is the absolute phase image after unwrapping the horizontal stripes, and (b) is the absolute phase image after unwrapping the vertical stripes.

[0147] Step S6, projector calibration and distortion correction;

[0148] At present, the coordinates of the center of the calibration plate in the camera pixel coordinates are known, and then the absolute phase in the horizontal direction corresponding to the 112 center points is found by indexing. and the absolute phase in the vertical direction Here, the projector is regarded as an inverse camera, and the projected image size is , we can first find the pixel position in the projected image corresponding to the center of the circle , Figure 6 is the position of the center of the calibration plate in the projector pixel coordinates.

[0149] Then use the camera calibration method in step S3 to calibrate the projector and calculate the homography matrix of the projector. Parameters arrive ; Then use the camera calibration method in step S3 to solve the internal parameters of the projector , external parameter matrix ;Finally, using the internal reference of the projector , external parameter matrix Calculate the radial distortion parameters and tangential distortion parameters of the projector and perform distortion correction.

[0150] The specific steps for projector calibration are:

[0151] Step 1: Through the above steps, the coordinates of 112 circle centers in projector pixel coordinates have been obtained. , and the world coordinates of the center of the circle are known , and calculate the homography matrix of the projector Parameters arrive ; The calculation formula is:

[0152]

[0153]

[0154]

[0155] Step 2: According to the homography matrix of the projector Parameters arrive , solve the internal parameters of the projector and the external parameter matrix , the solution formula is:

[0156]

[0157]

[0158]

[0159]

[0160] in, represents the scaling factor, ( , , 1) represents the pixel coordinates under homogeneous coordinates, ( , , , 1) represents the world coordinates under homogeneous coordinates, arrive Represents the homography matrix Parameters within; represents the projector intrinsic parameter matrix, represents the rotation matrix in the projector extrinsic parameters, represents the translation matrix in the projector extrinsic parameters, arrive Represents the parameters of the rotation matrix in the projector extrinsic parameters, arrive Represents the parameters of the translation matrix in the projector extrinsic parameters, , Respectively represent the normalized focal lengths on the x-axis and y-axis in the projector pixel coordinate system, ( , ) represents the coordinates of the image center in the projector pixel coordinate system, Represents the first row of the rotation matrix in the projector extrinsic parameters, Represents the second row of the rotation matrix in the projector extrinsic parameters, Represents the third row of the rotation matrix in the projector extrinsic parameters, The rotation matrix in the projector extrinsic parameter is represented by OK, The rotation matrix of the camera extrinsic parameter OK, The rotation matrix in the projector extrinsic parameter is represented by Line List, represents the translation matrix, represents the tilt factor of the projector, Represents the homography matrix The first line of Represents the homography matrix The second line, Represents the homography matrix The third line of Represents the homography matrix No. Line List.

[0161] The specific steps for projector distortion correction are:

[0162] Step 1: Project the pixel coordinate system of the center of the calibration plate to the normalized coordinate system, and project the world coordinate system to the projector coordinate system, which is specifically expressed as:

[0163]

[0164]

[0165] Step 2: coordinates ( , , ) is normalized, which is specifically expressed as:

[0166]

[0167] Step 3: Calculate the radial distortion parameters and tangential distortion parameters of the projector and perform distortion correction. The calculation formulas for the radial distortion parameters and tangential distortion parameters are:

[0168]

[0169]

[0170] in,( , , ) represents the coordinates of the projector coordinate system, ( , ) represents the coordinates of the center of the circle in the normalized coordinate system after distortion in the projector; , They represent the normalized focal lengths on the x-axis and y-axis in the projector pixel coordinate system, respectively. represents the tilt factor of the projector, ( , ) represents the coordinates of the image center in the projector pixel coordinate system, arrive Represents the parameters of the rotation matrix in the projector extrinsic parameters, arrive Represents the parameters of the translation matrix in the projector extrinsic parameters, ( , , 1) represents the pixel coordinates under homogeneous coordinates, ( , , , 1) represents the world coordinates under homogeneous coordinates; ( , ) represents the coordinates of the point in the normalized coordinate system without distortion in the projector; , , Both represent the radial distortion parameters of the projector, , Both represent the tangential distortion parameters of the projector, It represents the distance from the center of the circle to the origin in the normalized coordinate system without distortion. After testing, the distortion correction error of the projector was reduced from 8.72e-05 before correction to 8.03e-05 after correction, a decrease of 7.9%.

Claims

1. A method for correcting camera and projector distortion based on structured light, characterized in that: The specific steps are: Step S1, obtaining a fringe image; Using a projector to project sinusoidal stripes on the calibration plate in the vertical direction and the horizontal direction respectively; and using a camera to shoot the calibration plate with the sinusoidal stripes projected during the projection to obtain an image with stripes; Step S2, finding the center of the circle by affine transformation; Select 8 corner points in the image with stripes, perform affine transformation, identify the center of the circle on the image after affine transformation, and then perform inverse transformation to obtain the coordinates of the center of the circle in the camera pixel coordinate system. ; Calculate the modulated light intensity of an image with stripes , and modulate the light intensity As the original image for finding the center of the circle; Modulation light intensity The calculation method is: in, , Represent the phase shift steps and the total phase shift steps respectively, Indicates the sine wave light intensity at the camera's center coordinate; Step S3, camera calibration; The specific steps of camera calibration are: Step S3.1, based on the coordinates of the circle center in the camera pixel coordinate system , the coordinates of the center of the circle in world coordinates , calculate the camera homography matrix Parameters arrive , the calculation formula is: Step S3.2, according to the homography matrix Parameters arrive , solve the camera's internal parameters and the external parameter matrix , the solution formula is: in, represents the scaling factor, ( , , 1) represents the pixel coordinates under homogeneous coordinates, ( , , , 1) represents the world coordinates under homogeneous coordinates, arrive Represents the homography matrix Parameters within; represents the camera intrinsic parameter matrix, represents the rotation matrix in the camera extrinsic parameters, represents the translation matrix in the camera extrinsic parameters, arrive Represents the parameters of the rotation matrix in the camera extrinsic parameters, arrive Represents the parameters of the translation matrix in the camera extrinsic parameters, , Respectively represent the normalized focal length on the x-axis and y-axis in the camera pixel coordinate system, ( , ) represents the coordinates of the image center in the camera pixel coordinate system, Represents the first row of the rotation matrix in the camera extrinsic parameters, Represents the second row of the rotation matrix in the camera extrinsic parameters, Represents the third row of the rotation matrix in the camera extrinsic parameters, The rotation matrix of the camera extrinsic parameter OK, The rotation matrix of the camera extrinsic parameter OK, The rotation matrix of the camera extrinsic parameter Line List, represents the translation matrix, represents the tilt factor in the camera, Represents the homography matrix The first line of Represents the homography matrix The second line, Represents the homography matrix The third line of Represents the homography matrix No. Line List; Step S4, camera distortion correction; Using the camera’s intrinsic parameters , external parameter matrix Calculate the radial distortion parameters and tangential distortion parameters of the camera and perform distortion correction; Step S5, phase unwrapping; The coordinates of the center of the circle obtained in step S2 in the camera pixel coordinate system Perform phase unwrapping and normalize to 0-2π to obtain the absolute phase in the horizontal direction corresponding to the center of the circle and the absolute phase in the vertical direction ; Step S6, projector calibration and distortion correction; First find the pixel position in the projected image corresponding to the center of the circle ; Then use the camera calibration method in step S3 to calibrate the projector and calculate the homography matrix of the projector Parameters arrive ; Then use the camera calibration method in step S3 to solve the internal parameters of the projector , external parameter matrix ;Finally, using the internal reference of the projector , external parameter matrix Calculate the radial distortion parameters and tangential distortion parameters of the projector and perform distortion correction; in, , are the length and width of the projected image, respectively. represents the rotation matrix in the projector extrinsic parameters, Represents the translation matrix in the projector extrinsic parameters.

2. The method for correcting camera and projector distortion based on structured light according to claim 1, characterized in that: In step S1, the encoding of the projected sinusoidal stripes in the horizontal direction is: The encoding of the projected sinusoidal stripes in the vertical direction is: in, is the sine wave intensity at the coordinate in the projected image, are the coding pattern coordinates, , are the background light intensity and modulated light intensity of the projection, is the sine wave spatial frequency, , are the length and width of the projected image, respectively. , are the phase shift steps and the total phase shift steps respectively.

3. The method for correcting camera and projector distortion based on structured light according to claim 2, characterized in that: When projecting sinusoidal stripes in the vertical direction, the projector performs multiple projections with a projection frequency of 1 and a phase shift step of 4, a projection frequency of 8 and a phase shift step of 4, and a projection frequency of 64 and a phase shift step of 12, to obtain multiple vertical projection images with stripes; When projecting sinusoidal stripes in the horizontal direction, the projector uses the same projection frequency and phase shift steps as those used in the vertical direction to perform multiple projections to obtain multiple horizontal projection images with stripes.

4. The method for correcting camera and projector distortion based on structured light according to claim 1, characterized in that: In step S4, the specific steps of camera distortion correction are: Step S4.1, project the pixel coordinate system of the center of the calibration plate to the normalized coordinate system, and project the world coordinate system to the camera coordinate system, which is specifically expressed as: Step S4.2, coordinates of the camera coordinate system ( , , ) is normalized, which is specifically expressed as: Step S4.3, calculate the radial distortion parameter and tangential distortion parameter of the camera, and perform distortion correction; the calculation formula of the radial distortion parameter and tangential distortion parameter is: in,( , , ) represents the coordinates of the camera coordinate system, ( , ) represents the coordinates of the center of the circle in the normalized coordinate system after distortion in the camera; , Respectively represent the normalized focal length on the x-axis and y-axis in the camera pixel coordinate system, represents the tilt factor, ( , ) represents the coordinates of the image center in the camera pixel coordinate system, arrive Represents the parameters of the rotation matrix in the camera extrinsic parameters, arrive Represents the parameters of the translation matrix in the camera extrinsic parameters, ( , , 1) represents the pixel coordinates under homogeneous coordinates, ( , , , 1) represents the world coordinates under homogeneous coordinates; ( , ) represents the coordinates of the point in the normalized coordinate system without distortion in the camera; , , Both represent radial distortion parameters, , Both represent tangential distortion parameters, It represents the distance from the center of the circle to the origin in the normalized coordinate system without distortion.

5. The method for correcting camera and projector distortion based on structured light according to claim 1, characterized in that: In step S5, before phase unwrapping, the background intensity of the image with stripes is calculated. , modulated light intensity ; Background light intensity The calculation method is: Modulation light intensity The calculation method is: in, , Represent the phase shift steps and the total phase shift steps respectively, Represents the sine wave light intensity at the camera's center coordinate.

6. A method for correcting camera and projector distortion based on structured light as claimed in claim 5, characterized in that: The specific steps of phase unwrapping are: Step S5.1, respectively calculate the wrapping phase of the projection image with a projection frequency of 1 and a phase shift step of 4, a projection frequency of 8 and a phase shift step of 4, and a projection frequency of 64 and a phase shift step of 12, and obtain the wrapping phase , , , the specific calculation formula is: Step S5.2, wrapping the phase High frequency phase , Wrap Phase Low frequency phase , wrap the phase Unwrapping to Wrapping Phase , and the unwrapped phase , the specific calculation method is: Step S5.3, using the unwrapped phase obtained in step S5.2 High frequency phase , Wrap Phase Low frequency phase , the unwrapped phase obtained in step S5.2 Unpack to , and obtain the final unwrapped phase , the specific calculation method is: Step S5.4, the final unwrapped phase obtained in step S5.3 Normalized to 0-2π, the absolute phase in the horizontal direction corresponding to the center of the circle is obtained and the absolute phase in the vertical direction .