Three-camera structured light online measurement method based on CUDA acceleration

Abstract

The invention discloses a CUDA acceleration-based three-camera structured light online measurement method and system, a medium and equipment, and the method comprises the steps: synchronously collecting a three-frequency four-step phase shift vertical stripe pattern by using three cameras; performing stereo matching on the cameras C0 to C1 and the cameras C1 to C2 by adopting an epipolar search method; the three-camera structured light system performs online measurement, performs matching reconstruction in the 02 binocular system based on a CUDA parallel computing architecture to generate an image mask phase, performs matching reconstruction in the 01 binocular system and the 12 binocular system, and updates an image mask phase C0r, an image mask phase C1r and an image mask phase C2r; and according to the updated image mask phase C2r, de-overlapping is carried out on the 12 binocular system matching reconstruction point cloud, coordinate transformation is carried out on reconstruction results of the 01 binocular system and the 12 binocular system, and direct data fusion is carried out on reconstruction results of the 02 binocular system, the 01 binocular system and the 12 binocular system after transformation.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-camera structured light measurement, and in particular to a three-camera structured light online measurement method, system, medium and equipment based on CUDA acceleration. Background Art

[0002] The FPP (Fringe projection profilometry) structured light measurement method using three-dimensional vision is widely used in aviation manufacturing, automotive industry, medical and other fields due to its advantages such as fast, efficient, full-field measurement and high degree of automation.

[0003] However, with the ever-expanding demand for measurement technology, the current industrial intelligent manufacturing industry is more inclined to the development trend of precise, fast, and complex online measurement. In online measurement, there are many environmental interference factors, and there may be many random measurement problems. Therefore, for some industrial field measurement scenarios, facing complex and changeable working conditions, the measurement technology requirements are mainly reflected in measurement accuracy, efficiency, and measurement integrity. The technical problems they face are mainly the occlusion of the public measurement field of view and the local high reflection of the components to be measured.

[0004] In order to solve the problem of field of view occlusion, the most direct strategy is to use multi-view measurement for data compensation; for binocular structured light measurement, a single-binocular hybrid measurement strategy is used to compensate for missing data, and a multi-camera strategy is also used for collaborative measurement, which has achieved good measurement results in both measurement accuracy and measurement integrity;

[0005] To address the problem of local high reflection during measurement, high dynamic range (HDR) technology is often used to perform local compensation through multiple exposures or adaptive fringe projection technology. Studies have found that multi-view data compensation can also be performed by arranging different camera viewing angles, thereby improving the integrity of the measurement.

[0006] For the monocular and binocular hybrid method, theoretically, it is necessary to calculate three sets of measurement data for the binocular and two monocular cameras, while the multi-eye structured light strategy theoretically requires more calculations; HDR technology requires multiple data collection and fusion during measurement; although the existing strategies have made good progress in measurement accuracy and integrity, they still cannot meet the technical requirements of online measurement when using high-resolution cameras.

[0007] CUDA-based GPU parallel computing has been widely used in image processing, scientific computing, deep learning and other fields due to its powerful parallel processing capabilities, high memory bandwidth, high energy efficiency ratio, and strong programmability, and has achieved great progress. In the on-line measurement of industrial sites, it is required to obtain the highest possible accuracy, data integrity, and the fastest possible solution. Therefore, it is very important to fully consider the hardware cost and computational cost, adopt appropriate measurement strategies, and configure a reasonable number of CCDs and cameras.

[0008] In summary, the existing structured light measurement methods cannot simultaneously meet the measurement requirements of high precision, high speed, and high integrity. It is necessary to study a structured light measurement method with high measurement accuracy, good integrity, and high speed to achieve accurate, fast, and complete measurement of point cloud data in industrial sites.

[0009] The above information disclosed in the background art section is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0010] The present invention provides a three-camera structured light on-line measurement method, system, medium, and device based on CUDA acceleration to achieve accurate, fast, and complete measurement of point cloud data in industrial sites.

[0011] A three-camera structured light on-line measurement method based on CUDA acceleration includes:

[0012] Step S1, respectively, make the three cameras of camera C0, camera C1, and camera C2 form a 02 binocular system, a 01 binocular system, and a 12 binocular system in pairs and perform binocular vision system calibration respectively;

[0013] Step S2, measure a white plane at the center of the field of view, and use the three cameras to synchronously collect three-frequency four-step phase-shifted vertical stripe patterns; use the epipolar search method to perform stereo matching from camera C0 to camera C1 and from camera C1 to camera C2, and perform three-dimensional reconstruction in the 02 binocular system, the 01 binocular system, and the 12 binocular system, and calculate the coordinate transformation of the 01 binocular system and the 12 binocular system relative to the 02 binocular system according to the matching relationship for unifying the coordinate system of on-line measurement data;

[0014] Step S3, on-line measurement of the three-camera structured light system in the actual industrial site to obtain on-line measurement data of the multi-imaging system of the three-camera structured light system of the actual target to be measured, and generate image mask phases C0r, C1r, and C2r by matching and reconstructing in the 02 binocular system based on the CUDA parallel computing architecture;

[0015] Step S4: Use the image mask phases C0r, C1r, and C2r to perform matching and reconstruction in the 01 binocular system and the 12 binocular system, and update the image mask phases C0r, C1r, and C2r.

[0016] Step S5: De-overlap the matching and reconstructed point cloud of the 12 binocular system according to the updated image mask phase C2r, and perform coordinate transformation on the reconstruction results of the 01 binocular system and the 12 binocular system respectively. For the reconstruction results of the transformed 02 binocular system, 01 binocular system, and 12 binocular system, perform direct data fusion to obtain the measurement results of the multi-imaging system of the three-camera structured light system for the actual target to be measured in the 02 binocular system coordinate system.

[0017] In the described three-camera structured light online measurement method based on CUDA acceleration, step S1 includes

[0018] Step S11: The camera C0, camera C2, and the projector form a 02 binocular system.

[0019] Step S12: The camera C0, camera C1, and the projector form a 01 binocular system.

[0020] Step S13: The camera C1, camera C2, and the projector form a 12 binocular system.

[0021] In the described three-camera structured light online measurement method based on CUDA acceleration, step S2 includes

[0022] Step S20: After calibration, synchronously collect the fringe pattern of the white plane in the center of the field of view.

[0023] Step S21: Calculate the coordinate transformation between the 02 binocular system, 01 binocular system, and 12 binocular systems. Use the epipolar search method to match the phase diagrams of camera C0 and camera C2 to obtain the matching result M02. Among them, in the epipolar search method, for any pixel in the left camera, find a projection straight line in the right camera, and its corresponding point must be on this line, which satisfies the epipolar constraint:

[0024] ,

[0025] where F is a 3×3 matrix, called the fundamental matrix, representing the mapping from the pixels in the left camera imaging plane to the epipolar lines in the right camera imaging plane. In the formula, is the undistorted pixel coordinates of the left camera, is the undistorted pixel coordinates of the corresponding point in the right camera;

[0026] For the known pixel coordinates of the left camera, substituting into the epipolar constraint gives:

[0027] ,

[0028] In the formula, is the epipolar constraint vector of the left camera pixel coordinates;

[0029] The points with the same name in the right image satisfy the formula:

[0030] ,

[0031] So for any horizontal coordinate of the right image on the epipolar line, find the vertical coordinate:

[0032] ,

[0033] Step S22, using an epipolar line search method to match the phase images of camera C0 and camera C1, and obtaining a matching result M01;

[0034] Step S23, using the epipolar line search method, continue matching the coordinates of camera C1 in the matching result M01 in the phase map of camera C2 to obtain matching result M0-12;

[0035] Step S24, eliminating noise data according to the matching error threshold δ of the intersection of the matching result M02 and the matching result M0-12;

[0036] Step S25, traverse the matching results M02, matching results M01, matching results M0-12 to find the intersection of the successful matching results of the 02 binocular system, the 01 binocular system and the 12 binocular system;

[0037] Step S26, according to the corresponding relationship of the intersection matching, reconstruct the three-dimensional points in the 02 binocular system, the 01 binocular system and the 12 binocular system respectively, and according to the corresponding point relationship, use the SVD decomposition method to calculate the transformation matrix of the 01 binocular system and the 12 binocular system relative to the 02 binocular system respectively and :

[0038]

[0039] In the formula, , is the point set of the 02 binocular system in the intersection matching correspondence, is the point set of the 01 binocular system in the intersection matching correspondence, where It is the point set of 12 binocular systems in the intersection matching correspondence.

[0040] In the three-camera structured light online measurement method based on CUDA acceleration, the matching error threshold δ is 0.01 pixel<δ<0.05 pixel.

[0041] In the described online measurement method of three-camera structured light based on CUDA acceleration, step S3 includes:

[0042] Step S30, perform phase matching calculation of the epipolar line search method and subsequent reconstruction under the CUDA parallel computing framework;

[0043] Step S31, copy the phase diagrams of camera C0, camera C1, and camera C2 as image mask phases C0r, image mask phase C1r, and image mask phase C2r, and mark the image mask phases C0r and C2r at the coordinate positions where the matching is successful as invalid after each 02 binocular system matching;

[0044] Step S32, perform three-dimensional reconstruction using the least squares method:

[0045]

[0046] In the formula, is the three-dimensional coordinate to be solved, , are the pixel coordinates of the left camera and the right camera in the binocular system respectively, , are the parameter matrices of the left camera and the right camera in the binocular system respectively, and their forms are: , which can be transformed into: , that is, Ax = b;

[0047] Step S33, solve the matrix using the least squares reconstruction method based on LU decomposition; for the reconstruction method Ax = b using the least squares method, the matrix A is decomposed into the product of a unit lower triangular matrix L and an upper triangular matrix U, that is:

[0048] A = LU

[0049] Among them , , ,

[0050] The LU decomposition formula is as follows:

[0051]

[0052] After calculating L and U, substitute them into Ax = b to get LUx = b. Denote Ux = y, then Ly = b.

[0053] First, solve for y. Calculate Ly = b row by row from top to bottom:

[0054]

[0055] After obtaining y, x can be solved. Calculate Ux = y row by row from bottom to top:

[0056] 。

[0057] In the described online measurement method of three-camera structured light based on CUDA acceleration, step S4 includes:

[0058] Step S40: For the image mask phase C0r generated after the reconstruction of the 02 binocular system, perform binocular matching and reconstruction on the valid pixels therein with the image mask phase C1r, and update the image mask phase C1r with the coordinates of the successfully matched ones again;

[0059] Step S41: For the updated image mask phase C1r and the image mask phase C2r after the reconstruction of the 01 binocular system, perform binocular matching and reconstruction on the valid pixels therein, and save the result in the image matrix at the same coordinates as the image mask phase C2r;

[0060] Step S42: Perform data overlap removal processing on the reconstruction result of the 12 binocular system according to the pixel coordinates with the updated image mask phase C2r matched by the matching result M01.

[0061] In the described online measurement method of three-camera structured light based on CUDA acceleration, step S5 includes:

[0062] Step S50: represents the reconstruction result of the 02 binocular system, represents the reconstruction result of the camera C0 that was not successfully matched in the 02 binocular system but was reconstructed with the camera C1 in the 01 binocular system, represents the result of the reconstruction of the phase C2r where the camera C1 was not successfully matched in the 01 binocular system and was not successfully matched with C2 in the 02 binocular system in the 12 binocular system;

[0063] Step S51: According to the transformation matrices and of the 01 binocular system and the 12 binocular system relative to the 02 binocular system, directly perform coordinate transformation on the reconstruction results of the 01 binocular system and the 12 binocular system:

[0064] ,

[0065] Step S52: Directly perform data fusion on the reconstruction results of the three binocular systems after transformation:

[0066] , where in the formula, are respectively the point sets of the reconstruction results of the 01 binocular system and the 12 binocular system in the coordinate system of the 02 binocular system.

[0067] An online measurement of three-camera structured light based on CUDA acceleration includes:

[0068] A calibration system that respectively combines three cameras, namely camera C0, camera C1, and camera C2, in pairs to form a 02 binocular system, a 01 binocular system, and a 12 binocular system, and respectively performs binocular vision system calibration;

[0069] A 3D reconstruction unit that is used to measure a white plane at the center of the field of view, and uses three cameras to synchronously collect a three-frequency four-step phase-shifted vertical stripe pattern; adopts an epipolar search method to perform stereo matching between camera C0 and camera C1, and between camera C1 and camera C2, and performs 3D reconstruction in the 02 binocular system, the 01 binocular system, and the 12 binocular system, and calculates the coordinate transformation of the 01 binocular system and the 12 binocular system relative to the 02 binocular system according to the matching relationship to be used for unifying the coordinate system of on-line measurement data;

[0070] An on-line measurement unit that is used for on-line measurement of a three-camera structured light system in an actual industrial site to obtain on-line measurement data of a multi-imaging system of the three-camera structured light system of an actual target to be measured, and generates an image mask phase C0r, an image mask phase C1r, and an image mask phase C2r by performing matching and reconstruction in the 02 binocular system based on the CUDA parallel computing architecture;

[0071] An update unit that is used to perform matching and reconstruction in the 01 binocular system and the 12 binocular system by using the image mask phase C0r, the image mask phase C1r, and the image mask phase C2r, and update the image mask phase C0r, the image mask phase C1r, and the image mask phase C2r;

[0072] A fusion unit that is used to remove overlaps from the matching and reconstructed point cloud of the 12 binocular system according to the updated image mask phase C2r, and respectively perform coordinate transformation on the reconstruction results of the 01 binocular system and the 12 binocular system, and directly perform data fusion on the reconstruction results of the transformed 02 binocular system, 01 binocular system, and 12 binocular system to obtain the measurement results of the multi-imaging system of the three-camera structured light system of the actual target to be measured in the coordinate system of the 02 binocular system.

[0073] A computer storage medium, the storage medium includes computer instructions, when it runs on a computer, it causes the computer to execute the method described above.

[0074] An electronic device, characterized in that the electronic device includes:

[0075] A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein,

[0076] When the processor executes the program, it implements the method described above.

[0077] Compared with the prior art, the present invention has the following advantages: By adopting the phase matching and least squares reconstruction method of epipolar line search and the parallel computing method of CUDA, the computing speed is increased by two orders of magnitude compared with the single-threaded computing of CPU and by one order of magnitude compared with the multi-threaded computing of CPU, greatly improving the online computing speed. By continuously matching the corresponding points in the three views and performing three-dimensional reconstruction on the phase coordinates that satisfy the epipolar line constraint of the three views, and then calculating the system transformation matrix for the reconstructed coordinates, the accurate transformation of the coordinate systems between the three binocular systems is realized. By generating a mask of the phase diagram after each matching and reconstruction and storing the reconstruction results in the matrices corresponding to the phase coordinates respectively; after the data of the three binocular systems are reconstructed, data redundancy is removed according to the phase mask, improving the speed and quality of multi-system data fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] By reading the detailed description of the preferred specific embodiments below, various other advantages and benefits of the present invention will become clear to those of ordinary skill in the art. The accompanying drawings in the specification are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts. Moreover, throughout the drawings, the same reference numerals are used to represent the same components.

[0079] In the drawings:

[0080] Figure 1 is a flowchart of the online measurement method of three-camera structured light based on CUDA acceleration according to an embodiment of the present invention;

[0081] Figure 2 is a schematic diagram of the three-camera FPP configuration of the online measurement method of three-camera structured light based on CUDA acceleration according to an embodiment of the present invention;

[0082] Figure 3 is a schematic diagram of the matching of corresponding points of the three-camera structured light system of the online measurement method of three-camera structured light based on CUDA acceleration according to an embodiment of the present invention;

[0083] Figure 4 is a schematic diagram of the C0 phase mask of the online measurement method of three-camera structured light based on CUDA acceleration according to an embodiment of the present invention;

[0084] Figure 5 is a schematic diagram of the redundancy-removed reconstruction result of the online measurement method of three-camera structured light based on CUDA acceleration according to an embodiment of the present invention;

[0085] Figure 6The effect comparison diagram of the field of view occlusion measurement of the online measurement method of three-camera structured light based on CUDA acceleration according to the embodiment of the present invention;

[0086] Figure 7 The effect comparison diagram of the local high-reflectivity measurement of the online measurement method of three-camera structured light based on CUDA acceleration according to the embodiment of the present invention.

[0087] The present invention will be further explained below in conjunction with the accompanying drawings and embodiments. Specific embodiments

[0088] The specific embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although specific embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present invention can be more thoroughly understood and the scope of the present invention can be fully conveyed to those skilled in the art.

[0089] It should be noted that certain terms are used in the description and claims to refer to specific components. Those skilled in the art should understand that technicians may use different terms to refer to the same component. The description and claims of this specification do not use the difference in terms as a way to distinguish components, but use the difference in functions of components as the criterion for distinction. As mentioned throughout the description and claims, "comprising" or "including" is an open-ended term and should be interpreted as "including but not limited to". The subsequent description of the specification is the preferred embodiment for implementing the present invention, but the description is for the purpose of the general principles of the specification and is not intended to limit the scope of the present invention. The protection scope of the present invention shall be determined by the scope defined by the appended claims.

[0090] For the convenience of understanding the embodiments of the present invention, the following will further explain with specific embodiments as examples in conjunction with the accompanying drawings, and each accompanying drawing does not constitute a limitation to the embodiments of the present invention.

[0091] As Figures 1 to 7 shown, the online measurement method of three-camera structured light based on CUDA acceleration includes the following steps:

[0092] Step S1, respectively combine the three cameras, namely camera C0, camera C1, and camera C2, in pairs to form a 02 binocular system, a 01 binocular system, and a 12 binocular system, and respectively perform binocular vision system calibration;

[0093] Step S2: Measure the white plane at the center of the field of view, and use three cameras to synchronously collect three-frequency four-step phase-shifted vertical stripe patterns; adopt the epipolar search method to perform stereo matching on cameras C0 to C1 and cameras C1 to C2, and perform three-dimensional reconstruction in the 02 binocular system, 01 binocular system, and 12 binocular system. Calculate the coordinate transformation of the 01 binocular system and 12 binocular system relative to the 02 binocular system according to the matching relationship to unify the coordinate system for online measurement data.

[0094] Step S3: The online measurement of the three-camera structured light system in the actual industrial field to obtain the online measurement data of the multi-imaging system of the three-camera structured light system for the actual target to be measured. Based on the CUDA parallel computing architecture, perform matching and reconstruction in the 02 binocular system to generate the image mask phases C0r, C1r, and C2r. The CUDA parallel computing architecture is the NVIDIA parallel computing technology platform.

[0095] Step S4: Use the image mask phases C0r, C1r, and C2r to perform matching and reconstruction in the 01 binocular system and 12 binocular system, and update the image mask phases C0r, C1r, and C2r.

[0096] Step S5: Remove the overlap of the matching and reconstructed point cloud of the 12 binocular system according to the updated image mask phase C2r, and perform coordinate transformation on the reconstruction results of the 01 binocular system and 12 binocular system respectively. For the reconstruction results of the transformed 02 binocular system, 01 binocular system, and 12 binocular system, perform direct data fusion to obtain the measurement results of the multi-imaging system of the three-camera structured light system for the actual target to be measured in the coordinate system of the 02 binocular system.

[0097] In the preferred embodiment of the three-camera structured light online measurement method based on CUDA acceleration, step S1 includes:

[0098] Step S11: Cameras C0, C2, and the projector form the 02 binocular system.

[0099] Step S12: Cameras C0, C1, and the projector form the 01 binocular system.

[0100] Step S13: Cameras C1, C2, and the projector form the 12 binocular system.

[0101] In the preferred embodiment of the three-camera structured light online measurement method based on CUDA acceleration, step S2 includes:

[0102] Step S20: After calibration, synchronously collect the stripe pattern of the white plane at the center of the field of view.

[0103] Step S21, the coordinate transformation between the 02 binocular system, the 01 binocular system and the 12 binocular system is calculated, and the phase images of the camera C0 and the camera C2 are matched by the epipolar line search method to obtain the matching result M02. In the epipolar line search method, for any pixel in the left camera, a projection line is found in the right camera, and its related point must be on this line, which satisfies the epipolar line constraint:

[0104] ,

[0105] Where F is a 3×3 matrix, called the basic matrix, which represents the mapping from pixels in the left camera imaging plane to the epipolar lines in the right camera imaging plane. is the undistorted pixel coordinate of the left camera, are the undistorted pixel coordinates of the same-name points in the right camera;

[0106] For the known left camera pixel coordinates, substituting the epipolar constraints into:

[0107] ,

[0108] In the formula, is the epipolar constraint vector of the left camera pixel coordinates;

[0109] The points with the same name in the right image satisfy the formula:

[0110] ,

[0111] So for any horizontal coordinate of the right image on the epipolar line, find the vertical coordinate:

[0112] ,

[0113] Step S22, using an epipolar line search method to match the phase images of camera C0 and camera C1, and obtaining a matching result M01;

[0114] Step S23, using the epipolar line search method, continue matching the coordinates of camera C1 in the matching result M01 in the phase map of camera C2 to obtain matching result M0-12;

[0115] Step S24, eliminating noise data according to the matching error threshold δ of the intersection of the matching result M02 and the matching result M0-12;

[0116] Step S25, traverse the matching results M02, matching results M01, matching results M0-12 to find the intersection of the successful matching results of the 02 binocular system, the 01 binocular system and the 12 binocular system;

[0117] Step S26: Reconstruct three-dimensional points in the 02 binocular system, 01 binocular system, and 12 binocular system respectively according to the corresponding relationship of intersection matching, and calculate the transformation matrices of the 01 binocular system and 12 binocular system relative to the 02 binocular system respectively by using the SVD decomposition method according to the corresponding point relationship. and :

[0118]

[0119] In the formula, and are the point sets of the 02 binocular system in the corresponding relationship of intersection matching, is the point set of the 01 binocular system in the corresponding relationship of intersection matching. In the formula, is the point set of the 12 binocular system in the corresponding relationship of intersection matching.

[0120] In the preferred embodiment of the online measurement method of three-camera structured light based on CUDA acceleration, the matching error threshold δ is 0.01 pixel < δ < 0.05 pixel.

[0121] In the preferred embodiment of the online measurement method of three-camera structured light based on CUDA acceleration, Step S3 includes

[0122] Step S30: Perform phase matching calculation of the epipolar line search method and subsequent reconstruction under the CUDA parallel computing framework;

[0123] Step S31: Copy the phase diagrams of cameras C0, C1, and C2 to the image mask phases C0r, C1r, and C2r. After each matching of the 02 binocular system, mark the image mask phases C0r and C2r at the coordinate positions where the matching is successful as invalid;

[0124] Step S32: Perform three-dimensional reconstruction by using the least squares method:

[0125]

[0126] In the formula, is the three-dimensional coordinate to be solved, and are the pixel coordinates of the left camera and the right camera in the binocular system respectively, and are the parameter matrices of the left camera and the right camera in the binocular system respectively, and their forms are: , which can be transformed into: , that is, Ax = b;

[0127] Step S33, matrix solution of the least squares reconstruction method based on LU decomposition; for the reconstruction method Ax = b using the least squares method, matrix A is decomposed into the product of a unit lower triangular matrix L and an upper triangular matrix U, i.e.:

[0128] A = LU

[0129] where , , ,

[0130] The LU decomposition formula is as follows:

[0131]

[0132] After calculating L and U, substitute them into Ax = b to get LUx = b. Denote Ux = y, then Ly = b.

[0133] First, solve for y. Calculate Ly = b row by row from top to bottom:

[0134]

[0135] After obtaining y, x can be solved. Calculate Ux = y row by row from bottom to top:

[0136] .

[0137] In the preferred embodiment of the three-camera structured light online measurement method based on CUDA acceleration, step S4 includes

[0138] Step S40, for the image mask phase C0r generated after the reconstruction of the 02 binocular system, perform binocular matching and reconstruction on the valid pixels therein with the image mask phase C1r, and update the image mask phase C1r with the coordinates of the successfully matched ones.

[0139] Step S41, for the updated image mask phase C1r and the image mask phase C2r after the reconstruction of the 01 binocular system, perform binocular matching and reconstruction on the valid pixels therein, and save the result in the image matrix at the same coordinates as the image mask phase C2r.

[0140] Step S42, perform data overlap removal processing on the reconstruction result of the 12 binocular system according to the pixel coordinates of the updated image mask phase C2r matched by the matching result M01.

[0141] In the preferred embodiment of the three-camera structured light online measurement method based on CUDA acceleration, step S5 includes

[0142] Step S50, represents the reconstruction result of the 02 binocular system, Indicates the reconstruction result of camera C0 that was not successfully matched in the 02 binocular system but was reconstructed with camera C1 in the 01 binocular system. Indicates the reconstruction result of phase C2r where camera C1 was not successfully matched in the 01 binocular system and camera C2 was not successfully matched in the 02 binocular system, reconstructed in the 12 binocular system.

[0143] Step S51: According to the transformation matrices of the 01 binocular system and the 12 binocular system relative to the 02 binocular system and , directly perform coordinate transformation on the reconstruction results of the 01 binocular system and the 12 binocular system:

[0144] ,

[0145] Step S52: Directly perform data fusion on the reconstruction results of the three transformed binocular systems:

[0146]

[0147] In the formula, are respectively the point sets of the reconstruction results of the 01 binocular system and the 12 binocular system in the coordinate system of the 02 binocular system.

[0148] In one embodiment, as Figure 1 shown, a CUDA-accelerated three-camera structured light online measurement method includes the following steps:

[0149] S1 First, combine the three cameras in pairs to form the 02 binocular system, the 01 binocular system, and the 12 binocular system respectively, and perform binocular vision system calibration respectively.

[0150] S2 Then, measure a white plane at the center of the field of view, and use the three cameras to synchronously collect three-frequency four-step phase-shifted vertical stripe patterns; use the epipolar search method to perform stereo matching from camera C0 to camera C1 and from camera C1 to camera C2, and perform three-dimensional reconstruction in the three systems. Calculate the coordinate transformation of the 01 binocular system and the 12 binocular system relative to the 02 binocular system for unifying the coordinate systems of the online measurement data in the subsequent steps.

[0151] S3 The three-camera structured light system performs online measurement in the actual industrial field to obtain the measurement results of the multi-imaging system of the three-camera structured light system for the actual target to be measured. Based on the CUDA parallel computing architecture, generate image mask phases C0r, C1r, and C2r by performing matching and reconstruction in the 02 binocular system.

[0152] S4 Subsequently, perform matching and reconstruction using C0r, C1r, and C2r in the 01 binocular system and the 12 binocular system, and update the image mask phases C0r, C1r, and C2r.

[0153] Finally, S5 overlaps and removes the matched and reconstructed point cloud of the binocular system according to the updated C2r, and performs coordinate transformation on the reconstruction results of the 01 binocular system and the 12 binocular system respectively, so as to obtain the measurement results of the multi-imaging system of the three-camera structured light system of the actual target to be measured under the coordinate system of the 02 binocular system;

[0154] Among them, S10 is as Figure 2 shown. The three-camera structured light system consists of camera C0, camera C1, camera C2 and a projector. Specifically: S11 Camera C0, camera C2 and the projector form the 02 binocular system, S12 Camera C0, camera C1 and the projector form the 01 binocular system; S13 Camera C1, camera C2 and the projector form the 12 binocular system.

[0155] S20 is as Figure 3 shown, which is a schematic diagram of the corresponding point matching of the three-camera structured light system. After calibration, first synchronously collect the fringe pattern of the white plane in the center of the field of view, and calculate the coordinate transformation between each system. Specifically:

[0156] S21 uses the epipolar search method to match the phase diagrams of camera C0 and camera C2 to obtain the matching result M02;

[0157] For the camera imaging model without lens distortion, for any pixel in the left camera, theoretically, a projection straight line can be found in the right camera, and its corresponding point must be on this line, which satisfies the epipolar constraint:

[0158]

[0159] Among them, F is a 3×3 matrix, which is called the fundamental matrix. It represents the mapping from the pixel in the left camera imaging plane to the epipolar line in the right camera imaging plane. In the formula, is the undistorted pixel coordinate of the left camera, is the undistorted pixel coordinate of the corresponding point in the right camera.

[0160] For the known left camera pixel coordinates, substituting into the epipolar constraint is:

[0161]

[0162] In the formula, is the epipolar constraint vector of the left camera pixel coordinates;

[0163] The corresponding points in the right image should satisfy the formula:

[0164]

[0165] Therefore, for any horizontal coordinate of the right image on the epipolar line, the vertical coordinate can be obtained:

[0166]

[0167] S22 uses the epipolar search method to match the phase diagrams of camera C0 and camera C1, and obtains the matching result M01;

[0168] S23 uses the epipolar search method to continue the matching in the phase diagram of camera C2 for the coordinates of camera C1 in the matching result in S22, and obtains the matching result M0-12;

[0169] S24 removes the noise data according to the matching error threshold δ of the intersection of M02 and M0-12, where the matching error threshold can be selected as 0.01 < δ < 0.05 (pixels) in combination with engineering experience;

[0170] S25 traverses M02, M01, and M0-12 to find the intersection of the successful matching results of the 02 binocular system, the 01 binocular system, and the 12 binocular system;

[0171] S26 reconstructs the three-dimensional points in the 02 binocular system, the 01 binocular system, and the 12 binocular system respectively according to the matching correspondence relationship, and calculates the transformation matrices of the 01 binocular system and the 12 binocular system relative to the 02 binocular system respectively by using the SVD decomposition method according to the corresponding point relationship and :

[0172]

[0173] where 、 is the point set of the 02 binocular system in the intersection matching correspondence relationship, where is the point set of the 01 binocular system in the intersection matching correspondence relationship, where is the point set of the 12 binocular system in the intersection matching correspondence relationship.

[0174] S30 performs the phase matching calculation of the epipolar search method and subsequent reconstruction under the CUDA parallel computing framework;

[0175] Since the FPP measurement method is a structured light measurement method based on time-domain coding, the position of each pixel point on the image is independent in space, so parallel processing can be used for calculation acceleration; programming and development are carried out based on the CUDA parallel computing framework of NVIDIA graphics cards to achieve the calculation acceleration of the phase matching of the epipolar search method;

[0176] S31 copies the phase diagrams of cameras C0, C1, and C2 as C0r, C1r, and C2r, and marks C0r and C2r at the coordinate positions where the matching is successful as invalid after each matching of the 02 binocular system;

[0177] S32 performs three-dimensional reconstruction using the least squares method:

[0178]

[0179] In the formula, is the three-dimensional coordinate to be solved, , are the pixel coordinates of the left camera and the right camera in the binocular system respectively, , are the parameter matrices of the left camera and the right camera in the binocular system respectively;

[0180] Its form is: , which can be transformed into: , that is, Ax = b;

[0181] S33 Matrix solution of the least squares reconstruction method based on LU decomposition; for the reconstruction method Ax = b using the least squares, this method realizes the solution of the system of equations under CUDA through the LU decomposition method.

[0182] Matrix A can be decomposed into the product of a unit lower triangular matrix L and an upper triangular matrix U, that is:

[0183] A = LU

[0184] Where , ,

[0185] The LU decomposition formula is as follows:

[0186]

[0187] After calculating L and U, substitute them into Ax = b to get LUx = b. Denote Ux = y, then Ly = b.

[0188] First, solve for y. By calculating row by row from top to bottom for Ly = b, we can get:

[0189]

[0190] After obtaining y, x can be solved. Calculate row by row from bottom to top for Ux = y:

[0191]

[0192] S40 As Figure 4 shown, it is the comparison effect diagram of the phase diagram of camera C0 and the phase mask C0r. For the C0r phase mask diagram generated after the reconstruction of the 02 binocular system, binocular matching and reconstruction are performed on the valid pixels in it and C1r, and the coordinates of the successfully matched ones are used to update the phase C1r mask again;

[0193] S41 For the updated C1r and C2 phase mask images after binocular system reconstruction for 01, perform binocular matching and reconstruction on the valid pixels therein, and save the results in the image matrix with the same coordinates as C2r;

[0194] S42 According to the updated C2r for M01 matching, perform data overlap removal processing on the binocular system reconstruction results for 12 according to pixel coordinates.

[0195] S50 As Figure 5 shown, it is a schematic diagram of the redundancy removal reconstruction result of this method, representing the binocular system reconstruction result for 02, representing the result of C0 that was not successfully matched in 02 but reconstructed with C1 in the 01 binocular system, representing the result of C1 that was not successfully matched in 01 and reconstructed with C2r in the 12 binocular system; where C2r represents the phase that C2 failed to successfully match in the 02 system;

[0196] S51 According to the transformation matrices and calculated for the 01 binocular system and the 12 binocular system relative to the 02 binocular system in S26, directly perform coordinate transformation on the reconstruction results of the 01 binocular system and the 12 binocular system:

[0197]

[0198] S52 For the reconstruction results of the three transformed binocular systems, perform direct data fusion:

[0199] .

[0200] As Figure 6 shown, it is a comparison effect diagram of the piston part with field of view occlusion measured by the three-camera structured light system. The green is the reconstruction data of the 02 binocular system, the yellow is the reconstruction data of the 01 binocular system, and the blue is the reconstruction data of the 12 binocular system; based on the 02 reconstruction data, this method collects the fringe pattern through the camera C1 coaxial with the projector, and directly compensates for the missing part due to the field of view occlusion according to the phase characteristics. There is no obvious overlapping area for each part, and the multi-system point cloud data fusion quality is good.

[0201] As Figure 7 shown, it is a comparison effect diagram of the metal sheet with local high reflectivity measured by the three-camera structured light system. The green is the reconstruction data of the 02 binocular system, the yellow is the reconstruction data of the 01 binocular system, and the blue is the reconstruction data of the 12 binocular system; based on the 02 reconstruction data, this method compensates for the missing part of the phase information caused by the local high reflectivity of the part by collecting views from different directions, and the measurement integrity of the multi-system point cloud data is better.

[0202] Under the field of view of 200*200*100mm, the average error of the hybrid measurement of the three-camera FPP system is less than 15μm; for the three-camera FPP system with a 20-megapixel industrial camera, CUDA parallel computing is performed on the GPU platform of GTX1080. The binocular measurement of 11 million points of the plane takes less than 1.8s, and the three-camera FPP measurement of 3 million points of the piston component takes less than 1.5s.

[0203] A three-camera structured light on-line measurement based on CUDA acceleration includes

[0204] A calibration system that respectively combines three cameras, camera C0, camera C1, and camera C2, in pairs to form a 02 binocular system, a 01 binocular system, and a 12 binocular system, and respectively performs binocular vision system calibration;

[0205] A three-dimensional reconstruction unit that is used to measure a white plane at the center of the field of view and synchronously collect three-frequency four-step phase-shifted vertical stripe patterns using three cameras; uses the epipolar search method to perform stereo matching on camera C0 to camera C1 and camera C1 to camera C2, and performs three-dimensional reconstruction in the 02 binocular system, the 01 binocular system, and the 12 binocular system, and calculates the coordinate transformation of the 01 binocular system and the 12 binocular system relative to the 02 binocular system according to the matching relationship for unifying the coordinate system of on-line measurement data;

[0206] An on-line measurement unit that is used for the on-line measurement of the three-camera structured light system in an actual industrial site to obtain on-line measurement data of the multi-imaging system of the three-camera structured light system of the actual target to be measured, and generates image mask phases C0r, C1r, and C2r for matching and reconstruction in the 02 binocular system based on the CUDA parallel computing architecture;

[0207] An update unit that is used to perform matching and reconstruction in the 01 binocular system and the 12 binocular system using the image mask phases C0r, C1r, and C2r, and update the image mask phases C0r, C1r, and C2r;

[0208] A fusion unit that is used to remove overlaps from the matching and reconstructed point cloud of the 12 binocular system according to the updated image mask phase C2r, and respectively perform coordinate transformation on the reconstruction results of the 01 binocular system and the 12 binocular system. For the reconstruction results of the transformed 02 binocular system, 01 binocular system, and 12 binocular system, direct data fusion is performed to obtain the measurement results of the multi-imaging system of the three-camera structured light system of the actual target to be measured in the coordinate system of the 02 binocular system.

[0209] A computer storage medium, the storage medium includes computer instructions, when it runs on a computer, it causes the computer to execute the method.

[0210] An electronic device, the electronic device comprising:

[0211] a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein,

[0212] when the processor executes the program, the method is implemented.

[0213] Although the embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields. The above specific embodiments are merely illustrative and guiding, rather than restrictive. Those of ordinary skill in the art can also make many forms under the inspiration of this specification and without departing from the scope protected by the claims of the present invention, and all of these fall within the scope of protection of the present invention.

Claims

1. A three-camera structured light online measurement method based on CUDA acceleration, characterized in that: The steps include: Step S1, respectively combining camera C0, camera C1, and camera C2 in pairs to form a 02 binocular system, a 01 binocular system, and a 12 binocular system, and calibrating the binocular vision systems respectively; Step S2, measuring a white plane at the center of the field of view, using three cameras to synchronously collect a three-frequency four-step phase-shifted vertical stripe pattern; using an epipolar line search method to perform stereo matching from camera C0 to camera C1, and from camera C1 to camera C2, and performing three-dimensional reconstruction in the 02 binocular system, the 01 binocular system, and the 12 binocular system, and calculating the coordinate transformation of the 01 binocular system and the 12 binocular system relative to the 02 binocular system according to the matching relationship to unify the coordinate system of the online measurement data; Step S3, online measurement of the three-camera structured light system at an actual industrial site to obtain online measurement data of the multi-imaging system of the three-camera structured light system of the actual target to be measured, and matching and reconstructing the image mask phase C0r, image mask phase C1r, and image mask phase C2r in the 02 binocular system based on the CUDA parallel computing architecture; Step S4, using the image mask phase C0r, the image mask phase C1r, and the image mask phase C2r to match and reconstruct the 01 binocular system and the 12 binocular system, and updating the image mask phase C0r, the image mask phase C1r, and the image mask phase C2r; Step S5, de-overlapping the 12 binocular system matching and reconstructed point cloud according to the updated image mask phase C2r, and performing coordinate transformation on the reconstruction results of the 01 binocular system and the 12 binocular system respectively, performing direct data fusion on the transformed reconstruction results of the 02 binocular system, the 01 binocular system and the 12 binocular system, and obtaining the multi-imaging system measurement result of the three-camera structured light system of the actual target to be measured in the 02 binocular system coordinate system.

2. The three-camera structured light online measurement method based on CUDA acceleration according to claim 1, characterized in that: Preferably, step S1 comprises: Step S11, camera C0, camera C2 and projector form a binocular system 02, Step S12, camera C0, camera C1 and projector form a 01 binocular system; Step S13, the camera C1, the camera C2 and the projector form a binocular system 12.

3. The three-camera structured light online measurement method based on CUDA acceleration according to claim 1, characterized in that: Step S2 comprises, Step S20, after calibration, synchronously collect the fringe pattern of the white plane in the center of the field of view, Step S21, the coordinate transformation between the 02 binocular system, the 01 binocular system and the 12 binocular system is calculated, and the phase images of the camera C0 and the camera C2 are matched by the epipolar line search method to obtain the matching result M02. In the epipolar line search method, for any pixel in the left camera, a projection line is found in the right camera, and its related point must be on this line, which satisfies the epipolar line constraint: ， Where F is a 3×3 matrix, called the basic matrix, which represents the mapping from pixels in the left camera imaging plane to the epipolar lines in the right camera imaging plane. is the undistorted pixel coordinate of the left camera, are the undistorted pixel coordinates of the same-name points in the right camera; For the known left camera pixel coordinates, substituting the epipolar constraints into: ， In the formula, is the epipolar constraint vector of the left camera pixel coordinates; The points with the same name in the right image satisfy the formula: ， So for any horizontal coordinate of the right image on the epipolar line, find the vertical coordinate: ， Step S22, using an epipolar line search method to match the phase images of camera C0 and camera C1, and obtaining a matching result M01; Step S23, using the epipolar line search method, continue matching the coordinates of camera C1 in the matching result M01 in the phase map of camera C2 to obtain matching result M0-12; Step S24, eliminating noise data according to the matching error threshold δ of the intersection of the matching result M02 and the matching result M0-12; Step S25, traverse the matching results M02, matching results M01, matching results M0-12 to find the intersection of the successful matching results of the 02 binocular system, the 01 binocular system and the 12 binocular system; Step S26, according to the corresponding relationship of the intersection matching, reconstruct the three-dimensional points in the 02 binocular system, the 01 binocular system and the 12 binocular system respectively, and according to the corresponding point relationship, use the SVD decomposition method to calculate the transformation matrix of the 01 binocular system and the 12 binocular system relative to the 02 binocular system respectively and : ， In the formula, , is the point set of the 02 binocular system in the intersection matching correspondence, is the point set of the 01 binocular system in the intersection matching correspondence, where It is the point set of 12 binocular systems in the intersection matching correspondence.

4. The three-camera structured light online measurement method based on CUDA acceleration according to claim 3 is characterized in that: The matching error threshold δ is 0.01 pixel < δ < 0.05 pixel.

5. The three-camera structured light online measurement method based on CUDA acceleration according to claim 1, characterized in that: Step S3 comprises, Step S30, performing phase matching calculation and subsequent reconstruction using an epipolar line search method under a CUDA parallel computing framework; Step S31, the phase images of camera C0, camera C1 and camera C2 are copied into image mask phase C0r, image mask phase C1r and image mask phase C2r, and after each 02 binocular system matching, the image mask phase C0r and image mask phase C2r at the coordinate position where the matching is successful are marked as invalid; Step S32, using the least squares method to perform three-dimensional reconstruction: ， In the formula, is the three-dimensional coordinate to be solved, , They are the pixel coordinates of the left camera and the right camera in the binocular system, , They are the parameter matrices of the left camera and the right camera in the binocular system, respectively, and their forms are: , can be transformed into: , that is, Ax=b; Step S33, matrix solution of the least squares reconstruction method based on LU decomposition; for the least squares reconstruction method Ax=b, the matrix A is decomposed into the product of the unit lower triangular matrix L and the upper triangular matrix U, that is: A=LU in , , , The LU decomposition formula is as follows: ， After calculating L and U, substitute them into Ax=b to get LUx=b, and remember Ux=y, then Ly=b, First, solve for y, and calculate Ly=b row by row from top to bottom: ， After finding y, we can solve for x, and calculate Ux=y row by row from bottom to top: 。 6. The three-camera structured light online measurement method based on CUDA acceleration according to claim 1, characterized in that: Step S4 comprises, Step S40, for the image mask phase C0r generated after the reconstruction of the 02 binocular system, binocular matching and reconstruction are performed on the valid pixels and the image mask phase C1r, and the image mask phase C1r is updated again with the coordinates of the successful matching; Step S41, for the image mask phase C1r and the image mask phase C2r updated after the reconstruction of the 01 binocular system, binocular matching and reconstruction are performed on the valid pixels therein, and the results are stored in the image matrix with the same coordinates as the image mask phase C2r; Step S42, matching the updated image mask phase C2r according to the matching result M01, and performing data de-overlapping processing on the reconstruction result of the binocular system 12 according to the pixel coordinates.

7. The three-camera structured light online measurement method based on CUDA acceleration according to claim 1, characterized in that: Step S5 comprises, Step S50, Indicates the reconstruction result of 02 binocular system, Indicates that camera C0 is not successfully matched in the 02 binocular system but is reconstructed with camera C1 in the 01 binocular system. It indicates the result of reconstructing the phase C2r in the 12 binocular system when the camera C1 is not successfully matched in the 01 binocular system and C2 is not successfully matched in the 02 binocular system; Step S51, according to the transformation matrix of 01 binocular system and 12 binocular system relative to 02 binocular system and , directly transform the coordinates of the reconstruction results of the 01 binocular system and the 12 binocular system: ， Step S52, performing direct data fusion on the three binocular system reconstruction results after transformation: , where They are the point sets of the reconstruction results of the 01 binocular system and the 12 binocular system in the 02 binocular system coordinate system.

8. A three-camera structured light online measurement based on CUDA acceleration, characterized in that: These include, A calibration system, which respectively combines the three cameras C0, C1 and C2 in pairs to form a 02 binocular system, a 01 binocular system and a 12 binocular system and calibrates the binocular vision systems respectively; A three-dimensional reconstruction unit is used to measure a white plane at the center of the field of view, using three cameras to synchronously collect a three-frequency four-step phase-shifted vertical stripe pattern; using an epipolar line search method to perform stereo matching from camera C0 to camera C1, and from camera C1 to camera C2, and perform three-dimensional reconstruction in the 02 binocular system, the 01 binocular system, and the 12 binocular system; and according to the matching relationship, the coordinate transformation of the 01 binocular system and the 12 binocular system relative to the 02 binocular system is calculated to unify the coordinate system of the online measurement data; The online measurement unit is used for online measurement of the three-camera structured light system in the actual industrial site to obtain the online measurement data of the multi-imaging system of the three-camera structured light system of the actual target to be measured, and performs matching reconstruction in the 02 binocular system based on the CUDA parallel computing architecture to generate the image mask phase C0r, image mask phase C1r, and image mask phase C2r; An updating unit, which is used to use the image mask phase C0r, the image mask phase C1r, and the image mask phase C2r to match and reconstruct the 01 binocular system and the 12 binocular system, and to update the image mask phase C0r, the image mask phase C1r, and the image mask phase C2r; A fusion unit is used to de-overlap the 12 binocular system matching and reconstructed point cloud according to the updated image mask phase C2r, and to perform coordinate transformation on the reconstruction results of the 01 binocular system and the 12 binocular system respectively, and to perform direct data fusion on the transformed reconstruction results of the 02 binocular system, the 01 binocular system and the 12 binocular system, so as to obtain the multi-imaging system measurement result of the three-camera structured light system of the actual target to be measured in the 02 binocular system coordinate system.

9. A computer storage medium, characterized in that The storage medium includes computer instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The electronic device comprises: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images