A method for 3D point cloud reconstruction of highly reflective surface objects

Abstract

The present invention relates to the technical field of three-dimensional point cloud reconstruction, and specifically to a method for reconstructing three-dimensional point clouds of highly reflective surface objects, including obtaining left and right multi-step phase shift fringes of left and right cameras, generating left and right unfolded phase maps based on the phase shift fringes, and positioning left and right camera ROI areas based on the left and right unfolded phase maps; calibrating the screen extrinsic parameters and camera extrinsic parameters of the binocular camera system, and unifying the coordinates of the left and right cameras; based on the positioned left and right camera ROI areas and the coordinates of the left and right cameras after the unified coordinates, performing epipolar correction on the object images with reference point features obtained by the left and right cameras to obtain initial 3D coordinates; based on the initial 3D coordinates, obtaining the corresponding surface normal vectors of each pixel point on the left and right camera views; calculating each normal vector gradient based on the surface normal vector, calculating the height by integration, and obtaining the surface 3D point cloud coordinates. While improving the calculation accuracy, the present invention maintains reasonable calculation efficiency and meets actual use requirements.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional point cloud reconstruction, and in particular to a method for three-dimensional point cloud reconstruction of a high-reflective surface object. Background Art

[0002] In many industrial and scientific research fields, such as automobile manufacturing, robot vision, cultural heritage protection, and medical imaging, accurate 3D reconstruction of objects with mirror or high reflective properties is an important part of the measurement process. Due to their surface characteristics, these objects will cause strong reflection of incident light, making it difficult for traditional optical measurement technology to capture their surface details. For example, in the automotive industry, the metal body parts of vehicles are usually highly reflective, and in cultural relics protection, the surfaces of ancient bronzes, sculptures and other artworks often have mirror effects. Therefore, developing a 3D point cloud reconstruction method that can effectively handle mirror or highly reflective objects is of great significance to improving the technical level in the field of 3D measurement of highly reflective objects.

[0003] At present, the mainstream idea for the main method of 3D reconstruction of high-reflective surfaces is to build a binocular camera system, and then cooperate with the LCD display to project multiple phase-shifted fringe patterns to the left and right cameras to obtain the unfolded horizontal and vertical phase data. Since the phases of the same point of the high-reflective object projected to the left and right cameras do not match, it is impossible to reconstruct the three-dimensional points in space through the same corresponding phase values ​​of the left and right cameras. There is a characteristic for high-reflective objects, that is, the normal vectors calculated by the corresponding positions of the pixels obtained by the left and right cameras in the actual object in space are consistent. Based on this basic characteristic, the surface point cloud data of the high-reflective object can be reconstructed by matching the left and right normal vectors.

[0004] In the process of reconstructing surface data based on the above method, it is necessary to calculate the normal vectors of the corresponding points of the left and right cameras. The coordinates of three points in space need to be known in the calculation of the normal vectors: 1) the emission point is the coordinate of the light emitted from a point on the display screen; 2) the reflection point is the coordinate of the light reflected on the surface of the object; 3) the coordinate of the incident point where the camera receives the light. However, the reflection point itself is the result that needs to be calculated, which leads to the conditions of this calculation method not being met. At present, the left and right normal vectors are mainly calculated by estimating the initial point in space, and then gradually approaching the actual position through the matching process of the normal vector. The limitation of this method is that it will lead to a slow convergence speed of calculation. If the selection of the initial point deviates greatly from the actual position, it will also have a great impact on the calculation accuracy. In addition, after obtaining the surface normal vector, it is necessary to calculate the gradient and then restore the height information by integrating the gradient to finally realize the reconstruction of the surface point cloud. The current main reconstruction algorithms include: direct integration method, Fourier transform integration method, regional wavefront reconstruction method, spline function wavefront reconstruction method, etc. These methods have their own limitations. Among them, the direct reconstruction method has the fastest calculation speed and the largest error; while the spline function wavefront reconstruction method has the highest accuracy and is time-consuming. Therefore, a method for reconstructing three-dimensional point clouds of highly reflective surface objects is urgently needed to solve this problem. Summary of the invention

[0005] In view of the above problems, the present invention is proposed to provide a method for reconstructing a three-dimensional point cloud of a highly reflective surface object that overcomes the above problems or at least partially solves the above problems.

[0006] In order to solve the above technical problems, the embodiments of the present application disclose the following technical solutions:

[0007] The embodiment of the present invention discloses a method for reconstructing a three-dimensional point cloud of a high-reflective surface object, comprising:

[0008] Obtain left and right multi-step phase shift fringes of left and right cameras, generate left and right unfolded phase images based on the phase shift fringes, and locate the ROI areas of the left and right cameras based on the left and right unfolded phase images;

[0009] Calibrate the screen extrinsic parameters and camera extrinsic parameters of the binocular camera system, and unify the coordinates of the left and right cameras;

[0010] Based on the ROI areas of the left and right cameras after positioning and the coordinates of the left and right cameras after unified coordinates, the object images with reference point features acquired by the left and right cameras are corrected by epipolar lines to obtain the initial 3D coordinates;

[0011] Based on the initial 3D coordinates, the corresponding surface normal vector of each pixel point on the left and right camera views is obtained; based on the surface normal vector, each normal vector gradient is calculated, and the height is calculated by integration to obtain the surface 3D point cloud coordinates.

[0012] Further, the left and right multi-step phase shift fringes of the left and right cameras are obtained, the left and right unfolded phase images are generated based on the phase shift fringes, and the ROI areas of the left and right cameras are located based on the left and right unfolded phase images. The specific method includes:

[0013] Generate horizontal and vertical phase shift images in sequence on a display screen, project the phase shift images onto the photographed object, and obtain phase shift image projection data, which is used to obtain an unwrapped phase for a binocular camera;

[0014] Use the display screen to generate a feature point array map, project the feature point array map onto the photographed object, and obtain feature point array map projection data, which is used to calculate the subsequent initial spatial coordinate points. The size of the center point of the dot map needs to be adjusted according to the actual situation of the project;

[0015] Extract phase shift images from left and right cameras, and calculate horizontal and vertical unwrapped phases based on phase synthesis and phase unwrapping algorithms;

[0016] Using the characteristic that the brightness of the stripes in the phase shift image is different when they hit the background and the object, the ROI area projected onto the left and right cameras is obtained.

[0017] Furthermore, the screen extrinsic parameters and camera extrinsic parameters of the binocular camera system are calibrated. The specific method includes:

[0018] Use a checkerboard calibration plate or a circular calibration plate, and use the binocular camera system to shoot multiple calibration plate pose images to calibrate the memory and external parameters of the left and right cameras. Use a display screen to display the calibration pattern, set up a plane mirror in front, and use a binocular camera to shoot the calibration pattern to complete the calibration of the external parameters between the display screen and the main camera.

[0019] Furthermore, the coordinates of the left and right cameras are unified. The specific method includes: using the optical center of the left camera as the origin of the world coordinate system, and unifying the coordinates of the right camera and the display screen coordinates into the main camera coordinate system.

[0020] Furthermore, based on the ROI areas of the left and right cameras after positioning and the coordinates of the left and right cameras after unified coordinates, epipolar correction is performed on the object images with reference point features acquired by the left and right cameras to obtain initial 3D coordinates. The specific method includes: based on a visual algorithm, extracting the coordinates of other points outside the center point of the feature array; extracting the coordinates of the array center point of the left and right cameras according to the size characteristics of the pre-set feature array center; matching the coordinates extracted from the feature maps of the left and right cameras, first matching the position coordinates of the left and right center points; taking the center point as a reference, matching other feature points adjacent to the center point one by one to obtain a set of all feature points that can be matched; using the currently obtained set of matching points and the parameters obtained by the binocular camera calibration, the spatial coordinates of the initial reference matching point are obtained based on the triangulation method.

[0021] Further, based on the initial 3D coordinates, the surface normal vector corresponding to each pixel point on the left and right camera views is obtained, and the specific method includes:

[0022] Traverse the pixel point in the i-th row and j-th column of the camera view to determine whether the pixel point is matched for the first time. If the pixel point is matched for the first time, find a suitable spatial reference point based on the distance from the pixel point to the center of the image and the center position of the reference point.

[0023] Calculate the vector of reflected light by combining the camera pixel coordinates and space point coordinates;

[0024] Based on the camera pixel points, find the coordinates of the corresponding pixel points on the display screen according to the unfolded phase data;

[0025] Calculate the vector of the outgoing light by combining the camera pixel coordinates and the space point coordinates;

[0026] Combining the outgoing ray vector with the reflected ray vector, the normal vector is calculated.

[0027] Further, based on the initial 3D coordinates, a surface normal vector corresponding to each pixel point on the left and right camera views is obtained, and the specific method also includes:

[0028] Traverse the pixel point in the i-th row and j-th column of the camera view to determine whether the pixel point is matched for the first time. If the pixel point is not matched for the first time, use the point cloud data generated by the previous pixel to obtain the spatial coordinate point of the point cloud data at the current pixel point.

[0029] Calculate the vector of reflected light by combining the camera pixel coordinates and space point coordinates;

[0030] Based on the camera pixel points, find the coordinates of the corresponding pixel points on the display screen according to the unfolded phase data;

[0031] Calculate the vector of the outgoing light by combining the camera pixel coordinates and the space point coordinates;

[0032] Combining the outgoing ray vector with the reflected ray vector, the normal vector is calculated.

[0033] Furthermore, each normal vector gradient is calculated based on the surface normal vector, and the height is calculated by integration to obtain the surface 3D point cloud coordinates. The height calculation formula is as follows:

[0034]

[0035] Among them, Z cis the height value in the height matrix, Er is the coefficient matrix in the row direction corresponding to the differential operator, Ec is the coefficient matrix in the column direction corresponding to the differential operator, Sr is the differential vector estimated in the row direction, and Sc is the differential vector estimated in the column direction.

[0036] Furthermore, the calculation method of the coefficient matrix Er in the row direction corresponding to the difference operator and the coefficient matrix Ec in the column direction corresponding to the difference operator is as follows:

[0037]

[0038] Where m and n represent the row and column numbers, M1={1,2, ..., M(N-1)}, M2={1,2, ... ,(M-1) / N}, M and N are the maximum row and column widths of the camera, respectively. floor(a) means taking the nearest integer value that is less than or equal to the element a, and rem(a1,a2) means the remainder after dividing a1 and a2.

[0039] Furthermore, a method for reconstructing a three-dimensional point cloud of a highly reflective surface object also includes: using the updated surface 3D point cloud coordinates to repeatedly obtain the corresponding surface normal vector for each pixel point on the left and right camera views; calculating each normal vector gradient based on the surface normal vector, calculating the height by integration, and obtaining the surface 3D point cloud coordinates, until the difference between the two obtained 3D point cloud coordinates is less than a given threshold, as the final 3D point cloud coordinate result.

[0040] The beneficial effects of the above technical solution provided by the embodiment of the present invention include at least:

[0041] The present invention discloses a method for reconstructing three-dimensional point clouds of highly reflective surface objects based on reference point iteration and improved gradient integration algorithm. The method uses a feature point array to be projected onto the object, and then generates initial spatial coordinate points by matching feature points, thereby replacing random selection and obtaining initial coordinate points by 3D scanning and the like. Compared with the traditional method, this invention has the following advantages: 1. On the basis of shooting the phase-shift fringe pattern and the object view, only one more feature dot matrix pattern needs to be shot, which increases the time by a limited amount compared to 3D scanning the surface of the object; compared to randomly adding initial points in space, the initial spatial points obtained by this method are more reasonable. 2. Compared with the traditional gradient integration method, the present invention adds nonlinear calculation items, thereby improving the interference of noise in the integration process, so that the reconstructed height maintains a higher accuracy, while increasing the calculation time by very little. Through this method, it is possible to improve the calculation accuracy while maintaining a reasonable calculation efficiency, thereby meeting the actual use requirements.

[0042] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0044] Figure 1 This is a flow chart of a method for reconstructing a three-dimensional point cloud of a highly reflective surface object in Embodiment 1 of the present invention;

[0045] Figure 2 A flow chart of generating initial space points based on reference point matching in embodiment 1 of the present invention;

[0046] Figure 3 This is a normal vector calculation flow chart in Example 1 of the present invention. DETAILED DESCRIPTION

[0047] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0048] In order to solve the problems existing in the prior art, an embodiment of the present invention provides a method for reconstructing a three-dimensional point cloud of a highly reflective surface object.

[0049] Embodiment 1:

[0050] The present invention discloses a method for reconstructing a three-dimensional point cloud of a highly reflective surface object. Figure 1 ,include:

[0051] S100. Obtain the left and right multi-step phase shift fringes of the left and right cameras, generate left and right unfolded phase images based on the phase shift fringes, and locate the ROI areas of the left and right cameras based on the left and right unfolded phase images; wherein the phase shift method is a method of using multiple interference fringes to interpret the phase field by loading different conditions on the phase shifter to move the reference mirror, thereby forming moving fringes on the interference fringe image. Since multiple images have more advantages than one image, such as providing more information and there is no phase sign ambiguity problem, the calculation result is more accurate. This method is widely used in optical measurement and is one of the most important developments in this research field in modern times.

[0052] In a binocular camera system, the left and right cameras are two cameras placed opposite to each other, capturing images of the same scene from different perspectives. The two cameras usually have similar internal parameters (such as focal length, optical center position, distortion coefficient, etc.), but their external parameters (i.e., position and posture in the world coordinate system) are different. The configuration of the left and right cameras enables them to capture image pairs with parallax, which is the basis for stereo vision and 3D reconstruction.

[0053] In this embodiment S100, left and right multi-step phase shift fringes of left and right cameras are obtained, left and right unfolded phase images are generated based on the phase shift fringes, and the ROI areas of the left and right cameras are positioned based on the left and right unfolded phase images. The specific method includes:

[0054] Generate horizontal and vertical phase shift images in sequence on a display screen, project the phase shift images onto the photographed object, and obtain phase shift image projection data, which is used to obtain an unwrapped phase for a binocular camera;

[0055] Use the display screen to generate a feature point array map, project the feature point array map onto the photographed object, and obtain feature point array map projection data, which is used to calculate the subsequent initial spatial coordinate points. The size of the center point of the dot map needs to be adjusted according to the actual situation of the project;

[0056] Extract phase shift images from left and right cameras, and calculate horizontal and vertical unwrapped phases based on phase synthesis and phase unwrapping algorithms;

[0057] Using the characteristic that the brightness of the stripes in the phase shift image is different when they hit the background and the object, the ROI area projected onto the left and right cameras is obtained.

[0058] By obtaining the ROI area of ​​the object projected to the left and right cameras, it is convenient for subsequent calculations to only consider the object-related area.

[0059] S200. Calibrate the screen external parameters and camera internal and external parameters of the binocular camera system, and unify the coordinates of the left and right cameras; the calibration of the binocular camera system is an important step in computer vision, which involves determining the internal parameters (intrinsic parameters) and external parameters (extrinsic parameters) of the camera, as well as the relative position relationship between the left and right cameras. These parameters are crucial for subsequent stereo matching, 3D reconstruction and other tasks.

[0060] In S200 of this embodiment, the screen extrinsic parameters and the camera extrinsic parameters of the binocular camera system are calibrated, and the specific method includes:

[0061] Use a checkerboard calibration plate or a circular calibration plate, and use the binocular camera system to shoot multiple calibration plate pose images to calibrate the memory and external parameters of the left and right cameras. Use a display screen to display the calibration pattern, set up a plane mirror in front, and use a binocular camera to shoot the calibration pattern to complete the calibration of the external parameters between the display screen and the main camera.

[0062] In S200 of this embodiment, the coordinates of the left and right cameras are unified. The specific method includes: using the optical center of the left camera as the origin of the world coordinate system, and unifying the coordinates of the right camera and the display screen coordinates into the main camera coordinate system.

[0063] Through the above steps, the binocular camera system can be calibrated and the coordinates of the left and right cameras can be unified, which is crucial for subsequent stereo vision tasks.

[0064] S300. Based on the ROI areas of the left and right cameras after positioning and the coordinates of the left and right cameras after unification, epipolar correction is performed on the object images with reference point features obtained by the left and right cameras to obtain initial 3D coordinates; wherein, epipolar correction of the object images with reference point features obtained by the left and right cameras is an important step in stereo vision, which aims to correct the corresponding points in the left and right camera images to the same horizontal line (i.e., epipolar line), thereby simplifying the subsequent stereo matching process. Once the epipolar correction is completed, the initial 3D coordinates of the object can be calculated using the disparity information.

[0065] In S300 of this embodiment, based on the positioned ROI areas of the left and right cameras and the coordinates of the left and right cameras after the coordinates are unified, epipolar correction is performed on the object images with reference point features acquired by the left and right cameras to obtain the initial 3D coordinates, such as Figure 2 The specific method includes: extracting the coordinates of other points outside the center point of the feature array based on a visual algorithm; extracting the coordinates of the array center point of the left and right cameras according to the pre-set size characteristics of the center of the feature array; matching the coordinates extracted from the feature maps of the left and right cameras, first matching the position coordinates of the left and right center points; taking the center point as a reference, matching other feature points adjacent to the center point one by one to obtain a set of all feature points that can be matched; using the currently obtained set of matching points and the parameters obtained by the binocular camera calibration, based on the triangulation method, obtaining the spatial coordinates of the initial reference matching point.

[0066] S400. Based on the initial 3D coordinates, obtain the corresponding surface normal vector of each pixel point on the left and right camera views; calculate each normal vector gradient based on the surface normal vector, calculate the height by integration, and obtain the surface 3D point cloud coordinates.

[0067] Specifically, in S400 of this embodiment, based on the initial 3D coordinates, the surface normal vector corresponding to each pixel point on the left and right camera views is obtained, such as Figure 3, the specific methods include:

[0068] Traverse the pixel point in the i-th row and j-th column of the camera view to determine whether the pixel point is matched for the first time. If the pixel point is matched for the first time, find a suitable spatial reference point based on the distance from the pixel point to the center of the image and the center position of the reference point.

[0069] Calculate the vector of reflected light by combining the camera pixel coordinates and space point coordinates;

[0070] Based on the camera pixel points, find the coordinates of the corresponding pixel points on the display screen according to the unfolded phase data;

[0071] Calculate the vector of the outgoing light by combining the camera pixel coordinates and the space point coordinates;

[0072] Combining the outgoing ray vector with the reflected ray vector, the normal vector is calculated.

[0073] In some preferred embodiments, if the pixel point is not matched for the first time, the point cloud data generated by the previous pixel is used to obtain the spatial coordinate point of the point cloud data at the current pixel point;

[0074] Calculate the vector of reflected light by combining the camera pixel coordinates and space point coordinates;

[0075] Based on the camera pixel points, find the coordinates of the corresponding pixel points on the display screen according to the unfolded phase data;

[0076] Calculate the vector of the outgoing light by combining the camera pixel coordinates and the space point coordinates;

[0077] Combining the outgoing ray vector with the reflected ray vector, the normal vector is calculated.

[0078] In this embodiment, after obtaining the surface normal vector, each normal vector gradient is calculated based on the surface normal vector, and the height is calculated by integration to obtain the surface 3D point cloud coordinates. The height calculation formula is as follows:

[0079]

[0080] Among them, Z c is the height value in the height matrix, Er is the coefficient matrix in the row direction corresponding to the differential operator, Ec is the coefficient matrix in the column direction corresponding to the differential operator, Sr is the differential vector estimated in the row direction, and Sc is the differential vector estimated in the column direction.

[0081] In this embodiment, the calculation method of the coefficient matrix Er in the row direction corresponding to the difference operator and the coefficient matrix Ec in the column direction corresponding to the difference operator is as follows:

[0082]

[0083] Where m and n represent the row and column numbers, M1={1,2, ..., M(N-1)}, M2={1,2, ... ,(M-1) / N}, M and N are the maximum row and column widths of the camera, respectively. floor(a) means taking the nearest integer value that is less than or equal to the element a, and rem(a1,a2) means the remainder after dividing a1 and a2.

[0084] In this embodiment, the calculation method of the difference vectors Sr and Sc is:

[0085]

[0086] Where Zx and Zy are the gradient values ​​in the x-direction and y-direction respectively, and X and Y are the data coordinate values ​​in the x-direction and y-direction respectively.

[0087] In this embodiment, after obtaining the surface 3D point cloud coordinates, the updated surface 3D point cloud coordinates are used to repeatedly obtain the corresponding surface normal vectors for each pixel point on the left and right camera views; each normal vector gradient is calculated based on the surface normal vector, and the height is calculated by integration to obtain the surface 3D point cloud coordinates, until the difference between the two obtained 3D point cloud coordinates is less than a given threshold, as the final 3D point cloud coordinate result.

[0088] The method for reconstructing a three-dimensional point cloud of a highly reflective surface object based on reference point iteration and improved gradient integration algorithm disclosed in this embodiment uses a feature point array to be projected onto the object, and then generates initial spatial coordinate points by matching feature points, thereby replacing random selection and obtaining initial coordinate points by 3D scanning and the like. Compared with the traditional method, this invention has the following advantages: 1. On the basis of shooting the phase-shift fringe pattern and the object view, only one more feature dot matrix image needs to be shot, which increases the time by a limited amount compared to 3D scanning the surface of the object; compared to randomly adding initial points in space, the initial spatial points obtained by this method are more reasonable. 2. Compared with the traditional gradient integration method, the present invention adds nonlinear calculation items, thereby improving the interference of noise in the integration process, so that the reconstructed height maintains a higher accuracy, while increasing the calculation time very little. Through this method, it is possible to improve the calculation accuracy while maintaining a reasonable calculation efficiency, thereby meeting the actual use requirements.

[0089] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0090] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are clearly stated in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0091] Those skilled in the art will also appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments herein can all be implemented as electronic hardware, computer software, or a combination thereof. In order to clearly illustrate the interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above around their functions. Whether such functions are implemented as hardware or software depends on specific applications and the design constraints imposed on the entire system. A skilled person can implement the described functions in an alternative manner for each specific application, but such implementation decisions should not be interpreted as departing from the scope of protection of the present disclosure.

[0092] The steps of the method or algorithm described in conjunction with the embodiments herein may be directly embodied as hardware, a software module executed by a processor, or a combination thereof. The software module may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a mobile disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium may also be an integral part of the processor. The processor and the storage medium may be located in an ASIC. The ASIC may be located in a user terminal. Of course, the processor and the storage medium may also be present in a user terminal as discrete components.

[0093] For software implementation, the techniques described in this application can be implemented with modules (e.g., procedures, functions, etc.) that perform the functions described in this application. These software codes can be stored in a memory unit and executed by a processor. The memory unit can be implemented within the processor or outside the processor. In the latter case, it is coupled to the processor in a communication manner via various means, which are well known in the art.

[0094] The above description includes examples of one or more embodiments. Of course, it is impossible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but it should be recognized by those skilled in the art that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of protection of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, the word is covered in a manner similar to the term "including", just as "including," is explained as a transitional word in the claims. In addition, any term "or" used in the specification of the claims is intended to mean "non-exclusive or".

Claims

1. A method for reconstructing a three-dimensional point cloud of a highly reflective surface object, characterized in that: include: Obtain left and right multi-step phase shift fringes of left and right cameras, generate left and right unfolded phase images based on the phase shift fringes, and locate the ROI areas of the left and right cameras based on the left and right unfolded phase images; the specific method includes: Generate horizontal and vertical phase shift images in sequence on a display screen, project the phase shift images onto the photographed object, and obtain phase shift image projection data, which is used to obtain an unwrapped phase for a binocular camera; Use the display screen to generate a feature point array map, project the feature point array map onto the photographed object, and obtain feature point array map projection data, which is used to calculate the subsequent initial spatial coordinate points. The size of the center point of the dot map needs to be adjusted according to the actual situation of the project; Extract phase shift images from left and right cameras, and calculate horizontal and vertical unwrapped phases based on phase synthesis and phase unwrapping algorithms; Using the characteristic that the brightness of the stripes in the phase shift image is different when they hit the background and the object, the ROI area projected onto the left and right cameras is obtained; Calibrate the screen extrinsic parameters and camera extrinsic parameters of the binocular camera system, and unify the coordinates of the left and right cameras; Based on the ROI areas of the left and right cameras after positioning and the coordinates of the left and right cameras after unified coordinates, the object images with reference point features acquired by the left and right cameras are corrected by epipolar lines to obtain the initial 3D coordinates; Based on the initial 3D coordinates, a surface normal vector corresponding to each pixel point on the left and right camera views is obtained; based on the surface normal vector, each normal vector gradient is calculated, and the height is calculated by integration to obtain the surface 3D point cloud coordinates; Based on the initial 3D coordinates, the surface normal vector corresponding to each pixel point on the left and right camera views is obtained. The specific method includes: Traverse the pixel point in the i-th row and j-th column of the camera view to determine whether the pixel point is matched for the first time. If the pixel point is matched for the first time, find a suitable spatial reference point based on the distance from the pixel point to the center of the image and the center position of the reference point. Calculate the vector of reflected light by combining the camera pixel coordinates and space point coordinates; Based on the camera pixel points, find the coordinates of the corresponding pixel points on the display screen according to the unfolded phase data; Calculate the vector of the outgoing light by combining the camera pixel coordinates and the space point coordinates; Combine the vector of the outgoing ray and the vector of the reflected ray to calculate the normal vector; Based on the initial 3D coordinates, a surface normal vector corresponding to each pixel point on the left and right camera views is obtained, and the specific method further includes: Traverse the pixel point in the i-th row and j-th column of the camera view to determine whether the pixel point is matched for the first time. If the pixel point is not matched for the first time, use the point cloud data generated by the previous pixel to obtain the spatial coordinate point of the point cloud data at the current pixel point. Calculate the vector of reflected light by combining the camera pixel coordinates and space point coordinates; Based on the camera pixel points, find the coordinates of the corresponding pixel points on the display screen according to the unfolded phase data; Calculate the vector of the outgoing light by combining the camera pixel coordinates and the space point coordinates; Combine the vector of the outgoing ray and the vector of the reflected ray to calculate the normal vector; Based on the surface normal vector, each normal vector gradient is calculated, and the height is calculated by integration to obtain the surface 3D point cloud coordinates. The height calculation formula is as follows: Among them, Z c is the height value in the height matrix, Er is the coefficient matrix in the row direction corresponding to the differential operator, Ec is the coefficient matrix in the column direction corresponding to the differential operator, Sr is the differential vector estimated in the row direction, and Sc is the differential vector estimated in the column direction; The calculation method of the coefficient matrix Er in the row direction corresponding to the difference operator and the coefficient matrix Ec in the column direction corresponding to the difference operator are as follows: Where m and n represent the row and column numbers, M1 = {1, 2, ..., M(N-1)}, M2 = {1, 2, ..., (M-1) / N}, M and N are the maximum row and column widths of the camera respectively, floor(a) means taking the nearest integer value less than or equal to the element a, and rem(a1, a2) means the remainder after dividing a1 and a2.

2. A method for reconstructing a three-dimensional point cloud of a highly reflective surface object as claimed in claim 1, characterized in that: The screen extrinsic parameters and camera extrinsic parameters of the binocular camera system are calibrated. The specific methods include: Use a checkerboard calibration plate or a circular calibration plate, and use the binocular camera system to shoot multiple calibration plate pose images to calibrate the memory and external parameters of the left and right cameras. Use a display screen to display the calibration pattern, set up a plane mirror in front, and use a binocular camera to shoot the calibration pattern to complete the calibration of the external parameters between the display screen and the main camera.

3. The method for reconstructing a three-dimensional point cloud of a highly reflective surface object according to claim 1, characterized in that: The coordinates of the left and right cameras are unified. The specific method includes: using the optical center of the left camera as the origin of the world coordinate system, and unifying the coordinates of the right camera and the display screen into the main camera coordinate system.

4. The method for reconstructing a three-dimensional point cloud of a highly reflective surface object according to claim 1, characterized in that: Based on the ROI areas of the left and right cameras after positioning and the coordinates of the left and right cameras after unified coordinates, the polar line correction is performed on the object pictures with reference point features obtained by the left and right cameras to obtain the initial 3D coordinates. The specific method includes: based on the visual algorithm, the coordinates of other points outside the center point of the feature array are extracted; according to the size characteristics of the pre-set feature array center, the coordinates of the array center point of the left and right cameras are extracted; the coordinates extracted from the feature maps of the left and right cameras are matched, and the position coordinates of the left and right center points are matched first; with the center point as the reference, other feature points adjacent to the center point are matched one by one to obtain a set of all feature points that can be matched; using the currently obtained set of matching points and the parameters obtained by the binocular camera calibration, the spatial coordinates of the initial reference matching point are obtained based on the triangulation method.

5. The method for reconstructing a three-dimensional point cloud of a highly reflective surface object according to claim 1, characterized in that: Also includes: Using the updated surface 3D point cloud coordinates, repeatedly obtain the corresponding surface normal vectors for each pixel point on the left and right camera views; calculate each normal vector gradient based on the surface normal vector, calculate the height by integration, and obtain the surface 3D point cloud coordinates, until the difference between the two obtained 3D point cloud coordinates is less than a given threshold, as the final 3D point cloud coordinate result.

Images