Non-coding point matching and stereo reconstruction method and system based on spatial invariance

Abstract

The present invention discloses a non-coding point matching and stereo reconstruction method based on spatial invariance, which mainly includes that after the binocular camera obtains the image information of the object to be measured, the collected image is stereo corrected, and then the centroid of the non-coding points existing in the left and right same-frame images is extracted, and the two-dimensional same-name points of the left and right camera same-frame images are found according to the spatial invariance of the point space, and then the three-dimensional point cloud information of the non-coding points that are successfully matched in each frame is restored according to the two-dimensional same-name point matching results and the calibration parameters of the camera; then, based on the characteristics of multiple spatial invariances, the three-dimensional same-name points between different point clouds are found, the transformation relationship of different point clouds is calculated, and the non-coding point space three-dimensional model of the object to be measured is constructed, and then the three-dimensional reconstruction of the object can be realized by combining the structural information of the line structured light scanner. The present invention provides a real-time three-dimensional reconstruction method with strong robustness, accuracy and efficiency, which has broad application prospects in intelligent driving, smart retail, medical beauty and other aspects.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction of computer vision, and in particular to a non-coding point matching and three-dimensional reconstruction method and system based on space invariance. Background Art

[0002] When multi-line structured light binocular vision scanning equipment scans an object, it can usually only obtain less information per frame, and multiple frames of point cloud data need to be spliced ​​to complete the 3D reconstruction of the object. By pre-pasting non-coding points around the object, it can stably provide distinct feature points in each frame of the image, improving the speed and accuracy of 3D reconstruction. How to accurately and quickly obtain the 3D information of non-coding points and how to efficiently splice the restored multi-frame point cloud data have become urgent problems to be solved. Summary of the invention

[0003] The main purpose of the present invention is to provide a non-coding point matching and stereo reconstruction method and system based on spatial invariance with good robustness, high speed and high precision.

[0004] The technical solution adopted by the present invention is:

[0005] A non-coding point matching and stereo reconstruction method based on spatial invariance is provided, comprising the following steps:

[0006] S1. Use a binocular camera to collect image information of the object to be measured, and perform image epipolar correction on the left and right views of the same frame;

[0007] S2, extracting the centroid coordinates of the non-encoded points pre-pasted on the object to be measured in the left and right same frame views;

[0008] S3. According to the relationship between the polar lines between the non-coding points, two-dimensional same-name point matching is performed for the easy-to-match non-coding points without interference points; for the easy-to-match non-coding points with interference points, among the successfully matched non-coding points, the three points closest to the view where the easy-to-match non-coding points to be tested are located are found according to the coordinates of the centroid of the non-coding points, forming a triangle, and calculating the deviation distance from the easy-to-match non-coding points to be tested to the centroid of the triangle; and calculating the centroid of the corresponding triangle in another view, the point with the closest distance to the centroid of the corresponding triangle and the deviation distance is taken as the two-dimensional same-name point of the easy-to-match non-coding points to be tested;

[0009] S4, based on the 2D same-name point matching results and the camera calibration parameters, restore the 3D point cloud information of the non-coded points of the current frame, and calculate the Euclidean distance set between each point in the point cloud and other points, and process all frames in the same way;

[0010] S5. Find a pair of 3D points of the same name from two different point clouds through the constraints of Euclidean distance and relative position between points, regard the pair of 3D points of the same name as reference points of all non-coded points in their respective frames, extract the distance subset corresponding to the reference point from the Euclidean distance set, and obtain the 3D points of the same name for other non-coded points in the two point clouds according to the spatial relative positions of other points and the reference point. Similarly, find the 3D points of the same name for all non-coded points through other point clouds;

[0011] S6. Construct a three-dimensional spatial model of the non-coding points of the object to be tested based on the three-dimensional same-name point matching information.

[0012] Following the above technical solution, in step S1, distortion correction is first performed on the collected image information before the image epipolar correction.

[0013] Following the above technical solution, the distortion correction process is as follows:

[0014] Use Zhang's calibration method to obtain the camera parameters, including the intrinsic parameter matrix, distortion coefficient and rotation and translation matrix of the left camera coordinate system to the right camera coordinate system of each camera.

[0015] Stereo correction is performed according to the camera parameters to align the left and right camera coordinate systems in the same plane, and the rotation matrix between the coordinate systems of the two cameras before correction and the coordinate system after correction and the projection matrix of the coordinate system after correction are obtained. According to the projection matrix and the rotation matrix after correction, the image is mapped to cut off the distorted part.

[0016] Following the above technical solution, step S2 is specifically as follows:

[0017] S21, using the sobel discrete difference operator to convolve the image after epipolar correction, obtain the horizontal gradient and vertical gradient of the image, and calculate the overall gradient to achieve edge extraction of multi-pixel width;

[0018] S22, using the Zhang-Suen thinning algorithm, traversing each non-zero pixel in the extracted edge, focusing on the connected domain of neighboring points around it, deleting points that meet the preset conditions, and saving points that do not meet the conditions, so as to further extract the edge with multi-pixel width into an edge with single-pixel width;

[0019] S23. Filter the contours of non-coding points from the image according to the properties of the circle, and calculate the centroid of the non-coding points.

[0020] Following the above technical solution, step S6 is specifically as follows:

[0021] S61, calculating the rotation and translation transformation matrix set of the object to be measured by least squares fitting according to the three-dimensional same-name point matching information;

[0022] S62. Eliminate the errors of the three-dimensional same-name points between different frames due to the least square method according to the rotation and translation transformation matrix set, and construct a stable three-dimensional space model of the non-coding points of the object to be measured.

[0023] Continuing with the above technical solution, step S62 is specifically as follows: transform the three-dimensional same-name points in one view to another view according to the rotation transformation matrix set; if they do not coincide with the corresponding three-dimensional same-name points in the other view, use the middle point between the two points as a correction point, and construct a three-dimensional spatial model of the non-coding points of the object to be measured based on the correction point.

[0024] The present invention also provides a non-coding point matching and stereo reconstruction system based on spatial invariance, comprising:

[0025] The image acquisition and correction module is used to use a binocular camera to acquire image information of the object to be measured and perform image epipolar correction on the left and right views of the same frame;

[0026] A non-coding point extraction module is used to extract the centroid coordinates of the non-coding points pre-pasted on the object to be measured in the left and right same frame views;

[0027] The non-coding point two-dimensional matching module is used to perform two-dimensional same-name point matching for easy-to-match non-coding points without interference points according to the relationship between the polar lines between the non-coding points; for easy-to-match non-coding points with interference points, among the successfully matched non-coding points, the three points closest to the view where the easy-to-match non-coding point to be tested is located are found according to the coordinates of the centroid of the non-coding point, forming a triangle, and calculating the deviation distance from the easy-to-match non-coding point to be tested to the centroid of the triangle; and calculating the centroid of the corresponding triangle in another view, and the point with the closest distance to the centroid of the corresponding triangle and the deviation distance is used as the two-dimensional same-name point of the easy-to-match non-coding point to be tested;

[0028] The non-coding point reconstruction module is used to restore the non-coding point 3D point cloud information of the current frame based on the 2D same-name point matching results and the camera calibration parameters, and calculate the Euclidean distance set between each point in the point cloud and other non-coding points. All frames are processed in the same way.

[0029] The non-coding point 3D matching module is used to find a pair of 3D homonymous points from two different point clouds through the constraints of Euclidean distance and relative position between points, and regard the pair of 3D homonymous points as reference points for all non-coding points in their respective frames. The distance subset corresponding to the reference point is extracted from the Euclidean distance set, and the 3D homonymous points of other non-coding points in the two point clouds are obtained according to the spatial relative positions of other points and the reference points. Similarly, the 3D homonymous points of all non-coding points are found through other point clouds.

[0030] The spatial model building module is used to build a three-dimensional spatial model of the non-coded points of the object to be tested based on the matching information of three-dimensional points with the same name.

[0031] In accordance with the above technical solution, the system further includes: a distortion correction module, which is used to perform distortion correction on the collected image information before image epipolar correction.

[0032] Following the above technical solution, the spatial model construction module specifically includes:

[0033] A fitting module is used to calculate the rotation and translation transformation matrix set of the object to be tested by least squares fitting based on the three-dimensional same-name point matching information;

[0034] An error elimination module is used to transform the three-dimensional same-name points in one view to the other view according to the rotation transformation matrix set. If the three-dimensional same-name points do not coincide with the corresponding three-dimensional same-name points in the other view, the middle point between the two points is used as a correction point to eliminate the errors of the three-dimensional same-name points between different frames caused by the least square method;

[0035] The construction module is used to construct a stable three-dimensional spatial model of non-coding points of the object to be tested based on the calibration points.

[0036] The present invention also provides a computer storage medium, which stores a computer program that can be executed by a processor, and the computer program executes the non-coding point matching and stereo reconstruction method based on spatial invariance described in the above technical solution.

[0037] The beneficial effects of the present invention are as follows: when performing two-dimensional homonymous point matching, the present invention divides the points to be matched into easy-to-match non-coding points without interference points and easy-to-error non-coding points with interference points, matches the former directly based on the epipolar relationship, and adds the parallax factor to the latter, calculates the parallax distance between the point to be tested and the centroid of the triangle according to the triangle formed by the three closest points that have been matched, finds the corresponding three constituent triangles in another view, and then calculates the distances between other points and the centroid of the triangle, and uses the points corresponding to the distances closest to the parallax distance as matching homonymous points, thereby achieving the matching of two-dimensional homonymous points; in the matching process of three-dimensional homonymous points, the possible reference points of two point clouds are screened out by the Euclidean distance, and under the constraint of the relative positions between the points, the real reference points are found, and then the reference points are used to find other three-dimensional homonymous points to achieve the matching of three-dimensional points; the present invention can effectively offset the influence of calibration parameter errors and exclude the influence of most similar points by using the spatial invariance of the point space, and more accurately restore the three-dimensional information of non-coding points.

[0038] Furthermore, when constructing a three-dimensional space model, in order to eliminate the errors caused by the least squares method of three-dimensional homonymous points between different point clouds, the midpoints of the lines connecting the points after rotation and translation transformation and the original corresponding homonymous points are found as non-coding points to construct the three-dimensional space model, thereby improving the accuracy of three-dimensional reconstruction.

[0039] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 is a flow chart of a non-coding point matching and stereo reconstruction method based on spatial invariance according to an embodiment of the present invention;

[0042] FIG2 (a) is a picture taken by a binocular camera according to an embodiment of the present invention;

[0043] FIG2( b ) is a comparison diagram of the image taken by the binocular camera before correction according to an embodiment of the present invention;

[0044] FIG2 (c) is a comparison diagram of the image taken by the binocular camera after correction according to an embodiment of the present invention;

[0045] FIG3 (a) is a diagram showing the horizontal and vertical convolution results of the Sobel operator according to an embodiment of the present invention;

[0046] FIG3( b ) is a multi-pixel width edge result image and a single-pixel width edge result image according to an embodiment of the present invention;

[0047] FIG3 (c) is a diagram showing the centroid extraction result of non-coding points according to an embodiment of the present invention;

[0048] FIG4 (a) is a diagram showing the two-dimensional matching result of the same-frame images of the left and right cameras according to an embodiment of the present invention;

[0049] FIG4( b ) is a schematic diagram of a point cloud after single-frame reconstruction according to an embodiment of the present invention;

[0050] FIG5( a ) is a schematic diagram of a point cloud according to an embodiment of the present invention;

[0051] FIG5( b ) is a schematic diagram of different point clouds of an embodiment of the present invention;

[0052] FIG5 (c) is a schematic diagram of the result of constructing two point clouds according to an embodiment of the present invention;

[0053] FIG6 (a) is a schematic diagram of an object according to an embodiment of the present invention;

[0054] FIG6( b ) is a schematic diagram of a non-coding point space model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0056] It should be noted that the illustrations provided in the embodiments of the present invention are only used to illustrate the basic concept of the present invention in a schematic manner. Therefore, the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout type may also be more complicated.

[0057] In the present invention, it is also necessary to explain that, if the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, if the terms "first" and "second" appear, they are only used for description and distinction purposes, and cannot be understood as indicating or implying relative importance.

[0058] Example 1

[0059] like Figure 1 As shown, the non-coding point matching and stereo reconstruction method based on spatial invariance of an embodiment of the present invention includes the following steps:

[0060] S1. Use a binocular camera to collect image information of the object to be measured, and perform image epipolar correction on the left and right views of the same frame;

[0061] S2, extracting the centroid coordinates of the non-encoded points pre-pasted on the object to be measured in the left and right same frame views;

[0062] S3. According to the relationship between the polar lines between the non-coding points, two-dimensional same-name point matching is performed for the easy-to-match non-coding points without interference points; for the easy-to-match non-coding points with interference points, among the successfully matched non-coding points, the three points closest to the view where the easy-to-match non-coding points to be tested are located are found according to the coordinates of the centroid of the non-coding points, forming a triangle, and calculating the deviation distance from the easy-to-match non-coding points to be tested to the centroid of the triangle; and calculating the centroid of the corresponding triangle in another view, the point with the closest distance to the centroid of the corresponding triangle and the deviation distance is taken as the two-dimensional same-name point of the easy-to-match non-coding points to be tested;

[0063] S4, based on the 2D same-name point matching results and the camera calibration parameters, restore the 3D point cloud information of the non-coded points of the current frame, and calculate the Euclidean distance set between each point in the point cloud and other points, and process all frames in the same way;

[0064] S5. Find a pair of 3D points of the same name from two different point clouds through the constraints of Euclidean distance and relative position between points, and regard the pair of 3D points of the same name as reference points of all non-coded points in their respective frames. Extract the distance subset corresponding to the reference point from the Euclidean distance set, and obtain the 3D points of the same name for other non-coded points in the two point clouds according to the spatial relative positions of other points and the reference point. Similarly, find the 3D points of the same name for all non-coded points through other point clouds.

[0065] S6. Construct a three-dimensional spatial model of the non-coding points of the object to be tested based on the three-dimensional same-name point matching information.

[0066] Furthermore, in step S1, before the image epipolar correction, the collected image information is firstly subjected to distortion correction.

[0067] The distortion correction process is:

[0068] Use Zhang's calibration method to obtain the camera parameters, including the intrinsic parameter matrix, distortion coefficient and rotation and translation matrix of the left camera coordinate system to the right camera coordinate system of each camera.

[0069] Stereo correction is performed according to the camera parameters to align the left and right camera coordinate systems in the same plane, and the rotation matrix between the coordinate systems of the two cameras before correction and the coordinate system after correction and the projection matrix of the coordinate system after correction are obtained. According to the projection matrix and the rotation matrix after correction, the image is mapped to cut off the distorted part.

[0070] Furthermore, step S2 is specifically as follows:

[0071] S21, using the sobel discrete difference operator to convolve the image after epipolar correction, obtain the horizontal gradient and vertical gradient of the image, and calculate the overall gradient to achieve edge extraction of multi-pixel width;

[0072] S22, using the Zhang-Suen thinning algorithm, traversing each non-zero pixel in the extracted edge, focusing on the connected domain of neighboring points around it, deleting points that meet the preset conditions, and saving points that do not meet the conditions, so as to further extract the edge with multi-pixel width into an edge with single-pixel width;

[0073] S23. Filter the contours of non-coding points from the image according to the properties of the circle, and calculate the centroid of the non-coding points.

[0074] Step S6 is specifically as follows:

[0075] S61, calculating the rotation and translation transformation matrix set of the object to be measured by least squares fitting according to the three-dimensional same-name point matching information;

[0076] S62. Eliminate the errors of the three-dimensional same-name points between different frames due to the least square method according to the rotation and translation transformation matrix set, and construct a stable three-dimensional space model of the non-coding points of the object to be measured.

[0077] Among them, step S62 is specifically: transform the three-dimensional same-name points in one view to another view according to the rotation transformation matrix set. If they do not coincide with the corresponding three-dimensional same-name points in the other view, the middle point between the two points is used as a correction point, and a three-dimensional spatial model of the non-coding points of the object to be measured is constructed based on the correction point.

[0078] After constructing the three-dimensional spatial model of the object surface, the subsequently reconstructed three-dimensional point cloud of non-coded points can also find the corresponding three-dimensional points of the same name on the spatial model based on the Euclidean distance constraint and the relative position relationship between the points, and calculate the corresponding posture transformation relationship based on this. Combined with the structural information of the line structured light scanner, the three-dimensional reconstruction of the object can be achieved by performing a rotation transformation based on the transformation relationship.

[0079] When performing two-dimensional matching of points with the same name, the present invention divides the points to be matched into easy-to-match non-coded points without interference points and easy-to-error non-coded points with interference points. The former are matched by the distance between corresponding epipolar lines, and the parallax factor is added to the latter and combined with the centroid matching of the triangle, thereby realizing a highly robust matching method that is not easily affected by calibration accuracy, and improving the accuracy of subsequent three-dimensional point clouds.

[0080] In the process of matching three-dimensional homonymous points, the possible reference points of the two point clouds are pre-screened by Euclidean distance, and under the constraint of the relative position between the points, the real reference points are found, and the Euclidean distance subset corresponding to the reference point is extracted from the Euclidean distance set, thereby reducing the search space of other homonymous points between subsequent matching frames. Because of the constraints of the Euclidean distance and relative position between other points and the reference point, the accuracy of three-dimensional homonymous point matching can be effectively improved.

[0081] In the process of building a spatial model, in order to eliminate the errors caused by the least squares method, the midpoint of the line connecting the point after rotation and translation transformation and the original corresponding point of the same name is found as a non-coding point to build a stable three-dimensional spatial model. The subsequently reconstructed three-dimensional point cloud of non-coding points can efficiently find the corresponding point of the same name in the spatial model, calculate a more accurate posture transformation relationship, and improve the accuracy of object reconstruction.

[0082] The non-coding point matching and stereo reconstruction method based on spatial invariance of the embodiment of the present invention can be built using C++ and QT on the Windows 10 platform. It is understandable that the algorithm of the present invention can also be transplanted to other programming languages ​​or hardware devices.

[0083] Example 2

[0084] This embodiment is the same as the embodiment 1, except that each step is applied to a specific example. Specifically, the following steps are included:

[0085] In step S1, Zhang’s calibration method can be used to obtain the intrinsic parameter matrices of the two cameras. , and distortion coefficient , And the rotation matrix from the left camera coordinate system to the right camera coordinate system and translation matrix As shown in Figure 2 (a) and 2 (b), due to the influence of external factors such as industrial precision, the collected images will inevitably produce radial distortion and tangential distortion.

[0086]

[0087]

[0088] The distorted point can be calculated. Assume that there is a point on the normalized plane , polar coordinates available In the above formula, , , is the radial distortion coefficient, , is the tangential distortion coefficient, yes The distance from the origin of the normalized plane. Then calculate the pixel coordinates of these distorted points and put them back to their original positions, that is, the distortion is removed. The pixel coordinate calculation formula is as follows:

[0089]

[0090]

[0091] In the formula, and is the focal length of the camera, and is the pixel coordinate of the camera principal point. Because the degree of distortion at different positions of the image is not consistent in reality, only the effect of distortion is eliminated to the maximum extent, not completely.

[0092] Ideally, only one camera needs to be calibrated , The left and right cameras can be aligned in the same plane by transforming the matrix, but it will produce a large reprojection distortion. Therefore, if (R, T) is split into two parts, the left camera is The right camera is , which will minimize the reprojection distortion. At this time, you need to construct a rotation matrix

[0093]

[0094] Because the new x-axis is parallel to the baseline, , assuming the baseline vector is ,and and In the coordinate system, the XYZ axes are orthogonal to each other, so we can get , . Then the final rotation matrix of the left and right imaging planes can be obtained , .

[0095]

[0096]

[0097] according to , , it is not difficult to get the corrected projection matrix , The dedistorted image is based on , , , Perform epipolar correction, as shown in Figure 2 (c), so that the images captured by the left and right cameras are on the same epipolar line, which can greatly reduce the matching space when matching the two-dimensional same-name points of non-encoded points.

[0098] In step S2, the Sobel discrete difference operator is used on the corrected image, which includes two sets of template matrices, which are matrices for horizontal processing of the image. , and the matrix of longitudinal processing . When the template matrix size is 3,

[0099]

[0100] For a single image , the template matrix and the image By performing convolution respectively, as shown in Figure 3 (a), the lateral gradient of the image can be obtained and longitudinal gradient . Then calculate the overall gradient of the image , as shown in Figure 3(b), edge extraction with multi-pixel width is achieved.

[0101]

[0102]

[0103]

[0104] The essence is based on the gradient formula

[0105]

[0106] Because the image is discrete, the minimum value of h can only be 1, so the gradient of a pixel in the image (for example, in the X direction) is equal to half the difference between the pixels on its left and right, and in the Y direction it is the difference between the pixels above and below.

[0107] In order to ensure the accuracy of the proposed centroid, the Zhang-Suen thinning algorithm is used to traverse each non-zero pixel in the proposed edge and record it as , The number of non-zero pixels in the eight-neighborhood is recorded as , The number of alternations from foreground pixels to background pixels is recorded as The Zhang-Suen algorithm can be divided into two steps:

[0108] Stage 1: Loop through all foreground pixels and mark the P1 pixels that meet the following conditions as deleted;

[0109] :

[0110] : =1;

[0111] :

[0112] :

[0113] Stage 2: Loop through all foreground pixels and mark the pixel P1 that meets the following condition as deleted.

[0114] :

[0115] : )=1;

[0116] :

[0117] :

[0118] The difference between Phase 1 and Phase 2 is , In two steps, stage 1 is to remove points on the east and south boundary lines, and corner points on the northwest corner. Stage 2 is to remove points on the west and north boundary lines, and corner points on the southeast corner. The algorithm runs iteratively, as shown in Figure 3 (b), until no points are deleted in both stages.

[0119] At this time, the image is full of edges with a single pixel width. By calculating the roundness, convexity and eccentricity of each edge line, it is not difficult to filter out the contours of non-coded points from the image, as shown in Figure 3 (c), and then calculate the centroid of the non-coded points.

[0120] In step S3, after the stereo calibration in step S1, the left and right cameras are ideally aligned in the same plane. If there is a pair of points in the left and right views , , and the basic matrix between the cameras is known , then we can know The corresponding epiline in the right figure , The corresponding epiline in the left figure for:

[0121]

[0122]

[0123] like and is a synonym, then it should satisfy:

[0124]

[0125]

[0126] According to the above formula, it is not difficult to deduce

[0127]

[0128] However, due to the influence of noise and accuracy, the two cameras often cannot be aligned in practice, resulting in errors. , so the above relationship should be:

[0129]

[0130]

[0131]

[0132] because It cannot be determined due to various factors. If there is another pair of points with the same name in both the left and right views, , ,but and Also satisfies the above relationship, then there should be:

[0133]

[0134]

[0135]

[0136] From the above formula, we can know and The basic matrix The accuracy of

[0137]

[0138] in is the translation matrix The antisymmetric matrix of is, according to S1, thus , and after stereo correction, the image coordinate systems of the left and right cameras are in the same plane. , , , Approximately parallel to each other, the distance between the two polar lines in the left view The distance between the corresponding polar lines in the right view The difference can be approximately regarded as and The difference between .

[0139] On this basis, the matching of the same-name points in the left and right views can be completed, and the robustness to factors such as noise and accuracy can be greatly improved. However, there are also new problems. Because the horizontal relationship between the same-name points is not established, if there are interference points on the same epipolar line, the left and right views will be Figure 1 The point corresponds to the situation of multiple points in the right figure. Therefore, the points without interference are matched first. And according to the binocular camera model, the parallax is used to limit the interference points. The parallax can be expressed as:

[0140]

[0141] in is the focal length of the camera (in pixels), is the baseline length, is the depth, and it is not difficult to conclude that the farther the point is from the optical center of the camera, the smaller its parallax is. It can be further deduced that the points at the same distance from the optical center of the camera have the same parallax.

[0142] If there is a point in the left figure now , there is a point to be tested in the right figure Among the points that have been successfully matched in the left figure, find The three closest points form , the reference points are denoted as , , , then the centroid of the triangle is:

[0143]

[0144] Similarly, we can obtain , , Corresponding three points in the right view , , The centroid of the triangle . Because at this time the centroid of the triangle on the left is The distance to the left camera is The distance to the camera optical center is the most similar, then we only need to calculate and the measured point relative to The parallax deviation of the right figure can be found by recording the two points with the most similar deviations as the same-name points. The distance between and centroid The closest distance between the points to be tested , that is Two-dimensional points of the same name.

[0145] The two-dimensional same-name points of the easy-to-match non-coding points of the current frame can be obtained by matching the non-coding points easily, thereby obtaining the two-dimensional same-name point set of the current frame. ,in, and The points in are one-to-one corresponding, that is, the non-encoded points in the world coordinate system are mapped to the points in the left and right camera coordinate systems. Similarly, through the above method, the two-dimensional points with the same name in all frames can be obtained, denoted as , as shown in Figure 4 (a), it is the matching result of the same-name points in the same frame of the left and right cameras.

[0146] like , Corresponding to the world coordinate system point, in the known , In the case of image coordinates, combined with the camera parameters obtained by camera calibration, the triangulation principle can be used to calculate According to the above process, we can calculate The three-dimensional coordinates of the non-coded points corresponding to each pair of points with the same name in the current world coordinate system are restored to the three-dimensional point cloud data of the current frame. , as shown in Figure 4(b). And calculate the Euclidean distance between each point in the point cloud and other points:

[0147]

[0148] at this time , we get the Euclidean distance set

[0149]

[0150] Can be simplified to

[0151]

[0152] in and It is a subset of the Euclidean distances of different reference points to other points.

[0153] Similarly, the point cloud data of all frames can be restored and recorded as , and the calculated Euclidean distance set for each point cloud .

[0154] In step S4, two point clouds are obtained according to the above steps , After that, and there are points with the same name in the two point clouds, , As shown in Figure 5 (a) and Figure 5 (b), the boxed parts in the figure are the same-name point sets of the two point clouds. , The corresponding Euclidean distance sets are and ,set up yes Members of Filter out Members whose distance difference is within the threshold range , The corresponding point is , , correspond , Ideally, if , and , are points of the same name in two point clouds, then they should satisfy:

[0155]

[0156] in express arrive The vector of express arrive When the vector satisfies the above vector relationship and the constraint relationship of the Euclidean distance between points, it is considered that , and , are points with the same name in two point clouds, and It can be used as a reference point for subsequent operations.

[0157] Once the reference point is found, and Find the and A subset of distances calculated for the reference points and , then search for points in the distance subset of the reference point that satisfy the above relative position relationship with the reference point and the constraint relationship of the Euclidean distance between points, and other points of the same name in the two point clouds can be found. Similarly, according to the above steps, we can get and , and ,..., and , and The set of points with the same name between .

[0158] In step S5, and ,yes and The set of points with the same name between and The point in and The three-dimensional points with the same name in one-to-one correspondence are known. The rotation matrix between the two point clouds can be solved by fitting the result with the minimum cost according to the least squares method. And the translation matrix Similarly, according to and The corresponding rotation transformation matrix can also be calculated from the point sets with the same name between them. After obtaining the transformation relationship between all point clouds, the rotation and translation matrix set is obtained. ,in , , , They are and , and , and and and The transformation matrix between .

[0159] In step S6, it is assumed that The world coordinate system is the main one, and its point cloud information is , the rotation and translation matrix set is known to be , point cloud , then the point cloud Transform to In the world coordinate system, the transformation result is recorded as

[0160]

[0161] By rotating the other point clouds in the above method, a complete non-coding point space model of the object can be gradually constructed. Ideally, the corresponding points with the same name should exist after the rotation transformation.

[0162]

[0163]

[0164] in yes Hit a point, Should be The points that coincide after the rotation transformation, but because , is the result obtained by least squares fitting, so

[0165]

[0166] At this time, you need to build and The center point of the connecting line ,

[0167]

[0168] by replace To build a spatial model, this can effectively reduce the impact of errors and thus build a stable spatial model of the object. As shown in Figure 5 (c), the model result constructed by two point clouds.

[0169] Ideally, six pieces of point cloud data can construct a spatial model of a six-sided object, as shown in FIG6 (a) for the measured object in this embodiment, and FIG6 (b) for the constructed non-coded point spatial model of the object. The non-coded point three-dimensional point cloud reconstructed subsequently can also find the corresponding three-dimensional points of the same name on the spatial model based on the Euclidean distance constraint and the relative position relationship between the points to calculate the pose transformation relationship, and combine the structural information of the line structured light scanner to perform a rotation transformation based on the transformation relationship to achieve three-dimensional reconstruction of the object.

[0170] Example 3

[0171] This embodiment is a non-coding point matching and stereo reconstruction method based on spatial invariance for implementing the above method embodiment, including:

[0172] The image acquisition and correction module is used to use a binocular camera to acquire image information of the object to be measured and perform image epipolar correction on the left and right views of the same frame;

[0173] A non-coding point extraction module is used to extract the centroid coordinates of the non-coding points pre-pasted on the object to be measured in the left and right same frame views;

[0174] The non-coding point two-dimensional matching module is used to perform two-dimensional same-name point matching for easy-to-match non-coding points without interference points according to the relationship between the polar lines between the non-coding points; for easy-to-match non-coding points with interference points, among the successfully matched non-coding points, the three points closest to the view where the easy-to-match non-coding point to be tested is located are found according to the coordinates of the centroid of the non-coding point, forming a triangle, and calculating the deviation distance from the easy-to-match non-coding point to be tested to the centroid of the triangle; and calculating the centroid of the corresponding triangle in another view, and the point with the closest distance to the centroid of the corresponding triangle and the deviation distance is used as the two-dimensional same-name point of the easy-to-match non-coding point to be tested;

[0175] The non-coding point reconstruction module is used to restore the non-coding point 3D point cloud information of the current frame based on the 2D same-name point matching results and the camera calibration parameters, and calculate the Euclidean distance set between each point in the point cloud and other non-coding points. All frames are processed in the same way.

[0176] The non-coding point 3D matching module is used to find a pair of 3D homonymous points from two different point clouds through the constraints of Euclidean distance and relative position between points, and regard the pair of 3D homonymous points as reference points for all non-coding points in their respective frames. The distance subset corresponding to the reference point is extracted from the Euclidean distance set, and the 3D homonymous points of other non-coding points in the two point clouds are obtained according to the spatial relative positions of other points and the reference points. Similarly, the 3D homonymous points of all non-coding points are found through other point clouds.

[0177] The spatial model building module is used to build a three-dimensional spatial model of the non-coded points of the object to be tested based on the matching information of three-dimensional points with the same name.

[0178] The spatial model building module is used to build a three-dimensional spatial model of the non-coded points of the object to be tested based on the matching information of three-dimensional points with the same name.

[0179] Furthermore, the system also includes: a distortion correction module, which is used to perform distortion correction on the collected image information before the image epipolar correction.

[0180] Furthermore, the spatial model building module specifically includes:

[0181] A fitting module is used to calculate the rotation and translation transformation matrix set of the object to be tested by least squares fitting based on the three-dimensional same-name point matching information;

[0182] An error elimination module is used to transform the three-dimensional same-name points in one view to the other view according to the rotation transformation matrix set. If the three-dimensional same-name points do not coincide with the corresponding three-dimensional same-name points in the other view, the middle point between the two points is used as a correction point to eliminate the errors of the three-dimensional same-name points between different frames caused by the least square method;

[0183] The construction module is used to construct a stable three-dimensional spatial model of non-coding points of the object to be tested based on the calibration points.

[0184] Each module is mainly used to implement each step of the method embodiment, which will not be described in detail here.

[0185] Example 4

[0186] The present application also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (for example

[0187] Such as SD or DX memory, random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, CD, server, App application store, etc., which store computer programs, and the corresponding functions are realized when the program is executed by the processor. The computer-readable storage medium of this embodiment is executed by the processor to realize the non-coding point matching and stereo reconstruction method based on spatial invariance of the method embodiment.

[0188] It should be pointed out that, according to the needs of implementation, the various steps / components described in this application can be split into more steps / components, and two or more steps / components or partial operations of steps / components can be combined into new steps / components to achieve the purpose of the present invention.

[0189] The order of execution of each step in the above embodiment does not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0190] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A non-coding point matching and stereo reconstruction method based on spatial invariance, characterized in that: The following steps are involved: S1. Use a binocular camera to collect image information of the object to be measured, and perform image epipolar correction on the left and right views of the same frame; S2, extracting the centroid coordinates of the non-encoded points pre-pasted on the object to be measured in the left and right same frame views; S3. According to the relationship between the polar lines between the non-coding points, two-dimensional same-name point matching is performed for the easy-to-match non-coding points without interference points; for the easy-to-match non-coding points with interference points, among the successfully matched non-coding points, the three points closest to the view where the easy-to-match non-coding points to be tested are located are found according to the coordinates of the centroid of the non-coding points, forming a triangle, and calculating the deviation distance from the easy-to-match non-coding points to be tested to the centroid of the triangle; and calculating the centroid of the corresponding triangle in another view, the point with the closest distance to the centroid of the corresponding triangle and the deviation distance is taken as the two-dimensional same-name point of the easy-to-match non-coding points to be tested; S4, based on the 2D same-name point matching results and the camera calibration parameters, restore the 3D point cloud information of the non-coded points of the current frame, and calculate the Euclidean distance set between each point in the point cloud and other points, and process all frames in the same way; S5. Find a pair of 3D points of the same name from two different point clouds through the constraints of Euclidean distance and relative position between points, and regard the pair of 3D points of the same name as reference points of all non-coded points in their respective frames. Extract the distance subset corresponding to the reference point from the Euclidean distance set, and obtain the 3D points of the same name for other non-coded points in the two point clouds according to the spatial relative positions of other points and the reference point. Similarly, find the 3D points of the same name for all non-coded points through other point clouds. S6. Construct a three-dimensional spatial model of the non-coding points of the object to be tested based on the three-dimensional same-name point matching information.

2. The non-coding point matching and stereo reconstruction method based on spatial invariance according to claim 1 is characterized in that: In step S1, before the image epipolar correction, the collected image information is firstly subjected to distortion correction.

3. The non-coding point matching and stereo reconstruction method based on spatial invariance according to claim 2 is characterized in that: The distortion correction process is: Use Zhang's calibration method to obtain the camera parameters, including the intrinsic parameter matrix, distortion coefficient and rotation and translation matrix of the left camera coordinate system to the right camera coordinate system of each camera. Stereo correction is performed according to the camera parameters to align the left and right camera coordinate systems in the same plane, and the rotation matrix between the coordinate systems of the two cameras before correction and the coordinate system after correction and the projection matrix of the coordinate system after correction are obtained. According to the projection matrix and the rotation matrix after correction, the image is mapped to cut off the distorted part.

4. The non-coding point matching and stereo reconstruction method based on spatial invariance according to claim 1 is characterized in that: Step S2 is specifically as follows: S21, using the sobel discrete difference operator to convolve the image after epipolar correction, obtain the horizontal gradient and vertical gradient of the image, and calculate the overall gradient to achieve edge extraction of multi-pixel width; S22, using the Zhang-Suen thinning algorithm, traversing each non-zero pixel in the extracted edge, focusing on the connected domain of neighboring points around it, deleting points that meet the preset conditions, and saving points that do not meet the conditions, so as to further extract the edge with multi-pixel width into an edge with single-pixel width; S23. Filter the contours of non-coding points from the image according to the properties of the circle, and calculate the centroid of the non-coding points.

5. The non-coding point matching and stereo reconstruction method based on spatial invariance according to claim 1 is characterized in that: Step S6 is specifically as follows: S61, calculating the rotation and translation transformation matrix set of the object to be measured by least squares fitting according to the three-dimensional same-name point matching information; S62. Eliminate the errors of the three-dimensional same-name points between different frames caused by the least square method according to the rotation and translation transformation matrix set, and construct a stable three-dimensional space model of the non-coding points of the object to be measured.

6. The non-coding point matching and stereo reconstruction method based on spatial invariance according to claim 5 is characterized in that: Step S62 is specifically as follows: transform the three-dimensional same-name points in one view to another view according to the rotation transformation matrix set; if they do not coincide with the corresponding three-dimensional same-name points in the other view, use the middle point between the two points as a correction point, and construct a three-dimensional spatial model of the non-coding points of the object to be measured based on the correction point.

7. A non-coding point matching and stereo reconstruction system based on spatial invariance, characterized in that: include: The image acquisition and correction module is used to use a binocular camera to acquire image information of the object to be measured and perform image epipolar correction on the left and right views of the same frame; A non-coding point extraction module is used to extract the centroid coordinates of the non-coding points pre-pasted on the object to be measured in the left and right same frame views; The non-coding point two-dimensional matching module is used to perform two-dimensional same-name point matching for easy-to-match non-coding points without interference points according to the relationship between the polar lines between the non-coding points; for easy-to-match non-coding points with interference points, among the successfully matched non-coding points, the three points closest to the view where the easy-to-match non-coding point to be tested is located are found according to the coordinates of the centroid of the non-coding point, forming a triangle, and calculating the deviation distance from the easy-to-match non-coding point to be tested to the centroid of the triangle; and calculating the centroid of the corresponding triangle in another view, and the point with the closest distance to the centroid of the corresponding triangle and the deviation distance is used as the two-dimensional same-name point of the easy-to-match non-coding point to be tested; The non-coding point reconstruction module is used to restore the non-coding point 3D point cloud information of the current frame based on the 2D same-name point matching results and the camera calibration parameters, and calculate the Euclidean distance set between each point in the point cloud and other non-coding points. All frames are processed in the same way. The non-coding point 3D matching module is used to find a pair of 3D homonymous points from two different point clouds through the constraints of Euclidean distance and relative position between points, and regard the pair of 3D homonymous points as reference points for all non-coding points in their respective frames. The distance subset corresponding to the reference point is extracted from the Euclidean distance set, and the 3D homonymous points of other non-coding points in the two point clouds are obtained according to the spatial relative positions of other points and the reference points. Similarly, the 3D homonymous points of all non-coding points are found through other point clouds. The spatial model building module is used to build a three-dimensional spatial model of the non-coded points of the object to be tested based on the matching information of three-dimensional points with the same name.

8. The non-coding point matching and stereo reconstruction system based on spatial invariance according to claim 7, characterized in that: The system also includes: a distortion correction module, which is used to perform distortion correction on the collected image information before the image epipolar line correction.

9. The non-coding point matching and stereo reconstruction system based on spatial invariance according to claim 7, characterized in that: The spatial model building module specifically includes: A fitting module is used to calculate the rotation and translation transformation matrix set of the object to be tested by least squares fitting based on the three-dimensional same-name point matching information; An error elimination module is used to transform the three-dimensional same-name points in one view to the other view according to the rotation transformation matrix set. If the three-dimensional same-name points do not coincide with the corresponding three-dimensional same-name points in the other view, the middle point between the two points is used as a correction point to eliminate the errors of the three-dimensional same-name points between different frames caused by the least square method; The construction module is used to construct a stable three-dimensional spatial model of non-coding points of the object to be tested based on the calibration points.

10. A computer storage medium, characterized in that: A computer program executable by a processor is stored therein, and the computer program executes the non-encoded point matching and stereo reconstruction method based on spatial invariance as described in any one of claims 1 to 6.

Images