Digital image calculation method and system based on variable template for RGB-D camera multi-view matching

Abstract

The present invention discloses a digital image calculation method and system for multi-view matching of RGB-D cameras based on a variable template, comprising six steps: data acquisition, point cloud data preprocessing, feature point matching, re-registration of a variable template, calculation of point cloud data transformation relationship between large-viewing angle images, and point cloud fusion. The size of non-adjacent image matching templates is adjusted by the registration results of adjacent viewing angles, so that correct registration of feature points of non-adjacent viewing angle images is achieved, matching accuracy is improved, accumulated errors of image sets are eliminated, relatively accurate initial values ​​are provided for subsequent iterations of point cloud fusion, the number of iterations is reduced, and three-dimensional reconstruction of images is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-view three-dimensional imaging and machine vision, and mainly relates to a digital image calculation method and system for multi-view matching of RGB-D cameras based on variable templates. Background Art

[0002] With the popularization of 3D printing and virtual reality technology, the demand for 3D reconstruction technology is also increasing. Since the cost of manual modeling using 3D modeling software is too high, people focus on the camera's multi-view shooting of objects to reconstruct the objects in three dimensions. The 3D reconstruction methods based on RGB-D cameras can be roughly divided into two categories: point cloud and image. Although there are some relatively good results in the 3D reconstruction algorithms based on point clouds, such as the famous ICP algorithm and improvements based on this algorithm, there are still some limitations. The processing ability for disordered point cloud scenes is weak, and an initial value close to the true value is often required, otherwise it is easy to fall into the local optimal solution. The 3D reconstruction algorithm based on the image relies on the matching effect of adjacent images. Generally speaking, multi-view 3D reconstruction is divided into the following steps: 1. The camera obtains a multi-view image sequence and matches the feature points in the image sequence. 2. Calculate the transformation relationship between images and align the point cloud data of the image sequence. Therefore, the quality of 3D reconstruction puts higher requirements on the quality of feature point pairs and the accuracy of feature point matching.

[0003] In the process of digital image correlation matching of image sequences, some problems are usually encountered. For example, excessive rotation or scaling between non-adjacent images will increase the number of mismatched points and make it impossible to match directly. With the increase of images with different perspectives, the matching errors between adjacent images will accumulate, resulting in bifurcation after point cloud alignment. In fact, relevant scholars have given some solutions to related problems. In the process of binocular stereo matching, the method of limiting geometric constraints is used to reduce mismatches. However, since the relative position of the two camera perspectives needs to be known in advance through calibration methods, this method cannot be used in single-camera multi-perspective scenarios. For two images that cannot be matched directly, previous related studies have proposed inserting a series of intermediate images to incrementally accumulate the results of guided matching. This is effective in most cases, but in the process of multi-perspective matching, introducing too many intermediate images will increase the computational cost and it is difficult not to cause cumulative errors. Some feature point matching methods such as sift and its improved methods have also achieved relatively good results. However, there are still limitations. These methods are more dependent on the number of feature points. If there are too few feature point pairs in the two images to be matched, matching cannot be performed. Related studies have also shown that matching at too large an angle will cause large errors, and it is necessary to limit the viewing angle of adjacent images. However, in scenarios that require a large viewing angle, introducing multiple small viewing angles will undoubtedly lead to the accumulation of matching errors. Summary of the invention

[0004] The present invention aims at the problems of matching error accumulation and mismatching existing in the multi-view matching of a single RGB-D camera in the prior art, and provides a digital image calculation method and system for multi-view matching of an RGB-D camera based on a variable template, including six steps: data acquisition, point cloud data preprocessing, feature point matching, re-registration of a variable template, calculation of the point cloud data transformation relationship between large-view images, and point cloud fusion. The size of the non-adjacent image matching template is adjusted according to the registration results of adjacent views, so as to achieve correct registration of feature points of non-adjacent view images, improve matching accuracy, eliminate accumulated errors of image sets, provide relatively accurate initial values ​​for subsequent iterations of point cloud fusion, reduce the number of iterations, and achieve three-dimensional reconstruction of the image.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is: a digital image calculation method based on multi-view matching of RGB-D cameras with variable templates, comprising the following steps:

[0006] S1, data acquisition: collect image information data through RGB-D camera, the image information data includes color information and depth data, and align the three-dimensional information and color information of the point cloud of the object to be measured by calibrating the camera or aligning the internal parameters of the depth camera of the RGB-D camera and the depth data with the color data;

[0007] S2, point cloud data preprocessing: perform straight-through filtering on the three-dimensional information obtained in step S1, and use a random sampling consensus algorithm to determine the optimal ground cloud and remove it; perform cluster analysis on the processed point cloud, use a density algorithm to retain point cloud data with similar density, and obtain the area to be registered;

[0008] S3, feature point matching: track the feature points of adjacent images in the image sequence, and calculate the relative position transformation relationship of the point cloud coordinates under each viewing angle through the position of the feature points;

[0009] S4, re-registration of the variable template: adjusting the scale and direction of the variable template according to the calculation result of the relative position and posture transformation relationship of the point cloud coordinates obtained in step S3;

[0010] S5, calculating the point cloud data transformation relationship between the large viewing angle images: aligning the image feature points according to the variable template obtained in step S4 to obtain the corresponding point coordinate transformation relationship;

[0011] S6, point cloud fusion: Calculate the pose transformation relationship of the matched feature points in a set of point cloud data, use it as the initial value of the iterative closest point algorithm for iterative calculation, complete point cloud fusion, and realize three-dimensional reconstruction of the image.

[0012] As an improvement of the present invention, the specific method for calculating the three-dimensional information of the object under test in step S1 is:

[0013]

[0014] Where: d i Indicates the distance between the target point and the camera plane; c x ,c y ,f x ,f y Represents the internal parameters of the depth camera; X iw ,Y iw ,Z iw Indicates the position of the target point in the world coordinate system;

[0015] The color information is obtained through the color data stream of the RBG-D camera. The specific method of obtaining is: Color = (R, G, B)

[0016] Where: R, G, B represent the corresponding red, green, and blue channel values ​​in the RGB-D camera respectively.

[0017] As an improvement of the present invention, in the step S2, the background area beyond the set range is filtered out during the through filtering of the three-dimensional information, specifically:

[0018]

[0019] Among them, X min ,X max ,Y min ,Y max ,Z min ,Z max Represents the detection threshold in the world coordinate system;

[0020] In the random sampling consensus algorithm, three points are randomly selected from the filtered point cloud. After multiple repetitions, the cloud with the largest number of points in the fitted plane is the optimal ground cloud.

[0021] As another improvement of the present invention, in step S3, correlation calculation is performed on the image sequence for tracking image feature points, and the correlation formula is:

[0022]

[0023] in: Represents the mean grayscale value of the pixel of the reference subset and the deformed subset; u, v represent the horizontal and vertical offsets of the feature point in the deformed image; r(x,y), d(x+u,y+v) represent the grayscale values ​​of the pixel of the reference subset and the deformed subset at (x,y), (x+u,y+v) in the image coordinate system.

[0024] As another improvement of the present invention, in step S3, the relative position transformation relationship of the point cloud coordinates under each viewing angle is calculated by the position of the feature point: the rotation matrix R and the translation matrix T, and the coordinate transformation relationship of the corresponding point is:

[0025]

[0026] Among them, P1 and P2 represent the positions of the feature points in the image before rotation and translation and the image after rotation and translation respectively; The rotation matrix and translation matrix representing the feature points in the image after rotation and translation relative to the feature points in the image before rotation and translation;

[0027] The rotation matrix based on the coordinate transformation relationship between all adjacent images in the image sequence and translation matrix The relative pose transformation of the feature points between the kth image and the nth image is calculated and converted into Euler angles as follows:

[0028]

[0029] Among them, θ x ,θ y and θ z They are the scale parameter and direction parameter of the variable template respectively.

[0030] As another improvement of the present invention, in step S4, the sampling of the reference template is described in polar coordinates, specifically:

[0031]

[0032] Where R represents the sampling radius and θ represents the sampling angle. When the shooting angle changes, the deformed image rotates around the x, y, and z axes in space relative to the reference image. x θ y θ z Angle, then the sampling of the deformed template is:

[0033]

[0034] The sampling radius and sampling angle of the deformation template are set to be consistent with those of the reference template.

[0035] As another improvement of the present invention, in step S5, the coordinate transformation relationship of the corresponding points is specifically:

[0036]

[0037] Among them, P k ,P nRespectively represent the position of the feature point in image k and image n; P k ,P n Represents the rotation matrix and translation matrix of the feature points in image n relative to the feature points in image k;

[0038] As a further improvement of the present invention, the calculation of the iteration termination in step S6 is specifically:

[0039]

[0040] Among them, A current ,B current Respectively represent the coordinates of the target point cloud of the current iteration and the point cloud after the change; A last ,B last Respectively represent the coordinates of the target point cloud of the previous iteration and the point cloud after the change; m, n represent the number of corresponding points in the point cloud of the current iteration and the previous iteration; error represents the threshold for iteration termination;

[0041] After iterative convergence, we get Realize the registration and fusion of the k-th point cloud and the n-th point cloud data to complete multi-perspective 3D reconstruction.

[0042] In order to achieve the above-mentioned purpose, the present invention also adopts a technical solution: a digital image computing system for multi-view matching of RGB-D cameras based on variable templates, including a computer program, which implements the steps of any of the above-mentioned methods when executed by a processor.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] (1) The present invention proposes to use a variable template to align images of different perspectives. Compared with the existing direct matching method, the method proposed in this invention takes into account the changes in the area to be aligned caused by changes in perspective. The method is intuitive, simple and relatively accurate, which improves the accuracy of matching and reduces the probability of mismatching.

[0045] (2) The present invention uses the normalized cross-correlation function as an image registration determination method, which reduces the impact of illumination problems under different viewing angles. For images under non-adjacent viewing angles, the variable template scale and direction are obtained by using the intermediate image calculation results, thereby avoiding the time loss caused by obtaining the variable template parameters through a loop traversal method, and is also more accurate.

[0046] (3) The variable template in the present invention can directly align images with relatively large changes in viewing angle. Compared with the traditional guided matching method, this method eliminates the cumulative error caused by guided matching.

[0047] (4) In the present invention, the relative pose transformation of the point cloud is calculated by the matching result of the final feature point and brought into the iterative closest point algorithm as the initial value of the iteration, so that the iteration is not easy to fall into the local optimal solution, and the iteration converges quickly, reducing the number of iterations and improving the iteration efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of the steps of the method is shown in FIG.

[0049] Figure 2 Schematic diagram of multi-view matching of RGB-D cameras in the method of the present invention;

[0050] Figure 3 Schematic diagram of variable template sampling in the method of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0052] Example 1

[0053] Since a single RGB-D camera has a limited shooting angle, only local information of the object can be obtained by shooting from one angle, while in some specific scenes, shooting from multiple angles and performing multi-angle matching are required. Matching image feature points from different angles is the key to three-dimensional reconstruction, but as the camera angle changes, digital image correlation methods will experience accumulation of matching errors and increase in mismatches, which in turn affects the three-dimensional reconstruction effect. The present invention discloses a digital image calculation method based on multi-angle matching of RGB-D cameras with variable templates. First, data acquisition is performed. The data acquisition device is an RGB-D camera, which includes a color camera and a depth camera. The collected data includes color information and depth data, wherein the depth data represents the distance of the target point from the camera plane. By calibrating the color camera and the depth camera or the internal parameters of the depth camera of the RGB-D camera and the alignment method of the depth data and the color data, the three-dimensional information and corresponding color information of the object under test can be obtained.

[0054] Then, the point cloud data is preprocessed, and the acquired depth image is filtered directly to filter the background area beyond the set range, and the random sampling consensus algorithm is used to find the optimal ground point cloud and remove it. After that, cluster analysis is performed, and the density clustering algorithm is used to retain point cloud data with similar density, and the outlier point cloud is filtered out to retain the point cloud belonging to the measured object. The retained point cloud area is the area for subsequent feature point selection and registration.

[0055] Then perform feature point matching, and use the digital image correlation method to track the feature points of adjacent images in the image sequence. The relative pose transformation under each viewing angle is calculated by the position of the feature point, and the calculation result of the relative pose change is used to adjust the scale and direction of the variable template. The variable template is used to calculate the correlation of feature points between non-adjacent images. The adjusted variable template makes the pixel points sampled in the reference image and the deformed image correspond as much as possible, reducing the impact of the wrong matching part and ensuring that the correct match can be found even if the viewing angle changes too much. The point with the largest correlation is selected as the matched feature point.

[0056] Finally, the pose transformation relationship of the matched feature points in a set of point cloud data is calculated and used as the initial value of the iterative closest point algorithm for iterative calculation. When the iterative convergence condition is reached, the point cloud fusion is completed.

[0057] Digital image calculation method based on variable template RGB-D camera multi-view matching, such as Figure 1 As shown, the method specifically includes:

[0058] Step S1: Image sequence acquisition, acquisition and alignment of color data and depth data.

[0059] The three-dimensional information and corresponding color information of the object under test are obtained through the internal parameters of the depth camera provided by the RGB-D camera and the alignment method of the depth data and color data. The calculation formula of the three-dimensional information is as follows:

[0060]

[0061] Where: d i Represents the depth value, that is, the distance between the target point and the camera plane; c x ,c y ,f x ,f y Represents the internal parameters of the depth camera; X iw ,Y iw ,Z iw Indicates the position of the target point in the world coordinate system;

[0062] The color information is obtained through the color data stream of the RBG-D camera. The specific method of obtaining is: Color = (R, G, B)

[0063] Where: R, G, B represent the corresponding red, green, and blue channel values ​​in the RGB-D camera respectively.

[0064] Step S2: Preprocess the point cloud data, remove the ground and outlier points, and obtain the area to be registered.

[0065] Based on the three-dimensional information of the point cloud, the image is filtered directly, and the threshold is Xmin ,X max ,Y min ,Y max ,Z min ,Z max , filter out the background area beyond the set range, as follows:

[0066]

[0067] Among them, X min ,X max ,Y min ,Y max ,Z min ,Z max They represent the detection thresholds in the world coordinate system respectively.

[0068] Using the random sampling consensus algorithm, we randomly select 3 points from the filtered point cloud, repeat k times, and the plane with the most points in the fitted plane is the optimal ground, which is subtracted from the filtered point cloud to remove the ground point cloud data. We perform cluster analysis on the processed point cloud, and use the density algorithm to retain point cloud data with similar density to eliminate outliers.

[0069] Step S3: feature point matching is performed on adjacent images in the image sequence, and the transformation relationship of the point cloud coordinates under different viewing angles is calculated.

[0070] The correlation calculation of the image sequence is used to track the image feature points. The correlation formula uses the normalized cross-correlation formula, which can make the registration under multi-viewing angles not affected by lighting:

[0071]

[0072] in: Represents the pixel grayscale mean of the reference subset and the deformed subset. The point with the largest correlation is selected as the matched feature point. The relative pose transformation under each viewing angle is calculated through the position of the feature point, that is, the rotation matrix R and the translation matrix T.

[0073] by Figure 2 For example, Figure 2 In the figure, the coordinate transformation relationship between the corresponding points of the first and second images is calculated by the following formula:

[0074]

[0075] Similarly, the coordinate transformation relationship between all adjacent images in the image sequence can be calculated and translation matrix Then we can calculate the relative pose transformation of the feature points of the k-th image and the n-th image and convert it into Euler angles:

[0076]

[0077] θ x θ y and θ z They are the scale parameter and direction parameter of the variable template respectively.

[0078] Step S4: re-register the variable template corresponding to the non-adjacent large-viewing angle image by adjusting the scale and direction of the variable template according to the calculation results.

[0079] like Figure 3 As shown, the sampling of the reference template is described in polar coordinates:

[0080]

[0081] Where R represents the sampling radius and θ represents the sampling angle. When the shooting angle changes, it is assumed that the deformed image is rotated by θ around the x, y, and z axes in space relative to the reference image. x θ y θ z Angle, then the sampling of the deformation template is as follows:

[0082]

[0083] Through the above formula, the sampling radius of the deformed template is set to be consistent with the sampling angle and the reference template.

[0084] Among them, for θ x θ y θ z The value of This method can make the pixel points obtained by the two sampling templates correspond to each other as much as possible, reduce the impact of the wrong matching part, and ensure that the correct match can be found even if the viewing angle changes greatly.

[0085] Step S5: Calculate the point cloud data transformation relationship between the large-viewing angle images.

[0086] By registering the feature points of the k-th image and the n-th image through a variable template, the coordinate transformation relationship between the corresponding points of the k-th image and the n-th image can be obtained as follows:

[0087]

[0088] Step S6: point cloud fusion, calculating the pose transformation relationship of the matched feature points in a set of point cloud data, and performing iterative calculation as the initial value of the iterative closest point algorithm to complete the point cloud fusion.

[0089] Will Bring it into the iterative closest point algorithm as the initial value of the iteration, and set the termination condition of the I CP iteration to reach num times or reach the termination condition:

[0090]

[0091] After iterative convergence, we get Realize the registration and fusion of the k-th point cloud and the n-th point cloud data to complete the multi-perspective 3D reconstruction.

[0092] Therefore, the present invention adjusts the size of the non-adjacent image matching template through the registration results of adjacent viewpoints, and can directly match the feature points of images with large viewpoint changes, thereby achieving correct registration of feature points of non-adjacent viewpoint images, reducing the cumulative errors caused by mismatching and intermediate calculations, improving matching accuracy, and eliminating the cumulative errors of the image set. This method provides a more accurate initial value for subsequent point cloud fusion iterations, reduces the number of iterations, and achieves three-dimensional reconstruction.

[0093] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A digital image calculation method for multi-view matching of RGB-D cameras based on a variable template, characterized in that: The steps include: S1, data acquisition: collect image information data through RGB-D camera, the image information data includes color information and depth data, and align the three-dimensional information and color information of the point cloud of the object to be measured by calibrating the camera or aligning the internal parameters of the depth camera of the RGB-D camera and the depth data with the color data; S2, point cloud data preprocessing: perform straight-through filtering on the three-dimensional information obtained in step S1, and use a random sampling consensus algorithm to determine the optimal ground cloud and remove it; Perform cluster analysis on the processed point cloud, use density algorithm to retain point cloud data with similar density, and obtain the area to be registered; S3, feature point matching: track the feature points of adjacent images in the image sequence, and calculate the relative position transformation relationship of the point cloud coordinates under each viewing angle through the position of the feature points; S4, re-registration of the variable template: adjusting the scale and direction of the variable template according to the calculation result of the relative position and posture transformation relationship of the point cloud coordinates obtained in step S3; specifically, the sampling of the reference template is described in polar coordinates: Where R represents the sampling radius and θ represents the sampling angle. When the shooting angle changes, the deformed image rotates around the x, y, and z axes in space relative to the reference image. x θ y θ z Angle, then the sampling of the deformation template is: Among them, the sampling radius and sampling angle of the deformation template are set to be consistent with the reference template; S5, calculating the point cloud data transformation relationship between the large viewing angle images: aligning the image feature points according to the variable template obtained in step S4 to obtain the corresponding point coordinate transformation relationship; S6, point cloud fusion: Calculate the pose transformation relationship of the matched feature points in a set of point cloud data, use it as the initial value of the iterative closest point algorithm for iterative calculation, complete point cloud fusion, and realize three-dimensional reconstruction of the image.

2. The digital image calculation method based on variable template RGB-D camera multi-view matching according to claim 1, characterized in that: The specific method for calculating the three-dimensional information of the object under test in step S1 is: Where: d i Indicates the distance between the target point and the camera plane; c x ,c y ,f x ,f y Represents the internal parameters of the depth camera; X iw ,Y iw ,Z iw Respectively represent the position of the target point in the world coordinate system; The color information is obtained through the color data stream inside the RBG-D camera, and the specific method of obtaining is as follows: Color = (R, G, B) Where: R, G, B represent the corresponding red, green, and blue channel values ​​in the RGB-D camera respectively.

3. The digital image calculation method based on variable template RGB-D camera multi-view matching as claimed in claim 2, characterized in that: In the step S2, the background area beyond the set range is filtered out in the through filtering of the three-dimensional information, specifically: Among them, X min ,X max ,Y min ,Y max ,Z min ,Z max They represent the detection thresholds in the world coordinate system respectively; In the random sampling consensus algorithm, three points are randomly selected from the filtered point cloud. After multiple repetitions, the cloud with the largest number of points in the fitted plane is the optimal ground cloud.

4. The digital image calculation method based on variable template RGB-D camera multi-view matching as claimed in claim 3, characterized in that: In step S3, correlation calculation is performed on the image sequence for tracking image feature points, and the correlation formula is: in: Represents the mean grayscale value of the pixel of the reference subset and the deformed subset; u, v represent the horizontal and vertical offsets of the feature point in the deformed image; r(x,y), d(x+u,y+v) represent the grayscale values ​​of the pixel of the reference subset and the deformed subset at (x,y), (x+u,y+v) in the image coordinate system.

5. The digital image calculation method based on variable template RGB-D camera multi-view matching as claimed in claim 3, characterized in that: In step S3, the relative position transformation relationship of the point cloud coordinates under each adjacent viewing angle is calculated by the position of the feature point: the rotation matrix R and the translation matrix T, and the coordinate transformation relationship of the corresponding point is: Among them, P1 and P2 represent the positions of the feature points in the image before rotation and translation and the image after rotation and translation respectively; R2 1 ,T2 1 The rotation matrix and translation matrix representing the feature points in the image after rotation and translation relative to the feature points in the image before rotation and translation; The rotation matrix R based on the coordinate transformation relationship between all adjacent images in the image sequence n n-1 and translation matrix The relative pose transformation of the feature points between the kth image and the nth image is calculated and converted into Euler angles as follows: Among them, θ x ,θ y and θ z They are the scale parameter and direction parameter of the variable template respectively.

6. The digital image calculation method based on variable template RGB-D camera multi-view matching as claimed in claim 5, characterized in that: In step S5, the coordinate transformation relationship of the corresponding points is specifically: Among them, P k ,P n Respectively represent the position of the feature point in image k and image n; R n k , T n k Represents the rotation matrix and translation matrix of the feature points in image n relative to the feature points in image k.

7. The digital image calculation method based on variable template RGB-D camera multi-view matching according to claim 6, characterized in that: The calculation of iteration termination in step S6 is specifically as follows: Among them, A current ,B current Respectively represent the coordinates of the target point cloud of the current iteration and the point cloud after the change; A last ,B last Respectively represent the coordinates of the target point cloud of the previous iteration and the point cloud after the change; m, n represent the number of corresponding points in the point cloud of the current iteration and the previous iteration; error represents the threshold for iteration termination; After iterative convergence, we get Realize the registration and fusion of the k-th point cloud and the n-th point cloud data to complete multi-perspective 3D reconstruction.

8. A digital image computing system for multi-view matching of RGB-D cameras based on a variable template, including a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method as described in any one of claims 1 to 7 are implemented.

Images