Digital image calculation method and system for deformable template-based RGB-d camera multi-view matching

Abstract

Disclosed in the present invention are a digital image calculation method and system for deformable template-based RGB-D camera multi-view matching. The method comprises the six steps of data acquisition, point cloud data preprocessing, feature point matching, re-registration of deformable templates, calculation of a point cloud data transformation relationship between large-view images, and point cloud fusion. The size of a non-adjacent image matching template is adjusted on the basis of a registration result of adjacent views to achieve correct registration of feature points of non-adjacent view images, thereby improving the matching precision, eliminating the accumulative error of an image set, providing a relatively accurate initial value for subsequent iteration of point cloud fusion, reducing the number of iterations, and implementing three-dimensional reconstruction of images.

Description

Digital image calculation method and system for multi-view matching of RGB-D cameras based on variable templates Technical Field

[0001] The present invention belongs to the technical field of multi-view three-dimensional imaging and machine vision, and mainly relates to a digital image calculation method and system for multi-view matching of RGB-D cameras based on variable templates. Background Art

[0002] With the increasing popularity of 3D printing and virtual reality technologies, the demand for 3D reconstruction technology is growing. Because manual modeling using 3D modeling software is prohibitively expensive, research has focused on reconstructing objects from multiple camera perspectives. RGB-D camera-based 3D reconstruction methods can be broadly categorized into two categories: point cloud and image-based. While point cloud-based 3D reconstruction algorithms currently offer some promising results, such as the well-known ICP algorithm and its improvements, they still have limitations. They struggle with unordered point cloud scenes and often require an initial value close to the ground truth, otherwise they can easily fall into local optima. Image-based 3D reconstruction algorithms, on the other hand, rely on matching adjacent images. Generally speaking, multi-view 3D reconstruction involves the following steps: 1. The camera acquires a multi-view image sequence and matches feature points within the image sequence. 2. The transformation relationship between the images is calculated, and the point cloud data within the image sequence is aligned. Therefore, the quality of 3D reconstruction places higher demands on the quality of feature point pairs and the accuracy of feature point matching.

[0003] Digital image correlation matching in image sequences often encounters several challenges. For example, excessive rotation or scaling between non-adjacent images can lead to an increase in mismatches, making direct matching impossible. Furthermore, as the number of images from different viewpoints increases, matching errors between adjacent images accumulate, resulting in bifurcations after point cloud alignment. Researchers have proposed solutions to these issues. In binocular stereo matching, extreme geometric constraints are used to reduce mismatches. However, this approach cannot be applied to single-camera, multi-view scenarios because it requires prior knowledge of the relative pose of the two camera viewpoints through calibration. For images that cannot be directly matched, previous research has proposed inserting a series of intermediate images and incrementally accumulating the results of the guided matching. This approach is effective in most cases, but in multi-view matching, the inclusion of too many intermediate images increases computational cost and makes it difficult to avoid cumulative errors. Some feature point matching methods, such as SIFT and its modifications, have also achieved promising results. However, these methods still have limitations. They rely heavily on the number of feature points. If the number of feature point pairs between the two images to be matched is too small, matching may fail. Related studies have also shown that matching at too large an angle will cause large errors, and it is necessary to limit the viewing angles of adjacent images. However, in scenarios that require a large viewing angle, introducing multiple small viewing angles will undoubtedly lead to the accumulation of matching errors.

[0004] Summary of the Invention

[0005] The present invention addresses the problems of accumulated matching errors and mismatching in the existing technology of multi-view matching with a single RGB-D camera. It provides a digital image calculation method and system for multi-view matching with RGB-D cameras based on a variable template. The method includes six steps: data acquisition, point cloud data preprocessing, feature point matching, re-registration of the variable template, calculation of the point cloud data transformation relationship between large-view images, and point cloud fusion. The registration results of adjacent viewpoints are used to adjust the size of the matching template of non-adjacent images to achieve correct registration of feature points of non-adjacent viewpoint images, improve matching accuracy, eliminate accumulated errors of the image set, provide more accurate initial values ​​for subsequent point cloud fusion iterations, reduce the number of iterations, and achieve three-dimensional reconstruction of the image.

[0006] To achieve the above object, the technical solution adopted by the present invention is: a digital image calculation method for multi-view matching of RGB-D cameras based on a variable template, comprising the following steps:

[0007] S1, data acquisition: using an RGB-D camera to acquire image information data, including color information and depth data. Aligning the three-dimensional information and color information of the object point cloud by calibrating the camera or aligning the depth data with the color data using the RGB-D camera's built-in depth camera parameters.

[0008] S2, point cloud data preprocessing: perform straight-through filtering on the 3D information obtained in step S1, and use a random sampling consensus algorithm to determine the optimal ground cloud and remove it; perform cluster analysis on the processed point cloud, use a density algorithm to retain point cloud data with similar density, and obtain the area to be registered;

[0009] S3, feature point matching: Track the feature points of adjacent images in the image sequence, and calculate the relative position transformation relationship of the point cloud coordinates under each view angle based on the position of the feature points;

[0010] S4, re-registration of the variable template: adjusting the scale and direction of the variable template according to the calculation result of the relative pose transformation relationship of the point cloud coordinates obtained in step S3;

[0011] S5, calculating the point cloud data transformation relationship between the wide-view images: registering the image feature points according to the variable template obtained in step S4 to obtain the corresponding point coordinate transformation relationship;

[0012] S6, point cloud fusion: Calculate the pose transformation relationship of the matched feature points in a set of point cloud data, use it as the initial value of the iterative closest point algorithm for iterative calculation, complete point cloud fusion, and realize 3D reconstruction of the image.

[0013] As an improvement of the present invention, the specific method for calculating the three-dimensional information of the measured object in step S1 is:

[0014] Where: d i Indicates the distance between the target point and the camera plane; c x ,c y ,f x ,f y Represents the internal parameters of the depth camera; X iw ,Y iw ,Z iw Indicates the position of the target point in the world coordinate system;

[0015] The color information is obtained through the color data stream of the RBG-D camera. The specific method of obtaining is: Color = (R, G, B)

[0016] Where: R, G, B represent the corresponding red, green, and blue channel values ​​in the RGB-D camera respectively.

[0017] As an improvement of the present invention, in step S2, the background area beyond the set range is filtered out during the through filtering of the three-dimensional information, specifically:

[0018] Among them, X min ,X max ,Ymin ,Y max ,Z min ,Z max Represents the detection threshold in the world coordinate system;

[0019] In the random sampling consensus algorithm, three points are randomly selected from the filtered point cloud. After repeated multiple times, the cloud with the largest number of points in the fitted plane is the optimal ground cloud.

[0020] As another improvement of the present invention, in step S3, correlation calculation is performed on the image sequence for tracking image feature points. The correlation formula is:

[0021] in: Represents the mean grayscale value of the reference subset and the deformed subset; u, v represent the horizontal and vertical offsets of the feature point in the deformed image; r(x,y), d(x+u,y+v) represent the grayscale values ​​of the reference subset and the deformed subset at (x,y), (x+u,y+v) in the image coordinate system.

[0022] As another improvement of the present invention, in step S3, the relative position transformation relationship of the point cloud coordinates under each viewing angle is calculated by the position of the feature point: the rotation matrix R and the translation matrix T, and the coordinate transformation relationship of the corresponding point is:

[0023] Among them, P1 and P2 represent the positions of the feature points in the image before rotation and translation and the image after rotation and translation respectively; The rotation matrix and translation matrix representing the feature points in the image after rotation and translation relative to the feature points in the image before rotation and translation;

[0024] The rotation matrix based on the coordinate transformation relationship between all adjacent images in the image sequence and translation matrices The relative pose transformation of the feature points of the k-th image and the n-th image is calculated and converted into Euler angles as follows:

[0025] Among them, θ x ,θ y and θ z are the scale parameter and direction parameter of the variable template respectively.

[0026] As another improvement of the present invention, in step S4, the sampling of the reference template is described in polar coordinates, specifically:

[0027] Where R represents the sampling radius and θ represents the sampling angle. When the shooting angle changes, the deformed image rotates around the x, y, and z axes relative to the reference image in space by θ. x θ y θ z Angle, then the sampling of the deformation template is:

[0028] The sampling radius and sampling angle of the deformation template are set to be consistent with those of the reference template.

[0029] As another improvement of the present invention, in step S5, the coordinate transformation relationship of the corresponding points is specifically:

[0030] Among them, P k ,P n Represent the positions of feature points in image k and image n respectively; P k ,P n Represents the rotation matrix and translation matrix of the feature points in image n relative to the feature points in image k;

[0031] As a further improvement of the present invention, the calculation of the iteration termination in step S6 is specifically as follows:

[0032] Among them, A current ,B current Respectively represent the coordinates of the target point cloud of the current iteration and the point cloud after the change; A last ,B last Represent the coordinates of the target point cloud and the changed point cloud of the previous iteration respectively; m, n represent the number of corresponding points in the point cloud of the current iteration and the previous iteration respectively; error represents the threshold for iteration termination;

[0033] After iterative convergence, we get Realize the registration and fusion of the k-th point cloud and the n-th point cloud data to complete multi-perspective 3D reconstruction.

[0034] In order to achieve the above-mentioned purpose, the present invention also adopts a technical solution: a digital image computing system for multi-view matching of RGB-D cameras based on a variable template, including a computer program, which, when executed by a processor, implements the steps of any of the methods described above.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] (1) The present invention proposes to use a variable template to align images of different perspectives. Compared with the existing direct matching method, the method proposed in this invention takes into account the changes in the area to be registered caused by changes in perspective. The method is intuitive, simple and relatively accurate, which improves the accuracy of matching and reduces the probability of mismatching.

[0037] (2) The present invention uses the normalized cross-correlation function as an image registration determination method, which reduces the impact of illumination problems under different viewing angles. For images under non-adjacent viewing angles, the variable template scale and direction are obtained by calculating the results of the intermediate image, thereby avoiding the time loss caused by obtaining the variable template parameters through a cyclic traversal method, and is also more accurate.

[0038] (3) The variable template in the present invention can directly align images with relatively large changes in perspective. Compared with the traditional guided matching method, this method eliminates the cumulative error caused by guided matching.

[0039] (4) In the present invention, the relative pose transformation of the point cloud is calculated based on the matching result of the final feature point and brought into the iterative closest point algorithm as the initial value of the iteration, so that the iteration is not easily trapped in the local optimal solution, and the iteration converges quickly, reducing the number of iterations and improving the iteration efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] FIG1 is a flow chart of the steps of the present invention;

[0041] FIG2 is a schematic diagram of multi-view matching of RGB-D cameras in the method of the present invention;

[0042] FIG3 is a schematic diagram of variable template sampling in the method of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0044] Example 1

[0045] Since a single RGB-D camera has a limited shooting angle, only local information of the object can be obtained by shooting from one angle, while in some specific scenes, it is necessary to shoot from multiple angles and perform multi-angle matching. Matching image feature points from different angles is the key to three-dimensional reconstruction, but as the camera angle changes, digital image correlation methods will experience accumulation of matching errors and increase in mismatches, which in turn affects the three-dimensional reconstruction effect. The present invention discloses a digital image calculation method for multi-angle matching of RGB-D cameras based on variable templates. First, data acquisition is performed. The data acquisition device is an RGB-D camera, which includes a color camera and a depth camera. The collected data includes color information and depth data, wherein the depth data represents the distance of the target point from the camera plane. By calibrating the color camera and the depth camera or aligning the internal parameters of the depth camera of the RGB-D camera and the depth data with the color data, the three-dimensional information and corresponding color information of the object under test can be obtained.

[0046] The point cloud data is then preprocessed. A straight-through filter is applied to the acquired depth image to filter out background areas outside the set range. A random sampling consensus algorithm is then used to find and remove the optimal ground point cloud. Cluster analysis is then performed, using a density clustering algorithm to retain point cloud data with similar density. Outlier point clouds are filtered out, retaining the point cloud belonging to the object being measured. The retained point cloud area is then used for subsequent feature point selection and registration.

[0047] Next, feature point matching is performed, using digital image correlation methods to track feature points in adjacent images within the image sequence. The relative pose transformation under each viewpoint is calculated from the position of the feature points. The calculated relative pose transformation is used to adjust the scale and orientation of the variable template. A variable template is used to calculate the correlation between feature points in non-adjacent images. This adjusted variable template ensures that the pixels sampled from the reference image and the deformed image are as closely aligned as possible, minimizing the impact of incorrect matches and ensuring accurate matches even with large viewpoint changes. The point with the highest correlation is selected as the matched feature point.

[0048] Finally, the pose transformation relationship of the matched feature points in a set of point cloud data is calculated and used as the initial value of the iterative closest point algorithm for iterative calculation. When the iterative convergence condition is reached, the point cloud fusion is completed.

[0049] A digital image calculation method for multi-view matching of RGB-D cameras based on a variable template is shown in FIG1 , and the method specifically includes:

[0050] Step S1: Image sequence acquisition, acquisition and alignment of color data and depth data.

[0051] The 3D information and corresponding color information of the object being measured are obtained by using the internal parameters of the depth camera provided by the RGB-D camera and the alignment method of the depth data and color data. The calculation formula of the 3D information is as follows:

[0052] Where: d i Represents the depth value, that is, the distance between the target point and the camera plane; c x ,c y ,f x ,f y Represents the internal parameters of the depth camera; X iw ,Y iw ,Z iw Indicates the position of the target point in the world coordinate system;

[0053] The color information is obtained through the color data stream of the RBG-D camera. The specific method of obtaining is: Color = (R, G, B)

[0054] Where: R, G, B represent the corresponding red, green, and blue channel values ​​in the RGB-D camera respectively.

[0055] Step S2: Preprocess the point cloud data, remove the ground and outlier points, and obtain the area to be registered.

[0056] Based on the three-dimensional information of the point cloud, the image is filtered directly with a threshold of X min ,X max ,Y min ,Y max ,Z min ,Z max , filter out the background area beyond the set range, as follows:

[0057] Among them, X min ,X max ,Y min ,Y max ,Z min ,Z max They represent the detection thresholds in the world coordinate system respectively.

[0058] Using a random sampling consensus algorithm, we randomly select three points from the filtered point cloud, repeating this process k times. The plane fitted with the largest number of points is considered the optimal ground plane, and the ground point cloud data is removed by performing a set difference with the filtered point cloud. We then perform cluster analysis on the processed point cloud, using a density algorithm to retain point cloud data with similar density, thereby eliminating outliers.

[0059] Step S3: Match feature points of adjacent images in the image sequence and calculate the transformation relationship of point cloud coordinates under different viewing angles.

[0060] The correlation calculation of the image sequence is used to track the image feature points. The correlation formula uses the normalized cross-correlation formula, which can make the registration under multi-view not affected by lighting:

[0061] in: Represents the pixel grayscale mean of the reference subset and the deformed subset, selects the point with the largest correlation as the matched feature point, and calculates the relative pose transformation under each viewing angle through the position of the feature point, that is, the rotation matrix R and the translation matrix T.

[0062] Taking Figure 2 as an example, in Figure 2, the coordinate transformation relationship between the corresponding points in the first and second images is calculated by the following formula:

[0063] Similarly, the coordinate transformation relationship between all adjacent pictures in the picture sequence can be calculated and translation matrices Then we can calculate the relative pose transformation of the feature points of the k-th image and the n-th image and convert it into Euler angles:

[0064] θ x θ y and θ z are the scale parameter and direction parameter of the variable template respectively.

[0065] Step S4: re-register the variable templates corresponding to the non-adjacent large-viewing angle images by adjusting the scale and direction of the variable templates according to the calculation results.

[0066] As shown in Figure 3, the sampling of the reference template is described using polar coordinates:

[0067] Where R represents the sampling radius and θ represents the sampling angle. When the shooting angle changes, it is assumed that the deformed image is rotated by θ around the x, y, and z axes relative to the reference image in space. x θ y θ z Angle, then the sampling of the deformed template is as follows:

[0068] Through the above formula, the sampling radius of the deformed template is set to be consistent with the sampling angle and the reference template.

[0069] Among them, for θ x θ y θ z The value of This method ensures that the pixels obtained from the two sampling templates are as close as possible to each other, reducing the impact of incorrect matches and ensuring that the correct match can be found even when the viewing angle changes greatly.

[0070] Step S5: Calculate the point cloud data transformation relationship between the wide-view images.

[0071] By registering the feature points of the k-th image and the n-th image through a variable template, the coordinate transformation relationship of the corresponding points of the k-th image and the n-th image can be obtained as follows:

[0072] Step S6: point cloud fusion, calculate the pose transformation relationship of the matched feature points in a set of point cloud data, use it as the initial value of the iterative closest point algorithm for iterative calculation, and complete the point cloud fusion.

[0073] Will Bring it into the iterative closest point algorithm as the initial value of the iteration, and set the termination condition of the ICP iteration to reach num times or reach the termination condition:

[0074] After iterative convergence, we get Realize the registration and fusion of the k-th point cloud and the n-th point cloud data to complete the multi-view 3D reconstruction.

[0075] Therefore, the present invention adjusts the size of the non-adjacent image matching template through the registration results of adjacent perspectives, and can directly match the feature points of images with large perspective changes, thereby achieving correct registration of feature points of non-adjacent perspective images, reducing the cumulative errors caused by mismatching and intermediate calculations, improving matching accuracy, and eliminating the cumulative errors of the image set. This method provides a more accurate initial value for subsequent point cloud fusion iterations, reduces the number of iterations, and achieves three-dimensional reconstruction.

[0076] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. A digital image calculation method for multi-view matching of RGB-D cameras based on a variable template, characterized in that: The steps include: S1, data acquisition: collect image information data through RGB-D camera, the image information data includes color information and depth data, and align the three-dimensional information and color information of the point cloud of the object to be measured by calibrating the camera or aligning the internal parameters of the depth camera of the RGB-D camera and the depth data with the color data; S2, point cloud data preprocessing: perform straight-through filtering on the three-dimensional information obtained in step S1, and use a random sampling consensus algorithm to determine the optimal ground cloud and remove it; Perform cluster analysis on the processed point cloud, use density algorithm to retain point cloud data with similar density, and obtain the area to be registered; S3, feature point matching: track the feature points of adjacent images in the image sequence, and calculate the relative position transformation relationship of the point cloud coordinates under each viewing angle through the position of the feature points; S4, re-registration of the variable template: adjusting the scale and direction of the variable template according to the calculation result of the relative position and posture transformation relationship of the point cloud coordinates obtained in step S3; S5, calculating the point cloud data transformation relationship between the large viewing angle images: aligning the image feature points according to the variable template obtained in step S4 to obtain the corresponding point coordinate transformation relationship; S6, point cloud fusion: Calculate the pose transformation relationship of the matched feature points in a set of point cloud data, use it as the initial value of the iterative closest point algorithm for iterative calculation, complete point cloud fusion, and realize three-dimensional reconstruction of the image.

2. The digital image calculation method based on variable template RGB-D camera multi-view matching according to claim 1, characterized in that: The specific method for calculating the three-dimensional information of the object under test in step S1 is: Where: d i Indicates the distance between the target point and the camera plane; c x ,c y ,f x ,f y Represents the internal parameters of the depth camera; X iw ,Y iw ,Z iw Respectively represent the position of the target point in the world coordinate system; The color information is obtained through the color data stream inside the RBG-D camera, and the specific method of obtaining is as follows: Color = (R, G, B) Where: R, G, B represent the corresponding red, green, and blue channel values ​​in the RGB-D camera respectively.

3. The digital image calculation method based on variable template RGB-D camera multi-view matching as claimed in claim 2, characterized in that: In the step S2, the background area beyond the set range is filtered out in the through filtering of the three-dimensional information, specifically: Among them, X min ,X max ,Y min ,Y max ,Z min ,Z max They represent the detection thresholds in the world coordinate system respectively; In the random sampling consensus algorithm, three points are randomly selected from the filtered point cloud. After multiple repetitions, the cloud with the largest number of points in the fitted plane is the optimal ground cloud.

4. The digital image calculation method based on variable template RGB-D camera multi-view matching as claimed in claim 3, characterized in that: In step S3, correlation calculation is performed on the image sequence for tracking image feature points, and the correlation formula is: in: Represents the mean grayscale value of the pixel of the reference subset and the deformed subset; u, v represent the horizontal and vertical offsets of the feature point in the deformed image; r(x,y), d(x+u,y+v) represent the grayscale values ​​of the pixel of the reference subset and the deformed subset at (x,y), (x+u,y+v) in the image coordinate system.

5. The digital image calculation method based on variable template RGB-D camera multi-view matching as claimed in claim 3, characterized in that: In step S3, the relative position transformation relationship of the point cloud coordinates under each adjacent viewing angle is calculated by the position of the feature point: the rotation matrix R and the translation matrix T, and the coordinate transformation relationship of the corresponding point is: Among them, P1 and P2 represent the positions of the feature points in the image before rotation and translation and the image after rotation and translation respectively; The rotation matrix and translation matrix representing the feature points in the image after rotation and translation relative to the feature points in the image before rotation and translation; The rotation matrix based on the coordinate transformation relationship between all adjacent images in the image sequence and translation matrix The relative pose transformation of the feature points between the kth image and the nth image is calculated and converted into Euler angles as follows: Among them, θ x ,θ y and θ z They are the scale parameter and direction parameter of the variable template respectively.

6. The digital image calculation method based on variable template RGB-D camera multi-view matching as claimed in claim 5, characterized in that: In step S4, the sampling of the reference template is described in polar coordinates, specifically: Where R represents the sampling radius and θ represents the sampling angle. When the shooting angle changes, the deformed image rotates around the x, y, and z axes in space relative to the reference image by θ. x θ y θ z Angle, then the sampling of the deformation template is: The sampling radius and sampling angle of the deformation template are set to be consistent with those of the reference template.

7. The digital image calculation method based on variable template RGB-D camera multi-view matching according to claim 6, characterized in that: In step S5, the coordinate transformation relationship of the corresponding points is specifically: Among them, P k ,P n Respectively represent the position of the feature point in image k and image n; P k ,P n Represents the rotation matrix and translation matrix of the feature points in image n relative to the feature points in image k.

8. The digital image calculation method based on variable template RGB-D camera multi-view matching according to claim 7, characterized in that: The calculation of iteration termination in step S6 is specifically as follows: Among them, A current ,B current Respectively represent the coordinates of the target point cloud of the current iteration and the point cloud after the change; A last ,B last Respectively represent the coordinates of the target point cloud of the previous iteration and the point cloud after the change; m, n represent the number of corresponding points in the point cloud of the current iteration and the previous iteration; error represents the threshold for iteration termination; After iterative convergence, we get Realize the registration and fusion of the k-th point cloud and the n-th point cloud data to complete multi-perspective 3D reconstruction.

9. A digital image computing system for multi-view matching of RGB-D cameras based on a variable template, including a computer program, characterized in that: When the computer program is executed by a processor, the steps of any of the above methods are implemented.

Images