A method and system for initial registration of augmented reality targets

Abstract

The present invention discloses an augmented reality target initial registration method and system, which relates to the field of augmented reality technology. The method comprises: sampling the surface of a target model to obtain a model dense point cloud; downsampling the model dense point cloud to obtain a model point cloud; calculating the normal vector of each point in the model point cloud based on the model dense point cloud and the model point cloud; calculating the hash value of the point pair feature in the model point cloud based on the normal vector of each point in the model point cloud; determining the supervoxels of the target scene point cloud and clustering the supervoxels according to a preset threshold; applying a voting algorithm based on the hash value to determine the corresponding pose of each supervoxel in the clustering result; applying a clustering algorithm to the pose set, and using the pose with the highest number of votes in the clustered pose set as the initial pose estimation result; calculating the precise pose of the target, and using the precise pose as the target initial registration pose. The present invention can shorten the scene matching time and improve the estimation accuracy.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality technology, and in particular to an augmented reality target initial registration method and system. Background Art

[0002] Augmented reality (AR) is a technology that fuses virtual and real life, encompassing multiple research fields such as computer vision, computer graphics, and optics. It has widespread application in scenarios such as smart healthcare, industrial manufacturing, assisted assembly, robotic path planning, and natural sciences. AR is a system that integrates virtual and real life, creating an interactive, aligned virtual and real world. Accurately aligning the virtual and real worlds is a fundamental challenge for AR systems, and achieving accurate registration is a key technology limiting their development. Currently, relatively mature initial registration methods include those based on manual landmarks and those based on the natural features of the target model. Manual landmark deployment and alignment require significant resources, and the accuracy of landmark recognition directly impacts registration accuracy. Natural feature-based landmarks rely on the visual characteristics of the target model itself, making them ineffective in scenarios where the target model has weak or even no texture, such as in assisted assembly. With the increasing popularity of depth sensors, object registration methods based on stereoscopic vision perception are gaining increasing attention.

[0003] Target registration methods based on stereoscopic visual perception can overcome the problems of weak texture and uncertain scale transformation of target models. They are suitable for application scenarios where artificial identification is prohibited and the surface texture of target models is weak. Traditional stereo matching methods based on 3D feature correspondence extract 3D feature points from the target model point cloud and the scene point cloud, match the feature points in the model point cloud and the scene point cloud, and use the RANSAC algorithm to perform correspondence screening. Based on the stable correspondences after screening, ICP iteration is performed to calculate the transformed pose between the model point cloud and the target surface point cloud in the scene point cloud, which is used as the pose for model registration in the camera coordinate system. The extraction and matching of 3D features is time-consuming and can easily fall into local optimality in complex scenes, resulting in matching failure.

[0004] The target pose estimation method based on point-pair feature voting uses point-pair features to describe the local features of the target model. It matches a point on the target surface in the scene point cloud with a point on the target model generated point cloud, and then calculates the Hough transform between the two to estimate the target pose. Processing 3D data requires significant computing resources, often requiring downsampling to simplify the data processing process. However, in augmented reality applications, target models often have variable scales. Blind downsampling can lead to the failure of point-pair algorithms due to the lack of valid target information in the downsampled result. Summary of the Invention

[0005] The purpose of the present invention is to provide an augmented reality target initial registration method and system, which can shorten the scene matching time and improve the estimation accuracy.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] An augmented reality target initial registration method, the registration method comprising:

[0008] Sample the surface of the target model to obtain a dense point cloud of the model;

[0009] Calculating a diameter of the target model, and downsampling the model dense point cloud based on the diameter of the target model to obtain a model point cloud;

[0010] Calculate the normal vector of each point in the model point cloud by applying a Kd-tree algorithm based on the model dense point cloud and the model point cloud;

[0011] Calculating hash values ​​of point pair features in the model point cloud according to the normal vectors of each point in the model point cloud;

[0012] determining supervoxels of a target scene point cloud generated by the RGB-D image of the target, and clustering the supervoxels according to a preset threshold to obtain a clustering result;

[0013] According to the hash value of the point pair feature, a voting algorithm is applied to determine the corresponding pose of each supervoxel in the clustering result to obtain a pose set;

[0014] Applying a clustering algorithm to the pose set, and taking the pose with the highest number of votes in the clustered pose set as the initial pose estimation result;

[0015] The initial pose estimation result is used as the initial value of the ICP algorithm to calculate the precise pose of the target, and the precise pose is used as the initial registration pose of the target.

[0016] Optionally, a Poisson sampling method is applied to sample the surface of the target model to obtain a dense point cloud of the model.

[0017] Optionally, calculating the diameter of the target model and downsampling the model dense point cloud based on the diameter of the target model to obtain the model point cloud specifically includes:

[0018] Calculating a diameter of the target model;

[0019] Determining a downsampling step size according to the diameter of the target model;

[0020] The downsampling step size is used as a sampling radius, and a Poisson disk sampling method is applied to downsample the model dense point cloud to obtain a model point cloud.

[0021] Optionally, applying a Kd-tree algorithm based on the model dense point cloud and the model point cloud to calculate the normal vector of each point in the model point cloud specifically includes:

[0022] Constructing a Kd-tree according to the model dense point cloud and the model point cloud;

[0023] Querying the neighborhood point set corresponding to each point of the model point cloud in the Kd-tree;

[0024] Applying the PCA algorithm to calculate the eigenvectors of each point in the model point cloud and each point in the corresponding neighborhood point set;

[0025] The eigenvalues ​​of the eigenvectors are calculated and the unit vector in the direction of the eigenvector corresponding to the minimum eigenvalue is used as the normal vector of the point of the model point cloud corresponding to the minimum eigenvalue.

[0026] Optionally, determining supervoxels of a target scene point cloud generated by the RGB-D image of the target, and clustering the supervoxels according to a preset threshold to obtain a clustering result, specifically includes:

[0027] Registering the RGB-D image of the target with a preset coordinate system to generate a target scene point cloud;

[0028] Applying the VCCS algorithm to determine supervoxels of the target scene point cloud;

[0029] Merging the corresponding supervoxel with its adjacent supervoxels according to a similarity between the supervoxel and the adjacent supervoxel and a preset similarity threshold to obtain a merged supervoxel;

[0030] Supervoxels that meet a preset outlier condition are removed from the merged supervoxels to obtain remaining supervoxels, and the remaining supervoxels are used as clustering results.

[0031] Optionally, a voting algorithm is applied according to the hash value of the point pair feature to determine the corresponding pose of each supervoxel in the clustering result to obtain a pose set, specifically including:

[0032] Selecting a first preset number of points as a candidate point set and a second preset number of points as a construction point set from the supervoxels of the clustering result; the value of the first preset number is greater than the value of the second preset number;

[0033] According to each point in the candidate point set and each point in the construction point set, construct a scene point pair set corresponding to each point in the candidate point set;

[0034] A voting algorithm is applied according to the hash value of the point pair feature corresponding to each point in the scene point pair set to determine a candidate pose corresponding to each point in the candidate point set, thereby obtaining a pose set.

[0035] Optionally, a clustering algorithm is applied to the pose set, and the pose with the highest number of votes in the clustered pose set is used as the initial pose estimation result, specifically including:

[0036] Eliminate candidate poses from the pose set whose votes are lower than a preset number to obtain a eliminated pose set;

[0037] Selecting the candidate pose with the highest number of votes from the eliminated pose set as the reference pose;

[0038] Select candidate poses whose rotation angle and translation distance from the reference pose meet preset screening conditions from the eliminated pose set as a screening pose set;

[0039] Clustering the screened pose set according to the mean of the rotation angles and the mean of the translation distances corresponding to all candidate poses in the screened pose set to obtain a clustered pose, and taking the number of votes for all candidate poses in the clustered pose as the number of votes for the clustered pose;

[0040] Deleting the filtered pose set from the eliminated pose set to obtain an updated pose set;

[0041] Determine whether the updated pose set is an empty set;

[0042] When the updated pose set is not an empty set, the updated pose set is used as the eliminated pose set, and the process returns to the step of "selecting the candidate pose with the highest number of votes from the eliminated pose set as the reference pose";

[0043] When the updated pose set is an empty set, the cluster pose with the highest number of votes is selected as the initial pose estimation result.

[0044] An augmented reality target initial registration system is applied to the above-mentioned augmented reality target initial registration method, and the registration system includes:

[0045] The sampling module is used to sample the surface of the target model to obtain a dense point cloud of the model;

[0046] a downsampling module, configured to calculate a diameter of the target model and downsample the model dense point cloud based on the diameter of the target model to obtain a model point cloud;

[0047] A first calculation module is used to calculate the normal vector of each point in the model point cloud by applying a Kd-tree algorithm based on the model dense point cloud and the model point cloud;

[0048] A second calculation module is used to calculate the hash value of the point pair feature in the model point cloud according to the normal vector of each point in the model point cloud;

[0049] a clustering module, configured to determine supervoxels of a target scene point cloud generated by the RGB-D image of the target, and cluster the supervoxels according to a preset threshold to obtain a clustering result;

[0050] a pose set determination module, configured to determine the corresponding pose of each supervoxel in the clustering result by applying a voting algorithm according to the hash value of the point pair feature, and obtain a pose set;

[0051] An estimation result determination module is used to apply a clustering algorithm to the pose set, and use the pose with the highest number of votes in the clustered pose set as the initial pose estimation result;

[0052] The third calculation module is used to use the initial pose estimation result as the initial value of the ICP algorithm to calculate the precise pose of the target, and use the precise pose as the initial registration pose of the target.

[0053] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0054] The present invention provides a method and system for initial registration of augmented reality objects. This method addresses the issue of increased density leading to excessive matching times. By using a feature-fusion-based scene point cloud clustering algorithm, points with similar features in the scene point cloud are clustered. Within each cluster, feature correspondence voting is performed to estimate the object's pose. The introduction of the point cloud clustering algorithm limits the construction of scene point pairs, reducing the number of scene point pairs constructed on different object surfaces, thereby improving matching speed and matching estimation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 This is a flow chart of the method for initial registration of an augmented reality target of the present invention;

[0057] Figure 2 This is a flowchart of the application of the augmented reality target initial registration method of the present invention;

[0058] Figure 3 is a flow chart of the supervoxel merging algorithm of the present invention;

[0059] Figure 4 This is an algorithm flow chart of the pose voting strategy based on point cloud clustering of the present invention. DETAILED DESCRIPTION

[0060] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0061] The purpose of the present invention is to provide an augmented reality target initial registration method and system, which can shorten the scene matching time and improve the estimation accuracy.

[0062] The present invention constructs a scene point cloud based on the RGB-D image transmitted by the depth camera. Aiming at estimating the initial camera pose of the target in augmented reality applications, an initial registration method for augmented reality targets is designed to provide an initialization environment for subsequent virtual-real alignment. The entire process is divided into two stages: model training and scene matching. Specifically, the method includes: in the model training stage, uniformly sampling the visible surface of a given CAD model to obtain a dense point cloud of the model, estimating the normal vector of the model point cloud on the dense point cloud, and downsampling the model point cloud based on the model diameter. The geometric features of the model point cloud are quantized using point pair features and stored in a hash table for scene matching. In the scene matching stage, the RGB-D image transmitted by the alignment camera is used to construct a scene point cloud. The scene point cloud is clustered based on fused feature constraints, and candidate points are selected within the clustering results to construct scene point pairs. Candidate poses are estimated by voting. After screening and clustering the candidate pose set, the rough pose of the target model in the camera coordinate system is output. The output rough pose is used as input and the accurate camera pose of the model is iteratively obtained using the ICP algorithm. The present invention avoids the problem of a large number of invalid point pair constructions in the point pair voting process, improves the scene matching time, reduces the initial registration time of the target object in the 3D background, and improves the accuracy of the initial registration.

[0063] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0064] Example 1

[0065] like Figure 1 and Figure 2 As shown, the present invention provides an augmented reality target initial registration method, the registration method comprising:

[0066] Step S1: Sampling the surface of the target model to obtain a dense point cloud of the model. Specifically, a Poisson sampling method is applied to sample the surface of the target model to obtain a dense point cloud of the model.

[0067] In practical applications, the number of grid sampling points is determined based on the ratio of the area of ​​each grid on the model surface to the model surface area and the number of sampling targets. Dense sampling is performed on the model of a given target to obtain a dense point cloud of the model.

[0068] Furthermore, the model grid sampling intensity is set according to the relationship between the area of ​​each grid in the given CAD model and the visible surface area of ​​the model and the target sampling quantity, specifically including:

[0069] Step 1: Calculate the target grid sampling intensity based on the target grid area and the model visible area:

[0070]

[0071] Among them A Mesh is the target grid area; A Model is the surface area of ​​the model; N sum is the target number of samples.

[0072] Step 2: Use the Poisson random number with intensity λ calculated by formula (1) as the number of sampling points of the target grid, and randomly sample on the target grid.

[0073] Step 3: Execute Step 1 and Step 2 on all the faces of the target model to obtain the model dense point cloud;

[0074] Step S2: Calculate the diameter of the target model and downsample the model dense point cloud based on the diameter of the target model to obtain a model point cloud. S2 specifically includes:

[0075] Step S21: Calculate the diameter of the target model.

[0076] Step S22: Determine the downsampling step size according to the diameter of the target model.

[0077] Step S23: using the downsampling step size as a sampling radius, applying a Poisson disk sampling method to downsample the model dense point cloud to obtain a model point cloud.

[0078] In practical applications, the model diameter d is calculated M , and calculate the downsampling step size kd based on the model diameter M ; Use kd MThe dense point cloud of the model obtained by sampling is downsampled by Poisson disk sampling with a sampling radius to obtain the model point cloud M. Specifically, k is 0.05.

[0079] Step S3: Based on the model dense point cloud and the model point cloud, apply the Kd-tree algorithm to calculate the normal vector of each point in the model point cloud. S3 specifically includes:

[0080] Step S31: constructing a Kd-tree based on the model dense point cloud and the model point cloud. Specifically, the dense point cloud and the model point cloud are merged and a Kd-tree is constructed to structure the point cloud.

[0081] Step S32: Query the Kd-tree for the neighborhood point set corresponding to each point in the model point cloud. Specifically, for a point p in the model point cloud M, query the Kd-tree to obtain its neighborhood point set.

[0082] Step S33: Applying the PCA algorithm to calculate the feature vectors of each point in the model point cloud and each point in the corresponding neighborhood point set. Specifically, the feature vectors of the neighborhood point set are calculated using the PCA algorithm.

[0083] Step S34: Calculate the eigenvalues ​​of the eigenvectors and use the unit vector in the direction of the eigenvector corresponding to the minimum eigenvalue as the normal vector for the point in the model point cloud corresponding to the minimum eigenvalue. Specifically, use the unit vector in the direction of the eigenvector corresponding to the minimum eigenvalue as the normal vector for the current point. Calculate the corresponding normal vector for each point in the model point cloud M.

[0084] Step S4: Calculate the hash value of the point pair feature in the model point cloud based on the normal vector of each point in the model point cloud. Specifically including:

[0085] Step S41: For any point in the model point cloud M, calculate the point pair features between it and other points in M, where for any two points m i 、m j ∈M, point pair feature F(m i ,m j ) is defined as:

[0086] F(m i ,m j )=[||d||2,∠(n i ,d),∠(n j ,d),∠(n i ,n j )] T (2)

[0087] Where: d = m j -m i ;||d||2 is m i 、mj The Euclidean distance between them; ∠(a,b) represents the angle; n i 、n j is m i 、m j Corresponding normal vector.

[0088] Step S42: Calculate the point pair features of all points in the model point cloud M, and construct all corresponding model point pairs in the model point cloud M.

[0089] Step S43: quantize all corresponding model point pairs in the model point cloud M calculated in step S42 and calculate the hash value, where any point pair F(m i ,m j )The corresponding hash value is calculated as:

[0090]

[0091] Where: d dist is the spacing quantization step size; d ang Quantify the amount for the angle; Indicates rounding down; specifically, d dist =0.05d M ;d ang =30.

[0092] Step S44: Construct a hash table, group point pairs with the same hash value into a point pair set, and store them under the corresponding hash value.

[0093] Step S5: determining the supervoxels of the target scene point cloud generated by the RGB-D image of the target, and clustering the supervoxels according to a preset threshold to obtain a clustering result. S5 specifically includes:

[0094] Step S51: aligning the RGB-D image of the target with a preset coordinate system to generate a target scene point cloud. S51 specifically includes:

[0095] Step S511: Calibrate the RGB and depth cameras to obtain the RGB and depth camera intrinsic parameter matrices CR and DR respectively.

[0096] Step S512: aligning the RGB and depth cameras to obtain an extrinsic parameter matrix W between the two.

[0097] Step S513: Align the RGB and depth images according to equation (4):

[0098]

[0099] Where: u D and v D are all depth image pixel coordinates; u C and v CAll are RGB image pixel coordinates; z D and z C Both are pixel depths in depth and RGB images.

[0100] Step S514: According to formula (5), the scene point cloud is constructed by combining RGB and depth information:

[0101]

[0102] Where: x, y, z are the spatial coordinates of the point cloud; d is the pixel depth; r (u,v) 、g (u,v) 、b (u,v) All are aligned pixel RGB information.

[0103] Step S52: applying the VCCS algorithm to determine the supervoxels of the target scene point cloud.

[0104] In practical applications, the VCCS algorithm is used to supervoxelize the scene point cloud, and formula (6) is used as the distance metric between points:

[0105]

[0106] Where: w c 、w n 、w s are color distance, geometric distance, and spatial distance weight respectively; D c is the CIEDE2000 standard color difference normalized distance; D n is the normal vector similarity, D n (n i ,n j )=1-|cos(n i ,n j )|;D s is the spatial Euclidean distance. Specifically, w c =0.2, w n =0.4, w n =0.4.

[0107] Step S53 : merging the corresponding supervoxel with its adjacent supervoxels according to the similarity between the supervoxel and the adjacent supervoxels of the supervoxel and a preset similarity threshold to obtain a merged supervoxel.

[0108] In practical applications, similar adjacent supervoxels are iteratively merged until the maximum similarity is less than a threshold ξ, where any adjacent supervoxel r i 、r j The similarity between them is defined as follows:

[0109]

[0110] Where:

[0111]

[0112] Among them, p i and p j is the adjacent voxel; c ij =p i -p j ;λ is the custom weight;D c is the CIEDE2000 standard color difference normalized distance; D R is the geometric distance; specifically, λ=0.5.

[0113] Step S531: Set the VCCS supervoxel clustering result R m , according to R m The undirected adjacency graph G is constructed by the adjacency relationship between supervoxels in m ={R m ,W m}, where W m is the similarity set between adjacent supervoxels.

[0114] Step S532: W m The similarities within are sorted from most significant to least significant.

[0115] Step S533: If W m If the maximum similarity within a voxel is greater than a threshold ξ, the corresponding two supervoxels are merged and the supervoxel center is recalculated according to the mean value, otherwise the step ends. Specifically, ξ = 0.5.

[0116] Step S534: Update the supervoxel set and similarity set in the undirected weighted graph:

[0117]

[0118]

[0119] Among them, R n is the supervoxel set in the nth iteration; R m-(k+1) is the supervoxel set in the m-(k+1)th iteration process; R m-k is the supervoxel set in the mk-th iteration process; represents any supervoxel in the nth iteration; represents any supervoxel in the m-(k+1)th iteration process; Represents any supervoxel in the mkth iteration process; adds the merged supervoxel to the supervoxel set, updates the similarity set, and updates the clustering state; p and q are arbitrary indices is the adjacent supervoxel, Step S54: remove the supervoxels that meet the preset outlier condition from the merged supervoxels to obtain the remaining supervoxels, and use the remaining supervoxels as the clustering result.

[0120] Specifically, outliers whose data volume is less than 1 times the scene point cloud data volume are eliminated from the clustering results, and the remaining supervoxels are output as the clustering results. Furthermore, 1=0.005.

[0121] In practical applications, such as Figure 3 As shown, the supervoxel merging algorithm is as follows:

[0122] Input: undirected weighted graph G m-k ={R m-k ,W m-k}; Output: point cloud clustering result S′={s1,s2,...,s N}.

[0123] Step 1: Merge W m-k The maximum similarity w max The corresponding adjacent supervoxels are constructed to obtain supervoxels;

[0124] Step 2: Add the constructed supervoxel to the supervoxel set and update the undirected weighted graph;

[0125] Step 3: Calculate the similarity between the constructed supervoxel and the adjacent supervoxel as the adjacency weight;

[0126] Step 4: Repeat the above distance until w max Greater than the threshold ξ;

[0127] Step 5: Propose outliers and output R t .

[0128] Step S6: Based on the hash value of the point pair feature, a voting algorithm is applied to determine the corresponding pose of each supervoxel in the clustering result to obtain a pose set. S6 specifically includes:

[0129] Step S61: selecting a first preset number of points as a candidate point set and a second preset number of points as a construction point set from the supervoxels of the clustering result; the value of the first preset number is greater than the value of the second preset number.

[0130] Step S62: Based on each point in the candidate point set and each point in the construction point set, construct a set of scene point pairs corresponding to each point in the candidate point set.

[0131] In practical applications, let the clustering result S′, for any pair s∈S′, uniformly select N in s min candidate points form a candidate point set P s , where Nmin is the minimum amount of data in S′; for candidate point p s ∈P s , select N in s min -1 point constitutes the construction point set Q s , combined with the candidate point p s And construct point set Q s Construct p s Corresponding scene point pair set

[0132] Step S63: Based on the hash value of the point pair feature corresponding to each point in the scene point pair set, a voting algorithm is applied to determine the candidate pose corresponding to each point in the candidate point set to obtain a pose set. S63 specifically includes:

[0133] Step S631: s Maintain an N M ×d ang The accumulator ACC, where N M is the number of point clouds in M; d ang Quantize the amount for the angle.

[0134] Step S632: s Any point pair (p s ,p t ), let K[F(p s ,p t )] corresponds to the point pair set M[F(p s ,p t )], for any (m i ,m j )∈M[F(p s ,p t )], calculate its difference with (p s ,p t ) t , and the ACC (m i ,α t ) plus one.

[0135] Step S633: traverse M[F(p s ,p t )] Perform Step 3-2 on the midpoint pair to complete the point pair (p s ,p t ) corresponding vote.

[0136] Step S634: In C s After all the internal points vote, the corresponding point with the highest number of votes in ACC is selected (m max ,α max ) as a candidate point p sThe corresponding voting results, the corresponding posture [R max |t max ] as p s Corresponding posture [R s |t s ].

[0137] Step S635: Execute Steps 2-4 on all cluster point sets in S′, and form a candidate pose set T with candidate poses corresponding to each candidate point.

[0138] In practical applications, such as Figure 4 As shown in Figure 2, the algorithm of the pose voting strategy based on point cloud clustering is as follows:

[0139] Input: Point cloud clustering result S′={s1,s2,...,s N}; Output: estimated pose [R|t].

[0140] FOR s∈S′ THEN

[0141] Uniformly select N within s min candidate points as the candidate point set P s

[0142] FOR p s ∈P s THEN

[0143] Construct accumulator ACC

[0144] Take another (N min -1) points as the construction point set Q s

[0145] FOR p t ∈Q s THEN

[0146] Construct point pair feature F(p s ,p t ), and calculate the hash value K[F(p s ,p t )]

[0147] Query the feature set to obtain K[F(p s ,p t )] corresponds to the set M[F(p s ,p t )]

[0148] FOR(m i ,m j )∈M[F(p s ,p t )]THEN

[0149] Calculate its value with (p s ,p t ) t

[0150] ACC (m i ,α t ) plus one vote

[0151] END FOR

[0152] END FOR

[0153] The Hough transform corresponding to the highest number of votes [R s |t s ]Add pose set T

[0154] END FOR

[0155] END FOR

[0156] Pose screening and clustering

[0157] ICP precise alignment

[0158] Output alignment result [R|t]

[0159] Figure 4 The specific steps are:

[0160] Step 1: Determine whether s∈S′ is true. When s∈S′ is true, uniformly select N in s. min candidate points as the candidate point set P s .

[0161] Step 2: Determine p s ∈P s Is it true when p s ∈P s When it is established, construct the accumulator ACC and take (N min -1) points as the construction point set Q s .

[0162] Step 3: Determine p t ∈Q s Is it true when p t ∈Q s When it is established, construct the point pair feature F(p s ,p t ), and calculate the hash value K[F(p s ,p t )], query the feature set to obtain K[F(p s ,p t )] corresponds to the set M[F(p s ,p t )], when (mi ,m j )∈M[F(p s ,p t )] is established, calculate its difference with (p s ,p t ) t ACC (m i ,α t ) plus one, and the Hough transform corresponding to the highest number of votes [R s |t s ]Add to the pose set T.

[0163] Step 4: When p t ∈Q s If not, the candidate point index is incremented and the process returns to step 2.

[0164] Step 5: When p s ∈P s If not, the clustering result index is incremented and the process returns to step 1.

[0165] Step 6: When s∈S′ does not hold, pose screening and clustering, ICP precise alignment, and output alignment result [R|t].

[0166] Step S7: Applying a clustering algorithm to the pose set, and taking the pose with the highest number of votes in the clustered pose set as the initial pose estimation result. S7 specifically includes:

[0167] Step S71: Eliminate candidate poses from the pose set that have votes lower than a preset number to obtain a eliminated pose set. Specifically, eliminate poses from the candidate pose set T that have votes lower than half of the highest number of votes.

[0168] Step S72: Select the candidate pose with the highest number of votes from the eliminated pose set as the reference pose.

[0169] Step S73: Filtering candidate poses whose rotation angle and translation distance with the reference pose meet preset filtering conditions from the eliminated pose set as a filtered pose set.

[0170] In practical applications, the pose with the highest number of votes in T is selected [R i |t i ], filter T and [R i |t i ] The pose set T whose rotation angle is less than ε and translation distance is less than σ c ={[R i |t i ],[R i2 |t i2 ],...,[R ix |tix ]}, for any pose [R i |t i ]、[R j |t j ], the rotation and translation clustering conditions between them are calculated as (R and t represent the rotation matrix and translation vector respectively):

[0171]

[0172] Step S74: Cluster the filtered pose set according to the mean of the rotation angles and the mean of the translation distances corresponding to all candidate poses in the filtered pose set to obtain a clustered pose, and use the number of votes for all candidate poses in the clustered pose as the number of votes for the clustered pose.

[0173] Step S75: deleting the filtered pose set from the eliminated pose set to obtain an updated pose set.

[0174] In practical applications, take T c The rotation matrix and translation vector mean of all poses in T are taken as the final cluster pose, and T c The votes of all the poses in the cluster are taken as the votes of the cluster pose; excluding T c , T=TT c .

[0175] Step S76: Determine whether the updated pose set is an empty set.

[0176] Step S77: When the updated pose set is not an empty set, the updated pose set is used as the eliminated pose set, and the process returns to step S72.

[0177] Step S78: When the updated pose set is an empty set, the cluster pose with the highest number of votes is selected as the initial pose estimation result.

[0178] Step S8: Using the initial pose estimation result as the initial value of the ICP algorithm, calculating the precise pose of the target, and using the precise pose as the initial registration pose of the target.

[0179] The whole process provided by the present invention is divided into two stages: model training and scene matching. In the model training stage, the model dense point cloud is obtained by sampling the visual surface of a given CAD model, and the model point cloud is obtained by downsampling. The normal vector of each point in the model point cloud is estimated based on the model dense point cloud and then used for model quantization. The model point cloud is quantized using point pair features and stored in a feature table for query in the scene matching stage. In the scene matching stage, the RGB-D image returned by the depth camera is aligned to generate a scene point cloud, and the scene point cloud is supervoxelized using the VCCS algorithm. Similar supervoxels are iteratively merged to perform supervoxel clustering. Candidate points are selected from the supervoxel results to construct scene point pairs, and the candidate poses are estimated by voting. After the candidate pose set completes the screening and clustering, the clustered pose with the highest number of votes is selected as the initial pose of the ICP algorithm, the precise pose is iteratively obtained, and the estimated result is output as the initial registration pose of the target.

[0180] The above-mentioned model training phase specifically includes:

[0181] 1) Steps for generating dense point cloud of model surface: Based on the ratio between the area of ​​each grid on the model surface and the model surface area, the number of grid sampling points is determined in combination with the number of sampling targets, and dense sampling is performed on the model of a given target to obtain the model dense point cloud.

[0182] 2) Sampling dense point cloud downsampling step: with kd M Downsample the model point cloud by the sampling step.

[0183] 3) Downsampled sparse point cloud normal vector estimation step: Based on the dense point cloud, estimate the normal vector of each point in the downsampled result.

[0184] Among them, the scene matching stage includes the point cloud clustering part and the pose voting part.

[0185] Specifically, the point cloud clustering part includes the following steps:

[0186] Step 1: Obtain the scene point cloud and align the RGB-D image sent back by the camera to generate the scene point cloud S.

[0187] Step 2: Supervoxelize the scene point cloud. Combine the spatial and fusion features of the Euclidean distance to supervoxelize the scene point cloud and obtain the supervoxel set R m .

[0188] Step 3: Construction of the supervoxel result adjacency graph, based on the supervoxel set R m The adjacency relationship between super voxels constructs an undirected adjacency graph G m ={R m ,W m},w ij ∈W m is the fusion feature similarity between adjacent supervoxels.

[0189] Step 4: Merge supervoxels and merge supervoxel set R m The adjacent supervoxel with the highest internal feature similarity is W m The similarities of the adjacent supervoxels in W are sorted from large to small, and W m The two supervoxels corresponding to the maximum similarity in are merged, and the similarity between the supervoxel center and the adjacent supervoxel is recalculated, and the undirected adjacency graph G is updated. m-1 ={R m-1 ,W m-1}.

[0190] Step 5: Supervoxel clustering, repeat the above process until W m-k The maximum similarity w max <ξ, where ξ is the similarity threshold, and the output corresponds to R m-k As supervoxel clustering results.

[0191] Specifically, the posture voting part includes the following steps:

[0192] Step 1: Remove outliers and cluster the result R output in Step 5 of the above point cloud clustering part m-k The supervoxels whose data volume is less than k times of the data volume of S are eliminated, and S′={s1,s2,...,s N}.

[0193] Step 2: Select candidate points and construct scene point pairs. For any s∈S′, uniformly select N points within s. min candidate points form a candidate point set P s , where N min is the minimum amount of data in S′; for candidate point p s ∈P s , select N in s min -1 point constitutes the construction point set Q s , combined with the candidate point p s With Q s Scene point pair collection

[0194] Step 3: Vote for candidate poses and maintain an N M ×d ang The accumulator ACC, where N M is the number of point clouds in M; d ang is the angle quantization quantity; for any point pair in Cs (p s ,p t ), let K[F(p s ,p t )] corresponds to the point pair set M[F(p s ,pt )], for any (m i ,m j )∈M[F(p s ,p t )], calculate its difference with (p s ,p t ) t , and the ACC (m i ,α t ) plus one vote; traverse M[F(p s ,p t )] Perform the above operations on the midpoint pair to complete the point pair (p s ,p t ) corresponding votes; after all the votes are cast in Cs, the corresponding point with the highest number of votes in ACC is selected (m max ,α max ) as a candidate point p s The corresponding voting results, the corresponding posture [R max |t max ] as p s Corresponding posture [R s |t s ].

[0195] Step 4: Pose screening. Each set in S′ executes Step 2 and Step 3 to construct a candidate pose set. The poses with votes less than half of the highest votes in the set are eliminated to obtain the candidate pose set T.

[0196] Step 5: Pose clustering, let the pose with the highest number of votes in T be [R i |t i ], put T in [R i |t i ] are clustered with the positions whose rotation angle is less than ε and translation distance is less than σ, and the clustered position set T is obtained. c ={[R i |t i ],[R i2 |t i2 ],...,[R ix |t ix ]}, where ε and σ are the rotation angle threshold and translation distance threshold, respectively, which are set to 75° and 0.1d M , where d M is the model diameter; calculate T c The cluster center R c1 |t c1 ], as candidate estimated pose, T c The sum of the votes of all poses in [R c1 |t c1 ] weight; take T1 = TTc , repeat the above operation for T1 until all poses in T are clustered, and output the cluster center with the largest weight as the pose estimation result.

[0197] The advantages and beneficial effects of the present invention are as follows:

[0198] This invention addresses the problems in augmented reality applications where traditional 2D visual perception methods are unable to cope with weak texture and multi-scale targets, and where 3D visual perception methods have slow runtimes and low matching accuracy. It uses point-pair features to describe model geometry, matches scene points with model points based on the model's local geometric features, and estimates the target pose by solving the Hough transform between corresponding points. To mitigate the impact of the variable scale of targets in augmented reality applications, a scene perception module based on feature fusion constraint clustering is added. Point clouds with similar features in the scene point cloud are clustered, and scene point pairs are constructed within the clustering results to describe the characteristics of the scene point sets, accelerating the matching process and improving matching accuracy.

[0199] Example 2

[0200] In order to execute the method corresponding to the above embodiment 1 and achieve the corresponding functions and technical effects, the following provides an augmented reality target initial registration system, which includes:

[0201] The sampling module is used to sample the surface of the target model to obtain a dense point cloud of the model.

[0202] A downsampling module is used to calculate the diameter of the target model and downsample the model dense point cloud based on the diameter of the target model to obtain a model point cloud.

[0203] The first calculation module is used to apply the Kd-tree algorithm according to the model dense point cloud and the model point cloud to calculate the normal vector of each point in the model point cloud.

[0204] The second calculation module is used to calculate the hash value of the point pair feature in the model point cloud according to the normal vector of each point in the model point cloud.

[0205] The clustering module is used to determine the supervoxels of the target scene point cloud generated by the RGB-D image of the target, and cluster the supervoxels according to a preset threshold to obtain a clustering result.

[0206] The pose set determination module is used to apply a voting algorithm according to the hash value of the point pair feature to determine the corresponding pose of each supervoxel in the clustering result to obtain a pose set.

[0207] The estimation result determination module is used to apply a clustering algorithm to the pose set and take the pose with the highest number of votes in the clustered pose set as the initial pose estimation result.

[0208] The third calculation module is used to use the initial pose estimation result as the initial value of the ICP algorithm to calculate the precise pose of the target, and use the precise pose as the initial registration pose of the target.

[0209] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0210] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A method for initial registration of an augmented reality target, characterized in that: The registration method includes: Sample the surface of the target model to obtain a dense point cloud of the model; Calculating a diameter of the target model, and downsampling the model dense point cloud based on the diameter of the target model to obtain a model point cloud; Calculate the normal vector of each point in the model point cloud by applying a Kd-tree algorithm based on the model dense point cloud and the model point cloud; Calculating hash values ​​of point pair features in the model point cloud according to the normal vectors of each point in the model point cloud; determining supervoxels of a target scene point cloud generated by the RGB-D image of the target, and clustering the supervoxels according to a preset threshold to obtain a clustering result; According to the hash value of the point pair feature, a voting algorithm is applied to determine the corresponding pose of each supervoxel in the clustering result to obtain a pose set; Applying a clustering algorithm to the pose set, and taking the pose with the highest number of votes in the clustered pose set as the initial pose estimation result; The initial pose estimation result is used as the initial value of the ICP algorithm to calculate the precise pose of the target, and the precise pose is used as the initial registration pose of the target.

2. The method for initial registration of an augmented reality target according to claim 1, wherein: The Poisson sampling method is applied to sample the surface of the target model to obtain a dense point cloud of the model.

3. The method for initial registration of an augmented reality target according to claim 1, wherein: Calculating the diameter of the target model and downsampling the model dense point cloud based on the diameter of the target model to obtain a model point cloud, specifically including: Calculating a diameter of the target model; Determining a downsampling step size according to the diameter of the target model; The downsampling step size is used as a sampling radius, and a Poisson disk sampling method is applied to downsample the model dense point cloud to obtain a model point cloud.

4. The method for initial registration of an augmented reality target according to claim 1, wherein: Applying a Kd-tree algorithm based on the model dense point cloud and the model point cloud to calculate the normal vector of each point in the model point cloud specifically includes: Constructing a Kd-tree according to the model dense point cloud and the model point cloud; Querying the neighborhood point set corresponding to each point of the model point cloud in the Kd-tree; Applying the PCA algorithm to calculate the eigenvectors of each point in the model point cloud and each point in the corresponding neighborhood point set; The eigenvalues ​​of the eigenvectors are calculated and the unit vector in the direction of the eigenvector corresponding to the minimum eigenvalue is used as the normal vector of the point of the model point cloud corresponding to the minimum eigenvalue.

5. The method for initial registration of an augmented reality target according to claim 1, wherein: Determining supervoxels of a target scene point cloud generated by the RGB-D image of the target, and clustering the supervoxels according to a preset threshold to obtain a clustering result, specifically comprising: Registering the RGB-D image of the target with a preset coordinate system to generate a target scene point cloud; Applying the VCCS algorithm to determine supervoxels of the target scene point cloud; Merging the corresponding supervoxel with its adjacent supervoxels according to a similarity between the supervoxel and the adjacent supervoxel and a preset similarity threshold to obtain a merged supervoxel; Supervoxels that meet a preset outlier condition are removed from the merged supervoxels to obtain remaining supervoxels, and the remaining supervoxels are used as clustering results.

6. The method for initial registration of an augmented reality target according to claim 1, wherein: According to the hash value of the point pair feature, a voting algorithm is applied to determine the corresponding pose of each supervoxel in the clustering result to obtain a pose set, which specifically includes: Selecting a first preset number of points as a candidate point set and a second preset number of points as a construction point set from the supervoxels of the clustering result; the value of the first preset number is greater than the value of the second preset number; According to each point in the candidate point set and each point in the construction point set, construct a scene point pair set corresponding to each point in the candidate point set; A voting algorithm is applied according to the hash value of the point pair feature corresponding to each point in the scene point pair set to determine a candidate pose corresponding to each point in the candidate point set, thereby obtaining a pose set.

7. The method for initial registration of an augmented reality target according to claim 6, wherein: Applying a clustering algorithm to the pose set, and taking the pose with the highest number of votes in the clustered pose set as the initial pose estimation result, specifically including: Eliminate candidate poses from the pose set whose votes are lower than a preset number to obtain a eliminated pose set; Selecting the candidate pose with the highest number of votes from the eliminated pose set as the reference pose; Select candidate poses whose rotation angle and translation distance from the reference pose meet preset screening conditions from the eliminated pose set as a screening pose set; Clustering the screened pose set according to the mean of the rotation angles and the mean of the translation distances corresponding to all candidate poses in the screened pose set to obtain a clustered pose, and taking the number of votes for all candidate poses in the clustered pose as the number of votes for the clustered pose; Deleting the filtered pose set from the eliminated pose set to obtain an updated pose set; Determine whether the updated pose set is an empty set; When the updated pose set is not an empty set, the updated pose set is used as the eliminated pose set, and the process returns to step "selecting the candidate pose with the highest number of votes from the eliminated pose set as the reference pose"; When the updated pose set is an empty set, the cluster pose with the highest number of votes is selected as the initial pose estimation result.

8. An augmented reality target initial registration system, characterized in that: The registration system includes: The sampling module is used to sample the surface of the target model to obtain a dense point cloud of the model; a downsampling module, configured to calculate a diameter of the target model and downsample the model dense point cloud based on the diameter of the target model to obtain a model point cloud; A first calculation module is used to calculate the normal vector of each point in the model point cloud by applying a Kd-tree algorithm based on the model dense point cloud and the model point cloud; A second calculation module is used to calculate the hash value of the point pair feature in the model point cloud according to the normal vector of each point in the model point cloud; a clustering module, configured to determine supervoxels of a target scene point cloud generated by the RGB-D image of the target, and cluster the supervoxels according to a preset threshold to obtain a clustering result; a pose set determination module, configured to determine the corresponding pose of each supervoxel in the clustering result by applying a voting algorithm according to the hash value of the point pair feature, and obtain a pose set; An estimation result determination module is used to apply a clustering algorithm to the pose set, and use the pose with the highest number of votes in the clustered pose set as the initial pose estimation result; The third calculation module is used to use the initial pose estimation result as the initial value of the ICP algorithm to calculate the precise pose of the target, and use the precise pose as the initial registration pose of the target.

Images