Virtual model and real object registration method and device, equipment and storage medium

Abstract

The invention relates to the technical field of point cloud registration, and discloses a virtual model and real object registration method and device, equipment and a storage medium. The method comprises the following steps: firstly, performing single-view segmentation on virtual model point clouds, and performing feature extraction on each single-view point cloud obtained by segmentation to obtain a view global feature corresponding to each single-view point cloud; and performing feature extraction on the point cloud of the real registration object to obtain real global features. And then, based on the matching degree between the visual angle global feature corresponding to each single visual angle point cloud and the real global feature, determining a target visual angle point cloud from the plurality of single visual angle point clouds. And finally, performing registration based on the target visual angle point cloud and the real registration object point cloud, and registering the virtual model to a real space coordinate to obtain a target virtual model. According to the method, the interference of exterior points is effectively reduced, the number of required iterations is reduced, the accuracy of a registration result is remarkably improved, and the problem that registration needs multiple iterations when the pose of a real object is unknown is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud registration, and in particular to a registration method, device, equipment and storage medium for a virtual model and a real object. Background Art

[0002] Due to the field of view and occlusion effects, scanning devices can only obtain a single layer of point cloud images within a limited field of view when capturing real 3D scene data. Therefore, it is necessary to restore the 3D scene or determine the position information.

[0003] In the related art, the rotation transformation matrix is ​​calculated by matching the most likely correspondence between the scanned point cloud and the virtual model. However, when the point cloud contains many external points, additional iterations or additional markers are required to improve the accuracy of the registration, which increases the registration time and cost. Summary of the invention

[0004] The embodiments of this specification are intended to solve at least one of the technical problems in the related art to a certain extent. To this end, the embodiments of this specification propose a method, device, equipment and storage medium for registering a virtual model with a real object.

[0005] The embodiment of this specification provides a method for registering a virtual model with a real object, the method comprising:

[0006] Extracting features from the point cloud data to obtain global features; wherein the global features include perspective global features and real global features corresponding to each single-view point cloud, the perspective global features corresponding to each single-view point cloud are obtained by performing single-view segmentation on the virtual model point cloud and extracting features from each single-view point cloud obtained by segmentation, and the real global features are obtained by performing feature extraction on the real registration object point cloud;

[0007] Determining a target perspective point cloud from a plurality of single-perspective point clouds based on a matching degree between the perspective global feature corresponding to each of the single-perspective point clouds and the reality global feature;

[0008] Based on the registration of the target viewpoint point cloud and the real registration object point cloud, the virtual model is registered to the real space coordinates to obtain the target virtual model.

[0009] In one embodiment, the virtual model point cloud is obtained by:

[0010] Resample the virtual model to obtain a model sampling point cloud;

[0011] The model sampling point cloud is spatially segmented and processed to obtain the virtual model point cloud.

[0012] In one embodiment, the real-world registration object point cloud is obtained by:

[0013] Preprocess the scanned reality point cloud, and perform foreground extraction based on the preprocessing result to obtain the registered object point cloud;

[0014] The registered object point cloud is spatially segmented and processed to obtain the real registered object point cloud.

[0015] In one embodiment, the preprocessing of the scanned reality point cloud and the foreground extraction according to the preprocessing result to obtain the registered object point cloud include:

[0016] Downsampling the scanned actual point cloud to obtain a downsampled point cloud;

[0017] Removing outliers from the downsampled point cloud to obtain the preprocessing result;

[0018] Perform coordinate comparison based on the preprocessing results to determine the basic foreground point cloud;

[0019] The basic foreground point cloud is screened according to an object quantity threshold to obtain the registered object point cloud.

[0020] In one embodiment, removing outliers from the downsampled point cloud to obtain the preprocessing result includes:

[0021] Performing statistical analysis on the downsampled point cloud, determining and removing outliers in the downsampled point cloud, and obtaining a first processing result;

[0022] Removing abnormal points from the first processing result in units of points to obtain a second processing result;

[0023] The second processing result is subjected to removal of outliers in units of classes to obtain the preprocessing result.

[0024] In one embodiment, removing abnormal points in units of points from the first processing result to obtain a second processing result includes:

[0025] Determine, according to the first distance, a first statistical space corresponding to each point in the first processing result;

[0026] Calculate the number of points in the first statistical space corresponding to each point in the first processing result;

[0027] When the number of points does not satisfy the point screening condition, the corresponding points are removed from the first processing result to obtain the second processing result.

[0028] In one embodiment, removing outliers from the second processing result in units of classes to obtain the preprocessing result includes:

[0029] Calculating the number of points in the second statistical space corresponding to each point in the second processing result; wherein the second statistical space corresponding to each point in the second processing result is determined according to the second distance;

[0030] In the case where the number of points meets the adjacent point screening condition, the corresponding points are used as core points in the second processing result;

[0031] In the case that the number of points does not satisfy the adjacent point screening condition, the corresponding points are used as boundary points in the second processing result;

[0032] Based on the core points and the boundary points in the second statistical space corresponding to the core points, classification is performed to obtain noise points;

[0033] The noise points are removed from the second processing result to obtain the preprocessing result.

[0034] In one embodiment, the registering the target viewpoint point cloud with the real registration object point cloud, registering the virtual model to the real space coordinates, and obtaining the target virtual model includes:

[0035] Downsampling the target viewpoint point cloud and the actual registered object point cloud to obtain a target downsampled point cloud and a registered downsampled point cloud;

[0036] Performing rough point cloud registration on the target downsampled point cloud and the registered downsampled point cloud to determine an initial registration matrix;

[0037] Based on the initial registration matrix, performing point cloud fine registration on the target viewpoint point cloud and the actual registration object point cloud to determine a target registration matrix;

[0038] The virtual model is registered to the real space coordinates based on the target registration matrix to obtain a target virtual model.

[0039] In one embodiment, the step of performing rough registration of the target reduced-sample point cloud and the registered reduced-sample point cloud to determine an initial registration matrix includes:

[0040] Perform feature extraction based on the normal vector corresponding to each point in the target downsampling point cloud to obtain a local feature of the viewing angle;

[0041] Perform feature extraction based on the normal vector corresponding to each point in the registered downsampled point cloud to obtain a realistic local feature;

[0042] An initial registration matrix is ​​determined based on the perspective local features and the reality local features.

[0043] In one embodiment, determining an initial registration matrix based on the perspective local feature and the reality local feature includes:

[0044] Selecting a target point in the registered and reduced sampling point cloud;

[0045] Based on the local features of the viewing angle and the local features of the reality, a pairing point of the target point is determined in the target point cloud to obtain a feature point pair;

[0046] Based on the feature point pairs, the initial registration matrix is ​​determined.

[0047] In one embodiment, determining the initial registration matrix based on the feature point pairs includes:

[0048] Determine a basic registration matrix using the feature point pairs;

[0049] Based on the basic registration matrix, determining the matching degree between the target reduced mining point cloud and the registered reduced mining point cloud;

[0050] In the case where the matching degree satisfies a matching condition, determining the basic registration matrix as the initial registration matrix;

[0051] When the matching degree does not satisfy the matching condition, the target view point cloud is re-determined from the single view point cloud.

[0052] In one embodiment, the target registration matrix includes a rotation matrix and a translation matrix; and the step of registering the virtual model to the real space coordinates based on the target registration matrix to obtain the target virtual model includes:

[0053] Performing an inverse calculation on the rotation matrix to obtain a target rotation matrix;

[0054] Performing an inverse calculation on the translation matrix to obtain a target translation matrix;

[0055] The virtual model is aligned to the real space coordinates based on the target rotation matrix and the target translation matrix to obtain a target virtual model.

[0056] In one embodiment, the step of performing point cloud fine registration on the target viewpoint point cloud and the real registration object point cloud based on the initial registration matrix to determine the target registration matrix includes:

[0057] Using the initial registration matrix, coarsely registering the target viewpoint point cloud to obtain a point cloud to be corrected;

[0058] The initial registration matrix is ​​used as an initial value, and the point cloud to be corrected and the point cloud of the actual registration object are iteratively registered to obtain the target registration matrix.

[0059] The embodiment of this specification provides a virtual model and a real object registration device, the device comprising:

[0060] A feature extraction module is used to extract features from point cloud data to obtain global features; wherein the global features include perspective global features and real global features corresponding to each single-view point cloud, wherein the perspective global features corresponding to each single-view point cloud are obtained by performing single-view segmentation on the virtual model point cloud and extracting features from each single-view point cloud obtained by segmentation, and the real global features are obtained by performing feature extraction on the real registration object point cloud;

[0061] A feature matching module, configured to determine a target view point cloud from a plurality of single view point clouds based on a matching degree between a view global feature corresponding to each of the single view point clouds and the reality global feature;

[0062] The model registration module is used to perform registration based on the target viewpoint point cloud and the real registration object point cloud, register the virtual model to the real space coordinates, and obtain the target virtual model.

[0063] An embodiment of the present specification provides a computer device, which includes: a memory, and one or more processors communicatively connected to the memory; the memory stores instructions executable by the one or more processors, and the instructions are executed by the one or more processors to enable the one or more processors to implement the steps of the method described in any of the above embodiments.

[0064] The embodiments of this specification provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0065] An embodiment of the present specification provides a computer program product, wherein the computer program product includes instructions, and when the instructions are executed by a processor of a computer device, the computer device is enabled to perform the steps of the method described in any one of the above embodiments.

[0066] In the implementation manner of the above specification, first, the virtual model point cloud is segmented into single perspectives, and features are extracted for each single perspective point cloud obtained by the segmentation to obtain the perspective global features corresponding to each single perspective point cloud. Feature extraction is performed on the point cloud of the real registration object to obtain the real global features. Then, based on the matching degree between the perspective global features corresponding to each single perspective point cloud and the real global features, the target perspective point cloud is determined from multiple single perspective point clouds. Finally, based on the target perspective point cloud and the real registration object point cloud, the virtual model is aligned to the real space coordinates to obtain the target virtual model. The interference of external points is effectively reduced, the number of iterations required is reduced, the accuracy of the registration results is significantly improved, and the problem of multiple iterations required for registration when the pose of the real object is unknown is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A flowchart of a method for aligning a virtual model with a real object provided in an embodiment of this specification;

[0068] Figure 2a A schematic diagram of a flow chart of a method for registering a virtual model with a real object provided in an embodiment of this specification;

[0069] Figure 2b A schematic diagram of multiple single-view point clouds provided for implementation of this specification;

[0070] Figure 2c A schematic diagram of the Euclidean distance corresponding to each single-view point cloud provided in the implementation manner of this specification;

[0071] Figure 3a A schematic diagram of a process for obtaining a virtual model point cloud provided in an embodiment of this specification;

[0072] Figure 3b A schematic diagram of a virtual model point cloud provided in an embodiment of this specification;

[0073] Figure 4 A schematic diagram of a process for obtaining a point cloud of a real-world registered object provided in an embodiment of this specification;

[0074] Figure 5a A schematic diagram of a process for obtaining a point cloud of a registered object provided in an embodiment of this specification;

[0075] Figure 5b A schematic diagram of registering an object point cloud provided in an embodiment of this specification;

[0076] Figure 6 A schematic diagram of a process for obtaining a preprocessing result provided in an embodiment of this specification;

[0077] Figure 7aA schematic diagram of a process for obtaining a second processing result provided in an embodiment of this specification;

[0078] Figure 7b A schematic diagram of a second processing result provided in an embodiment of this specification;

[0079] Figure 8a A schematic diagram of a process for obtaining a preprocessing result provided in an embodiment of this specification;

[0080] Figure 8b A schematic diagram of the preprocessing results provided in the embodiments of this specification;

[0081] Figure 9a A schematic diagram of a process for obtaining a target virtual model provided in an embodiment of this specification;

[0082] Figure 9b A schematic diagram of the registration and reduction point cloud provided in the implementation manner of this specification;

[0083] Fig.10a A schematic diagram of a process for determining an initial registration matrix provided in an embodiment of this specification;

[0084] Fig.10b A schematic diagram of the result of rough registration of the target reduced sampling point cloud and the registered reduced sampling point cloud provided in the implementation manner of this specification;

[0085] Fig.11 A schematic diagram of a process for determining an initial registration matrix provided in an embodiment of this specification;

[0086] Fig.12 A schematic diagram of a process for re-determining a target viewpoint point cloud provided in an embodiment of this specification;

[0087] Fig.13a A schematic diagram of a process for obtaining a target virtual model provided in an embodiment of this specification;

[0088] Fig.13b A schematic diagram of a target virtual model aligned to real space coordinates provided in an embodiment of this specification;

[0089] Fig.14a A schematic diagram of a process for obtaining a target registration matrix provided in an embodiment of this specification;

[0090] Fig.14b A schematic diagram of the result of fine registration of point clouds between a target viewpoint point cloud and a real registration object point cloud provided in an embodiment of this specification;

[0091] Fig.15 A schematic diagram of a flow chart of a method for registering a virtual model with a real object provided in an embodiment of this specification;

[0092] Fig.16 A schematic diagram of a virtual model and a real object registration device provided in an embodiment of this specification;

[0093] Fig.17 An internal structural diagram of a computer device provided for an embodiment of this specification. DETAILED DESCRIPTION

[0094] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0095] Due to the field of view and occlusion effects, scanning equipment can only obtain a single layer of point cloud images within a limited field of view when capturing real 3D scene data. Therefore, in order to restore the 3D scene or determine the location information, it is necessary to solve the transformation matrix (including the rotation matrix R and the translation matrix T) for the point cloud data of the overlapping area of ​​the two frames to map the point cloud of the overlapping area to a unified coordinate system. Point cloud registration plays an important role in the construction of digital twin cities, autonomous driving, augmented / virtual reality and other fields.

[0096] In related technologies, the initial pose of the point cloud has a great impact on the time required for registration and the quality of the results. Since the pose information of the real object is unknown, the rotation transformation matrix can only be obtained by finding the most likely correspondence between the scanned point cloud and the virtual model. However, when the point cloud contains many external points, in order to improve the accuracy of the registration, additional iterations or additional markers are required, which increases the registration time and cost.

[0097] Based on this, the embodiments of this specification provide a method for aligning a virtual model with a real object. First, the virtual model point cloud is segmented into single perspectives, and features are extracted for each single perspective point cloud obtained by the segmentation to obtain the perspective global features corresponding to each single perspective point cloud. Feature extraction is performed on the point cloud of the real object to be registered to obtain the real global features. Then, based on the degree of match between the perspective global features corresponding to each single perspective point cloud and the real global features, the target perspective point cloud is determined from multiple single perspective point clouds. Finally, based on the target perspective point cloud and the real object point cloud, the virtual model is aligned to the real space coordinates to obtain the target virtual model. The interference of external points is effectively reduced, the number of iterations required is reduced, the accuracy of the registration results is significantly improved, and the problem of multiple iterations required for registration when the pose of the real object is unknown is solved.

[0098] In the application scenarios of the methods provided in the embodiments of this specification, please refer to Figure 1First, obtain the scanned reality point cloud and perform preprocessing on the scanned reality point cloud, including downsampling, denoising and clustering, to extract the objects to be registered from the scanned reality point cloud in the real space and obtain the reality registration object point cloud. Then, perform feature extraction on the reality registration object point cloud, calculate the viewpoint feature histogram (VFH) feature of the reality registration object point cloud, and obtain the reality global feature.

[0099] Acquire the virtual model, resample it, and perform spatial segmentation to obtain the virtual model point cloud. Perform single-view segmentation on the virtual model point cloud to obtain each single-view point cloud. Perform feature extraction on each single-view point cloud obtained by segmentation, calculate the VFH features of each single-view point cloud, and obtain the perspective global features corresponding to each single-view point cloud. Use KD-Tree to store the perspective global features corresponding to each single-view point cloud. Determine the target perspective point cloud from multiple single-view point clouds by comparing the matching degree between the perspective global features corresponding to each single-view point cloud and the actual global features. Calculate the normal vector corresponding to each point in the target perspective point cloud and the actual registration object point cloud. Downsample the target perspective point cloud and the actual registration object point cloud to obtain the target downsampled point cloud and the registration downsampled point cloud.

[0100] The FPFH (Fast Point Feature Histograms) and random sampling consistency algorithm are used to perform rough registration of the target downsampled point cloud and the registration downsampled point cloud to determine the initial registration matrix. Then, based on the initial registration matrix, the point-to-surface ICP algorithm and normal vector are used to perform fine registration of the target viewpoint point cloud and the actual registration object point cloud to determine the target registration matrix.

[0101] The target registration matrix is ​​transformed, that is, the rotation matrix included in the target registration matrix is ​​inverted to obtain the target rotation matrix, and the translation matrix included in the target registration matrix is ​​inverted to obtain the target translation matrix. Finally, the virtual model is aligned to the real space coordinates based on the target rotation matrix and the target translation matrix to obtain the target virtual model.

[0102] This specification provides a method for registering a virtual model with a real object. Figure 2a The registration method of the virtual model and the real object may include the following steps:

[0103] S210, extracting features from the point cloud data to obtain global features.

[0104] Among them, the global features include the perspective global features corresponding to each single-perspective point cloud and the real global features. The perspective global features corresponding to each single-perspective point cloud are obtained by performing single-perspective segmentation on the virtual model point cloud and extracting features from each segmented single-perspective point cloud. The real global features are obtained by performing feature extraction on the real registered object point cloud.

[0105] Specifically, point cloud data is an effective way to represent three-dimensional shapes and spatial information by capturing discrete points on the surface of an object. Feature extraction of point cloud data aims to capture the geometric and spatial characteristics of point cloud data, thereby obtaining global features for subsequent analysis and processing. Global features can include perspective global features and reality global features corresponding to each single-view point cloud.

[0106] Feature extraction is performed on the point cloud of the real registered object to capture the geometric and spatial information of the point cloud of the real registered object and obtain the real global features.

[0107] The virtual model point cloud is segmented into single-view points according to different perspectives to obtain multiple single-view point clouds. Each segmented single-view point cloud can represent the part of the virtual model point cloud observed from a certain perspective. Next, feature extraction is performed on each segmented single-view point cloud to capture the geometric and spatial information of each single-view point cloud, and obtain the perspective global features corresponding to each single-view point cloud. For example, when performing single-view segmentation on the virtual model point cloud, different virtual camera positions can be set to simulate the point cloud at each perspective in the actual acquisition process. Select a virtual camera position, determine the visible area of ​​the virtual camera at this position, and delete the invisible points at this position to obtain a single-view point cloud.

[0108] For example, see Figure 2b , perform single-view segmentation on the virtual model point cloud to obtain multiple single-view point clouds. After segmenting the complete model, when extracting features from each single-view point cloud obtained by segmentation, in order to accurately describe the geometric shape and spatial distribution of each single-view point cloud, the viewpoint feature histogram (Viewpoint Feature Histogram, VFH) is used as a description method for global features. During the calculation of the VFH descriptor, the global features of each single-view point cloud are represented as a 308-dimensional feature vector to obtain the perspective global features. And when extracting features from the point cloud of the real registered object, in order to accurately describe the geometric shape and spatial distribution of the point cloud of the real registered object, the viewpoint feature histogram (Viewpoint Feature Histogram, VFH) is used as a description method for global features. During the calculation of the VFH descriptor, the global features of the point cloud of the real registered object are represented as a 308-dimensional feature vector to obtain the real global features.

[0109] It should be noted that the perspective global features and the actual global features can be stored so that they can be easily called and accessed during subsequent operations and analysis. For example, the perspective global features corresponding to each single-view point cloud can be used as nodes of the KD-Tree, and then the KD-Tree is constructed and the construction results are stored. Through the spatial partitioning of the KD-Tree, the nearest neighbor query and other spatial operations can be quickly performed.

[0110] S220 , determining a target perspective point cloud from multiple single-perspective point clouds based on the matching degree between the perspective global features corresponding to each single-perspective point cloud and the actual global features.

[0111] S230 , aligning the target viewpoint point cloud with the real registration object point cloud, aligning the virtual model to the real space coordinates, and obtaining the target virtual model.

[0112] Specifically, a similarity metric (such as cosine similarity, Euclidean distance, etc.) is used to calculate the difference between the perspective global features corresponding to each single-view point cloud and the real global features, and the matching degree between the perspective global features corresponding to each single-view point cloud and the real global features is obtained. Compare the obtained matching degrees, and select the single-view point cloud with the highest matching degree from all single-view point clouds as the target perspective point cloud. Then, select a suitable registration algorithm to register the target perspective point cloud with the real registration object point cloud to determine the registration result. Then, use the registration result to register the virtual model to the real space coordinates to obtain the target virtual model.

[0113] For example, the Euclidean distance between the perspective global feature corresponding to each single-view point cloud and the real global feature is calculated to evaluate the accuracy of the scan perspective. The smaller the Euclidean distance, the higher the matching degree between the perspective and the real perspective. Figure 2c ,The Euclidean distance corresponding to each single-view point cloud is shown in the figure. The Euclidean distance between the perspective global features corresponding to each single-view point cloud and the actual global features is compared. According to the comparison result, the single-view point cloud with the smallest Euclidean distance is selected as the target perspective point cloud.

[0114] In the above implementation, first, the virtual model point cloud is segmented into single perspectives, and features are extracted for each single perspective point cloud obtained by the segmentation, so as to obtain the perspective global features corresponding to each single perspective point cloud. Feature extraction is performed on the point cloud of the real object to be registered, so as to obtain the real global features. Then, based on the matching degree between the perspective global features corresponding to each single perspective point cloud and the real global features, the target perspective point cloud is determined from multiple single perspective point clouds. Finally, based on the target perspective point cloud and the real object point cloud, registration is performed, and the virtual model is registered to the real space coordinates to obtain the target virtual model. The interference of external points is effectively reduced, the number of iterations required is reduced, the accuracy of the registration results is significantly improved, and the problem of multiple iterations required for registration when the pose of the real object is unknown is solved.

[0115] In some embodiments, see Figure 3a , the virtual model point cloud is obtained by:

[0116] S310: resample the virtual model to obtain a model sampling point cloud.

[0117] S310 , spatially segmenting and processing the model sampling point cloud to obtain a virtual model point cloud.

[0118] Specifically, according to the actual situation and specific needs, select a suitable resampling algorithm, such as uniform sampling and random sampling, and resample the virtual model to obtain a model sampling point cloud, which is convenient for subsequent analysis and processing. Use voxel gridding or a segmentation method based on spatial features to spatially segment the model sampling point cloud, divide the model sampling point cloud into different voxel grids, and perform subsequent processing on each voxel grid to ensure that it meets the needs of analysis and processing, and obtain a virtual model point cloud. For example, a CT device can be used to scan and obtain a virtual model. Among them, the virtual model can be in a grid form.

[0119] For example, for a virtual model, 500,000 points are uniformly resampled on its surface to obtain a model sampling point cloud. The model sampling point cloud is then spatially segmented with the same grid size to obtain multiple voxel blocks. For each segmented voxel block, the mean value of all point coordinates of the voxel block is calculated, and the mean value is used as a new point, and all original points in the voxel block are replaced with the mean value. The above operation is repeated for each segmented voxel block to obtain a uniformly distributed virtual model point cloud. Please refer to Figure 3b , Figure 3b The virtual model point cloud is shown.

[0120] In the above implementation, the virtual model is resampled to obtain a model sampling point cloud, and the model sampling point cloud is spatially segmented and processed to obtain a virtual model point cloud, thereby reducing the number of points and eliminating certain errors.

[0121] In some embodiments, see Figure 4 , the actual registration object point cloud is obtained by:

[0122] S410, preprocessing the scanned reality point cloud, and performing foreground extraction according to the preprocessing result to obtain a registered object point cloud.

[0123] S420 , spatially segment and process the registration object point cloud to obtain a real registration object point cloud.

[0124] Specifically, point cloud data may be very large, and the computational complexity of processing and registration is high. In addition, scanned reality point cloud data usually contains noise, outliers or incomplete information, which will affect the accuracy of the data and subsequent processing. Therefore, the scanned reality point cloud is preprocessed (such as downsampling, outlier removal), and the foreground is extracted based on the preprocessing results to obtain the registered object point cloud. Due to the noise in the process of obtaining the virtual model and generating the scanned reality point cloud and the uneven distribution of points between different devices, the registered object point cloud is spatially segmented using the same voxel gridding or spatial feature-based segmentation method as the spatial segmentation of the virtual model, and the registered object point cloud is divided into different voxel grids, and each voxel grid is subsequently processed to ensure that it meets the analysis and processing requirements to obtain the real registered object point cloud. For example, a scanned reality point cloud can be obtained using a binocular camera.

[0125] Exemplarily, the scanned real point cloud is preprocessed, and the foreground is extracted according to the preprocessing result to obtain the registered object point cloud. Then the registered object point cloud is spatially segmented with the same grid size as the model sampling point cloud to obtain multiple voxel blocks. For each segmented voxel block, the mean value of all point coordinates of the voxel block is calculated, and the mean result is used as a new point, and the mean result is used to replace all the original points in the voxel block. Repeat the above operation for each segmented voxel block to obtain a uniformly distributed real registered object point cloud.

[0126] In the above implementation, the scanned real point cloud is preprocessed, and the foreground is extracted based on the preprocessing result to obtain the registered object point cloud, and the registered object point cloud is spatially segmented and processed to obtain the real registered object point cloud while reducing the number of points and eliminating certain errors.

[0127] In some embodiments, see Figure 5a , preprocessing the scanned reality point cloud, and extracting the foreground according to the preprocessing result to obtain the registered object point cloud, which can include the following steps:

[0128] S510 , downsampling the scanned actual point cloud to obtain a downsampled point cloud.

[0129] S520: Remove outliers from the downsampled point cloud to obtain a preprocessing result.

[0130] S530: Perform coordinate comparison based on the preprocessing results to determine the basic foreground point cloud.

[0131] S540 , filtering the basic foreground point cloud according to the object quantity threshold to obtain the registered object point cloud.

[0132] Specifically, a scanning device (such as a binocular camera) is used to scan a real object to obtain a scanned real point cloud. After obtaining the scanned real point cloud, the scanned real point cloud is first downsampled, and a certain proportion of points are selected from the scanned real point cloud to reduce the number of points but try to maintain the original structural features to obtain a downsampled point cloud. Then, in the downsampled point cloud, there may be noise points or outliers, which are irrelevant to the real object. Therefore, the downsampled point cloud is removed from the abnormal points to improve the data quality and obtain the preprocessing result. By comparing the coordinates of the point clusters in the preprocessing results, the point clusters closer to the scanning device are determined and extracted to obtain the basic foreground point cloud. After obtaining the basic foreground point cloud, it needs to be further analyzed. According to the characteristics of the real object, the number of object clusters in the point cloud is determined. By setting an object number threshold, the number of each point cluster in the basic foreground point cloud is compared with the threshold, and the part with a point cluster number greater than the threshold is screened out. These point clusters usually correspond to real objects, while the point clusters with a smaller number may belong to the background or other small objects. After screening out a sufficient number of point clusters, the registered object point cloud is obtained.

[0133] Exemplarily, after obtaining the scanned real point cloud, the scanned real point cloud is first randomly downsampled. A downsampled point cloud is obtained by randomly selecting a certain proportion of points in the point cloud. Outliers are removed from the downsampled point cloud to obtain a preprocessing result. In the absence of occlusion, the real object to be registered should be closer to the scanning device. By comparing the z coordinate values ​​of the point clusters in the preprocessing results, points with smaller z coordinate values ​​(i.e., points closer to the scanning device) are screened out to extract foreground objects and determine the basic foreground point cloud. A minimum point number threshold is set according to the size of the real object to screen the basic foreground point cloud, identify the actual object, and exclude background noise or small objects to obtain the registered object point cloud. Please refer to Figure 5b , Figure 5b Shows the registered object point cloud.

[0134] In the above implementation, the scanned actual point cloud is downsampled to obtain a downsampled point cloud, the number of points in the downsampled point cloud is reduced, and the distribution density of the points in the downsampled point cloud is maintained while the calculation speed is improved. The downsampled point cloud is subjected to outliers removal to obtain a preprocessing result, thereby improving the quality of the point cloud data. Coordinate comparison is performed based on the preprocessing result to determine the basic foreground point cloud, and the basic foreground point cloud is screened according to the object quantity threshold to obtain the registered object point cloud, identify the actual object, and exclude background noise or small objects, thereby improving the accuracy of subsequent matching.

[0135] In some embodiments, see Figure 6 , removing outliers from the downsampled point cloud to obtain a preprocessing result, may include the following steps:

[0136] S610: Perform statistical analysis on the downsampled point cloud, determine and remove outliers in the downsampled point cloud, and obtain a first processing result.

[0137] S620: Remove abnormal points from the first processing result in units of points to obtain a second processing result.

[0138] S630: Remove outliers from the second processing result in units of classes to obtain a preprocessing result.

[0139] Specifically, in the downsampled point cloud, there may be noise points or outliers, which are irrelevant to real objects. Therefore, outliers are removed from the downsampled point cloud to improve data quality. Statistical analysis is performed on the downsampled point cloud using statistical methods to determine the outliers in the downsampled point cloud. Then, outliers are removed from the downsampled point cloud to obtain a first processing result. There are small point clusters in the first processing result, which may represent noise. Therefore, outliers are removed from the first processing result in units of points to obtain a second processing result. There will also be noise in the second processing result, so outliers are removed from the second processing result in units of classes to obtain a preprocessing result.

[0140] In some embodiments, the downsampled point cloud is analyzed based on the Gaussian distribution, and the outliers in the downsampled point cloud are determined and removed to obtain a first processing result. First, for each point in the downsampled point cloud, the K neighborhood corresponding to the point is determined. Then the average distance between the point and other points in its K neighborhood is calculated. Repeat the above operation for each point in each point in the downsampled point cloud to obtain the average distance corresponding to each point. The average distance corresponding to all points in the downsampled point cloud is regarded as a Gaussian distribution to obtain the mean μ and standard deviation σ of the Gaussian distribution. According to the Gaussian distribution, points whose average distance is outside the range of (μ-σ, μ+σ) are identified, and these points are considered to be outliers. Remove outliers from the downsampled point cloud to obtain a first processing result.

[0141] In the above implementation, based on Gaussian distribution, outliers in the downsampled point cloud are determined and removed to obtain a first processing result, outliers are removed from the first processing result in units of points to obtain a second processing result, outliers are removed from the second processing result in units of classes to obtain a preprocessing result, and the quality of the point cloud is improved to improve the accuracy of the registration.

[0142] In some embodiments, see Figure 7a , removing abnormal points in units of points from the first processing result to obtain a second processing result, may include the following steps:

[0143] S710: Determine, according to the first distance, a first statistical space corresponding to each point in the first processing result.

[0144] S720: Calculate the number of points in the first statistical space corresponding to each point in the first processing result.

[0145] S730: When the number of points does not satisfy the point screening condition, remove the corresponding points from the first processing result to obtain a second processing result.

[0146] Specifically, there are smaller point clusters in the first processing result, which may represent noise, so the first processing result is subjected to point-by-point removal of abnormal points. According to the actual situation and the characteristics of real objects, a suitable distance threshold, namely the first distance, is pre-set. For each point in the first processing result, the first distance is used as the radius to determine the spherical neighborhood corresponding to each point in the first processing result, namely the first statistical space. The number of points in the first statistical space corresponding to each point in the first processing result is calculated to indicate the density of the first statistical space corresponding to each point in the first processing result. A point screening condition is pre-set. For example, if the number of points in the first statistical space corresponding to a point in the first processing result is lower than a pre-set point number threshold, the point is considered to be an abnormal point. Therefore, when the number of points does not meet the point screening condition, the points that do not meet the point screening condition are removed from the first processing result to obtain the second processing result. For example, please refer to Figure 7b , Figure 7b The second processing result is shown.

[0147] In the above implementation, based on the first distance, the first statistical space corresponding to each point in the first processing result is determined, and the number of points in the first statistical space corresponding to each point in the first processing result is calculated. When the number of points does not meet the point screening condition, the corresponding points are removed from the first processing result to obtain the second processing result, thereby improving the quality of the point cloud and improving the accuracy of the registration.

[0148] In some embodiments, see Figure 8a, removing outliers from the second processing result in units of classes to obtain a preprocessing result, may include the following steps:

[0149] S810, calculating the number of points in the second statistical space corresponding to each point in the second processing result.

[0150] S820: When the number of points meets the adjacent point screening condition, the corresponding points are used as core points in the second processing result.

[0151] S830: When the number of points does not satisfy the adjacent point screening condition, the corresponding points are used as boundary points in the second processing result.

[0152] S840: Classify the core points and the boundary points in the second statistical space corresponding to the core points to obtain noise points.

[0153] S850: Remove noise points from the second processing result to obtain a preprocessing result.

[0154] The second statistical space corresponding to each point in the second processing result is determined according to the second distance.

[0155] In some cases, there may be noise in the second processing result, so the second processing result is removed by class. A density-based spatial clustering algorithm (such as DBSCAN) is used to classify the second processing result to eliminate unnecessary point clusters and retain important point clusters.

[0156] Specifically, a suitable distance threshold, i.e., the second distance, is pre-set. For each point in the second processing result, the second distance is used as the radius to determine the spherical neighborhood corresponding to each point in the second processing result, i.e., the second statistical space. The number of points in the second statistical space corresponding to each point in the second processing result is calculated. An adjacent point screening condition is pre-set, for example, the adjacent point screening condition can be a threshold value of the number of points in the second statistical space corresponding to a point in the second processing result, which is used to classify the total points of the second processing result. When the number of points meets the adjacent point screening condition, the corresponding point is used as a core point. When the number of points does not meet the adjacent point screening condition, the corresponding point is used as a boundary point. For each core point, all boundary points in its second statistical space are determined. Then, for each boundary point in the second statistical space of the core point, the second statistical space corresponding to each boundary point is used to expand the cluster, and then expand outward in sequence until no more boundary points can be added. The points clustered together during the expansion process are defined as a point cluster. The above process is repeated for each core point. After clustering is completed, if there is a boundary point that is not expanded into the point cluster, the boundary point is used as a noise point. Therefore, the noise points are removed from the second processing result to obtain the preprocessing result. For example, see Figure 8b , Figure 8b The preprocessing results are shown.

[0157] In some embodiments, the adjacent point screening condition may be a threshold value of the number of points in the second statistical space corresponding to a point in the second processing result. Points whose number of points in the second statistical space corresponding to the points in the second processing result is greater than or equal to the threshold are taken as core points. Points whose number of points in the second statistical space corresponding to the points in the second processing result is less than the threshold are taken as demarcation points.

[0158] In the above implementation, the number of points in the second statistical space corresponding to each point in the second processing result is calculated. When the number of points meets the adjacent point screening condition, the corresponding points are used as core points in the second processing result. When the number of points does not meet the adjacent point screening condition, the corresponding points are used as boundary points in the second processing result. Based on the core points and the boundary points in the second statistical space corresponding to the core points, classification is performed to obtain noise points. The noise points are removed from the second processing result to obtain a preprocessing result, thereby improving the quality of the point cloud and improving the accuracy of the registration.

[0159] In some embodiments, see Figure 9a , based on the target viewpoint point cloud and the real registration object point cloud, registering the virtual model to the real space coordinates to obtain the target virtual model, can include the following steps:

[0160] S910 , downsampling the target viewpoint point cloud and the actual registered object point cloud to obtain a target downsampled point cloud and a registered downsampled point cloud.

[0161] Specifically, before downsampling the target view point cloud and the real registered object point cloud, for a point in the target view point cloud, select the neighborhood points of the point within a specified radius. Use principal component analysis (PCA) to fit the neighborhood points corresponding to the point, so as to calculate the local plane of the point and estimate the normal vector corresponding to the point. Repeat the above operation for each point in the target view point cloud to obtain the normal vector corresponding to each point in the target view point cloud. For a point in the real registered object point cloud, select the neighborhood points of the point within a specified radius. Use principal component analysis (PCA) to fit the neighborhood points corresponding to the point, so as to calculate the local plane of the point and estimate the normal vector corresponding to the point. Repeat the above operation for each point in the real registered object point cloud to obtain the normal vector corresponding to each point in the real registered object point cloud. Then use a suitable downsampling method (such as voxel downsampling) to downsample the target view point cloud and the real registered object point cloud to speed up the time required for feature calculation and registration, and obtain the target downsampled point cloud and the registered downsampled point cloud.

[0162] For example, see Figure 9b , Figure 9b The registered downsampled point cloud obtained by normal estimation and voxel downsampling is shown.

[0163] S920, performing rough registration of the target downsampled point cloud and the registration downsampled point cloud to determine an initial registration matrix.

[0164] S930: Based on the initial registration matrix, perform point cloud fine registration on the target viewpoint point cloud and the actual registration object point cloud to determine the target registration matrix.

[0165] S940: align the virtual model to the real space coordinates based on the target alignment matrix to obtain the target virtual model.

[0166] Specifically, the target downsampled point cloud and the registration downsampled point cloud are first coarsely registered based on the feature matching method to determine the initial registration matrix. The purpose of estimating the initial registration matrix is ​​to provide a good starting point for the subsequent fine registration. Based on the initial registration matrix obtained by coarse registration, an optimization algorithm is used to perform fine point cloud registration on the target viewpoint point cloud and the real registration object point cloud to minimize the distance error between the two point clouds and obtain a more accurate target registration matrix. Finally, the target registration matrix is ​​used to register the virtual model to the real space coordinates to obtain the target virtual model. The target model can accurately reflect the position and posture of the virtual model in the real space coordinate system.

[0167] In the above implementation, the target viewpoint point cloud and the real registration object point cloud are downsampled to obtain the target downsampled point cloud and the registration downsampled point cloud, the target downsampled point cloud and the registration downsampled point cloud are coarsely registered, and the initial registration matrix is ​​determined. Based on the initial registration matrix, the target viewpoint point cloud and the real registration object point cloud are finely registered to determine the target registration matrix. Based on the target registration matrix, the virtual model is registered to the real space coordinates to obtain the target virtual model, so that the target model can accurately reflect the position and posture of the virtual model in the real space coordinate system, which helps to realize the combination of the virtual model and the real space.

[0168] In some embodiments, see Fig.10a , performing rough registration of the target downsampled point cloud and the registration downsampled point cloud to determine the initial registration matrix may include the following steps:

[0169] S1010, extracting features based on the normal vector corresponding to each point in the target downsampled point cloud to obtain a local feature of the viewing angle.

[0170] S1020, extracting features based on the normal vector corresponding to each point in the registered downsampled point cloud to obtain actual local features.

[0171] S1030: Determine an initial registration matrix based on the local features of the viewing angle and the local features of the reality.

[0172] Specifically, the normal vector and neighborhood point information corresponding to the target downsampled point cloud are used to extract features from the target downsampled point cloud, capture the geometric and spatial information of the target downsampled point cloud, and obtain the local features of the perspective. The normal vector and neighborhood point information corresponding to the registered downsampled point cloud are used to extract features from the registered downsampled point cloud, capture the geometric and spatial information of the registered downsampled point cloud, and obtain the local features of the reality. The local features of the perspective and the local features of the reality are matched to determine the initial registration matrix corresponding to the coarse registration. For example, please refer to Fig.10b , Fig.10b The results of rough point cloud registration of target downsampled point cloud and registered downsampled point cloud are shown.

[0173] For example, when extracting features based on the normal vector corresponding to each point in the target downsampling point cloud, in order to accurately describe the geometric shape and spatial distribution of the target downsampling point cloud, the Fast Point Feature Histogram (FPFH) is used as a description method of local features. During the calculation of the FPFH descriptor, the local features of the target downsampling point cloud are represented as a 33-dimensional feature vector to obtain the local features of the viewing angle.

[0174] When extracting features based on the normal vector corresponding to each point in the registered and subtracted point cloud, in order to accurately describe the geometric shape and spatial distribution of the registered and subtracted point cloud, the fast point feature histogram (FPFH) is used as a description method of local features. During the calculation of the FPFH descriptor, the local features of the registered and subtracted point cloud are represented as a 33-dimensional feature vector to obtain the actual local features.

[0175] In the above implementation, feature extraction is performed based on the normal vector corresponding to each point in the target downsampled point cloud to obtain the local feature of the viewing angle, and feature extraction is performed based on the normal vector corresponding to each point in the registered downsampled point cloud to obtain the local feature of the reality, and based on the local feature of the viewing angle and the local feature of the reality, the initial registration matrix is ​​determined to provide a good starting point for the subsequent fine registration.

[0176] In some embodiments, see Fig.11 , based on the local features of the perspective and the local features of the reality, determining the initial registration matrix may include the following steps:

[0177] S1110, selecting a target point in the registered point cloud.

[0178] S1120. Based on the local features of the viewing angle and the local features of the reality, the pairing points of the target points are determined in the target sampling point cloud to obtain feature point pairs.

[0179] S1130: Determine an initial registration matrix based on the feature point pairs.

[0180] Specifically, a certain number of target points are selected from the registered downsampled point cloud, and these points will serve as the basis for feature point pairs. When selecting target points, it should be ensured that these points are representative in space and can be effectively matched and registered in the subsequent process. The actual local features of a target point are compared with the local features of the viewpoint in the target downsampled point cloud. The similarity between the actual local features of the target point and the local features of the viewpoint in the target downsampled point cloud is calculated using feature descriptors and similarity metrics (such as Euclidean distance or other similarity metrics). According to the similarity calculation results, the points corresponding to the local features of the viewpoint with the highest similarity are taken as potential corresponding points of the target point. Repeat the above operation for each target point to obtain the potential corresponding points corresponding to each target point. After completing the matching of the potential corresponding points, determine the target distance between the two target points, and determine the potential distance between the potential corresponding points corresponding to the two target points. These distances are used to evaluate the accuracy of the matching. Compare the size between the target distance and the potential distance. If the distance difference between the target distance and the potential distance exceeds a predetermined threshold, the paired points that do not meet the requirements are rejected in advance. The above operations are performed sequentially, and the remaining target points and their corresponding potential corresponding points are taken as feature point pairs. Based on the feature point pairs, the initial registration matrix is ​​determined.

[0181] For example, the target points selected in the registration point cloud are target point A, target point B, and target point C. After feature matching, the potential corresponding point a corresponding to target point A, the potential corresponding point b corresponding to target point B, and the potential corresponding point c corresponding to target point C are determined in the target point cloud. The distance AB between target point A and target point B is calculated. The distance ab between the potential corresponding point a and the potential corresponding point b is calculated. The distance AB is compared with the distance ab. If the difference between the distance between point AB and the distance between point ab exceeds a predetermined threshold, it is rejected in advance.

[0182] Exemplarily, the Euclidean distance between the actual local feature of the target point and the local feature of the view in the target point cloud is calculated using the feature descriptor and the similarity measurement method, and the point corresponding to the local feature of the view of the Euclidean distance is used as the potential corresponding point of the target point.

[0183] In the above implementation, a target point is selected in the registration point cloud, and a pairing point of the target point is determined in the target point cloud based on the local features of the perspective and the local features of reality to obtain a feature point pair. Based on the feature point pair, an initial registration matrix is ​​determined to provide a good starting point for subsequent fine registration.

[0184] In some embodiments, see Fig.12 , based on the feature point pairs, determining the initial registration matrix can include the following steps:

[0185] S1210: Determine a basic registration matrix using feature point pairs.

[0186] S1220: Determine the degree of match between the target reduced sampling point cloud and the registered reduced sampling point cloud based on the basic registration matrix.

[0187] S1230: When the matching degree satisfies the matching condition, the basic registration matrix is ​​determined as the initial registration matrix.

[0188] S1240: When the matching degree does not satisfy the matching condition, re-determine the target view point cloud from the single view point cloud.

[0189] Specifically, the selected feature point pairs are used to calculate the basic registration matrix through the registration algorithm (such as the least squares method, the random sampling consensus algorithm (RANSAC), etc.). After determining the basic registration matrix, the basic registration matrix is ​​applied to transform the target downsampled point cloud into the coordinate system of the registered downsampled point cloud. The matching degree between the transformed point cloud and the registered downsampled point cloud is calculated using a similarity measurement method (such as the mean square error (MSE)). Check whether the matching degree meets the preset matching condition. If the matching degree meets the matching condition, the basic registration matrix is ​​determined as the initial registration matrix. If the matching degree does not meet the matching condition, it means that the current basic registration matrix fails to effectively align the point cloud, and the target view point cloud needs to be re-determined. Based on the matching degree between the perspective global features corresponding to each single view point cloud and the actual global features, the single view point cloud with the second smallest matching degree is determined as the target view point cloud. For each newly selected target view point cloud, the above operation is repeated until the matching degree meets the matching condition, and its corresponding basic registration matrix is ​​determined as the initial registration matrix.

[0190] It should be noted that when the matching degree meets the matching condition or does not meet the matching condition, the above process can be repeated multiple times to gradually optimize the registration result. During each repetition, the matching result and the corresponding basic registration matrix are recorded.

[0191] In some implementations, the stopping condition for repeating the above process multiple times may be that the matching degree reaches a specified threshold, indicating that the effect of the current basic registration matrix meets the requirements and can be used as the initial registration matrix.

[0192] In other embodiments, the stopping condition for repeating the above process multiple times may be that a specified number of times is reached, and the basic registration matrix corresponding to the highest matching degree is selected as the initial registration matrix.

[0193] In the above implementation, feature point pairs are used to determine a basic registration matrix, and based on the basic registration matrix, the degree of matching between the target downsampled point cloud and the registered downsampled point cloud is determined. When the degree of matching meets the matching conditions, the basic registration matrix is ​​determined as the initial registration matrix. When the degree of matching does not meet the matching conditions, the target perspective point cloud is re-determined from the single perspective point cloud to improve the accuracy of point cloud registration, thereby improving the accuracy of registering the virtual model to the real space coordinates.

[0194] In some embodiments, see Fig.13a , the target registration matrix includes a rotation matrix and a translation matrix; based on the target registration matrix, the virtual model is registered to the real space coordinates to obtain the target virtual model, which may include the following steps:

[0195] S1310, performing inverse calculation on the rotation matrix to obtain a target rotation matrix.

[0196] S1320, performing inverse calculation on the translation matrix to obtain a target translation matrix.

[0197] S1330 , aligning the virtual model to the real space coordinates based on the target rotation matrix and the target translation matrix to obtain the target virtual model.

[0198] Specifically, the role of the target registration matrix is ​​to align the scanned point cloud in the real space coordinates to the coordinate system of the virtual model. However, in actual use, the virtual model needs to be registered from the virtual coordinate system back to the real space coordinate system. Therefore, it is necessary to invert the rotation matrix to obtain the target rotation matrix. The translation matrix is ​​inverted to obtain the target translation matrix. Using the adjusted target rotation matrix and target translation matrix, the virtual model is registered from the virtual coordinate system to the real space coordinates to obtain the target virtual model. For example, please refer to Fig.13b , Fig.13b The target virtual model is shown aligned to the real space coordinates.

[0199] In the above implementation, the rotation matrix is ​​inverted to obtain the target rotation matrix, the translation matrix is ​​inverted to obtain the target translation matrix, and the virtual model is aligned to the real space coordinates based on the target rotation matrix and the target translation matrix to obtain the target virtual model, so that the virtual model can be accurately aligned in the real space.

[0200] In some embodiments, see Fig.14a Based on the initial registration matrix, the target viewpoint point cloud and the actual registration object point cloud are finely registered to determine the target registration matrix, which may include the following steps:

[0201] S1410 , using the initial registration matrix, coarsely register the target viewpoint point cloud to obtain the point cloud to be corrected.

[0202] S1420 , using the initial registration matrix as the initial value, iteratively registering the point cloud to be corrected and the actual registration object point cloud to obtain a target registration matrix.

[0203] Specifically, first, the target viewpoint point cloud is coarsely registered using the initial registration matrix to roughly align the initial position of the target viewpoint point cloud to the reference coordinate system to obtain the point cloud to be corrected. Although the point cloud to be corrected has been preliminarily aligned, there may still be some errors. In the fine registration stage, the point cloud to be corrected after coarse registration is used as the basis, and the initial registration matrix is ​​used as the initial value. The point cloud to be corrected is fine-tuned through high-precision algorithms (such as the least squares method and the iterative closest point algorithm). The position and direction of the point cloud are optimized through continuous iterations to make it highly matched with the point cloud in the reference coordinate system. Finally, the target registration matrix obtained after fine adjustment can minimize the error between point clouds, achieve precise alignment, and provide an accurate data basis for subsequent analysis or processing.

[0204] For example, the point-to-plane iterative closest point (ICP) method is used to replace the straight-line distance between points with the distance from a point in the actual registration object point cloud to the fitting plane of each point in the point cloud to be corrected, that is, the distance along the normal vector, as the error and perform iterative optimization to finally obtain the transformation matrix. Fig.14b , Fig.14b The results of fine point cloud registration between the target view point cloud and the real registration object point cloud are shown.

[0205] In the above implementation, the initial registration matrix is ​​used to perform a rough registration on the target perspective point cloud to obtain the point cloud to be corrected. The initial registration matrix is ​​used as the initial value to iteratively align the point cloud to be corrected and the point cloud of the real registration object to obtain the target registration matrix, so that the target model can accurately reflect the position and posture of the virtual model in the real space coordinate system, which helps to realize the combination of the virtual model and the real space.

[0206] The embodiment of this specification also provides a method for registering a virtual model with a real object. The target registration matrix includes a rotation matrix and a translation matrix. For example, see Fig.15 The registration method of the virtual model and the real object may include the following steps:

[0207] S1502 , resampling the virtual model to obtain a model sampling point cloud, and spatially segmenting and processing the model sampling point cloud to obtain a virtual model point cloud.

[0208] S1504: downsample the scanned actual point cloud to obtain a downsampled point cloud, determine and remove outliers in the downsampled point cloud based on Gaussian distribution, and obtain a first processing result. Determine a first statistical space corresponding to each point in the first processing result according to the first distance, and calculate the number of points in the first statistical space corresponding to each point in the first processing result.

[0209] S1506. When the number of points does not satisfy the point screening condition, remove the corresponding points in the first processing result to obtain a second processing result, and calculate the number of points in the second statistical space corresponding to each point in the second processing result.

[0210] The second statistical space corresponding to each point in the second processing result is determined according to the second distance. When the number of points meets the adjacent point screening condition, the corresponding point is used as a core point in the second processing result; when the number of points does not meet the adjacent point screening condition, the corresponding point is used as a boundary point in the second processing result.

[0211] S1508: Classify the core points and the boundary points in the second statistical space corresponding to the core points to obtain noise points. Remove the noise points from the second processing result to obtain a preprocessing result.

[0212] S1510: Perform coordinate comparison based on the preprocessing result to determine the basic foreground point cloud. Filter the basic foreground point cloud according to the object quantity threshold to obtain the registration object point cloud.

[0213] S1512: spatially segment and process the registration object point cloud to obtain a real registration object point cloud.

[0214] S1514, performing single-view segmentation on the virtual model point cloud, and performing feature extraction on each single-view point cloud obtained by the segmentation to obtain the perspective global features corresponding to each single-view point cloud, and performing feature extraction on the real registration object point cloud to obtain the real global features.

[0215] S1516. Determine a target perspective point cloud from multiple single-perspective point clouds based on the degree of matching between the perspective global features corresponding to each single-perspective point cloud and the actual global features.

[0216] S1518. Downsampling the target viewpoint point cloud and the actual registered object point cloud to obtain the target downsampled point cloud and the registered downsampled point cloud.

[0217] S1520, extracting features based on the normal vector corresponding to each point in the target reduced sampling point cloud to obtain a local feature of the viewing angle. Extracting features based on the normal vector corresponding to each point in the registration reduced sampling point cloud to obtain a local feature of the reality.

[0218] S1522, selecting a target point in the registered reduced sampling point cloud. Based on the local features of the viewing angle and the local features of the reality, a pairing point of the target point is determined in the target reduced sampling point cloud to obtain a feature point pair.

[0219] S1524: Determine a basic registration matrix using the feature point pairs. Determine the degree of match between the target downsampled point cloud and the registered downsampled point cloud based on the basic registration matrix.

[0220] S1526: If the matching degree meets the matching condition, the basic registration matrix is ​​determined as the initial registration matrix. If the matching degree does not meet the matching condition, the target view point cloud is re-determined from the single view point cloud.

[0221] S1528, using the initial registration matrix, roughly register the target viewpoint point cloud to obtain the point cloud to be corrected. Using the initial registration matrix as the initial value, iteratively register the point cloud to be corrected and the actual registration object point cloud to obtain the target registration matrix.

[0222] S1530, performing inverse calculation on the rotation matrix to obtain a target rotation matrix. Performing inverse calculation on the translation matrix to obtain a target translation matrix.

[0223] S1532. Align the virtual model to the real space coordinates based on the target rotation matrix and the target translation matrix to obtain the target virtual model.

[0224] The embodiment of this specification provides a virtual model and real object registration device 1600, see Fig.16 The virtual model and real object registration device 1600 includes: a feature extraction module 1610, a feature matching module 1620, and a model registration module 1630.

[0225] The feature extraction module 1610 is used to extract features from the point cloud data to obtain global features; wherein the global features include perspective global features and real global features corresponding to each single-view point cloud, wherein the perspective global features corresponding to each single-view point cloud are obtained by performing single-view segmentation on the virtual model point cloud and extracting features from each single-view point cloud obtained by segmentation, and the real global features are obtained by performing feature extraction on the real registration object point cloud;

[0226] A feature matching module 1620, configured to determine a target view point cloud from a plurality of single view point clouds based on a matching degree between the view global feature corresponding to each of the single view point clouds and the reality global feature;

[0227] The model registration module 1630 is used to perform registration based on the target viewpoint point cloud and the real registration object point cloud, register the virtual model to the real space coordinates, and obtain the target virtual model.

[0228] For a detailed description of the virtual model and real object registration device, please refer to the description of the virtual model and real object registration method above, which will not be repeated here.

[0229] An embodiment of the present specification provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method in any of the above embodiments are implemented.

[0230] One embodiment of the present specification provides a computer program product, which includes instructions. When the instructions are executed by a processor of a computer device, the computer device can perform the steps of the method of any of the above embodiments.

[0231] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.17 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for aligning a virtual model with a real object is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0232] Those skilled in the art will understand that Fig.17 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution disclosed in this specification, and does not constitute a limitation on the computer device to which the solution disclosed in this specification is applied. Specifically, the computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0233] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

Claims

1. A method for registering a virtual model with a real object, characterized in that: The method comprises: Extracting features from the point cloud data to obtain global features; wherein the global features include perspective global features and real global features corresponding to each single-view point cloud, the perspective global features corresponding to each single-view point cloud are obtained by performing single-view segmentation on the virtual model point cloud and extracting features from each single-view point cloud obtained by segmentation, and the real global features are obtained by performing feature extraction on the real registration object point cloud; Determining a target perspective point cloud from a plurality of single-perspective point clouds based on a matching degree between the perspective global feature corresponding to each of the single-perspective point clouds and the reality global feature; Based on the registration of the target viewpoint point cloud and the real registration object point cloud, the virtual model is registered to the real space coordinates to obtain the target virtual model.

2. The method according to claim 1, characterized in that The virtual model point cloud is obtained by: Resample the virtual model to obtain a model sampling point cloud; The model sampling point cloud is spatially segmented and processed to obtain the virtual model point cloud.

3. The method according to claim 1, characterized in that The actual registration object point cloud is obtained by: Preprocess the scanned reality point cloud, and perform foreground extraction based on the preprocessing result to obtain the registered object point cloud; The registered object point cloud is spatially segmented and processed to obtain the real registered object point cloud.

4. The method according to claim 3, characterized in that The method of preprocessing the scanned real point cloud and extracting the foreground according to the preprocessing result to obtain the registered object point cloud includes: Downsampling the scanned actual point cloud to obtain a downsampled point cloud; Removing outliers from the downsampled point cloud to obtain the preprocessing result; Perform coordinate comparison based on the preprocessing results to determine the basic foreground point cloud; The basic foreground point cloud is screened according to an object quantity threshold to obtain the registered object point cloud.

5. The method according to claim 4, characterized in that The removing of outliers from the downsampled point cloud to obtain the preprocessing result includes: Performing statistical analysis on the downsampled point cloud, determining and removing outliers in the downsampled point cloud, and obtaining a first processing result; Removing abnormal points from the first processing result in units of points to obtain a second processing result; The second processing result is subjected to removal of outliers in units of classes to obtain the preprocessing result.

6. The method according to claim 5, characterized in that The removing of abnormal points from the first processing result in units of points to obtain a second processing result includes: Determine, according to the first distance, a first statistical space corresponding to each point in the first processing result; Calculate the number of points in the first statistical space corresponding to each point in the first processing result; When the number of points does not satisfy the point screening condition, the corresponding points are removed from the first processing result to obtain the second processing result.

7. The method according to claim 5, characterized in that The removing of outliers from the second processing result in units of classes to obtain the preprocessing result includes: Calculating the number of points in the second statistical space corresponding to each point in the second processing result; wherein the second statistical space corresponding to each point in the second processing result is determined according to the second distance; In the case where the number of points meets the adjacent point screening condition, the corresponding points are used as core points in the second processing result; In the case that the number of points does not satisfy the adjacent point screening condition, the corresponding points are used as boundary points in the second processing result; Based on the core points and the boundary points in the second statistical space corresponding to the core points, classification is performed to obtain noise points; The noise points are removed from the second processing result to obtain the preprocessing result.

8. The method according to claim 1, characterized in that: The step of registering the target viewpoint point cloud with the real registration object point cloud, registering the virtual model to the real space coordinates, and obtaining the target virtual model includes: Downsampling the target viewpoint point cloud and the actual registered object point cloud to obtain a target downsampled point cloud and a registered downsampled point cloud; Performing rough point cloud registration on the target downsampled point cloud and the registered downsampled point cloud to determine an initial registration matrix; Based on the initial registration matrix, performing point cloud fine registration on the target viewpoint point cloud and the actual registration object point cloud to determine a target registration matrix; The virtual model is registered to the real space coordinates based on the target registration matrix to obtain a target virtual model.

9. The method according to claim 8, characterized in that The step of performing rough registration of the target reduced sampling point cloud and the registered reduced sampling point cloud to determine an initial registration matrix includes: Perform feature extraction based on the normal vector corresponding to each point in the target downsampling point cloud to obtain a local feature of the viewing angle; Perform feature extraction based on the normal vector corresponding to each point in the registered downsampled point cloud to obtain a realistic local feature; An initial registration matrix is ​​determined based on the perspective local features and the reality local features.

10. The method according to claim 9, characterized in that The determining of an initial registration matrix based on the perspective local feature and the reality local feature comprises: Selecting a target point in the registered and reduced sampling point cloud; Based on the local features of the viewing angle and the local features of the reality, a pairing point of the target point is determined in the target point cloud to obtain a feature point pair; Based on the feature point pairs, the initial registration matrix is ​​determined.

11. The method according to claim 10, characterized in that The determining the initial registration matrix based on the feature point pairs comprises: Determine a basic registration matrix using the feature point pairs; Based on the basic registration matrix, determining the matching degree between the target reduced mining point cloud and the registered reduced mining point cloud; In the case where the matching degree satisfies a matching condition, determining the basic registration matrix as the initial registration matrix; When the matching degree does not satisfy the matching condition, the target view point cloud is re-determined from the single view point cloud.

12. The method according to claim 8, characterized in that The target registration matrix includes a rotation matrix and a translation matrix; and the step of registering the virtual model to the real space coordinates based on the target registration matrix to obtain the target virtual model includes: Performing an inverse calculation on the rotation matrix to obtain a target rotation matrix; Performing an inverse calculation on the translation matrix to obtain a target translation matrix; The virtual model is aligned to the real space coordinates based on the target rotation matrix and the target translation matrix to obtain a target virtual model.

13. The method according to claim 8, characterized in that The step of performing point cloud fine registration on the target viewpoint point cloud and the actual registration object point cloud based on the initial registration matrix to determine the target registration matrix includes: Using the initial registration matrix, coarsely registering the target viewpoint point cloud to obtain a point cloud to be corrected; The initial registration matrix is ​​used as an initial value, and the point cloud to be corrected and the point cloud of the actual registration object are iteratively registered to obtain the target registration matrix.

14. A virtual model and real object registration device, characterized in that: The device comprises: A feature extraction module is used to extract features from point cloud data to obtain global features; wherein the global features include perspective global features and real global features corresponding to each single-view point cloud, wherein the perspective global features corresponding to each single-view point cloud are obtained by performing single-view segmentation on the virtual model point cloud and extracting features from each single-view point cloud obtained by segmentation, and the real global features are obtained by performing feature extraction on the real registration object point cloud; A feature matching module, configured to determine a target view point cloud from a plurality of single view point clouds based on a matching degree between a view global feature corresponding to each of the single view point clouds and the reality global feature; The model registration module is used to register the target viewpoint point cloud with the real registration object point cloud, register the virtual model to the real space coordinates, and obtain the target virtual model.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 13 are implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images