Point Cloud Density Upsampling Method, System and Medium Based on Cross-Modal Data Registration

Abstract

The present invention discloses a point cloud density upsampling method, system and medium based on cross-modal data registration. The present invention includes collecting three-dimensional point cloud data of a target object and images from multiple perspectives; respectively extracting image features, registering and calculating the spatial association between the image and the image features under any perspective and constructing a three-dimensional point cloud skeleton model; extracting an original three-dimensional skeleton model from the three-dimensional point cloud data, and overlapping it with the three-dimensional point cloud skeleton model to obtain an overlapping skeleton model; for the points in the overlapping skeleton model, reversely obtaining the pixel points in the corresponding perspective image, and copying and expanding the obtained pixel points according to a specified multiple, and finally obtaining a complete three-dimensional point cloud density upsampling result. The present invention aims to solve the problems of large computational amount and low accuracy of the output model encountered in the existing two methods of point cloud densification, and has the advantages of simple implementation method and low cost, and can quickly densify low-density three-dimensional point clouds based on images.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and particularly relates to a point cloud density upsampling method, system and medium based on cross-modal data registration. Background Art

[0002] A point cloud is a data set, and each point in the data set represents a set of X, Y, Z geometric coordinates and an intensity value, which records the intensity of the returned signal according to the surface reflectivity of the object. When these points are combined together, a point cloud is formed, that is, a set of data points representing a 3D shape or object in space. The point cloud can also be automatically colored to achieve more realistic visualization. Currently, there are two ways to densify the point cloud: (1) Semantic-based point cloud densification method: Upsampling is performed on the point cloud, that is, an input point cloud is input, and a denser point cloud is output, and it falls on the implicit geometric body (such as the surface) of the input point cloud. The core idea of PU-Net is to learn the features of each point at multiple granularities (from local to global), then expand the point set in the feature space, and finally map the expanded point set back to three dimensions. (2) Geometry-based point cloud densification method: Whether it is a semantic-based or geometry-based point cloud densification method, they have high complexity in modeling objects, and have high prior requirements for the geometric shape of the model itself, and lack accuracy and robustness in modeling the densification of the point cloud. In actual working conditions, for the initial point cloud obtained by scanning in a short time and limited environment, complete and accurate semantic prior information cannot be obtained, and there is a lack of conditions with sufficient computing resources for geometric upsampling. Summary of the Invention

[0003] In view of the above technical problems existing in the prior art, the technical problem to be solved by the present invention is to provide a point cloud density upsampling method, system and medium based on cross-modal data registration. The present invention aims to solve the problems of large computational amount and low accuracy of the output model in the existing two methods of point cloud densification, and provide a point cloud density upsampling method with simple implementation method and low cost, which can quickly densify the low-density three-dimensional point cloud based on the image.

[0004] To solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0005] A point cloud density upsampling method based on cross-modal data registration includes:

[0006] S101, collecting three-dimensional point cloud data of the target object and images from multiple perspectives;

[0007] S102, respectively extracting the features of the images from multiple perspectives, registering the image features from multiple perspectives, and calculating the spatial association between the image and the image features in any perspective;

[0008] S103. Construct a 3D point cloud skeleton model through the spatial correlation between the image features of any two perspectives;

[0009] S104. Extract the original 3D skeleton model from the 3D point cloud data, and overlap the extracted original 3D skeleton model with the constructed 3D point cloud skeleton model and the original 3D skeleton model to obtain an overlapping skeleton model;

[0010] S105. For the points in the overlapping skeleton model, reversely obtain the pixel points in the corresponding perspective image, and copy and expand the obtained pixel points according to a specified multiple to finally obtain the complete upsampling result of the 3D point cloud density.

[0011] Optionally, the 3D point cloud data of the target object collected in step S101 is a sparse point cloud model P L .

[0012] Optionally, when collecting images of multiple perspectives in step S101, the included angle between adjacent perspectives is fixed, and all perspectives cover a 360-degree range of the target object, and finally n perspective images I1 to I n .

[0013] Optionally, calculating the spatial correlation between the image and the image features in any perspective in step S102 means calculating the distance d i between the image I i in any perspective i and the image feature F i = d(I i , F i ) as the spatial correlation.

[0014] Optionally, the distance d i between the image I i in any perspective i and the image feature F i = d(I i , F i ) refers to the Euclidean distance.

[0015] Optionally, step S103 includes:

[0016] S201. Using the distance d i between the image I i in any perspective i and the image feature F i = d(I i , F i ) as the association weight between the image and the feature in perspective i, aggregate the image I i in perspective i and the image feature F i to construct the binary group data (I1, F1), (I2, F2),..., (I n , Fn );

[0017] S202. Reconstruct a three-dimensional skeleton model from the binary data of each perspective image and its features (I1, F1), (I2, F2),..., (I n , F n ), where each point in the image I i and the image feature F i at any perspective i is used as a point (x, y, z) in the three-dimensional skeleton, thereby obtaining the finally reconstructed overlapping skeleton model P I .

[0018] Optionally, in step S104, overlapping the extracted original three-dimensional skeleton model with the constructed three-dimensional point cloud skeleton model and the original three-dimensional skeleton model means superimposing the extracted original three-dimensional skeleton model with the constructed three-dimensional point cloud skeleton model and the original three-dimensional skeleton model in three-dimensional space to obtain an overlapping skeleton model P1.

[0019] Optionally, step S105 includes:

[0020] S301. For each point (x, y, z) in the overlapping skeleton model, reversely obtain the pixel point (x, y) in the corresponding perspective image and the adjacent 8 pixel points (x - 1, y - 1), (x - 1, y), (x - 1, y + 1), (x, y - 1), (x, y + 1), (x + 1, y - 1, z), (x + 1, y), and (x + 1, y + 1);

[0021] S302. Assign the z value of the coordinate of the point (x, y, z) to the pixel point (x, y) in the corresponding perspective image and the adjacent 8 pixel points (x - 1, y - 1), (-1, y), (x - 1, y + 1), (x, y - 1), (x, y + 1), (x + 1, y - 1, z), (x + 1, y), and (x + 1, y + 1), a total of nine pixel points, to expand them into the coordinate points of 9 point clouds. Finally, expand the overlapping skeleton model by 9 times to obtain a densified point cloud model P2 as the final complete three-dimensional point cloud density upsampling result.

[0022] In addition, the present invention also provides a point cloud density upsampling system based on cross-modal data registration, including a microprocessor and a memory connected to each other. The microprocessor is programmed or configured to execute the point cloud density upsampling method based on cross-modal data registration.

[0023] In addition, the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and the computer program is used to be programmed or configured by a microprocessor to execute the point cloud density upsampling method based on cross-modal data registration.

[0024] Compared with the prior art, the present invention mainly has the following advantages: The present invention includes collecting three-dimensional point cloud data of a target object and images from multiple perspectives; respectively extracting image features, registering, calculating the spatial association between the image and the image features at any perspective, and constructing a three-dimensional point cloud skeleton model; extracting an original three-dimensional skeleton model from the three-dimensional point cloud data, and overlapping it with the three-dimensional point cloud skeleton model to obtain an overlapping skeleton model; for the points in the overlapping skeleton model, reversely obtaining the pixel points in the corresponding perspective image, and copying and expanding the obtained pixel points according to a specified multiple, and finally obtaining a complete upsampling result of the three-dimensional point cloud density. The present invention can solve the problems of large computational amount and low accuracy of the output model encountered in the existing two methods for point cloud densification, has the advantages of simple implementation method and low cost, and can quickly densify low-density three-dimensional point clouds based on images. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the basic process of the method in the embodiment of the present invention.

[0026] Figure 2 It is the three-dimensional point cloud data of the target object collected in the embodiment of the present invention.

[0027] Figure 3 It is the complete upsampling result of the three-dimensional point cloud density finally obtained in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] As Figure 1 shown, the method for upsampling the point cloud density based on cross-modal data registration in this embodiment includes:

[0029] S101, collecting three-dimensional point cloud data of a target object and images from multiple perspectives;

[0030] S102, respectively extracting the features of the images from multiple perspectives, registering the image features from multiple perspectives, and calculating the spatial association between the image and the image features at any perspective;

[0031] S103, constructing a three-dimensional point cloud skeleton model through the spatial association between the image features of any two perspectives;

[0032] S104, extracting an original three-dimensional skeleton model from the three-dimensional point cloud data, and overlapping the extracted original three-dimensional skeleton model with the constructed three-dimensional point cloud skeleton model and the original three-dimensional skeleton model to obtain an overlapping skeleton model;

[0033] S105, for the points in the overlapping skeleton model, reversely obtaining the pixel points in the corresponding perspective image, and copying and expanding the obtained pixel points according to a specified multiple, and finally obtaining a complete upsampling result of the three-dimensional point cloud density.

[0034] The point cloud density upsampling method based on cross-modal data registration in this embodiment is achieved through the registration processes of calculating spatial association and generating a point cloud skeleton model in steps S102 and S103. A three-dimensional point cloud skeleton model is constructed through the spatial association between image features from any two perspectives in step S103, and the original three-dimensional skeleton model is extracted from the three-dimensional point cloud data in step S104, obtaining two cross-modal skeleton models. By overlapping the extracted original three-dimensional skeleton model with the constructed three-dimensional point cloud skeleton model and the original three-dimensional skeleton model, an overlapping skeleton model is obtained, realizing the registration and fusion of cross-modal vector data. It can solve the problems of large computational amount and low accuracy of the output model encountered in the existing two methods for point cloud densification, has the advantages of simple implementation method and low cost, and can quickly densify low-density three-dimensional point clouds based on images.

[0035] In this embodiment, the three-dimensional point cloud data of the target object collected in step S101 is a sparse point cloud model P L .

[0036] In this embodiment, when collecting images from multiple perspectives in step S101, the included angle between adjacent perspectives is fixed, and all perspectives cover the 360-degree range of the target object, and finally n images I1 to I of perspectives are obtained n .

[0037] In this embodiment, calculating the spatial association between the image and the image features from any perspective in step S102 means calculating the distance d i between the image I i from any perspective i and the image feature F i =x(I i , F i ) as the spatial association.

[0038] In this embodiment, the distance d i between the image I i from any perspective i and the image feature F i =d(I i , F i ) refers to the Euclidean distance. The features in the multi-perspective images are extracted respectively, and the pairwise spatial associations are obtained by registering the multi-perspective image features. The edge pixel point features in the images I1, I2,... n collected from different perspectives are extracted, and the features extracted from each image are denoted as F1, F2,..., F n , and the binary data (I1, F1), (I2, F2),...,(I n , F n ) of the images and features are calculated for the Euclidean distance between the features and the image data pairwise to obtain d n =d(In , F n )。

[0039] In this embodiment, step S103 includes:

[0040] S201. Take the distance d i between the image I i and the image feature F i = d(I i , F i ) as the association weight between the image and the feature at view angle i, and aggregate (extract features and combine them) the image I i and the image feature F i at view angle i to construct the binary data of the image and the feature for each view angle (I1, F1), (I2, F2),..., (I n , F n );

[0041] S202. Reconstruct the 3D skeleton model for the binary data of the image and the feature for each view angle (I1, F1), (I2, F2),..., (I n , F n ). Among them, each point in the image I i and the image feature F i at any view angle i serves as a point (x, y, z) in the 3D skeleton. Combine all the points to obtain the finally reconstructed overlapping skeleton model P I .

[0042] In this embodiment, the overlap of the extracted original 3D skeleton model with the constructed 3D point cloud skeleton model and the original 3D skeleton model in step S104 means superimposing the extracted original 3D skeleton model with the constructed 3D point cloud skeleton model and the original 3D skeleton model in the 3D space to obtain the overlapping skeleton model P1.

[0043] In this embodiment, step S105 includes:

[0044] S301. For each point (x, y, z) in the overlapping skeleton model, reversely obtain the pixel point (x, y) and the adjacent 8 pixel points (x - 1, y - 1), (x - 1, y), (x - 1, y + 1), (x, y - 1), (x, y + 1), (x + 1, y - 1, z), (x + 1, y), and (x + 1, y + 1) in the corresponding view image;

[0045] In S302, the pixel point (x, y) in the image of the corresponding perspective and the adjacent 8 pixel points (x - 1, y - 1), (x - 1, y), (x - 1, y + 1), (x, y - 1), (x, y + 1), (x + 1, y - 1, z), (x + 1, y), and (x + 1, y + 1), a total of nine pixel points, are given the coordinate z value of the point (x, y, z) to expand the coordinate points of 9 point clouds. Finally, the overlapping skeleton model is expanded by 9 times to obtain the densified point cloud model P2 as the final complete three-dimensional point cloud density upsampling result.

[0046] In this embodiment, the three-dimensional point cloud data of the target object collected in step S101 is as Figure 2 shown, and the final complete three-dimensional point cloud density upsampling result in step S105 is as Figure 3 shown. It can be seen that this embodiment can realize the rapid densification of low-density three-dimensional point clouds based on images.

[0047] In summary, the point cloud density upsampling method based on cross-modal data registration in this embodiment includes collecting the three-dimensional point cloud data of the target object and images of multiple perspectives; respectively extracting image features, registering and calculating the spatial association between the image and the image features under any perspective, and constructing a three-dimensional point cloud skeleton model; extracting the original three-dimensional skeleton model from the three-dimensional point cloud data, overlapping it with the three-dimensional point cloud skeleton model to obtain an overlapping skeleton model; for the points in the overlapping skeleton model, reversely obtaining the pixel points in the image of the corresponding perspective, and copying and expanding the obtained pixel points according to the specified multiple to finally obtain the complete three-dimensional point cloud density upsampling result. The point cloud density upsampling method based on cross-modal data registration in this embodiment can effectively solve the problems of large computational amount and low accuracy of the output model encountered in the existing two methods of point cloud densification, has the advantages of simple implementation method and low cost, and can realize the rapid densification of low-density three-dimensional point clouds based on images.

[0048] In addition, this embodiment also provides a point cloud density upsampling system based on cross-modal data registration, including a microprocessor and a memory connected to each other. The microprocessor is programmed or configured to execute the point cloud density upsampling method based on cross-modal data registration.

[0049] In addition, this embodiment also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and the computer program is used to be programmed or configured by the microprocessor to execute the point cloud density upsampling method based on cross-modal data registration.

[0050] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for realizing the functions specified in the flow Figure 1 one or more flows and / or blocks Figure 1 These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that realizes the functions specified in the flow Figure 1 one or more flows and / or blocks Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for realizing the functions specified in the flow Figure 1 one or more flows and / or blocks Figure 1 one or more blocks.

[0051] The above are only the preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions falling within the idea of the present invention belong to the protection scope of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements should also be regarded as within the protection scope of the present invention.

Claims

1. A point cloud density upsampling method based on cross-modal data registration, characterized in that, Including: S101, acquiring three-dimensional point cloud data of a target object and images from multiple perspectives; S102, respectively extracting features of images from multiple perspectives, registering the image features from multiple perspectives, and calculating the spatial association between the image and the image features at any perspective; S103, constructing a three-dimensional point cloud skeleton model through the spatial association between the image features of any two perspectives; S104, extracting an original three-dimensional skeleton model from the three-dimensional point cloud data, and overlapping the extracted original three-dimensional skeleton model with the constructed three-dimensional point cloud skeleton model and the original three-dimensional skeleton model to obtain an overlapping skeleton model; S105, for the points in the overlapping skeleton model, reversely obtaining the pixel points in the corresponding perspective image, and copying and expanding the obtained pixel points according to a specified multiple, finally obtaining a complete upsampling result of the three-dimensional point cloud density; Step S103 includes: S201, taking the distance between the image I i and the image feature F i as the association weight between the image and the feature at the viewing angle i, aggregating the image I at the viewing angle i and the image feature F i and constructing the binary data of the images and features at each viewing angle i ; ; S202, perform three-dimensional skeleton model reconstruction on the binary data of each perspective image and its features where each point in the image I i and the image feature F i under any perspective i is used as a point in the three-dimensional skeleton to obtain the finally reconstructed overlapping skeleton model .

2. The method for upsampling point cloud density based on cross-modal data registration according to claim 1, wherein The three-dimensional point cloud data of the target object collected in step S101 is a sparse point cloud model .

3. The method for upsampling point cloud density based on cross-modal data registration according to claim 1, wherein When collecting images from multiple perspectives in step S101, the included angle between adjacent perspectives is fixed, and all perspectives cover a 360-degree range of the target object, and finally images I1 to I of n perspectives are obtained n .

4. The method for upsampling point cloud density based on cross-modal data registration according to claim 1, wherein Calculating the spatial correlation between the image and the image features at any perspective in step S102 means calculating the distance between the image I i and the image feature F i as the spatial correlation. ​ 5. The method for upsampling point cloud density based on cross-modal data registration according to claim 4, wherein The distance between the image I i and the image feature F i at any arbitrary viewing angle i refers to the Euclidean distance.

6. The method for upsampling point cloud density based on cross-modal data registration according to claim 1, wherein In step S104, the overlap of the extracted original three-dimensional skeleton model with the constructed three-dimensional point cloud skeleton model and the original three-dimensional skeleton model means the superposition of the extracted original three-dimensional skeleton model with the constructed three-dimensional point cloud skeleton model and the original three-dimensional skeleton model in three-dimensional space to obtain an overlapping skeleton model .

7. The method for upsampling point cloud density based on cross-modal data registration according to claim 1, characterized in that Step S105 includes: S301, for each point in the overlapping skeleton model , inversely obtain the pixel points in the image corresponding to the perspective and the adjacent 8 pixel points , , , , , , and ; S302, assign the coordinate z value of the nine pixel points, namely the pixel point in the image of the corresponding perspective and the adjacent eight pixel points , , , , , , and to the point , and expand the coordinate z value of the point to the coordinate points of 9 point clouds. Finally, expand the overlapping skeleton model by 9 times to obtain a densified point cloud model as the final complete three-dimensional point cloud density upsampling result.

8. A point cloud density upsampling system based on cross-modal data registration, comprising a microprocessor and a memory connected to each other, characterized in that, The microprocessor is programmed or configured to execute the point cloud density upsampling method based on cross-modal data registration according to any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that, The computer program is used to be programmed or configured by the microprocessor to execute the point cloud density upsampling method based on cross-modal data registration according to any one of claims 1 to 7.

Images