Point cloud feature enhancement method and device based on multi-scale fusion

Abstract

The present invention provides a point cloud feature enhancement method and device based on multi-scale fusion. The method includes: constructing at least one sub-point cloud of a target object of different scales based on the original point cloud of the target object; wherein the number of points in the sub-point cloud of different scales is less than the number of points in the original point cloud; determining the neighborhood points of each point in at least one sub-point cloud of different scales based on the number of pre-set neighborhood points; determining the local features of each point in at least one sub-point cloud of different scales based on the determined neighborhood points; determining the global features of each sub-point cloud based on the local features of all points in each sub-point cloud of at least one sub-point cloud of different scales, and fusing the original features of each point in the original point cloud based on the global features of each sub-point cloud of at least one sub-point cloud of different scales, to obtain a more expressive target point cloud. The present invention can enhance the feature description capability of point clouds.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a point cloud feature enhancement method and device based on multi-scale fusion. Background Art

[0002] With the advancement of scanning technology, point cloud data has increasingly become the main data form in the field of autonomous driving and drone remote sensing mapping. Unlike traditional two-dimensional image data, point cloud data is a collection of points that express the surface features of the target object and the spatial distribution of the xyz three-dimensional position coordinates in the same spatial reference system. Because it can provide accurate spatial position coordinate information, point cloud data is also widely used in many fields such as electricity, construction, industry, automobiles, games, criminal investigation, etc., and has important use value and research significance.

[0003] However, compared with two-dimensional image data, point cloud data has sparse points and disordered storage format, which makes point cloud data lack the ability to describe features during use, making it difficult to be applied to three-dimensional object recognition and scene segmentation. Therefore, research on enhancing point cloud feature description has become an urgent problem to be solved.

[0004] The existing processing methods for point cloud feature enhancement mainly include: 1. Rotation transformation of point cloud data (mainly rotation and translation operations on the coordinates of each point in the point cloud); 2. Random noise perturbation of point cloud data (mainly adding a small Gaussian noise increment to the coordinates of each point in the point cloud in the three dimensions of xyz). However, the existing processing methods for point cloud feature enhancement, due to the relatively simple data enhancement method, only perform simple transformation processing on the coordinates, and only retain the enhanced data, essentially do not expand the data scale, and therefore are insufficient to improve the expression of point cloud diversity, and cannot describe point cloud features under different point densities. Summary of the invention

[0005] The present invention provides a point cloud feature enhancement method and device based on multi-scale fusion, which is used to solve the defects of the prior art that the point cloud data enhancement method is single, the expression of point cloud diversity is insufficient, and the point cloud features under different point densities cannot be described. The feature description capability of the point cloud can be enhanced by fusing the multi-dimensional global features of the point cloud at different scales with the original features of the point cloud.

[0006] In a first aspect, the present invention provides a point cloud feature enhancement method based on multi-scale fusion, comprising:

[0007] Based on the original point cloud of the target object, construct at least one sub-point cloud of the target object of different scales; wherein the number of points in the sub-point clouds of different scales is less than the number of points in the original point cloud;

[0008] Based on a preset number of neighborhood points, determining the neighborhood points of each point in the at least one sub-point cloud of different scales respectively;

[0009] Based on the determined neighborhood points, respectively determine the local features of each point in the at least one sub-point cloud of different scales;

[0010] Based on the local features of all points in each sub-point cloud of the at least one sub-point cloud of different scale, the global features of each sub-point cloud are determined; based on the determined global features of each sub-point cloud of the at least one sub-point cloud of different scale, the original features of each point in the original point cloud are fused to obtain a more expressive target point cloud.

[0011] According to the point cloud feature enhancement method based on multi-scale fusion provided by the present invention, the method of constructing at least one sub-point cloud of a target object of different scales based on the original point cloud of the target object includes:

[0012] Based on the farthest point sampling, determine the sampling points from the original point cloud to construct a sampling point set;

[0013] It is determined that the number of sampling points in the sampling point set reaches a preset number of points in the sub-point cloud, and the sub-point cloud is constructed based on the sampling point set.

[0014] According to the point cloud feature enhancement method based on multi-scale fusion provided by the present invention, the neighborhood points of each point in the at least one sub-point cloud of different scales are determined based on the preset number of neighborhood points, and for each point in the at least one sub-point cloud of different scales, the method includes:

[0015] Determine the distance between it and other points in the sub-point cloud;

[0016] Sort the distances between the determined point and other points in the sub-point cloud in ascending order;

[0017] Based on the distance sorting, the point whose distance is closest to the point and meets the number of neighborhood points is selected as the neighborhood point.

[0018] According to the point cloud feature enhancement method based on multi-scale fusion provided by the present invention, the local features of each point in the at least one sub-point cloud of different scales are determined based on the determined neighborhood points, including:

[0019] By using a graph convolutional network, the neighborhood points of each point in the at least one sub-point cloud of different scales are processed respectively to obtain a first feature of each point;

[0020] The first feature of each point in the at least one sub-point cloud of different scales is processed respectively through the maximum pooling layer to obtain the local feature of each point.

[0021] According to the point cloud feature enhancement method based on multi-scale fusion provided by the present invention, the global features of each sub-point cloud are determined based on the local features of all points in each sub-point cloud of at least one sub-point cloud of different scales, and the original features of each point in the original point cloud are fused based on the determined local features of each point in the at least one sub-point cloud of different scales to obtain a more expressive target point cloud, including:

[0022] For each sub-point cloud in the at least one sub-point cloud of different scales, the local features of all points are processed by a maximum pooling layer to obtain global features of the sub-point cloud;

[0023] The global features of each sub-point cloud in the at least one sub-point cloud of different scales are respectively fused with the original features of each point in the original point cloud to obtain the more expressive target point cloud.

[0024] According to the point cloud feature enhancement method based on multi-scale fusion provided by the present invention, the neighborhood points of each point in the at least one sub-point cloud of different scales are determined based on the preset number of neighborhood points, including:

[0025] Based on a preset number of neighborhood points, determining the neighborhood points of each point in the at least one sub-point cloud of different scales and the original point cloud respectively;

[0026] The determining, based on the determined neighborhood points, respectively the local features of each point of the at least one sub-point cloud of different scales comprises:

[0027] Based on the determined neighborhood points, respectively determining local features of each point in the at least one sub-point cloud of different scales and the original point cloud;

[0028] The determining of the global features of each sub-point cloud based on the local features of all points of each sub-point cloud in the at least one sub-point cloud of different scales, and fusing the original features of each point in the original point cloud based on the determined global features of each sub-point cloud in the at least one sub-point cloud of different scales to obtain a more expressive target point cloud includes:

[0029] Based on the local features of all points in each sub-point cloud in the at least one sub-point cloud of different scale, the global features of each sub-point cloud are determined; based on the determined global features of each sub-point cloud in the at least one sub-point cloud of different scale, the local features of each point in the original point cloud are fused to obtain the more expressive target point cloud.

[0030] In a second aspect, the present invention further provides a point cloud feature enhancement device based on multi-scale fusion, comprising:

[0031] A point cloud construction module, configured to construct at least one sub-point cloud of a target object at a different scale based on an original point cloud of the target object; wherein the number of points in the sub-point clouds of different scales is less than the number of points in the original point cloud;

[0032] A neighborhood point determination module, configured to determine the neighborhood points of each point in the at least one sub-point cloud of different scales based on a preset number of neighborhood points;

[0033] A feature extraction module, configured to determine, based on the determined neighborhood points, the local features of each point in the at least one sub-point cloud of different scales;

[0034] A feature fusion module is used to determine the global features of each sub-point cloud based on the local features of all points in each sub-point cloud of at least one sub-point cloud of different scales, and to fuse the original features of each point in the original point cloud based on the determined global features of each sub-point cloud of at least one sub-point cloud of different scales to obtain a more expressive target point cloud.

[0035] In a third aspect, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the point cloud feature enhancement method based on multi-scale fusion as described above are implemented.

[0036] In a fourth aspect, the invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described point cloud feature enhancement methods based on multi-scale fusion.

[0037] In a fifth aspect, the invention further provides a computer program product having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described point cloud feature enhancement methods based on multi-scale fusion.

[0038] The point cloud feature enhancement method and device based on multi-scale fusion provided by the present invention can obtain the point cloud of the target object at different scales by constructing at least one sub-point cloud of a target object at a different scale according to the original point cloud of the target object, thereby enhancing the diversity of the point cloud at different density levels; by determining the neighborhood points of each point in at least one sub-point cloud of a different scale, determining the local features of each point in at least one sub-point cloud of a different scale according to the neighborhood points, and determining the global features of each sub-point cloud in at least one sub-point cloud of a different scale according to the local features of all points in each sub-point cloud in at least one sub-point cloud of a different scale, the multi-dimensional global features of the point cloud at different scales can be obtained; by fusing the original features of each point in the original point cloud with the global features of each sub-point cloud in at least one sub-point cloud of a different scale determined according to the neighborhood points, the information complementarity of the point cloud features at different scales can be achieved, the feature description capability of the point cloud can be enhanced, and the accuracy and robustness of recognition using the point cloud can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 It is a flow chart of a point cloud feature enhancement method based on multi-scale fusion provided by the present invention;

[0041] Figure 2 It is a schematic diagram of the process of constructing a sub-point cloud provided by the present invention;

[0042] Figure 3 It is a schematic diagram of a process for determining the neighborhood points of each point provided by the present invention;

[0043] Figure 4 It is a schematic diagram of a process of determining the local features of each point based on the neighborhood points provided by the present invention;

[0044] Figure 5 It is a schematic diagram of the process of feature fusion of point cloud provided by the present invention;

[0045] Figure 6 It is a flow chart of another point cloud feature enhancement method based on multi-scale fusion provided by the present invention;

[0046] Figure 7 is a schematic diagram of an original point cloud and sub-point clouds of different scales provided by the present invention;

[0047] Figure 8 is a schematic diagram of an application scenario of the point cloud feature enhancement method based on multi-scale fusion provided by the present invention;

[0048] Fig. 9 It is a schematic diagram of the composition structure of the point cloud feature enhancement device based on multi-scale fusion provided by the present invention;

[0049] Fig.10 It is a schematic diagram of the composition structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] Combine the following Figure 1-Figure 8 The point cloud feature enhancement method based on multi-scale fusion of the present invention is described.

[0052] See also Figure 1 , Figure 1 is a flow chart of a point cloud feature enhancement method based on multi-scale fusion provided by the present invention, Figure 1 The multi-scale fusion-based point cloud feature enhancement method shown can be performed by a multi-scale fusion-based point cloud feature enhancement device, which can be set on a client or a server. For example, the client can be a smart phone, a notebook computer, a vehicle-mounted computer, a robot, a wearable device, etc. The server can be a physical server including an independent host, a virtual server carried by a host cluster, a cloud server, etc., which is not limited in the embodiment of the present invention. Figure 1 As shown, the point cloud feature enhancement method based on multi-scale fusion at least includes:

[0053] 101. Construct at least one sub-point cloud of a target object at a different scale based on an original point cloud of the target object; wherein the number of points in the sub-point clouds at different scales is less than the number of points in the original point cloud.

[0054] In an embodiment of the present invention, the target object may be an object for obtaining a point cloud. The type of the target object may be determined according to the technical field of the point cloud application. For example, in the field of autonomous driving, the target object may be a vehicle, a pedestrian, etc., and in the field of drone remote sensing mapping, the target object may be an urban building, a farmland, etc. The embodiment of the present invention does not limit the type of the target object. The target object may be scanned by a three-dimensional laser scanner or a photographic scanner to obtain the original point cloud of the target object, or the original point cloud of the target object may be obtained from a database. The embodiment of the present invention does not limit the implementation method of obtaining the original point cloud of the target object. Optionally, after obtaining the original point cloud of the target object, the original point cloud of the target object may also be preprocessed to remove outliers, noise points, etc. in the original point cloud. The embodiment of the present invention does not limit the type and implementation method of preprocessing the original point cloud.

[0055] In an embodiment of the present invention, after obtaining the original point cloud of the target object, a sub-point cloud of the target object at a preset scale can be constructed according to the original point cloud of the target object and the pre-set scale of the sub-point cloud used to describe the target object. When the pre-set scale of the sub-point cloud used to describe the target object is multiple scales, sub-point clouds of multiple scales of the target object can be constructed. Among them, the scale is used to characterize the number of points in the point cloud, and different scales are used to characterize that the sub-point cloud has a different number of points or point density from the original point cloud relative to the original point cloud. The number of scales of the sub-point cloud used to describe the target object can be pre-set according to the application scenario of the point cloud. For example, the scale of the original point cloud of the target object is 2000 points, the pre-set scale of the sub-point cloud is 1500 points, 1000 points and 800 points, and the scales of the three sub-point clouds of the constructed target object are 1500 points, 1000 points and 800 points respectively. The method of constructing a sub-point cloud based on the original point cloud may adopt existing point cloud sampling methods, such as uniform sampling, geometric sampling, random sampling, grid sampling, etc., or may adopt a point cloud sampling method designed according to requirements, which is not limited in the embodiments of the present invention.

[0056] 102 , based on a preset number of neighborhood points, respectively determine a neighborhood point for each point in at least one sub-point cloud of different scales.

[0057] In an embodiment of the present invention, after constructing the sub-point cloud of the target object, the neighborhood points of each point in each constructed sub-point cloud can be determined respectively according to the number of preset neighborhood points. The number of neighborhood points can be preset according to factors such as the density and accuracy of the point cloud. For example, the number of neighborhood points can be 20 points, or 40 points, or 60 points. The embodiment of the present invention does not limit the number of preset neighborhood points. The method for determining the neighborhood points of each point in the sub-point cloud according to the number of preset neighborhood points can adopt the existing method for determining the neighborhood points of the point cloud, such as the K-nearest neighbor query method (KNN for short), etc., or can also adopt the method for determining the neighborhood points of the point cloud designed according to the requirements. The embodiment of the present invention does not limit the implementation method of determining the neighborhood points of each point in the sub-point cloud according to the number of preset neighborhood points.

[0058] 103 . Based on the determined neighborhood points, respectively determine a local feature of each point in at least one sub-point cloud of different scales.

[0059] In the embodiment of the present invention, after determining the neighborhood points of each point in the sub-point cloud, the local features of each point in each sub-point cloud can be determined respectively according to the determined neighborhood points of each point in each sub-point cloud. Among them, the method of determining the local features of each point in the sub-point cloud according to the neighborhood points of each point in the sub-point cloud can adopt the existing method of determining the local features according to the neighborhood points, or can also adopt the method of determining the local features according to the neighborhood points designed according to the requirements. The embodiment of the present invention does not limit the implementation method of determining the local features of each point in the sub-point cloud according to the neighborhood points of each point in the sub-point cloud.

[0060] 104. Determine the global features of each sub-point cloud based on the local features of all points in each sub-point cloud of at least one sub-point cloud of different scales, and fuse the original features of each point in the original point cloud based on the determined global features of each sub-point cloud of at least one sub-point cloud of different scales to obtain a more expressive target point cloud.

[0061] In an embodiment of the present invention, after determining the local features of each point in the sub-point cloud, the global features of each sub-point cloud can be determined according to the local features of all points in each sub-point cloud determined, and the original features of each point in the original point cloud can be fused through the global features of each sub-point cloud determined to obtain the target point cloud of the target object. Among them, the method of determining the global features of each sub-point cloud according to the local features of all points in each sub-point cloud can adopt the existing global feature determination method, or can also adopt the global feature determination method designed according to the requirements. The embodiment of the present invention does not limit the implementation method of determining the global features of each sub-point cloud according to the local features of all points in each sub-point cloud. The method of fusing the original features of each point in the original point cloud according to the global features of each sub-point cloud can adopt the existing feature fusion method, such as a multilayer perceptron (Multilayer Perceptron, MLP for short), etc., or can also adopt the feature fusion method designed according to the requirements. The embodiment of the present invention does not limit the implementation method of fusing the original features of each point in the original point cloud according to the global features of each sub-point cloud.

[0062] The point cloud feature enhancement method based on multi-scale fusion provided by an embodiment of the present invention can obtain point clouds of the target object at different scales by constructing at least one sub-point cloud of a target object at a different scale based on the original point cloud of the target object, thereby enhancing the diversity of the point cloud at different density levels; by determining the neighborhood points of each point in at least one sub-point cloud of a different scale, determining the local features of each point in at least one sub-point cloud of a different scale based on the neighborhood points, and determining the global features of each sub-point cloud in at least one sub-point cloud of a different scale based on the local features of all points in each sub-point cloud in at least one sub-point cloud of a different scale, multi-dimensional global features of the point cloud at different scales can be obtained; by fusing the original features of each point in the original point cloud with the global features of each sub-point cloud in at least one sub-point cloud of a different scale determined based on the neighborhood points, information complementarity of point cloud features at different scales can be achieved, the feature description capability of the point cloud can be enhanced, and the accuracy and robustness of recognition using the point cloud can be improved.

[0063] See also Figure 2 , Figure 2 is a schematic diagram of the process of constructing a sub-point cloud provided by the present invention, such as Figure 2 As shown, based on the original point cloud of the target object, constructing at least one sub-point cloud of a target object at a different scale includes at least:

[0064] 201, based on the farthest point sampling, determine the sampling points from the original point cloud to construct a sampling point set.

[0065] 202 , determining whether the number of sampling points in the sampling point set reaches a preset number of points in the sub-point cloud, and constructing a first sub-point cloud based on the sampling point set.

[0066] In an embodiment of the present invention, when constructing a sub-point cloud of a target object according to an original point cloud of the target object, a sampling point set of the original point cloud can be obtained by determining the sampling points according to the farthest point sampling method, and when the number of sampling points in the sampling point set reaches the number of points in the sub-point cloud that is preset, a sub-point cloud can be constructed according to the sampling point set. By presetting the number of points in multiple sub-point clouds, multiple sampling point sets with different numbers of sampling points can be obtained, thereby obtaining multiple sub-point clouds of different scales.

[0067] For example, for a sub-point cloud with a scale of l, the original point cloud P = {x1, x2, ..., x n} randomly select a sampling point x i , put into the sampling point set S l ={x i}, where x i is the 3D coordinate of the i-th point in the point cloud P, and then calculates the original point cloud after removing the sampling point x i The remaining points after the sampling point x is added to the sampling point set i The distances between them constitute a point-to-point distance set. Select the point x with the largest distance in the point-to-point distance set. i1 As a sampling point, add the sampling point set S l ={x i , x i1}, delete the sampling point x in the point pair distance set i1 Then calculate the distance of the original point cloud by removing the sampling point x i and x i1 The remaining points after the sampling point x is added to the sampling point set i1 The distance between the two points is calculated, and the point pair distance set is updated according to the minimum value principle, that is, only each remaining point and the sampling point x are retained in the updated point pair distance set. i The distance between the remaining point and the sampling point x i1 Select the point x with the largest distance in the updated point pair distance set. i2 As a sampling point, add the sampling point set S l ={x i , x i1 , x i2}, delete the sampling point x in the updated point pair distance set i2 Then repeat the process of calculating the distance and updating the point pair distance set until the sampling point set S l The number of midpoints meets the requirement of scale l S l={x i , x i1 , …, x nl}.

[0068] See also Figure 3 , Figure 3 : is a schematic diagram of the process of determining the neighborhood points of each point provided by the present invention, such as Figure 3 As shown, based on the preset number of neighborhood points, determining the neighborhood points of each point in at least one sub-point cloud of different scales respectively includes at least:

[0069] 301 , for each point in at least one sub-point cloud of different scale, respectively determine the distance between the point and other points in the sub-point cloud.

[0070] 302 , for each point in at least one sub-point cloud of different scales, sort the determined distances between the point and other points in the sub-point cloud in which the point is located in order of distance from small to large.

[0071] 303 , for each point in at least one sub-point cloud of different scales, based on the distance sorting, select the point that is closest to it and has the same number of neighboring points as the neighboring point.

[0072] In an embodiment of the present invention, when determining the neighborhood points of each point in a sub-point cloud according to a preset number of neighborhood points, the distance between each point in the sub-point cloud and other points in the sub-point cloud in which it is located can be determined according to the K nearest neighbor query method, and the distances between each point and other points in the sub-point cloud in which it is located are sorted in order from small to large. According to the sorting of the distances, the points that are closest to each point and meet the preset number of domain points are selected as the neighborhood points of each point.

[0073] For example, for any point x in the sub-point cloud i , we can calculate the point x i The distance between the point x and all other points in the sub-point cloud where it is located, and the calculated point x in ascending order i Sort the distances between the points and other points, and select the point x according to the distance sorting. i The K points closest to the point x i The domain point, construct point x i The local neighborhood R(x i ), and use this local neighborhood as the basis for determining local features in the next step.

[0074] See also Figure 4 , Figure 4 is a flow chart of determining the local features of each point based on the neighborhood points provided by the present invention, such as Figure 4As shown, based on the determined neighborhood points, local features of each point in at least one sub-point cloud of different scales are determined respectively, including at least:

[0075] 401 , respectively process the neighborhood points of each point in at least one sub-point cloud of different scales through a graph convolutional network to obtain a first feature of each point.

[0076] 402 , respectively process the first feature of each point in at least one sub-point cloud of different scales through a maximum pooling layer to obtain a local feature of each point.

[0077] In an embodiment of the present invention, when determining the local features of each point in the sub-point cloud based on the neighborhood points of each point in the sub-point cloud, the neighborhood points of each point in the sub-point cloud can be processed by a graph convolutional network (GCN for short), and then the features obtained by the graph convolutional network are processed by a maximum pooling layer to obtain the local features of each point in the sub-point cloud.

[0078] For example, for a sub-point cloud that has completed the construction of a local neighborhood, the multi-dimensional local features of each point in the sub-point cloud can be determined by combining a graph convolutional network with a maximum pooling layer. The determined multi-dimensional local features of each point can be expressed as x i ′, x can be calculated by formula 1 i ′, Formula 1 can be expressed as follows:

[0079] x i ′=max_pooling{GCN{R(x i )} (Formula 1)

[0080] See also Figure 5 , Figure 5 It is a schematic diagram of the process of feature fusion of point cloud provided by the present invention, such as Figure 5 As shown, based on the local features of all points in each sub-point cloud of at least one sub-point cloud of different scales, the global features of each sub-point cloud are determined, and based on the determined global features of each sub-point cloud of at least one sub-point cloud of different scales, the original features of each point in the original point cloud are fused to obtain a more expressive target point cloud, which at least includes:

[0081] 501 , for each sub-point cloud in at least one sub-point cloud of different scales, local features of all points are processed through a maximum pooling layer to obtain global features of the sub-point cloud.

[0082] 502 , the global features of each sub-point cloud in at least one sub-point cloud of different scales are respectively fused with the original features of each point in the original point cloud to obtain a more expressive target point cloud.

[0083] In an embodiment of the present invention, when the original features of each point in the original point cloud are fused according to the local features of each point in the determined sub-point cloud, the local features of all points in the sub-point cloud can be first processed through the maximum pooling layer for each sub-point cloud to obtain the global features of the sub-point cloud, and the global features of all sub-point clouds are merged with the original features of each point in the original point cloud. Then, the local features and the original features in the merged features are fused through a multi-layer perceptron, and the fused features are used as the enhanced features of each point in the original point cloud to obtain a feature-enhanced target point cloud. Among them, for sub-point clouds of multiple scales, global features of multiple scales can be obtained through the maximum pooling layer, and the global features of multiple scales are merged with the original features of each point in the original point cloud, which can enhance the point cloud's ability to express multi-scale features. The global features of multiple scales and the original features in the merged features are fused through the multi-layer perceptron, so that the fused features contain the point cloud's characterization of multiple scales and have stronger feature expression.

[0084] For example, for the sub-point cloud S with a scale of i after determining the local features i , the global feature f of this scale can be determined through the maximum pooling layer i , and f i Merge with the original features of each point in the original point cloud. The point cloud features after merging the original features and the global features of multiple scales can be expressed as x ie , x can be calculated by formula 2 and formula 3 ie , Formula 2 and Formula 3 can be expressed as follows:

[0085] f i =max_pooling{S i ′} (Formula 2)

[0086] x ie =concate(x i , f1, f2, …, f l ) (Formula 3)

[0087] The merged features x can be further processed by a multi-layer perceptron ie The original feature x in i and global features f1, f2, ..., f at multiple scales l Fusion is performed to obtain better feature expression. The fused point cloud feature can be expressed as x ie ′, x can be calculated by formula 4ie ′, Formula 4 can be expressed as follows:

[0088] x ie ′=MLP(x ie ) (Formula 4)

[0089] In other embodiments of the present invention, domain points are determined, local features are determined based on the domain points, and the point cloud fused with the original features of the point cloud includes not only the sub-point cloud constructed based on the original point cloud of the target object, but also the original point cloud. In this case, the point cloud feature enhancement method based on multi-scale fusion is as follows: Figure 6 As shown, the point cloud feature enhancement method based on multi-scale fusion at least includes:

[0090] 601. Construct at least one sub-point cloud of a target object at a different scale based on an original point cloud of the target object; wherein the number of points in the sub-point clouds at different scales is less than the number of points in the original point cloud.

[0091] 602 , based on a preset number of neighborhood points, determine neighborhood points of each point in at least one sub-point cloud of different scales and the original point cloud respectively.

[0092] 603 , based on the determined neighborhood points, respectively determine local features of each point in at least one sub-point cloud of different scales and the original point cloud.

[0093] 604, based on the local features of all points in each sub-point cloud of at least one sub-point cloud of different scale, determine the global features of each sub-point cloud, and based on the determined global features of each sub-point cloud of at least one sub-point cloud of different scale, fuse the local features of each point in the original point cloud to obtain a more expressive target point cloud.

[0094] In the embodiment of the present invention, the operations performed on the original point cloud in 602, 603 and 604 are the same as the operations performed on the sub-point cloud. The operations performed on the sub-point cloud in 602, 603 and 604 can be referred to in Figure 1 The descriptions about 102, 103 and 104 are given in the specification, so they will not be repeated here.

[0095] See also Figure 7 and Figure 8 As shown, Figure 7 is a schematic diagram of the original point cloud and sub-point clouds of different scales provided by the present invention, Figure 8 is a schematic diagram of an application scenario of the point cloud feature enhancement method based on multi-scale fusion provided by the present invention. Figure 7 As described above, the scale of the original point cloud may be 1024 points, and two sub-point clouds with scales of 512 points and 256 points are obtained by sampling the farthest point. Figure 8 As shown in the figure, the complete point cloud data of the input target object, i.e. the original point cloud, is output as the point cloud with enhanced features, i.e. the target point cloud, after the scale level library is constructed, the features of different scales are extracted and fused. Among them, the scale level library construction is to obtain the point cloud data corresponding to multiple scales, i.e. the sub-point cloud, from a complete input point cloud data by sampling the farthest point, and to construct a multi-scale point cloud set. The multi-scale feature extraction is to construct a local neighborhood through Knn for each scale of the point cloud data, and then obtain the local features of each point through the graph convolutional neural network and the maximum pooling on the neighborhood point channel, and finally extract the global features of multiple scales through the maximum pooling on all point channels. Feature fusion is to merge the global features corresponding to multiple scales with the original features of the point, and then fuse the merged features through a multi-layer perceptron.

[0096] The point cloud feature enhancement device based on multi-scale fusion provided by the present invention is described below. The point cloud feature enhancement device based on multi-scale fusion described below and the point cloud feature enhancement method based on multi-scale fusion described above can refer to each other.

[0097] See also Fig. 9 , Fig. 9 is a schematic diagram of the structure of a point cloud feature enhancement device based on multi-scale fusion provided by the present invention, Fig. 9 The point cloud feature enhancement device based on multi-scale fusion shown can be used to perform Figure 1 Point cloud feature enhancement methods based on multi-scale fusion, such as Fig. 9 As shown, the point cloud feature enhancement device based on multi-scale fusion at least includes:

[0098] The point cloud construction module 910 is used to construct at least one sub-point cloud of a target object at a different scale based on the original point cloud of the target object; wherein the number of points in the sub-point clouds of different scales is less than the number of points in the original point cloud.

[0099] The neighborhood point determination module 920 is used to determine the neighborhood points of each point in at least one sub-point cloud of different scales based on a preset number of neighborhood points.

[0100] The feature extraction module 930 is used to determine the local features of each point in at least one sub-point cloud of different scales based on the determined neighborhood points.

[0101] The feature fusion module 940 is used to determine the global features of each sub-point cloud based on the local features of all points in each sub-point cloud of at least one sub-point cloud of different scales, and to fuse the original features of each point in the original point cloud based on the determined global features of each sub-point cloud of at least one sub-point cloud of different scales to obtain a more expressive target point cloud.

[0102] Optionally, the point cloud construction module 910 includes:

[0103] The sampling unit is used to determine the sampling points from the original point cloud based on the farthest point sampling to construct a sampling point set.

[0104] The point cloud construction unit is used to determine whether the number of sampling points in the sampling point set reaches the number of points in the pre-set sub-point cloud, and to construct the sub-point cloud based on the sampling point set.

[0105] Optionally, the neighborhood point determination module 920 includes:

[0106] The distance calculation unit is used to determine, for each point in at least one sub-point cloud of different scales, the distance between the point and other points in the sub-point cloud.

[0107] The sorting unit is used to sort, for each point in at least one sub-point cloud of different scales, the determined distances between the point and other points in the sub-point cloud.

[0108] The neighborhood point selection unit is used to select, for each point in at least one sub-point cloud of different scales, points that are closest to it and meet the number of neighborhood points as neighborhood points.

[0109] Optionally, the feature extraction module 930 includes:

[0110] The graph convolutional network is used to process the neighborhood points of each point in at least one sub-point cloud of different scales through the graph convolutional network to obtain the first feature of each point.

[0111] The maximum pooling layer is used to process the first feature of each point in at least one sub-point cloud of different scales to obtain the local feature of each point.

[0112] Optionally, the feature fusion module 940 includes

[0113] The maximum pooling layer is used to process the local features of all points in each sub-point cloud of at least one sub-point cloud of different scales through the maximum pooling layer to obtain the global features of the sub-point cloud.

[0114] The feature fusion unit is used to fuse the global features of each sub-point cloud in at least one sub-point cloud of different scale with the original features of each point in the original point cloud to obtain a more expressive target point cloud.

[0115] Optionally, the neighborhood point determination module 920 is used to determine the neighborhood points of each point in the original point cloud and at least one sub-point cloud of different scales based on a preset number of neighborhood points.

[0116] The feature extraction module 930 is used to respectively determine the local features of each point in at least one sub-point cloud of different scales and the original point cloud.

[0117] The feature fusion module 940 is used to determine the global features of each sub-point cloud based on the local features of all points in each sub-point cloud of at least one sub-point cloud of different scales, and to fuse the local features of each point in the original point cloud based on the determined global features of each sub-point cloud of at least one sub-point cloud of different scales to obtain a more expressive target point cloud.

[0118] Fig.10 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.10 As shown, the electronic device may include: a processor 1010, a communication interface 1020, a memory 1030 and a communication bus 1040, wherein the processor 1010, the communication interface 1020, and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may call the logic instructions in the memory 1030 to execute a point cloud feature enhancement method based on multi-scale fusion, the method comprising:

[0119] Based on the original point cloud of the target object, construct at least one sub-point cloud of the target object of different scales; wherein the number of points in the sub-point clouds of different scales is less than the number of points in the original point cloud;

[0120] Based on the original point cloud of the target object, construct at least one sub-point cloud of the target object of different scales; wherein the number of points in the sub-point clouds of different scales is less than the number of points in the original point cloud;

[0121] Based on a preset number of neighborhood points, determining the neighborhood points of each point in the at least one sub-point cloud of different scales respectively;

[0122] Based on the determined neighborhood points, respectively determine the local features of each point in the at least one sub-point cloud of different scales;

[0123] Based on the local features of all points in each sub-point cloud of the at least one sub-point cloud of different scale, the global features of each sub-point cloud are determined; based on the determined global features of each sub-point cloud of the at least one sub-point cloud of different scale, the original features of each point in the original point cloud are fused to obtain a more expressive target point cloud.

[0124] In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0125] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, and when the program instructions are executed by a computer, the computer can execute the point cloud feature enhancement method based on multi-scale fusion provided by the above-mentioned method embodiments, the method comprising:

[0126] Based on the original point cloud of the target object, construct at least one sub-point cloud of the target object of different scales; wherein the number of points in the sub-point clouds of different scales is less than the number of points in the original point cloud;

[0127] Based on a preset number of neighborhood points, determining the neighborhood points of each point in the at least one sub-point cloud of different scales respectively;

[0128] Based on the determined neighborhood points, respectively determine the local features of each point in the at least one sub-point cloud of different scales;

[0129] Based on the local features of all points in each sub-point cloud of the at least one sub-point cloud of different scale, the global features of each sub-point cloud are determined; based on the determined global features of each sub-point cloud of the at least one sub-point cloud of different scale, the original features of each point in the original point cloud are fused to obtain a more expressive target point cloud.

[0130] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the point cloud feature enhancement method based on multi-scale fusion provided in the above-mentioned method embodiments, the method comprising:

[0131] Based on the original point cloud of the target object, construct at least one sub-point cloud of the target object of different scales; wherein the number of points in the sub-point clouds of different scales is less than the number of points in the original point cloud;

[0132] Based on a preset number of neighborhood points, determining the neighborhood points of each point in the at least one sub-point cloud of different scales respectively;

[0133] Based on the determined neighborhood points, respectively determine the local features of each point in the at least one sub-point cloud of different scales;

[0134] Based on the local features of all points in each sub-point cloud of the at least one sub-point cloud of different scale, the global features of each sub-point cloud are determined; based on the determined global features of each sub-point cloud of the at least one sub-point cloud of different scale, the original features of each point in the original point cloud are fused to obtain a more expressive target point cloud.

[0135] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0136] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A point cloud feature enhancement method based on multi-scale fusion, characterized in that: include: Based on the original point cloud of the target object, construct at least one sub-point cloud of the target object with different scales; wherein the number of points in the sub-point cloud with different scales is less than the number of points in the original point cloud, and the different scales are used to characterize that the sub-point cloud has a different number of points or point density than the original point cloud; Based on a preset number of neighborhood points, determining the neighborhood points of each point in the at least one sub-point cloud of different scales respectively; Based on the determined neighborhood points, respectively determine the local features of each point in the at least one sub-point cloud of different scales; Determine the global features of each of the sub-point clouds based on the local features of all points in each of the at least one sub-point cloud of different scales, and fuse the original features of each point in the original point cloud based on the determined global features of each of the at least one sub-point cloud of different scales to obtain a more expressive target point cloud; Wherein, based on the determined neighborhood points, respectively determining the local features of each point in the at least one sub-point cloud of different scales includes: By using a graph convolutional network, the neighborhood points of each point in the at least one sub-point cloud of different scales are processed respectively to obtain a first feature of each point; The first feature of each point in the at least one sub-point cloud of different scales is processed respectively through the maximum pooling layer to obtain the local feature of each point.

2. The point cloud feature enhancement method based on multi-scale fusion according to claim 1, characterized in that: The step of constructing at least one sub-point cloud of a target object of different scales based on the original point cloud of the target object includes: Based on the farthest point sampling, determine the sampling points from the original point cloud to construct a sampling point set; It is determined that the number of sampling points in the sampling point set reaches a preset number of points in the sub-point cloud, and the sub-point cloud is constructed based on the sampling point set.

3. The point cloud feature enhancement method based on multi-scale fusion according to claim 1, characterized in that: The determining of the neighborhood points of each point in the at least one sub-point cloud of different scales based on the preset number of neighborhood points, for each point in the at least one sub-point cloud of different scales, comprises: Determine the distance between it and other points in the sub-point cloud; Sort the distances between the determined point and other points in the sub-point cloud in ascending order; Based on the sorting of the distances, the points whose distances are closest to the points and meet the number of the neighborhood points are selected as neighborhood points.

4. The point cloud feature enhancement method based on multi-scale fusion according to claim 1, characterized in that: The determining of the global features of each sub-point cloud based on the local features of all points of each sub-point cloud in the at least one sub-point cloud of different scales, and fusing the original features of each point in the original point cloud based on the determined global features of each sub-point cloud in the at least one sub-point cloud of different scales to obtain a more expressive target point cloud includes: For each sub-point cloud in the at least one sub-point cloud of different scales, the local features of all points are processed by a maximum pooling layer to obtain global features of the sub-point cloud; The global features of each sub-point cloud in the at least one sub-point cloud of different scales are respectively fused with the original features of each point in the original point cloud to obtain the more expressive target point cloud.

5. The point cloud feature enhancement method based on multi-scale fusion according to any one of claims 1 to 4, characterized in that: The determining of the neighborhood points of each point in the at least one sub-point cloud of different scales based on the preset number of neighborhood points includes: Based on a preset number of neighborhood points, determining the neighborhood points of each point in the at least one sub-point cloud of different scales and the original point cloud respectively; The determining, based on the determined neighborhood points, respectively the local features of each point of the at least one sub-point cloud of different scales comprises: Based on the determined neighborhood points, respectively determining local features of each point in the at least one sub-point cloud of different scales and the original point cloud; The determining of the global features of each sub-point cloud based on the local features of all points of each sub-point cloud in the at least one sub-point cloud of different scales, and fusing the original features of each point in the original point cloud based on the determined global features of each sub-point cloud in the at least one sub-point cloud of different scales to obtain a more expressive target point cloud includes: Based on the local features of all points in each sub-point cloud of the at least one sub-point cloud of different scale, the global features of each sub-point cloud are determined; based on the determined global features of each sub-point cloud of the at least one sub-point cloud of different scale, the local features of each point in the original point cloud are fused to obtain the more expressive target point cloud.

6. A point cloud feature enhancement device based on multi-scale fusion, characterized in that: include: A point cloud construction module, configured to construct at least one sub-point cloud of a target object at a different scale based on an original point cloud of the target object; wherein the number of points in the sub-point cloud of the different scale is less than the number of points in the original point cloud, and the different scales are used to characterize that the sub-point cloud has a different number of points or point density than the original point cloud; A neighborhood point determination module, configured to determine the neighborhood points of each point in the at least one sub-point cloud of different scales based on a preset number of neighborhood points; A feature extraction module, configured to determine, based on the determined neighborhood points, the local features of each point in the at least one sub-point cloud of different scales; A feature fusion module, configured to determine the global features of each sub-point cloud based on the local features of all points in each sub-point cloud of the at least one sub-point cloud of different scales, and fuse the original features of each point in the original point cloud based on the determined global features of each sub-point cloud of the at least one sub-point cloud of different scales, so as to obtain a more expressive target point cloud; Among them, the feature extraction module includes: A graph convolutional network, used for processing the neighborhood points of each point in at least one sub-point cloud of different scales through the graph convolutional network to obtain a first feature of each point; The maximum pooling layer is used to process the first feature of each point in at least one sub-point cloud of different scales to obtain the local feature of each point.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the point cloud feature enhancement method based on multi-scale fusion as described in any one of claims 1 to 5 are implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the point cloud feature enhancement method based on multi-scale fusion as claimed in any one of claims 1 to 5 are implemented.

9. A computer program product having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the point cloud feature enhancement method based on multi-scale fusion as claimed in any one of claims 1 to 5 are implemented.

Images