A three-dimensional space object detection method based on scattered point cloud spectrum voting learning

Abstract

The application discloses a three-dimensional space object detection method based on a scattered point cloud spectrum voting learning, and comprises the following steps: step S1, scattered point cloud data is acquired through laser radar scanning; step S2, the point cloud data acquired in the step S1 is taken as the input of a spectrum voting network, wherein the spectrum voting network comprises a spectrum graph convolution module, a down-sampling module and a voting module; step S3, after the point cloud data is input into the spectrum voting network, feature extraction is carried out through the spectrum graph convolution module; step S4, the features extracted through the spectrum graph convolution module are taken as the input of the down-sampling module; step S5, the output of the down-sampling module is taken as the input of the voting module; step S6, the output of the spectrum voting network is clustered through sampling grouping; step S7, the voting clustering is taken as the input of a target generation module, and a target proposal is generated; and step S8, non-maximum suppression processing is carried out on the target proposal, and a three-dimensional object detection result is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning and three-dimensional point cloud target detection, and particularly relates to a three-dimensional space object detection method based on scattered point cloud spectrum voting learning. BACKGROUND

[0002] Nowadays, the society develops rapidly in science and technology, and each industry is changing day by day. With the rapid rise of artificial intelligence, computer vision has become a field of great concern in the community. In the field of computer vision, target detection is an indispensable link and is widely used in various industries.

[0003] In the target detection of traditional RGB images, Mono3D is relatively representative, but Mono3D can only obtain good detection results in the case of inputting monocular images. Then, Mousavian et al. proposed a method of using convolutional neural network for target detection in monocular images, which made three-dimensional target detection based on monocular images reach a new level. However, due to the lack of depth spatial information in RGB images, the detection effect is affected. Then, Deng et al. extracted two-dimensional feature representation from RGB-D images and added a description of spatial information. Then, through data reconstruction, the two-dimensional target detection result is converted into a three-dimensional target detection result. The target class is learned based on two-dimensional feature learning and the three-dimensional regression compensation estimates the target spatial information. However, the error between the calculated spatial information and the actual spatial information still affects the detection effect.

[0004] Compared with traditional RGB images, point cloud data contains color information and three-dimensional coordinates of the target. In three-dimensional target detection based on point cloud, MV3D proposed by Xiaozhi Chen et al. encodes three-dimensional point cloud data into a multi-view form for input, which indirectly converts the three-dimensional problem into a two-dimensional problem, sacrificing part of the geometric details of the point cloud. Then, Charles R. Qi et al. proposed VoteNet, which uses PointNet++ as the backbone to learn and extract point cloud data features, which leads to an independent feature learning process between points, reducing the correlation between neighborhoods. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a three-dimensional space object detection method based on scattered point cloud spectrum voting learning, which improves the neighborhood correlation of the feature learning process and improves the accuracy of the detection effect under the premise of ensuring the network training speed.

[0006] To solve the above technical problems, the technical scheme adopted by the present application is as follows: a three-dimensional space object detection method based on scattered point cloud spectrum voting learning, comprising the following steps:

[0007] Step S1: Acquire scattered point cloud data by scanning with lidar;

[0008] Step S2: Use the point cloud data obtained in step S1 as input to the spectral voting network, which includes a spectral convolution module, a downsampling module, and a voting module.

[0009] Step S3: After the point cloud data is input into the spectral voting network, feature extraction is performed through the spectral graph convolution module;

[0010] Step S4: The features extracted by the spectral convolution module are used as the input to the downsampling module;

[0011] Step S5: Use the output of the downsampling module as the input of the voting module;

[0012] Step S6: Cluster the output of the spectral voting network by sampling groups;

[0013] Step S7: Use the voting clustering as input to the target generation module to generate the target proposal;

[0014] Step S8: Perform nonmaximum suppression processing on the target proposal to obtain the 3D object detection results.

[0015] A further improvement to the technical solution of the present invention is that the feature extraction process of the spectral convolution module in step S3 is as follows:

[0016] In the spatial domain, convolution operations are described as follows:

[0017]

[0018] Where X represents the input point set information, g represents the spatial convolution kernel, and h represents the result of X after being filtered by g;

[0019] Analogous to the spectral domain:

[0020]

[0021] in, Graph Fourier transform representing click features This represents the filter kernel in the graphical Fourier domain. This represents the result after filtering;

[0022] Given k points x1, x2, ..., xk in a local neighborhood k The set of elements, which form a graph G. k In order to calculate graph G k The graph Fourier transform of any point in the graph defines a graph Laplace matrix:

[0023] L = ID1 / 2 WD 1 / 2 , (3)

[0024] where I is the identity matrix, D is a diagonal matrix, and D ii =∑ j W ij ; W is the adjacency matrix of graph G k , and contains W ij =dist(x i ,x j );

[0025] It can be seen that L is a real symmetric semi-definite matrix, and its complete standard orthogonal eigenvectors are:

[0026] U=[u0,u1,…,u k-1 ]∈R k×k , (4)

[0027] and the diagonal matrix composed of eigenvalues:

[0028] Λ=diag([λ0,λ1,…,λ k-1 ])∈R k×k , (5)

[0029] Therefore, L=UΛU T , and the graph Fourier transform of X can be defined as The inverse operation is

[0030] From the above change, if the graph convolution output at this time is The graph signal filtered by the filter g is represented as:

[0031]

[0032] From equation (6), it can be seen that in the process of spectral graph convolution, k points in the local neighborhood are treated as a whole.

[0033] Further improvement of the technical scheme of the application is that the downsampling module in step S4 includes two downsampling layers.

[0034] Further improvement of the technical scheme of the application is that in the voting module in step S5, the input features are recorded as spatial information x and feature information f, which are processed by MLP to output spatial offset Δx and feature offset Δf, and the generated vote is V:

[0035] V=[y;g], (7)

[0036] Where y=x+Δx; g=f+Δf.

[0037] A further improvement to the technical solution of the present invention is that: the specific process of step S6 is as follows: during the sampling grouping process, from a group of votes V i =[y i g i In the context of ], i = 1, ..., n, the use of {y} in 3D Euclidean space i The farthest point sampling of} samples a subset of K votes to obtain {V ik}, then through each V ik Cluster C is formed by grouping adjacent 3D locations through voting. i C i That is, V i The clustering output.

[0038] A further improvement to the technical solution of the present invention lies in the following: the specific process of step S7 is as follows: in the target generation module, the clustering output C = {C} obtained from the sampling group is obtained. i If i = 1, ..., n, then the generated target proposal is P(C):

[0039] P(C)=MLP2{max{MLP1[C]}}, (8)

[0040] Where MLP1 is for each C i The information from MLP1 is processed separately, then max-pooled and passed to MLP2, where it is further combined and processed.

[0041] The technological advancements achieved by this invention due to the adoption of the above technical solutions are as follows:

[0042] This invention constructs a 3D point cloud object detection network based on spectral graph convolution. This network model incorporates a spectral graph convolution model on top of VoteNet to enhance its point cloud feature learning capabilities. Through spectral graph convolution, the neighborhood relevance of the backbone network during point cloud feature learning is improved. Furthermore, by employing Hough voting and proposal classification, detection accuracy can be improved with a slight impact on detection efficiency. Compared to VoteNet, the network in this invention achieves better detection results. Attached Figure Description

[0043] Figure 1 This invention relates to a network structure for a three-dimensional spatial object detection method based on scattered point cloud spectrum voting learning.

[0044] Figure 2 This is a schematic diagram of the feature extraction process in the traditional MLP method;

[0045] Figure 3 This is a schematic diagram of the spectral convolution feature extraction process of the present invention. Detailed Implementation

[0046] The present invention will be further described in detail below with reference to embodiments:

[0047] like Figure 1 As shown, a 3D spatial object detection method based on scattered point cloud spectrum voting learning includes the following steps:

[0048] Step S1: Acquire scattered point cloud data by scanning with lidar;

[0049] Step S2: Use the point cloud data obtained in step S1 as the input to the spectral voting network. Input N*3 scattered point cloud data, where the spectral voting network includes a spectral convolution module, a downsampling module, and a voting module.

[0050] Step S3: After the point cloud data is input into the spectral voting network, feature extraction is performed through the spectral graph convolution module;

[0051] like Figure 3 As shown, the feature extraction process of the spectral convolution module is as follows:

[0052] In the spatial domain, convolution operations are described as follows:

[0053]

[0054] Where X represents the input point set information, g represents the spatial convolution kernel, and h represents the result of X after being filtered by g;

[0055] Analogous to the spectral domain:

[0056]

[0057] in, Graph Fourier transform representing click features This represents the filter kernel in the graphical Fourier domain. This represents the result after filtering;

[0058] Given k points x1, x2, ..., xk in a local neighborhood k The set of elements, which form a graph G. k In order to calculate graph G k The graph Fourier transform of any point in the graph defines a graph Laplace matrix:

[0059] L = ID 1 / 2 WD 1 / 2 (3)

[0060] Where I is the identity matrix; D is a diagonal matrix, and D ii =∑ j W ij W is diagram G kadjacency matrix of the graph G, and contains W ij = dist(x i , x j );

[0061] It is known that L is a real symmetric semi-definite matrix, and its complete standard orthogonal eigenvectors are:

[0062] U = [u0, u1, …, u k-1 ] ∈ R k×k , (4)

[0063] and the diagonal matrix composed of eigenvalues:

[0064] Λ = diag([λ0, λ1, …, λ k-1 ]) ∈ R k×k , (5)

[0065] Therefore, L = UΛU T , and the graph Fourier transform of X can be defined as The inverse operation is

[0066] According to the above transformation, if the graph convolution output at this time is then the graph signal filtered by the filter g is represented as:

[0067]

[0068] It can be seen from equation (6) that in the process of spectral graph convolution, the k points in the local neighborhood are treated as a whole, which improves the neighborhood correlation in the process of extracting features from the point cloud, and further enhances the feature extraction capability of the network.

[0069] In the convolution layer, the spectral graph convolution model is applied in step S3, and the spectral graph convolution method is used to replace the feature extraction means in the form of traditional MLP as shown in Figure 2 By improving the neighborhood correlation in the process of learning the features of the point cloud, the spectral graph convolution has better effect on capturing the features of the local neighborhood, so it can enhance the feature extraction capability of the network for the point cloud data.

[0070] Step S4, the features extracted by the spectral graph convolution module are input into the downsampling module, and further key features are extracted and redundant feature information is reduced to improve the learning efficiency of the network. The downsampling module includes two downsampling layers, which essentially perform maximum pooling operations.

[0071] Step S5, the output of the downsampling module is input into the voting module, and in the voting module, the MLP is used to perform Hough voting operation to realize feature aggregation. The output of the voting module is the output of the spectral voting network.

[0072] In the voting module, the input features are recorded as spatial information x and feature information f, which are processed by the MLP to output spatial offset Δx and feature offset Δf, and the voting is generated as V:

[0073] V=[y; g], (7)

[0074] Wherein, y=x+Δx; g=f+Δf.

[0075] Step S6, clustering the output of the spectral voting network by sampling grouping.

[0076] In the sampling grouping process, from a group of votes V i =[y i ;g i ], i=1,…,n, a subset of K votes is sampled in the 3D Euclidean space using the farthest point sampling based on {y i} to obtain {V ik}. Then, the clustering C ik is formed by grouping the adjacent votes of the 3D position of each V i . C i is the clustering output of V i ;

[0077] Step S7, taking the voting clustering as the input of the target generation module to generate the target proposal.

[0078] In the target generation module, the clustering output C={C i}, i=1,…,n obtained by sampling grouping is obtained. The generated target proposal P(C) is obtained as follows:

[0079] P(C)=MLP2{max{MLP1[C]}} (8)

[0080] Wherein, MLP1 processes each C i individually, and then is maximum-pooled and passed to MLP2. In MLP2, multiple information from MLP1 is further combined and processed.

[0081] Step S8, performing non-maximum suppression on the target proposal to obtain the three-dimensional object detection result.

[0082] The application constructs a three-dimensional point cloud target detection network based on spectral graph convolution. The network model adds a spectral graph convolution model on the basis of VoteNet to improve the ability of point cloud feature learning. Through the spectral graph convolution mode, the neighborhood correlation of the backbone network in the point cloud feature learning process is improved. Further, through Hough voting and proposal classification, the detection accuracy can be improved at the cost of slightly affecting the detection efficiency. Compared with VoteNet, the network in the application can obtain better detection effect.

Claims

1. A three-dimensional object detection method based on scattered point cloud spectrum voting learning, characterized in that: Includes the following steps: Step S1: Acquire scattered point cloud data by scanning with lidar; Step S2: Use the point cloud data obtained in step S1 as input to the spectral voting network, which includes a spectral convolution module, a downsampling module, and a voting module. Step S3: After the point cloud data is input into the spectral voting network, feature extraction is performed through the spectral graph convolution module; The feature extraction process performed by the spectral convolution module is as follows: In the spatial domain, convolution operations are described as follows: Where X represents the input point set information, g represents the spatial convolution kernel, and h represents the result of X after being filtered by g; Analogous to the spectral domain: in, Graph Fourier transform representing click features This represents the filter kernel in the graphical Fourier domain. This represents the result after filtering; Given k points x1, x2, ..., xk in a local neighborhood k The set of elements, which form a graph G. k In order to calculate graph G k The graph Fourier transform of any point in the graph defines a graph Laplace matrix: L=I-D 1 / 2 WD 1 / 2 , (3) Where I is the identity matrix; D is a diagonal matrix, and D ii =Σ j W ij W is diagram G k The adjacency matrix, And includes W ij =dist(x i ,x j ); Therefore, L is a real symmetric positive semi-definite matrix, and it consists of complete orthogonal eigenvectors: U=[u0,u1,…,u k-1 ]∈R k×k , (4) And a diagonal matrix composed of eigenvalues: Λ=diag([λ0,λ1,…,λ k-1 ])∈R k×k , (5) Therefore, L = UΛU T The graphical Fourier transform of X can be defined as follows: The reverse operation is Based on the above changes, if the output of the convolution of this image is... The graph signal filtered by filter g is represented as: As can be seen from equation (6), during the spectral convolution process, the k points in the local neighborhood are treated as a whole. Step S4: The features extracted by the spectral convolution module are used as the input to the downsampling module; Step S5: Use the output of the downsampling module as the input of the voting module; Step S6: Cluster the output of the spectral voting network by sampling groups; Step S7: Use the voting clustering as input to the target generation module to generate the target proposal; Step S8: Perform nonmaximum suppression processing on the target proposal to obtain the 3D object detection results.

2. The method for detecting three-dimensional objects based on scattered point cloud spectrum voting learning according to claim 1, characterized in that: The downsampling module in step S4 includes two downsampling layers.

3. The method for detecting three-dimensional objects based on scattered point cloud spectrum voting learning according to claim 1, characterized in that: In step S5, within the voting module, the input features are denoted as spatial information x and feature information f. After MLP processing, the spatial offset Δx and feature offset Δf are output, generating a vote V. V = [y; g], (7) Among them, y=x+Δx; g=f+Δf.

4. The three-dimensional spatial object detection method based on scattered point cloud spectrum voting learning according to claim 3, characterized in that: The specific process of step S6 is as follows: In the sampling grouping process, from a group of votes V i =[y i g i In the context of ], i = 1, ..., n, the use of {y} in 3D Euclidean space i The farthest point sampling of} samples a subset of K votes to obtain {V ik }, then through each V ik Cluster C is formed by grouping adjacent 3D locations through voting. i C i That is, V i The clustering output.

5. The three-dimensional spatial object detection method based on scattered point cloud spectrum voting learning according to claim 4, characterized in that: The specific process of step S7 is as follows: In the target generation module, obtain the clustering output C = {C} obtained from the sampling grouping. i If i = 1, ..., n, then the generated target proposal is P(C): P(C)=MLP2{max{MLP1[C]}}, (8) Where MLP1 is for each C i The information from MLP1 is processed separately, then max-pooled and passed to MLP2, where it is further combined and processed.

Images