A three-dimensional object detection method and system based on substream sparse convolution

Abstract

The present invention provides a three-dimensional target detection method and system based on substream sparse convolution, the method comprising: obtaining initial point cloud data of a target scene; obtaining target point cloud features based on the initial point cloud data and an optimized point cloud voxel convolutional neural network, the optimized point cloud voxel convolutional neural network is obtained by optimizing the point cloud voxel convolutional neural network through substream sparse convolution; and performing target detection on the target scene according to the target point cloud features. The embodiment of the present invention constructs a point cloud voxel convolutional neural network extraction method based on substream sparse convolution, and further extracts features by utilizing the sparsity of the point cloud, which can more efficiently perform deep extraction of local information, thereby improving the efficiency and accuracy of three-dimensional target detection.

Description

Technical Field

[0001] The present invention relates to the field of robot and computer vision technology, and in particular to a three-dimensional target detection method and system based on substream sparse convolution. Background Art

[0002] In the fields of autonomous driving and robotics, it is often necessary to directly acquire and process three-dimensional data information. Three-dimensional data information has more depth information than ordinary two-dimensional data information. This depth information can eliminate a lot of segmentation uncertainties in two-dimensional images and provide important geometric information.

[0003] Since the 1990s, 3D shape feature extraction algorithms have experienced more than 20 years of development, gradually transitioning from manual feature extraction to deep learning. There are four main ways to use deep learning for 3D: (a) multi-view-based convolution, (b) voxel-based convolution, (c) structured convolution, and (d) direct convolution on point clouds. Among them, the method based on direct processing of point clouds was first proposed in 2017 and has attracted much attention in recent years.

[0004] There are two general convolution methods used for 3D target detection: voxel-based convolution and point-based convolution.

[0005] Volume-based 3D convolution represented by VoxNet rasterizes point clouds into voxel grids, performs convolution in voxel units, and uses 3D filters to train CNNs. For voxel convolution, the degree of information loss in the process of converting point clouds into voxels is related to the size of the voxel resolution. As the voxel resolution increases, the amount of computation and memory usage increases cubically. Therefore, setting a high resolution will increase the computational cost and memory requirements, while lowering the resolution will lead to information loss. It is difficult to balance the resolution and computational cost and memory requirements. In 2017, Stanford University proposed PointNet, which allows the network to directly process point clouds. The PointNet network directly convolves each point, only extracting information from each point from low dimensions to high dimensions, and does not do any processing on the neighborhood information of the point. Subsequent improved methods require searching for neighboring points when extracting local information for the direct convolution method of point clouds, and the search for neighboring points often takes too much time.

[0006] Point-voxel convolutional neural network (PVCNN) proposes to combine the two methods, using the point-based method to extract global features and the low-resolution voxel-based method to extract local features. This method is called the PVCNN method. PVCNN solves the shortcomings of the above two convolution methods, but the detection efficiency is still not high when using the PVCNN method for target detection, and because the PVCNN method has a high memory usage, it has high hardware requirements. Summary of the invention

[0007] The present invention provides a three-dimensional target detection method and system based on substream sparse convolution, which are used to solve the defects of low efficiency and large memory space occupied by the PVCNN method in the prior art, and realize fast and efficient three-dimensional target detection.

[0008] The present invention provides a three-dimensional target detection method based on substream sparse convolution, comprising:

[0009] Obtain initial point cloud data of the target scene;

[0010] Based on the initial point cloud data and the optimized point cloud voxel convolutional neural network, the target point cloud features are obtained, and the optimized point cloud voxel convolutional neural network is obtained by optimizing the point cloud voxel convolutional neural network through substream sparse convolution;

[0011] According to the target point cloud features, target detection is performed on the target scene.

[0012] According to a three-dimensional target detection method based on substream sparse convolution provided by the present invention, the optimized point cloud voxel convolution neural network is obtained by optimizing the point cloud voxel convolution neural network by substream sparse convolution, which specifically includes:

[0013] The substream sparse convolution is used in the point cloud voxel convolutional neural network to extract local features.

[0014] According to the present invention, a three-dimensional target detection method based on substream sparse convolution is provided, wherein the target point cloud features are obtained based on the initial point cloud data and the optimized point cloud voxel convolution neural network, specifically including:

[0015] Convert the normalized initial point cloud data into voxels;

[0016] Extracting local information of the voxel using the substream sparse convolution to obtain local features;

[0017] The local features are devoxelized to obtain the target point cloud features.

[0018] According to the three-dimensional target detection method based on substream sparse convolution provided by the present invention, before converting the normalized initial point cloud data into voxels, the method further includes:

[0019] The initial point cloud data is normalized to obtain normalized initial point cloud data.

[0020] According to the present invention, a three-dimensional target detection method based on sub-stream sparse convolution is provided, and the specific calculation formula of the sub-stream sparse convolution is as follows:

[0021]

[0022] Among them, y k Represents local features, input(x i ) represents the input, K(x k ,x i ) represents the convolution kernel.

[0023] According to the three-dimensional target detection method based on substream sparse convolution provided by the present invention, the normalized initial point cloud data is converted into voxels, and the specific calculation formula is as follows:

[0024]

[0025] Where r represents the voxel resolution and Π[·] represents the normalized coordinates. Whether it belongs to the voxel grid (n,v,w) or not, f k,c Indicates that it corresponds to The relevant c-th channel feature, N u,v,w represents the normalization factor, (u,v,w) represents the normalized coordinates Coordinates that fall into the voxel grid.

[0026] The present invention also provides a three-dimensional target detection system based on substream sparse convolution, comprising:

[0027] Point cloud acquisition module, used to obtain the initial point cloud data of the target scene;

[0028] A feature extraction module, used for obtaining target point cloud features based on the initial point cloud data and the optimized point cloud voxel convolutional neural network, wherein the optimized point cloud voxel convolutional neural network is obtained by optimizing the point cloud voxel convolutional neural network through substream sparse convolution;

[0029] The target detection module is used to perform target detection on the target scene according to the target point cloud features.

[0030] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-described three-dimensional target detection methods based on substream sparse convolution are implemented.

[0031] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described three-dimensional target detection methods based on substream sparse convolution.

[0032] An embodiment of the present invention provides a three-dimensional target detection method and system based on substream sparse convolution, constructs a point cloud voxel convolution neural network extraction method based on substream sparse convolution, and uses the sparsity of the point cloud to perform further feature extraction, which can more efficiently perform deep extraction of local information. Compared with the original PVCNN method, the optimized PVCNN method can not only ensure time efficiency but also take into account its accuracy when processing highly complex and extremely challenging large-scene point cloud data, thereby improving the efficiency and accuracy of three-dimensional target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 A flowchart of a three-dimensional target detection method based on substream sparse convolution provided by the present invention;

[0035] Figure 2 Schematic diagram of the process of feature extraction using the PVCNN method in an embodiment of the present invention;

[0036] Figure 3 Schematic diagram of feature extraction process of substream sparse convolution in an embodiment of the present invention;

[0037] Figure 4 A schematic diagram of the structure of a three-dimensional object detection system based on substream sparse convolution provided by the present invention;

[0038] Figure 5 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] The embodiments of the present invention aim to solve the problems that the PVCNN method has poor performance in the local feature extraction part and has high time efficiency and GPU memory usage, which leads to low target detection efficiency and low accuracy.

[0041] Referring to substream sparse convolution, a faster and less memory-intensive PVCNN improvement method is proposed. The substream sparse convolution method that updates the hash table and feature matrix is ​​used to replace the simple voxel convolution in the existing PVCNN method, thereby achieving fast and effective feature extraction of large scenes and sparse point clouds, which can be applied to target detection.

[0042] In an embodiment of the present invention, substream sparse convolution is used to replace the existing local feature fusion method, and a substream sparse convolution improved PVCNN method is proposed to efficiently extract local features from the neighborhood information of each point on the Point-based branch. Due to the superiority of the method proposed in the embodiment of the present invention, compared with the original three-dimensional network, the replaced and modified three-dimensional network can not only ensure time efficiency but also take into account its accuracy when processing highly complex and challenging large scene point cloud data.

[0043] The embodiment of the present invention provides a three-dimensional target detection method based on sub-stream sparse convolution, such as Figure 1 As shown, the method includes:

[0044] S1, obtain the initial point cloud data of the target scene;

[0045] When a target scene needs to be detected, the target scene can be a video or an image, and point cloud data related to the target scene needs to be obtained, that is, the initial point cloud data. The initial point cloud data can be collected by a depth camera or synthesized by other monocular imaging systems or binocular imaging systems. Common depth cameras include Kinect cameras and TOF cameras.

[0046] S2, based on the initial point cloud data and the optimized point cloud voxel convolutional neural network, obtaining target point cloud features, the optimized point cloud voxel convolutional neural network is obtained by optimizing the point cloud voxel convolutional neural network through substream sparse convolution;

[0047] The initial point cloud data is input into the optimized point cloud voxel convolutional neural network to obtain the target point cloud features. It should be noted that the initial point cloud data only contains the three-dimensional coordinate information of the pixel points and does not contain other additional information. The target point cloud features also contain a lot of point cloud data. In addition to the three-dimensional coordinate information, it also contains other category information. For example, the target scene is usually divided into background and foreground. The target point cloud features contain the feature information of the point category. Through further processing, it can be determined whether the point belongs to the background or foreground.

[0048] In the field of three-dimensional vision, it is often necessary to process large-scale actual point cloud data and use the network to detect three-dimensional targets. The common fusion of neighbor point information adopts sampling, grouping and fusion methods, but the neighbor point search algorithm often takes a lot of time and there is also the problem of dynamic convolution kernel.

[0049] To address this problem, an optimized point cloud voxel convolutional neural network is used in an embodiment of the present invention, and substream sparse convolution is used to replace sampling and fusion. Based on the sparsity of point clouds, an embodiment of the present invention uses substream sparse convolution to transform the PVCNN method to construct a network.

[0050] like Figure 2 As shown, the optimized PVCNN method in the embodiment of the present invention has two branches: a Point-based backbone network based on PointNet is used to extract global features, and a Voxel-based branch PVSConv improved by substream sparse convolution is used to extract local features.

[0051] The traditional PVCNN method is divided into two branches. The voxel-based branch first converts these points into a low-resolution voxel grid, aggregates neighboring points through voxel-based convolution, and then de-voxelizes them to convert them into a sparse point structure. The point-based branch extracts features for each single point. Since it does not summarize neighborhood information, it does not take up a lot of memory. The voxel-based branch is used to fuse neighborhood information and extract local features. Since voxelization and de-voxelization only scan the points once, the memory cost is reduced.

[0052] Compared with the traditional PVCNN method, the optimized PVCNN method takes into account the sparsity of point clouds, so that local feature extraction can be achieved efficiently.

[0053] S3: Perform target detection on the target scene according to the target point cloud features.

[0054] According to the target point cloud features, the target scene can be detected because it contains the category information of the point cloud.

[0055] In summary, an embodiment of the present invention provides a three-dimensional target detection method based on substream sparse convolution, constructs a point cloud voxel convolution neural network extraction method based on substream sparse convolution, and uses the sparsity of the point cloud to perform further feature extraction, which can more efficiently perform deep extraction of local information. Compared with the original PVCNN method, the optimized PVCNN method can not only ensure time efficiency but also take into account its accuracy when processing highly complex and extremely challenging large-scene point cloud data, thereby improving the efficiency and accuracy of three-dimensional target detection.

[0056] On the basis of the above embodiment, preferably, the optimized point cloud voxel convolutional neural network is obtained by optimizing the point cloud voxel convolutional neural network through substream sparse convolution, which specifically includes:

[0057] The substream sparse convolution is used in the point cloud voxel convolutional neural network to extract local features.

[0058] Specifically, in an embodiment of the present invention, substream sparse convolution is used to improve the point cloud voxel convolutional neural network. Specifically, substream sparse convolution is used to replace the voxel convolution in the point cloud voxel convolutional neural network. The original PVCNN method simply fuses the voxel blocks in the voxel convolution, and does not use sparsity for further feature extraction. In the case of local features, more efficient deep feature extraction is not performed, which leads to poor performance of the traditional PVCNN method in target detection.

[0059] In the embodiment of the present invention, substream sparse convolution is used to fully consider the sparsity of point clouds and the correlation between point clouds, improve the time efficiency of the optimized PVCNN method, reduce the memory usage of the GPU, and reduce the hardware requirements.

[0060] On the basis of the above embodiment, preferably, the step of acquiring target point cloud features based on the initial point cloud data and the optimized point cloud voxel convolutional neural network specifically includes:

[0061] Convert the normalized initial point cloud data into voxels;

[0062] Extracting local information of the voxel using the substream sparse convolution to obtain local features;

[0063] The local features are devoxelized to obtain the target point cloud features.

[0064] Specifically, the optimized point cloud voxel convolutional neural network for feature extraction includes the following steps:

[0065] 1. Voxelization: convert the normalized initial point cloud data into voxels.

[0066] After the point cloud data is converted to a voxel grid, a 3D volume convolution stack is applied to aggregate features. Similar to the regular 3D voxel convolution model, batch normalization and nonlinear activation functions are applied after each 3D convolution to de-voxelize the voxel grid using trilinear interpolation to ensure that the features mapped to each point are different.

[0067] 2. Use sub-stream sparse convolution to extract local information of voxels and obtain local features.

[0068] Substream sparse convolution improves feature fusion, further extracts local information of points, and uses substream sparse convolution to improve time efficiency and reduce the GPU memory usage of the network. After testing, the MIou result on ShapeNet changed from 85.65% to 84.32% after reducing one layer of the network. The improvement method is to further use substream sparse convolution to efficiently extract local features of points when extracting local features to replace the simple voxel convolution of the traditional PVCNN method, while reducing the number of network layers of the Point-based branch to make the network more lightweight.

[0069] 3. Devoxelize the local features to obtain the target point cloud features.

[0070] Then the local features are de-voxelized, from Figure 2 It can be seen that the local features obtained in step 2 are voxel features, which need to be converted into point clouds. Therefore, the voxel features are devoxelized to obtain the target point cloud features.

[0071] Based on the above embodiment, preferably, before converting the normalized initial point cloud data into voxels, the method further includes:

[0072] The initial point cloud data is normalized to obtain normalized initial point cloud data.

[0073] Normalization is to normalize the coordinates of the point cloud {p k}First, normalize all points to a local coordinate system with the center of gravity as the origin. Then, divide all coordinates by Max||p k ||2, normalize the point cloud to the unit sphere, then scale and transform the points to the [0, 1] interval range.

[0074] Then convert the point cloud data into voxels. The specific conversion formula is as follows:

[0075]

[0076] Where r represents the voxel and Π[·] represents the normalized coordinate. Whether it belongs to the voxel grid (n,v,w) or not, f k,c Indicates that it corresponds to The relevant c-th channel feature, N u,v,w represents the normalization factor, (u,v,w) represents the normalized coordinates Coordinates that fall into the voxel grid.

[0077] Since the voxel resolution r is set very small, it does not introduce a large memory footprint.

[0078] Based on the above embodiment, preferably, the calculation formula of the sub-stream sparse convolution is as follows:

[0079]

[0080] Among them, y k Represents local features, input(x i ) represents the input, K(x k ,x i ) represents the convolution kernel.

[0081] VSC indicates that local feature extraction is to be performed, and multiple input neighborhood information is defined as input(x i ), the convolution kernel K(x k ,x i ) performs improved sub-stream sparse convolution and finally generates the corresponding output.

[0082] When performing three-dimensional convolution, the embodiment of the present invention uses substream sparse convolution. When extracting deep local information, the present invention uses the characteristic that the point cloud itself is sparse and uses substream sparse convolution, which can greatly improve the efficiency of convolution. Substream sparse convolution is suitable for efficiently processing problems such as target detection. The traditional convolution network implementation method is optimized for data with dense grids of filling points and cannot effectively process sparse data. The traditional PVCNN method simply uses a stack of 3D volume convolutions to aggregate other information within the neighborhood during voxel convolution. The embodiment of the present invention uses substream sparse convolution to obtain deeper feature information when fusing neighborhood information, replacing general voxel convolution to achieve the desired result.

[0083] like Figure 3As shown in the figure, submanifold sparse convolutional networks (SSCNs) use submanifold sparse convolution to fix the position of the active site, and regard the voxelized result as a block voxel model. The state of the input / hidden layer is stored using a hash table and a feature sparse matrix. In order to keep the sparsity constant for many layers, the convolution will fill the empty values ​​with zeros, and perform padding to keep the size of the voxel unchanged. After the convolution, the non-activated points are set to zero to maintain the sparse characteristics of the voxels.

[0084] First, the original voxel block’s non-zero features and position information are stored in a hash table and matrix, which is called an activate site. Each row of the matrix is ​​used to record the location of the active site. Each row stores the feature value of an active site.

[0085] If the input size is a×m, it means there are a active points and each point has m eigenvalues. Perform the following operations to initialize the input matrix and input hash table, perform convolution operations in VSC operations, and dynamically create the output matrix and hash table.

[0086] The specific operation of VSC is as follows: input(x i ) contains two values: input_marix records the feature value of each point, and input_hash_table represents the input hash table. The operation of VSC is divided into initialization and dynamic calculation.

[0087] Initialization will calculate the initial feature matrix and hash table for each point. Each row of the input feature matrix is ​​used to record the position of the active point. Each row stores the eigenvalue of an active point. The size of the feature matrix is ​​a×m, which means there are a active points and each point has m eigenvalues. The hash table contains the position and row mapping of all active points. The position is a tuple of integer coordinates, and the row number represents the corresponding row in the feature matrix.

[0088] Dynamic calculation dynamically creates the output matrix and hash table, and creates a rule table for each convolution, which includes three steps:

[0089] The first step is to pad the input to prevent the convolution from changing the size of each layer. The padding size is determined by the size of the convolution kernel, and f represents the size of the convolution kernel:

[0090] padding = (f-1) / 2,

[0091] In the second step, set the value of outmatrix to zero, find the input index and output index pair from the rulebook, and get the position of the input row from the inputmatrix. The eigenvalue can also be found through the input matrix. Each row of the inputmatrix corresponds to a point with a size of 1×m. Multiply it with the convolution kernel of size m×n to get the output value outputvalue of size 1×n and write it into the outputmatrix.

[0092] The third step is to calculate the active site and discard the non-activated site. When the middle point of the output is the active site, then this output site is the active site, and the non-activated site is discarded.

[0093] The size of the tensor output by sparse convolution is the same as that of the input tensor. The output will become larger and larger. The original zero value will be affected by the value area and convoluted into a non-zero value. In order to maintain the sparsity of the original feature, the area that was previously 0 is set to zero again, that is, forced to clear. It has less computational complexity than ordinary convolution.

[0094] Unlike the PVCNN method, because the data is sparse, there is no need to use activation functions for the empty positions, so only the activation functions are used for the positions with values, which reduces some of the network consumption. Such sparse convolution is performed multiple times, and then the average pooling is performed and input to the next layer to reconstruct the voxels and perform sparse convolution.

[0095] In summary, the embodiment of the present invention adopts the optimized PVCNN method of substream sparse convolution to process local information, and realizes fast and low memory usage convolution by updating the hash table and feature matrix, so as to extract the local features of the point cloud more efficiently and deeply.

[0096] Based on the sparsity characteristics of point clouds and the optimized PVCNN method proposed in this application, voxel convolution can be effectively used to extract local features and point-based methods can be used to extract global features. This allows effective feature extraction of three-dimensional data with high time efficiency and low memory usage, thereby improving the detection efficiency of three-dimensional target detection and reducing the requirements for memory hardware.

[0097] The following is a verification of 3D object detection using the KITTI dataset.

[0098] The embodiment of the present invention conducts experiments on target detection on the driving-oriented dataset KITTI. Table 1 is a comparison table of target detection on KITTI by the embodiment of the present invention. As shown in Table 1, a validation set is constructed from the training set, and there is no data overlapping with the training set in the validation set. The size of the validation set is 3769. Like PVCNN, the MLP layer in the F-PointNet instance segmentation network is replaced by PVS-Conv, and the backbone network is kept unchanged. The model is compared with F-PointNet (whose backbone is PointNet) and F-PVCNN (whose backbone is PVCNN). This application evaluates all models 20 times and takes their average 3D accuracy (mAP).

[0099] Table 1

[0100]

[0101] According to the results in Table 1, in terms of object detection, compared with previous methods, this application achieves the best overall performance. In nine categories, the average 3D accuracy of all categories in this application is the highest in the three categories of different difficulties. In terms of time efficiency and GPU occupancy, the model is close to PointNet, 0.7 times faster than F-PVCNN, and slightly lower in GPU memory occupancy than PVCNN.

[0102] The reasons why traditional PVCNN is not ideal for target detection are: (1) There is no deeper extraction of local information on voxel convolution. (2) Outputting keypoints based on FPS will reduce the resolution of the point cloud, which is faster than downsampling of voxel convolution. Both of these reasons are due to the inability to process the original points efficiently. The reason why this application is more suitable for target detection is that the proposed sub-stream sparse convolution reduces the network layers of the point-based branch but processes better on the voxel-based branch, performs deeper feature extraction, and thus better extracts local information, which is more suitable for the three-dimensional convolution problem of small targets.

[0103] The embodiment of the present invention provides a three-dimensional object detection system based on substream sparse convolution, such as Figure 4 As shown, the system includes a point cloud acquisition module 401, a feature extraction module 402 and a scene detection module 403, wherein:

[0104] The point cloud acquisition module 401 is used to acquire initial point cloud data of the target scene;

[0105] The feature extraction module 402 is used to obtain target point cloud features based on the initial point cloud data and the optimized point cloud voxel convolutional neural network, where the optimized point cloud voxel convolutional neural network is obtained by optimizing the point cloud voxel convolutional neural network through substream sparse convolution;

[0106] The target detection module 403 is used to perform target detection on the target scene according to the target point cloud features.

[0107] This embodiment is a system embodiment corresponding to the above method. For details, please refer to the above method embodiment. This system embodiment will not be described in detail here.

[0108] An electronic device provided by an embodiment of the present invention, such as Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute a three-dimensional target detection method based on substream sparse convolution, the method comprising:

[0109] Obtain initial point cloud data of the target scene;

[0110] Based on the initial point cloud data and the optimized point cloud voxel convolutional neural network, the target point cloud features are obtained, and the optimized point cloud voxel convolutional neural network is obtained by optimizing the point cloud voxel convolutional neural network through substream sparse convolution;

[0111] According to the target point cloud features, target detection is performed on the target scene.

[0112] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0113] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer, the computer can execute a three-dimensional object detection method based on substream sparse convolution provided by the above methods, the method comprising:

[0114] Obtain initial point cloud data of the target scene;

[0115] Based on the initial point cloud data and the optimized point cloud voxel convolutional neural network, the target point cloud features are obtained, and the optimized point cloud voxel convolutional neural network is obtained by optimizing the point cloud voxel convolutional neural network through substream sparse convolution;

[0116] According to the target point cloud features, target detection is performed on the target scene.

[0117] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the above-mentioned three-dimensional target detection method based on substream sparse convolution, the method comprising:

[0118] Obtain initial point cloud data of the target scene;

[0119] Based on the initial point cloud data and the optimized point cloud voxel convolutional neural network, the target point cloud features are obtained, and the optimized point cloud voxel convolutional neural network is obtained by optimizing the point cloud voxel convolutional neural network through substream sparse convolution;

[0120] According to the target point cloud features, target detection is performed on the target scene.

[0121] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0122] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional object detection method based on substream sparse convolution, characterized in that: include: Obtain initial point cloud data of the target scene; Based on the initial point cloud data and the optimized point cloud voxel convolutional neural network, the target point cloud features are obtained, and the optimized point cloud voxel convolutional neural network is obtained by optimizing the point cloud voxel convolutional neural network through substream sparse convolution; Performing target detection on the target scene according to the target point cloud features; The optimized point cloud voxel convolutional neural network includes a first branch and a second branch, wherein the first branch is a point-based branch for extracting global features and obtaining features of each point, and the second branch is a voxel-based branch based on substream sparse convolution for extracting local features; The optimized point cloud voxel convolutional neural network is obtained by optimizing the point cloud voxel convolutional neural network through substream sparse convolution, which specifically includes: Using the substream sparse convolution in the point cloud voxel convolutional neural network to extract local features; The step of obtaining target point cloud features based on the initial point cloud data and the optimized point cloud voxel convolutional neural network specifically includes: Convert the normalized initial point cloud data into voxels; Extracting local information of the voxel using the substream sparse convolution to obtain local features; The local features are devoxelized to obtain the target point cloud features.

2. The three-dimensional object detection method based on substream sparse convolution according to claim 1 is characterized in that: Before converting the normalized initial point cloud data into voxels, the method further includes: The initial point cloud data is normalized to obtain normalized initial point cloud data.

3. The three-dimensional object detection method based on substream sparse convolution according to claim 1, characterized in that: The specific calculation formula of the sub-stream sparse convolution is as follows: Among them, y k Represents local features, input(x i ) represents the input, K(x k ,x i ) represents the convolution kernel.

4. The three-dimensional object detection method based on substream sparse convolution according to claim 1, characterized in that: The normalized initial point cloud data is converted into voxels, and the specific calculation formula is as follows: Where r represents the voxel resolution and Π[·] represents the normalized coordinates. Whether it belongs to the voxel grid (n,v,w) or not, f k,c Indicates that it corresponds to The relevant c-th channel feature, N u,v,w represents the normalization factor, (u,v,w) represents the normalized coordinates Coordinates that fall into the voxel grid.

5. A three-dimensional object detection system based on substream sparse convolution, characterized in that: The three-dimensional object detection method based on substream sparse convolution as claimed in claim 1 comprises: Point cloud acquisition module, used to obtain the initial point cloud data of the target scene; A feature extraction module, used for obtaining target point cloud features based on the initial point cloud data and the optimized point cloud voxel convolutional neural network, wherein the optimized point cloud voxel convolutional neural network is obtained by optimizing the point cloud voxel convolutional neural network through substream sparse convolution; The target detection module is used to perform target detection on the target scene according to the target point cloud features.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the three-dimensional object detection method based on substream sparse convolution as described in any one of claims 1 to 4 are implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the three-dimensional object detection method based on substream sparse convolution as claimed in any one of claims 1 to 4 are implemented.

Images