Point cloud neural network model optimization structure and electronics for network acceleration on GPU

Abstract

The present invention provides a point cloud neural network model optimization structure and electronic device for network acceleration on a GPU. In the point cloud neural network model structure: the sampling layer is divided into two stages: a distance matrix calculation stage and a point selection stage; the clustering layer selects points adjacent to the center point according to the distance matrix; and the multi-layer SA module is improved according to the farthest point sampling of the spatial distance. The present invention achieves the effect of accelerating the calculation of the neural network by changing the neural network structure and the calculation method of certain operators in the neural network. It is a relatively general acceleration of the point cloud neural network and can be adapted to a variety of point cloud neural networks.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, in particular to the field of point cloud neural network technology. Background Art

[0002] With the rapid development of artificial intelligence in recent years, neural network technology has been widely used in various fields and has also emerged in many different forms. Point cloud neural network, as one of them, can also be applied to many fields such as autonomous driving. Unlike traditional convolutional neural networks, point cloud neural networks do not work on regular grid data structures, but on irregularly arranged point sets in three-dimensional space. Therefore, the processing of point sets is a key content of point cloud neural networks.

[0003] At present, the mainstream algorithm of point cloud neural network is proposed by PointNet++. PointNet++ network processes point cloud directly in the form of point data, and completes the final target detection and other tasks by sampling and clustering points and extracting features from the clustered points. Subsequent algorithms are based on this, and the processing of point sets basically follows the processing method of PointNet++, and has been improved to varying degrees. However, it was found in the experiment that the processing part of these networks for point sets has the problem of slow calculation speed. Due to the limitations of its network structure, some operators in the neural network cannot fully utilize the computing resources of the GPU, and because some data in the network cannot be reused, the calculation speed of the neural network is slow, which is particularly obvious in the reasoning stage of the neural network.

[0004] PointNet++ provides some groundbreaking designs for point cloud neural networks. Currently, PointNet++ proposes a multi-level feature extraction structure to process point set data, which is also a common approach in point cloud neural networks based on point sets. The following will introduce the specific structure of PointNet++.

[0005] PointNet++ is an improvement of PointNet. In PointNet, the author provides a method to extract features from unordered point sets. However, one of the shortcomings of PointNet is that it cannot obtain local features, which makes it difficult to analyze complex scenes. PointNet++ improves on this basis by clustering points hierarchically and extracting features from the clustered point sets. Each layer gradually expands the scope of clustering and abstraction, so that the entire network collects feature vectors of fine structure areas from small to large.

[0006] For each layer, the author calls it the SA (Set Abstraction) module. Each module is divided into three parts, and the process is sampling, clustering, and feature extraction. What sampling needs to do is to select representative central points from the point set. In PointNet++, the author uses the method of selecting the points with the farthest relative distance based on the mutual distance between each point, and the sampled points constitute a new point set. This method is called farthest point sampling. The specific process is as follows: First, randomly select an initial point to put into the new point set, then for each point in the original point set, calculate the distance from the point to the new point set, and then select the farthest point to add to the new point set. Here, the distance from the point to the new point set is the minimum value of the distance from the point to each point in the new point set. Iterate in this way until the target number of points is sampled. In PointNet++, the distance only considers the spatial distance between points. Representative central points are selected through farthest point sampling, and these central points are sent to the next step for clustering.

[0007] Clustering is to find the points around the central point according to the set radius value and cluster them to form a new point set. In this way, several point sets in local areas are obtained. These point sets are then used for feature extraction to obtain better local features.

[0008] PointNet++ directly uses the PointNet method for feature extraction. PointNet proposes a way to extract features from unordered point sets. By passing several point sets obtained by clustering through the PointNet network, the features of these point sets can be extracted. Since these point sets are composed of points adjacent to the center point, they can better reflect local features.

[0009] PointNet++ stacks SA modules in multiple levels, which enables it to extract feature vectors of each fine structure of the entire point cloud at different scales. By continuously extracting local features and then expanding the local range, a set of global features is finally obtained for classification. PointNet++'s SA module is also a common module in current point cloud networks.

[0010] Based on PointNet++, other point cloud neural networks have been further improved, such as 3DSSD. 3DSSD improves PointNet++ and proposes a 3D object detection network that does not rely on feature propagation by modifying the sampling method of the sampling layer. The author analyzed the performance of each functional layer of the previous classic point set-based network and found that the feature propagation layer and refinement stage take up a lot of time. The function of the feature propagation layer (FP layer) is to broadcast the features of the discarded points in the SA layer so that the features of these discarded points can be retained in the next level. The refinement stage further refines the prediction results of the region. These operations take up a lot of time and limit the efficiency of the network.

[0011] Therefore, 3DSSD considers removing the FP layer and the refinement module. If the FP layer is removed directly, the point set sent to the prediction will only contain the points selected by the SA layer. Since the classic SA layer adopts the farthest point sampling, the points screened out in this way will contain a large number of background points, resulting in a loss of accuracy. Therefore, the author improved the sampling method and proposed a method for farthest point sampling based on the feature distance of the points. By combining the feature vector extracted in the feature extraction stage of the SA layer with the spatial distance between points, the distance between the points in the physical coordinates and the distance between the feature vectors of the points are considered equally, and then the farthest point sampling is performed based on the combined data of the two, so that the accuracy of the network can be maintained.

[0012] The structure of PointNet++ will have a slow calculation speed when the number of points is large. On the one hand, the farthest point sampling module in PointNet++ fails to fully utilize the computing resources of the GPU. The farthest point sampling in PointNet++ calculates the distance between points and point sets each time a point is added to the point set, and updates the distance data after each addition. Therefore, on the GPU, due to data dependency, these operations must be completed in one thread block. However, most of the current GPUs have dozens of stream processor clusters, and each stream processor cluster can process one or two thread blocks at the same time. Therefore, the farthest point sampling is run in one thread block, and as a result, most of the GPU computing resources are idle. Therefore, sampling becomes a bottleneck for its calculation speed. On the other hand, when clustering, it is necessary to select points that are within a suitable distance from the center point. At this time, the distance data between these points is required. The distance data in the sampling cannot be inherited to the clustering operation, so the distance data in the clustering operation needs to be recalculated, which causes redundancy in calculation and also affects the calculation speed of the neural network. On the other hand, although 3DSSD changes the sampling method of the sampling layer, the overall framework of the SA layer is the same as PointNet++, so it also has the above-mentioned shortcomings of PointNet++.

[0013] In addition, the prior art uses the CUDA programming model when using GPU computing, among which CUDA's thread management is an important technology related to this patent. In CUDA, threads are divided into several levels: grid, block, and thread. All threads generated by a kernel startup are collectively referred to as a grid, and all threads in the same grid share the same global memory space. At the next level, a grid is composed of multiple thread blocks, and a thread block is composed of multiple threads. Threads in the same thread block share a shared memory space. Unlike global memory, this shared memory is only visible inside the thread block, and other thread blocks cannot access the shared memory of this thread block. Since the speed of shared memory is much faster than that of global memory, a common practice in GPU computing is to first have all threads in the thread block work together to move the data needed for the calculation of this thread block from the global memory to the shared memory. When the transfer is completed, the thread starts to execute the computing task, so that only the shared memory needs to be accessed during the calculation, which greatly improves the computing speed.

[0014] The number of thread blocks that a stream processor cluster can accommodate is related to certain parameters of the thread block, such as the size of the thread block, the required shared memory size, etc. Generally speaking, a stream processor cluster can accommodate multiple thread blocks, and modern GPUs have many stream processor clusters (such as Figure 3 Contains 84 SMs), so reasonable use will greatly improve the parallelism of the program.

[0015] In the above farthest point sampling algorithm, we can see that in each iteration, a farthest point must be selected from all points. Since thread blocks cannot access each other's shared memory space, the selection operation can only be completed by one thread block. Therefore, the parallelism of the original farthest point sampling is very low, and most SMs are idle, so the parallel characteristics of the GPU cannot be fully utilized. Summary of the invention

[0016] In view of the shortcomings of the prior art mentioned above, the purpose of the present invention is to provide a point cloud neural network model structure and electronic device to fully schedule computing resources and eliminate some redundant calculations to accelerate the computing speed in the reasoning stage.

[0017] To achieve the above-mentioned purpose and other related purposes, the present invention provides a point cloud neural network model structure, in which: the sampling layer is divided into two stages: the distance matrix calculation stage and the point selection stage; the clustering layer selects the adjacent points of the center point according to the distance matrix; and the multi-layer SA module is improved according to the farthest point sampling of the spatial distance.

[0018] In one embodiment of the present invention, the distance matrix calculation stage and the point selection stage include: the distance calculation stage obtains the distance matrix between all points in a parallel manner, and then sends the distance matrix to the point selection stage for farthest point sampling, and at the same time retains the distance matrix in the memory for reuse by operators that need to use distance.

[0019] In one embodiment of the present invention, before calculating the distance matrix, the point set data used for calculation is input twice, one is the original data, and the other is the transposed data.

[0020] In one embodiment of the present invention, the transposed point set data is arranged in the order of the dimensions of block, channel, and the number of points in the input point set.

[0021] In one embodiment of the present invention, in the distance calculation stage, the distance matrix is ​​decomposed into a plurality of sub-blocks, and each thread block is responsible for one of the sub-blocks.

[0022] In one embodiment of the present invention, the selection of adjacent points of the center point based on the distance matrix includes: passing the calculation results of the distance matrix stage and the calculation results of the point selection stage to the clustering layer, and performing clustering according to the data in the distance matrix; when selecting points, the clustering layer first consults the distance matrix, and then selects the adjacent points according to the set radius.

[0023] In one embodiment of the present invention, the center point index in the point selection stage is transmitted to the clustering layer, and the distance matrix in the distance matrix stage is transmitted to the clustering layer.

[0024] In one embodiment of the present invention, the clustering layer includes a first operator and a second operator; the first operator accepts two inputs: a distance matrix and a center point index, and searches for qualified points from the distance matrix according to a set radius range; the second operator extracts the coordinates of the point from the point set according to the index of the point, and inputs them into the feature extraction part in the clustering layer.

[0025] In one embodiment of the present invention, the improvement of the multi-layer SA module according to the farthest point sampling of the spatial distance includes: referring to the sampling result of the previous layer, selecting the points arranged relatively front, so as to omit the calculation.

[0026] To achieve the above-mentioned purpose and other related purposes, the present invention also provides an electronic device, including a memory for storing a computer program; and a processor for running the computer program to implement the point cloud neural network model structure as described above.

[0027] As described above, the point cloud neural network model structure and electronic device of the present invention have the following beneficial effects:

[0028] The present invention achieves the effect of accelerating neural network calculation by changing the neural network structure and the calculation method of certain operators in the neural network. It is a relatively general acceleration of point cloud neural networks and can be adapted to a variety of point cloud neural networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 Shown is a schematic diagram of the principle of the point cloud neural network model structure in one embodiment of the present application;

[0031] Figure 2 Shown is a schematic diagram of the farthest point sampling design in the point cloud neural network model structure in one embodiment of the present application;

[0032] Figure 3 Shown is a schematic diagram of a distance matrix operator generated in a point cloud neural network model structure in an embodiment of the present application;

[0033] Figure 4 Shown is a design principle diagram of the clustering layer in the point cloud neural network model structure in one embodiment of the present application;

[0034] Figure 5 Shown is a comparison diagram of the SA module in the point cloud neural network model structure in one embodiment of the present application.

[0035] Figure 6 Shown is a principle block diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0037] The purpose of this embodiment is to provide a point cloud neural network model structure and electronic device to fully allocate computing resources and save some redundant calculations to accelerate the computing speed in the reasoning stage.

[0038] This embodiment, based on PointNet++ and 3DSSD, proposes an optimized point cloud neural network structure, which fully schedules computing resources and eliminates some redundant calculations to speed up the calculation speed in the reasoning stage. This patent mainly improves the SA layer of PointNet++ and 3DSSD, and improves the performance of the SA layer by modifying the calculation method of the sampling layer and the data flow between the sampling layer and the clustering layer, as well as modifying the SA layer structure. Because the SA layer is a common practice in point cloud neural networks based on point sets, this embodiment provides a set of general optimization and acceleration methods for point set networks.

[0039] The principles and implementation methods of the point cloud neural network model structure and electronic device of the present invention will be explained in detail below, so that those skilled in the art can understand the point cloud neural network model structure and electronic device of the present invention without creative work.

[0040] Example 1

[0041] like Figure 1 As shown, this embodiment provides a point cloud neural network model structure 100, in which: the sampling layer 110 is divided into two stages: a distance matrix calculation stage and a point selection stage; the clustering layer 120 selects points adjacent to the center point according to the distance matrix; and the multi-layer SA module 130 is improved according to the farthest point sampling of the spatial distance.

[0042] The point cloud neural network model structure 100 of this embodiment is described in detail below.

[0043] In this embodiment, the sampling layer 110 is divided into two stages: a distance matrix calculation stage and a point selection stage.

[0044] Specifically, in this embodiment, the distance matrix calculation stage and the point selection stage include: the distance calculation stage obtains the distance matrix between all points in a parallel manner, and then sends the distance matrix to the point selection stage for farthest point sampling, and at the same time retains the distance matrix in the memory for reuse by operators that need to use distance.

[0045] PointNet++ and 3DSSD use a custom operator to implement farthest point sampling, whose input is a three-dimensional array with dimensions of batch, the number of points in the input point set (points_num), and channel. Since the farthest point sampling used by PointNet++ is based on the spatial position of the point, the number of channels is 3, representing the x, y, and z coordinates of the point. 3DSSD takes the feature vector of the point into consideration, so the number of channels will be larger. Both use L2 distance when calculating distance. When implemented on the GPU, a thread block is assigned to each block. Since the block size of the data is 1 most of the time during the inference phase, only one thread block is used to calculate the farthest point sampling during the inference phase, so many computing resources are not fully utilized.

[0046] Specifically, when calculating the farthest point sampling, an array of distances from each point to the sampling point set is maintained. Every time a new point is added, the largest point in the array is selected to enter the point set, and the distances from each point to this point are calculated to update the distance array. Due to this dependency, it is not possible to achieve parallelism by dividing the point set into small blocks, so the farthest point sampling is implemented in one thread block, which slows down the running speed.

[0047] Considering this situation, this embodiment divides the farthest point sampling into two parts: distance matrix calculation part and point selection part. Both are implemented by two operators, namely, generating distance matrix (GenDistMat) and generating sampling points (GenFpsIndexFromDm). Figure 2 The schematic diagram of the farthest point sampling design is shown.

[0048] When calculating the distance matrix, this embodiment calculates the distances between all the points in the point set, and then stores them in a matrix of size points_num × points_num. Although this method seems to be computationally intensive, it can be done in large-scale parallel because there is no dependency between threads when calculating the distance.

[0049] In this embodiment, in the distance calculation stage, the distance matrix is ​​decomposed into multiple sub-blocks, and each thread block is responsible for one of the sub-blocks. That is, the distance matrix is ​​decomposed into many small blocks, and each thread block is responsible for one of the blocks, which is similar to the idea of ​​matrix multiplication. Therefore, during calculation, a large number of thread blocks will calculate the distance in parallel on the GPU, fully calling the computing resources of the GPU. Figure 5 Shows the computation of a thread block. Figure 3 Shows the computational schematic of a thread block.

[0050] When selecting points, because each iteration is affected by the previous iteration, and each iteration needs to see the global distance data, point selection can only be implemented with one thread block. Therefore, after calculating the distance matrix, the distance matrix is ​​passed in and points are selected based on the distance matrix. The process is the same as the original farthest point sampling, but the operation of calculating the distance is omitted and replaced by directly looking up the table based on the distance matrix, which can shorten the calculation time.

[0051] Experiments show that on the Volta V100 GPU, because it contains 80 stream processor clusters, it can see obvious effects when processing a point set of 16K. The time to calculate the distance matrix is ​​about 1 millisecond, and the time to select the point is about 20 milliseconds. The original farthest point sampling takes more than 40 milliseconds, so this split saves half the time.

[0052] In this embodiment, before calculating the distance matrix, the point set data used for calculation is input twice, once as the original data and once as the transposed data. This makes the GPU more compliant with the memory access characteristics when reading the point set data. Experiments show that the time consumption of transposing the matrix is ​​at the microsecond level, so the overhead of transposing is extremely small.

[0053] In this embodiment, the transposed point set data is arranged in the order of the dimensions of block, channel, and the number of points in the input point set.

[0054] In the original network, the point set data is arranged in the order of the dimensions of batch, the number of points in the input point set (points_num), and channel. This embodiment adds a transposition operation in the sampling layer 110 to change the dimension order of the matrix to the order of batch, channel, and points in the input point set (points_num). In addition, in the subsequent operator operation of GenPointsByIndex in clustering, this embodiment also uses the transposed matrix.

[0055] The reason for using transposed data is mainly to consider the memory access characteristics of CUDA. When accessing the memory, elements in consecutive addresses will be read out at the same time, so the threads should try to make full use of these data in one read operation. Since the channel is used as the basis for loop iteration when implementing the code, after transposing the matrix, continuous data will be read each time it is read, which can reduce the memory access overhead.

[0056] Experimental data shows that transposition takes very little time, so increasing the transposition operation to reduce memory access overhead can reduce the final time consumption.

[0057] In this embodiment, the clustering layer 120 selects the neighboring points of the center point according to the distance matrix. The clustering layer 120 needs to select the neighboring points of the center point according to the distance matrix. Because the improved sampling layer 110 design has obtained the distance matrix. The distance matrix is ​​passed to the clustering layer 120, and clustering is performed according to the data in the distance matrix, which avoids repeated calculations during clustering and saves calculation time.

[0058] Among them, in this embodiment, the selection of adjacent points of the center point according to the distance matrix includes: transferring the calculation results of the distance matrix stage and the calculation results of the point selection stage to the clustering layer 120, and performing clustering according to the data in the distance matrix.

[0059] Specifically, in this embodiment, the center point index in the point selection stage is transmitted to the clustering layer 120 , and the distance matrix in the distance matrix stage is transmitted to the clustering layer 120 .

[0060] After the original farthest point sampling selects the center point, it is directly sent to the clustering layer 120, and then the adjacent points are selected according to the designed radius. When selecting adjacent points, the distance needs to be recalculated. Since the distance calculated in the sampling layer 110 is not retained, it needs to be recalculated in the clustering layer 120, which causes computational redundancy. The improved sampling layer 110 splits the farthest point sampling into two operators, so the results obtained by the two operators can be sent to the clustering layer 120.

[0061] Therefore, the clustering layer 120 can receive both the center point index provided by the operator for generating sampling points GenFpsIndexFromDm and the distance matrix provided by the operator for generating distance matrix GenDistMat. Therefore, after the improvement, when selecting points, the clustering layer 120 can directly refer to the distance matrix and then select the adjacent points according to the set radius. Figure 4 The structure of the clustering layer 120 is shown.

[0062] In this embodiment, the clustering layer 120 includes a first operator for generating neighboring points from a distance matrix (GenNeighbourIndexFromDm) and a second operator (clustering points according to indexes); the first operator accepts two inputs: a distance matrix and a center point index, and searches for qualified points from the distance matrix according to a set radius range; the second operator extracts the coordinates of a point from the point set according to the index of the point, and inputs the coordinates into the feature extraction part in the clustering layer 120.

[0063] In this embodiment, the clustering layer 120 selects points by first consulting the distance matrix and then selecting adjacent points according to a set radius.

[0064] from Figure 4It can be seen that in the clustering layer 120, the operator GenNeighbourIndexFromDm that generates neighboring points from the distance matrix accepts two inputs: the distance matrix and the center point index. Then the operator searches for eligible points from the distance matrix according to the set radius range. In this step, points are processed in an indexed manner, which avoids the trouble of considering the coordinates of the points when processing directly. The result given by this operator is the coordinates of the neighboring points around the center point. Then, the operator GenPointsByIndex that generates neighboring points from the distance matrix is ​​used to extract the coordinates of these points from the point set according to the index of the point and send them to the subsequent feature extraction part. In an actual network, the clustering layer 120 will have multiple radii, so there will be Figure 4 Same multiple branches, Figure 4 Only a simplified version of the clustering layer 120 structure is shown.

[0065] In this embodiment, the multi-layer SA module 130 is improved according to the farthest point sampling of the spatial distance.

[0066] Specifically, in this embodiment, the improvement of the multi-layer SA module according to the farthest point sampling of the spatial distance includes: referring to the sampling result of the previous layer, selecting the points arranged relatively front, so as to omit the calculation.

[0067] In the multi-layer SA module 130, since the initial point is fixed during sampling, for the farthest point sampling purely based on spatial distance, such as the farthest point sampling in PointNet++, after the first farthest point sampling, the farthest point sampling results of the remaining layers are fixed. Therefore, for the farthest point sampling based on spatial distance such as PointNet++, the calculation process of the farthest point sampling except the first layer can be omitted, and the sampling result can be directly obtained.

[0068] When sampling at the farthest point, the selection of the initial point will affect the final result, and different initial points often produce different point sets. However, when actually implementing the program, this embodiment does not randomly select the initial point, but selects the first point in the point set as the initial point. In this way, the result of the point set can be predicted to a certain extent.

[0069] Whether it is PointNet++ or 3DSSD, because of the hierarchical relationship, they both contain multiple SA modules, so multiple sampling is required. Due to the hierarchical relationship, the points to be sampled are gradually reduced. Taking a point set of 16K as an example, the first sampling is to select 4K points from 16K points, the second sampling is to select 1K points from the above 4K points, the third sampling is to select 512 points from the above 1K points, and so on.

[0070] Because when implementing the code, the initial point selected by each sampling layer 110 is the first point of the point set by default. Therefore, for a network like PointNet++ that only uses spatial distance as the selection criterion, the selection of the initial point in each sampling layer 110 is fixed. Therefore, in the above example, the sampling of the second layer is the first 1K points of the 4K points sampled for the first time, the sampling of the third layer is the first 512 points of the 1K points sampled for the second time, and so on. The experimental data also shows this point.

[0071] Therefore, for the subsequent layers, the steps of calculating the distance and sampling can be removed, and the sampling results of the previous layer can be directly referenced to select the top points. This can greatly shorten the sampling time in the subsequent layers.

[0072] It should be noted that this method is only valid for PointNet++ networks that rely purely on spatial distance sampling. For 3DSSD, since spatial distance and feature vectors are considered simultaneously during sampling, and each feature vector is provided by the feature extraction module of the previous level, the sampling results will be different and cannot be directly obtained by relying on the previously used data. Therefore, the sampling layer 110 in the subsequent levels of 3DSSD still needs to be calculated. Of course, the improved sampling layer 110 can be used in all levels, which can also save a lot of computing time in this respect.

[0073] Because of the memory access characteristics of CUDA, using a transposed matrix during data transmission is more efficient than the original method. In addition to the modifications to the sampling layer 110 and the clustering layer 120, the point cloud neural network model structure 100 of this embodiment has made a series of adjustments to the overall structure of the network. In addition to adding a transposition operation to the point cloud neural network, this embodiment has modified the sampling layer 110 and the clustering layer 120, and increased the data required for the clustering layer 120, so the structure of the point cloud neural network has been adjusted, replacing the original operator with the newly added operator. The above-mentioned point cloud neural network model structure 100 in this embodiment can be implemented in PointNet++ and 3DSSD.

[0074] This embodiment provides an improved model paradigm for accelerating the point cloud neural network in the inference stage, and improves resource usage and redundant calculations to enhance the overall performance of the network. This embodiment mainly implements the above-mentioned point cloud neural network model structure 100 based on PointNet++ and 3DSSD networks, and accelerates the network on a GPU. Since most point cloud neural networks based on point sets use this architecture, the above-mentioned point cloud neural network model structure 100 provides a more general optimization model paradigm. Figure 5 A comparison of the improved design and the original design is shown.

[0075] Example 2

[0076] like Figure 6 As shown, this embodiment provides an electronic device 101, and the electronic device 101 includes: a processor 1001 and a memory 1002; the memory 1002 is used to store a computer program; the processor 1001 is used to execute the computer program stored in the memory 1002, so that the electronic device 101 performs the steps of the point cloud neural network model structure 100 in Example 1. Since the specific implementation process of the steps of the point cloud neural network model structure 100 has been described in detail in Example 1, it will not be repeated here.

[0077] The processor 1001 is a central processing unit (CPU). The memory 1002 is connected to the processor 1001 through a system bus and communicates with each other. The memory 1002 is used to store computer programs, and the processor 1001 is used to run computer programs so that the processor 1001 executes the point cloud neural network model structure 100. The memory 1002 may include a random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage.

[0078] In addition, this embodiment further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by the processor 1001, the point cloud neural network model structure 100 described in Example 1 is implemented. Example 1 has already described the point cloud neural network model structure 100 in detail, which will not be repeated here.

[0079] A person skilled in the art can understand that all or part of the structures of the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the above-mentioned point cloud neural network model structure 100 is generated; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0080] In summary, the present invention achieves the effect of accelerating the calculation of the neural network by changing the neural network structure and the calculation method of certain operators in the neural network. It is a relatively universal acceleration of the point cloud neural network and can be adapted to a variety of point cloud neural networks. Therefore, the present invention effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.

[0081] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical ideas disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A point cloud neural network model optimization structure for network acceleration on GPU, characterized by: In the point cloud neural network model structure: The sampling layer is divided into two stages: a distance matrix calculation stage and a point selection stage; the distance matrix calculation stage and the point selection stage include: the distance calculation stage obtains the distance matrix between all points in a parallel manner, and then sends the distance matrix to the point selection stage for farthest point sampling, and at the same time retains the distance matrix in the memory for reuse by operators that need to use distance; in the distance calculation stage, the distance matrix is ​​decomposed into multiple sub-blocks, and each thread block is responsible for one of the sub-blocks; The clustering layer selects points close to the center point based on the distance matrix; Improve the multi-layer SA module based on the farthest point sampling of spatial distance.

2. The point cloud neural network model optimization structure for network acceleration on a GPU according to claim 1, characterized in that: Before calculating the distance matrix, the point set data used for calculation is input twice, one is the original data, and the other is the transposed data.

3. The point cloud neural network model optimization structure for network acceleration on a GPU according to claim 1, characterized in that: After transposition, the point set data is arranged in the order of the dimensions of block, channel, and the number of points in the input point set.

4. The point cloud neural network model optimization structure for network acceleration on a GPU according to claim 1 or 2, characterized in that: In the distance calculation stage, the distance matrix is ​​decomposed into multiple sub-blocks, and each thread block is responsible for one of the sub-blocks.

5. The point cloud neural network model optimization structure for network acceleration on a GPU according to claim 1, characterized in that: The selecting of the adjacent points of the center point according to the distance matrix comprises: The calculation results of the distance matrix calculation stage and the calculation results of the point selection stage are both transmitted to the clustering layer, and clustering is performed according to the data in the distance matrix.

6. The point cloud neural network model optimization structure for network acceleration on a GPU according to claim 5, characterized in that: The center point index in the point selection stage is passed to the clustering layer, and the distance matrix in the distance matrix calculation stage is passed to the clustering layer; when selecting points, the clustering layer first consults the distance matrix and then selects adjacent points according to the set radius.

7. The point cloud neural network model optimization structure for network acceleration on a GPU according to claim 5, characterized in that: The clustering layer includes a first operator and a second operator; the first operator accepts two inputs: a distance matrix and a center point index, and searches for qualified points from the distance matrix according to a set radius range; the second operator extracts the coordinates of a point from a point set according to the index of the point, and inputs the coordinates into a feature extraction part in the clustering layer.

8. The point cloud neural network model optimization structure for network acceleration on a GPU according to claim 1, characterized in that: The method of improving the multi-layer SA module based on the farthest point sampling of the spatial distance includes: referring to the sampling result of the previous layer, selecting the points arranged relatively front, so as to omit the calculation.

9. An electronic device, characterized in that: It includes a memory for storing a computer program; and a processor for running the computer program to implement the point cloud neural network model structure as described in any one of claims 1 to 8.

Images